The never-ending story of the hx4700 WM5 upgrade – a lot of new tests, tips and myth-busting!

UPDATE (08/01/2006): now that the WM5 2.0.1 ROM upgrade for the hx4700 AND a usable RAM disk are out, you may want to check out the following articles first:

The original article:

Avid readers of my blog may have noticed the additional patches and tweaks posted by readers to the inherent WM5 problems of the HP iPAQ hx4700 Windows Mobile 5 (WM5) upgrade. As there’s a big need for reports on them, and non-English forums has also proved to be worth checking out for fixes never posted to international, English sites (sometimes – for example, with Chinese forums - with a lot of guesswork :) – much as I understand spoken Japanese, Kanjis are an unknown area for me), I haven’t abandoned the question of the hx4700 WM5 upgrade. Two weeks ago, I’ve started testing all these tweaks to see whether they are of any good. Hereby I present the results.

Executive Summary

I have pretty good news for you all. With the (proper) hacks and clever usage of the PDA, you’ll be able to heavily reduce the load and greatly speed up your device. You will only need to make sure:

1. don’t install anything in the main memory and make sure you delete (for example, help files in \Windows) or relocate (for example, resource files) everything possible from there
2. use programs that don’t (mass-)write to the main memory (for example, don’t use Messaging and make sure you disable or upgrade DockWare)
3. as there is, contrary to what some people say, no RAMDisk or anything similar, you have to live with the system slowdowns, particularly after heavy storage writes like installing programs; this, as has already been pointed out, can be made pretty tolerable
4. you can definitely speed up the system with additional registry hacks

1. Background

You know it all if you upgraded your HP iPAQ hx4700. Slowing down to a crawl, getting very hot, really bad responsiveness – all so common with Windows Mobile 5-upgraded hx4700’s (and Dell Axim x50’s). And not at all common with native WM5 devices like the Dell Axim x51 series, the HTC Phone Edition devices or the new Pocket Loox N5x0/C550.

First and foremost, is this all Microsoft’s faults, you may ask. Nope, not in the least (and I’m not telling you this because I’m a Microsoft Windows - Mobile Devices MVP myself!) Much as the compaction algorithm (the most problems are caused by filesys.exe and the compaction it runs from time to time) they use is, without doubt, a little bit too cautious (also see the ‘Longer than we'd expect’ section in Some Compaction Answers), it’s mostly because of Dell and HP that the WM5 upgrade of the HP iPAQ hx4700 (and the Dell Axim x50) series are so bad.

OK, choosing the slow-to-write-and-very-slow-to-delete NOR flash ROM (please read the six Windows Mobile Team blog articles linked at the bottom of the article if you’re unsure what NOR and NAND are about) wasn’t a fault – after all, not many would have then predicted two years ago that NOR-based flash ROM will be close to useless under the forthcoming operating system version. Then, the advantages of NOR were clear under pre-WM5 operating systems (XIP, can be read faster than NAND, some CPU's even contain built-in NOR memory).

However, what HP did with the WM5 upgrade was, apart from the NOR question, to put it mildly, far from perfect. They not only left a lot of bugs in it (see the infamous NavPoint and the Compact Flash problems), but also, while refusing to make use of XIP which is otherwise supported by NOR-based ROM, they still left the sector size at 4kbytes. 4k sector sizes are only required by XIP and are much worse than the minimal, 512-byte sector sizes because they may result in a eight-fold slowdown of the deletion/write process. (To write out/delete one byte, you’ll need to write/delete 4k and not just 512 bytes. This takes eight times more time.)

1.1 Quantitive test results: What and how did I benchmark?

I’ve used a still not released task manager application to draw the CPU usage of the filesys process. The application is capable of showing CPU usage results between the last two minutes and six hours (this is shown on the horizontal axis). On the vertical axis, the summary of the CPU usage can be seen. The higher the peaks, the more CPU cycles have been consumed. I’ve always made sure no other tasks were running (except for the to-be-tested ones) during the tests so that the results are reproducible and reliable.

I’ve run several, really long tests to see the effects of a given set-up.

To emulate the effects of writing to the built-in storage, I’ve used the (original, with the WM5 upgrade shipped – the new version I’ve reported on here no longer does this) DockWare application. It constantly keeps putting its temporary files into the main storage, which results in a lot of filesys overhead.

For example, this screenshot:

shows the following case: in the last six hours, there were almost seven (meaning it “kicked in” every 52-53 minutes) filesys sessions. They were compatively short, some minutes long (as can also be seen in this screenshot, showing only the last hour of the above benchmark), but used 100% CPU time. In the other times, the CPU usage was pretty stochastic: between approximately 7 and 40, part (about 7%) of it caused by the task meter tool, part of it by other background processes/tasks and part of it by DockWare itself, the program running in the foreground.

2. Filesys.exe and compaction

Filesys.exe or, more precisely, the compaction algorithm it does, is our common enemy – therefore, I devote a lot of space to the discussion. (I also recommend my older articles and the Windows Mobile Team blog linked from section 5!)

I’ve made a lot of experiments with widely varying parameters and, based on all them, I think I’m able to formulate how, when and why compaction takes place.

First, you can not completely get rid of compaction. Sooner or later, it will inevitably kick in. You can only make it run as rarely as possible with clever program installation, manual cleanup and/or relocation of files installed into the main memory and (some) registry tweaking. With clever tweaking, program installation and PDA use, you will only “enjoy” the compaction process kicking in once every 9-18 (!) hours; and then, only for some minutes. If you, however, don’t do this, you can “enjoy” filesys even every half an hour – for, say, ten minutes.

In this (2.) section, I elaborate on all these questions.

2.1 The filesys throttler - benchmarks

Read this for more information on obtaining and installing the filesys throttler. In here, I only present some benchmark results.

In my benchmarks, the filesys throttler proved to be doing its stuff without severely slowing down the Pocket PC as can be seen in the following shot:

This shows three stages of the DockWare benchmark: first, without the throttler (100% CPU usage), second, with the throttler on external power (AC) (which means the throttler will throttle the filesys process to 80% CPU usage only) and the third on battery (DC) (~20% CPU usage). As can clearly be seen,

1. throttling didn’t result in increased time needed to run
2. the frequency of filesys.exe ‘kicking in’ didn’t increase. This also shows there are no hidden problems connected to the throttler
3. the throttler indeed delivers on its promises – that is, it works without – seeming - side effects. Running it didn’t result in added CPU usage / generic system slowdown.

There seems to be an explanation for all this. It’s time that needed while erasing NOR memory, not CPU cycles. That is, you can put to hold (this is how the filesys throttler works) filesys for quite a lot of time (in a CPU's life - we're speaking of milliseconds) for vastly increased CPU (but, unfortunately, not file system – while erasing is taking place, all other operations needing to read from/write to the storage memory will also slow down, independent of how much cycles the CPU is spending on a particular task) performance.

Some people over at AximSite ran a lot of tests on their WM5-upgraded x50v’s letting filesys only use 1-2% CPU time. They have reported generic system instability; therefore, I don’t think we can go much lower with the CPU usage than 20%.

Again and again, I must point out that my benchmark results are diametrically opposed to some people’s reports. That is, you may want to read all the above in conditional (“may be” instead of “is” and “are”). It’s completely up to you if you go for the filesys throttler or avoid it completely. After all, if you follow my tutorial, you will make a really usable Pocket PC of your hx4700 – a single filesys process taking some minutes every 10 hour isn’t that bad. Then, you’ll be able to live with filesys having even near-100% CPU usage without the need for using any throttler.

2.2 The effects of additional files in the Storage memory

Any sensible person (programmers included) would think compaction will only touch newly added, deleted or modified files. Unfortunately, this is not the case. If the main storage is not completely clean, the performance hit (the time filesys spends at compacting) will be enormous and it will be proportional with the main storage usage (that is, the cumulative size of user-installed files).

This is definitely caused by the ‘wear leveling’ feature of the compaction algorithm, which relocates already-written blocks too, not only currently written ones. The Microsoft WM Team has promised to reduce (see the ‘Longer than we'd expect’ section in Some Compaction Answers) this but it’s unlikely to happen in the near future and it’s even more dubious whether HP/Dell will also implement this in their bugfixes.

In the meantime, your only choice is keeping the main storage as clear as possible by installing everything on storage cards, relocating all caches (Opera Mobile – see this – and Internet Explorer Mobile) there, using mailer clients that don’t need to store the mail files downloaded in the main memory etc. Speaking of mailer clients: as far as Messaging, the built-in e-mailer client is concerned, while I know Microsoft is aware of the problem of ActiveSync’s only synchronizing Outlook mail to \Windows\ Messaging (that is, to the main storage) and hopefully will change this in the future, I, currently, recommend against any kind of Outlook mail synchronization. Stick to either Qmail, FlexMail 2006 or nPop (see this mailer roundup for a complete review and comparison of all of these titles) and only use direct POP3/IMAP mail access, keeping the mail files on a storage card.

I’ve made several tests to see the effects of the files in the main storage.

2.2.1 Almost full main storage without active writing operations

In this test, I’ve only left some 7.4 Mbytes free in the main memory and, then, ran a 12-hour-long test without any file system write operations (that is, without the old DockWare version. I did start the new DockWare version for about 1.5 hours in the second phase of this test, though. It, however, did not cause any additional filesys activation because the new version doesn’t write its temp files in the main storage.)

The CPU usage of filesys.exe in the first six hours is as follows:

The last peak took about 26 minutes (!) at 100% CPU usage as can be seen in here (using a 30-minute scale):

In the next 6 hours, there still was considerable CPU usage. The second compaction phase, for example, lasted for about 16 minutes, while the first for about two times more:

Now, compare this to the “only one, tolerably quick compaction every 9-18 hours” “clean” case (when there’s no user-initiated writing to the file system). It can clearly be seen one of the biggest enemies of the compaction process are static files – even if you never modify them, they cause tremendous compaction overheads and an, in general, much less usable device than with an almost-clean main storage.

2.2.2 Slightly and Half-full main storage with active writing operations

I’ve also made some other tests, now, with frequent writes to the storage memory, with it being filled up less. The results are as follows:

During making this test, I’ve started with 60Mbytes of free storage (that is, only slightly “polluted” storage) and, after about two hours (see the long slope showing the effect of the “time jump” – it was during that time that I’ve quickly copied a lot of files from the memory card to the main storage), I’ve quickly decreased this to about 35 Mbytes, keeping all the other parameters intact. As can clearly be seen, the frequency of the peaks every 45-47 minutes (note them being more frequent and taking more time than in the completely clean case – yes, even 5-6 Mbytes of storage ROM usage has really bad effects on the performance!) and the length of the peaks have both considerably increased.

Now, again, compare this to exactly the same setup (DockWare) with almost entirely free main memory:

See the difference? With completely clean main memory, filesys only kicks in slightly more than once an hour (about every 52-53 minutes) and only lasts about 6-8 minutes:

2.2.3 Verdict and another quick tip

All in all: do not let any files stored in the main storage!

Also, make sure you delete the contents of \My Documents\ My Pictures. It contains the DockWare source images (the pictures DockWare displays in row) – in the main storage. They take up almost one Megabyte! Note that if you really want to use DockWare, upgrade it to the latest version and, if you don’t plan to get the commercial version, also make sure you edit manually edit HKCU\ Software\ Ilium Software\ DockWare\ BitmapDir to point to your images on your storage cards. (If you’re afraid of this, use for example Tweaks2k2 to do the trick.)

2.3 Other filesys issues – try reducing write operations!

As has already been pointed out, you must reduce the number of files in the main storage. You, however, must also make sure you update/ delete/ write files in there as rarely as possible. For example, this is why I not only relocated the cache of Opera Mobile in my Opera Mobile relocation article, but also the small (for example URL history and cookie) files it constantly keeps updating. These files are some kilobytes only – still, Opera Mobile’s constant writing to them will surely cause much more filesys triggers than otherwise, when you keep them on a memory card.

Opera Mobile is just one of the applications that can be cleverly tweaked for much more WM5-friendliness. For example, ActiveSync is another application that keeps writing to the file system.

2.3.1 ActiveSync

As many have reported (for example, both the AximSite folks and PDAMania forum member pdafun), ActiveSync keeps updating its log file at \Windows\ Activesync\ CtrlLog.txt even when there’s no activity (that is, ActiveSync is not connected) and you’ve applied the (compulsory!) fake server hack. (The other files in the same directory aren’t updated if there’s no ActiveSync activity.) To combat this, you can make this file read-only (this tip was originally contributed by pdafun) with, say, Total Commander and the WinCE FS file system plug-in (installation mini-tutorial here). Then, AS will never ever update this file.

The same may stand for a lot of other applications. For example, several mobile phone handler applications, some Midlet managers (for example, IBM J9) constantly write to either the root directory of your (storage) file system or to \My Documents. You can try giving a try to making their files read only too.

3. Registry hacks

3.1 Reducing the WinCE DB / registry flushing

Please see this article for more information. In the provided registry import file, I use the values 256 for both ActivityThreshold’s and a 100-minute FlushPeriod. This has proved to be pretty usable.

3.2 File system caching

First and foremost, I must point out that caching is far more complicated than many think. Even I was mislead at first before really digging into how WM5 changed the caching parameters from the only-one-parameter-is-used pre-WM5 world. (See the separate FAT and data caching instead of a “common” “catch-up” one, for example).

Caching-wise, pdafun’s settings, with some additional tweaks (for example, I’ve completely abandoned the, under WM5, deprecated CacheSize value from his tweaks) have proved to be really usable and dependable in the last months of usage.

I’d like to point out that it seems caching has absolutely no effect on filesys kick-ins at all. That is, you’ll get the same results if you absolutely don’t use caching but still stick to the known “rules of thumb” of not putting any files in the storage ROM, heavily increasing the ObjectStore timings (see section 3.1) and minimizing writing operations, including, for example, making some always-accessed files read-only.

Caching, on the other hand, will help in better operating system speed in general – (cached) icons will be displayed much faster after reloading them, the FAT table of your storage card(s) will be cached, resulting in generic speedup. It’s, again, not essential to have caching - as opposed to sticking the tips in the previous sections (minimized writing; as few files in the main storage as possible).

Without further explanation, all these hacks are present in this registry file . I’ll provide some more explanation of it in the next section.

3.3 For techies/geeks only: what is in the registry file?

First and foremost, check out the official MSDN documentation of the FAT File System Registry Settings. It explains a lot; for example, why I've dropped the ‘plain’ CacheSize. I also provide some additional explanations in here:

The registry value HKEY_LOCAL_MACHINE\ System\ StorageManager\ FATFS\ Flags have traditionally caused a lot of confusion among PPC users. Many people with Axim x50 WM5 devices even recommend (see for example this post in this thread) using the Flags value 0x28. A quick glance in the MSDN document reveals that this enables verifying all writes (which is absolutely unnecessary – on the contrary!) as it sets the fourth bit of the flags word. This also shows there’re a lot of plain bad “hacks”, doing exactly the opposite what they should do, circulating all around.

You don’t necessarily need to touch its default value (0x64; note that, in this section, I only use hexa values and not decimal ones). I’ve been using it with the value 0x46, which disables automatic formatting of unformatted volumes (0x40), disables event logging, disables automatic calls to ScanVolume; does not updates access times (0x1), does not verify all writes (0x8), does not add a backup FAT to all formats (0x10), does not sets all files to WRITE_THROUGH, regardless of the parameters to CreateFile (you may also enable this), does not disable write through on the WriteFileWithSeek function (0x10000), does not disable format (0x20000), does not transact data on a write operation (0x40000); also see the explanation of the higher bits, all disabled.

HKEY_LOCAL_MACHINE\ System\ StorageManager\ Profiles\ MSFlash\ FATFS\ Flags, which overrides the above settings on the other hand for the built-in storage, uses the same settings but it disables write through on the WriteFileWithSeek function (0x10000) and disables format (0x20000) – this is why it uses the value 0x30000 larger than with the generic FAT case (that is, the case that also includes memory cards).

3.4 Other registry hacks

The most important of them is redirecting the system-level temp file to a storage card – that is, pointing HKEY_LOCAL_MACHINE\ System\ FileSys\ TempPath to a directory on a storage card. I’ll also elaborate on this question a bit later.

Unfortunately, it’s still not known how \Windows\ AppMgr\ Install can be redirected. Now, all ActiveSync-based installations result in the install CAB file being copied by ActiveSync to this directory, resulting in, later, a lengthy filesys.exe session. (Of course, as far as \Windows\ AppMgr\ Install is concerned, in the defense of Microsoft, it must be pointed out that installing applications should be done with the largest memory available possible because of a very bad, uninstallation & RAM memory usage-related bug in Windows Mobile 5 explained here (see Reed Robison’s (nick: reedr) explanation on Tue Feb 21, 2006). That is, in some (very few) cases, having the CAB file in the file system – that is, in the storage ROM – may be advantageous).

Also, you can use the traditional hack of increasing the GlyphCache limit and switching off animation (see this for more info).

3.5 Registry hacks that don’t seem to be usable

3.5.1 DirsToExclude hack

Many have tried to add the always-changing directories to HKEY_LOCAL_MACHINE\ System\ StorageManager\ Filters\ fsreplxfilt\ DirsToExclude.

Unfortunately, it doesn’t seem to help – all, say, program install operations will almost surely result in a heavy compaction process, showing that, if you put for example \Windows\ AppMgr\ Install in the list, writing to the given directory will still result in heavy compaction processes.

If you still choose to hack this (again, I could benchmark no difference between the unhacked and the hacked case), you’ll need to enter the directories into, all of them in a row of its own as can be seen in this screenshot and, then, increase the value NumDirsToExclude with the number of new directories added. With the three additional directories depicted in the above screenshot, the registry import file becomes the one provided, as far as the values HKEY_LOCAL_MACHINE\ System\ StorageManager\ Filters\ fsreplxfilt\ DirsToExclude and NumDirsToExclude are concerned. Feel free to delete these registry imports.

3.5.2 MountAsROM hacks

Many hacks are known that tamper with the different MountAsROM values in the Registry. Some people even say it makes the files stored in RAM.

I’d like to point out that there is no way to store any files in RAM under WM5. There is no RAMDisk under WM5 (it’d be a REAL lifesaver because, then, we could just run all the (temporary) writing operations in superfast, no-compation-needed RAM), no matter what some people say. Microsoft, one year ago, has included a barebone Ramdisk implementation in the Platform Builder but it hasn’t been implemented by any Pocket PC manufacturers. Please also see this article on the subject. Pocket PC manufacturers should implement this as soon as possible – along with, preferably, virtual memory. It’s worth pointing out that for example Nokia (a direct competitor to Microsoft) has just implemented this in the Debian Linux-based OS 2006 Beta (see Neil’s post dated June 10, 2006 here).

MountAsROM has absolutely no effect on anything and has nothing to do with the never-realized Ramdisk. That is, if someone states “with MountAsROM hacks, you can have the same speeds as with WM2003SE”, just don’t believe it.

These hacks can be found mostly in Chinese forums. On English forums/in my blog, they are posted by for example ‘liminwei’ ( see this post (05/31/06) and this thread).

3.5.3 The StorageManager\Profiles\MSFlash\Folder rename hack

This hack (‘just rename the above value from ‘NOR Flash’ to ‘NAND Flash’) was first sent to my blog by geostat (see this (05/26/06)).

Unfortunately, it absolutely no effect on the compaction. In one of my tests (with a lot of other hacks that otherwise had no effect, mainly from liminwei – see above), it even had a very adverse effect on the CPU usage: in the DockWare test, filesys.exe almost immediately (in 1-2 minutes) ‘kicked in’ and took a LOT of time to finish its work:

The hack may be related to the fact that NOR memory can be treated as NAND memory while writing/erasing (see the second and the third paragraph in the ‘Persistent Storage and NOR’ section in Mike’s article What Happened To NOR?). This is NOT the case – this hack won’t have any positive effect.

4. “Legacy” hacks revisited

Of the well-known “legacy” hacks I’ve described here, deleting the HKEY_LOCAL_MACHINE\ Services\ NavPointService, adding the “fake server” hack, the CF card fix and, probably (depending on your choice), the filesys throttler application are still very important. Applying the CacheSize hack (step 4), on the other hand, is no longer recommended – you will only need to import the only one registry file provided in this article.

5. Additional information

The hx4700 category in the Smartphone & Pocket PC magazine Expert Blogs.

Also make sure you scrutinize every relocation/cleanup-related article in my blog and my old homepage (these are older articles from my pre-blog times and, therefore, aren’t available in the blog; make sure you do a generic search with Ctrl-F for the word-stubs ‘relocat’ and ‘clean’). I publish a lot of information on relocation/cleanup issues for a lot of widely-used, well-known programs.

Don’t hesitate to ask me if you have a program with a lot of files installed into the main memory I haven’t published a relocation tutorial/tips on. Please do not send me private e-mails – private e-mails are of no benefit to others because they are read by only two people. Prefer posting your questions, for example, here in the blog.

"filesys.exe" and CompactionPrio256 - Are we close to fixing WM5? – Aximsite – this is Dell Axim x50-related and really long, but worth a read for people wanting to know more about the secrets of filesys.exe.

Finally, six excellent articles from the Windows Mobile Team:

What Happened To NOR?

Some Compaction Answers

Paging Dr. RAM

What's A "Compaction Thread"?

RAM, ROM, NAND, NOR--that's a lot of capital letters...

Beating up your PPC with the web... - an article about the lack of RAMDisk

Downloads

The registry import file can be found here. All you’ll need is importing them, while doing (some of) the “legacy” hacks (see section 4 above).

If you need a REGEDIT4-compliant version of the file (so that you can import it with, say, the free TRE – see this registry editor roundup for more info), let me know.

Acknowledgements

Many thanks goes to pdafun for his excellent tips. Also, Mike Calligaro’s articles were also pretty instructive.