How can you avoid the consequences of a new, file download-related bug I've just found in Pocket Internet Explorer?

http://smartphonemag.com/forum/topic.asp?TOPIC_ID=20477

Here at the Pocket PC Mag forums, I've just been reported a strange bug affecting Pocket Internet Explorer (PIE for short). The problem is as follows: PIE is unable to download files from various Web sites, one of them being Caiman. Give it a quick try if interested: try to click the "*Download from caiman.us" link at the bottom of, say, this page in PIE and check out what is actually downloaded (either a 0 or a 7x-byte HTML error report). Then, try to do the same in either NetFront (I've tested this with both version 3.2 and 3.3 TP 1.03) and the beta version of Opera – they will both work, unlike PIE. Note that as far as the the two alternative PPC browsers, Thunderhawk 2.1 and Minimo 0.012 are concerned, the former has absolutely no download capabilities and the latter is (still) unable to download anything. Please note that this bug, again, will no only affect Caiman downloads but all Web servers that offer binary (downloadable) files and check the original home page address in the request! There're numerous pages like these: Caiman is just an example. Affected Windows Mobile/Pocket PC operating system versions I've tested WM2003SE (Pocket Loox 720), WM2003 (iPAQ 2210) and Pocket PC 2002 (iPAQ 3660). The built-in PIE in all these three operating systems suffer from the problem. The solution on the client side One of the easiest solutions is using EZDownload (also see the official homepage of the application and this page for an alternative source). Download EZDownload, extract ARM\EzDownload.exe from the ZIP file and copy it to your PDA to, say, \Windows\Start Menu\Programs so that you can easily access it. Just paste the URL of the file you'd like to download to the clipboard (with any "Copy link" capable alternative Web browser (for example, NetFront 3.3+) or PIE plug-in (all the four of them are capable of this)) and paste it to EZDownload (the latter must be manually started as it offers no browser integration!). Then, click the Start button for initiating the download: click for screenshot. Please note that EZDownload auto-generates the Referer header (it just removes the trailing filename from the URL; that is, for example, for the URL http://masataka.sakura.ne.jp/ez/bin/EzDownload/EzDownload.115.zip it creates a Referer header with the value http://masataka.sakura.ne.jp/ez/bin/EzDownload/ and, from http://www.usedcarsbelgium.be/ZZZ/dtm.zip, http://www.usedcarsbelgium.be/ZZZ/), which isn't guaranteed to work with all sites that require a Referer header - for example, it won't work with sites that are linked from a completely different site and refuse local referrers. Caiman isn't one of these sites. (If you want to read about some advanced EZDownload and never-before-published hacks I've invented, please read the "advanced" section below for further tips.) Alternatively (especially if you don't like the, compared to most of the alternatives, definitely weak download speed, the pre-Pocket PC – that is, Palm-size PC-optimized – GUI of EZDownload, its grinding the whole PPC to halt sometimes during downloading; or, with sites where its 'chop off the trailing filename' Referer header auto-generation isn't sufficient), your other/only choice is switching to NetFront or Opera when you download from "problematic", Referer-checking sites. As far as PIE plug-ins are concerned, none of them work. Three of them (Spb Pocket Plus 3.1, MultiIE 3.10-d0059 and PIEPlus 1.3b2) don't support direct "Save target" downloads (note that the "Save page as" feature they all offer won't help because it would also require to be able to go to a given page and, only when you're here, initiate the save operation). The fourth, excellent and free PIE plug-in, ftxPBrowser (current, tested version: 0.1e/WM2003), has a "Save Link As..." feature, but it doesn't work either (for the reasons I'll elaborate on later, in the 'geek', 'techy' section for "geeks"). The current (2.2) version of VITO MobileDownloader is another PIE file saving option. It, however, doesn't pass the Referer tag to the server either and you can't force to do it in the settings dialogs (first, second) either. As you can see here, it didn't download the file either – just downloaded the returned error message. It's also worth knowing that adding a dword "EditFlags" value with the value 10010000 to [HKEY_CLASSES_ROOT\zipfile] in the Registry to disable the PIE target dialog window (see this blog entry for more information on the meaning of these flags if interested) won't fix this problem either. Then, no matter the transfer is auto-started (no target/open dialog box is presented), the second (no-Referer) request will still be sent out – totally in vain. The solution on the server side Before this bug is fixed by Microsoft, as a Web server administrator, what can you do if you want to avoid people directly linking your files (that is, you still want to check the Referer header to see that the file is indeed linked from a "friendly" page) BUT you still want to enable PIE users to download files from your server? Just add the following test(s) in your server engine: Look for the HTTP User-Agent header. If it clearly shows that the originator is PIE (its default User-Agent header is "Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320)", then, just ignore that no Referer header has been sent. Note that, in some cases, checking the User-Agent header isn't sufficient because many PIE users like to pretend to be using a desktop browser. In these cases, check for the existence of the "UA-Language: JavaScript" header. It's the only header sent back by PIE in the second request (unlike in the first, where a lot of other UA- custom headers are sent by it) that can be used to used by the server to be absolutely sure the client is a PIE. For geeks/ Microsoft Windows Mobile developers, geek stuff follows I've scrutinized the HTTP communication (as many of you may have know, I'm a TCP/IP / Networking Protocols expert) and found out that the bug is caused by the fact is that PIE sends out the request for the file two times. First, when you click a link of the file and, second, when you dispose (by clicking Yes/Change) to the "where to download" dialog box. The fact that the request is sent out two times (instead of one, which would be sufficient) isn't that problematic at the first glance. Yes, there's an overhead, particularly on the server side (because the HTTP server starts sending the first block(s) of the file, some dozens of kilobytes, back to the client, only to find out later that the data sent back was discarded). The most important difference between the two (the one sent when you click the link and the one when you dispose the 'where to' dialog box) requests is that, in the second request, the HTTP "Referer" header is not sent to the server. This means all file servers that do require the HTTP "Referer" header to enable downloads will actually refuse to download anything. This is what causes several file downloads to result in not downloading anything. A protocol-level communication script follows (showing the dowload from the above-linked page): Request 1 (you click the download link in the page): GET /ZZZ/dtm.zip HTTP/1.0 Accept: */* Accept-Language: en-us UA-OS: Windows CE (Pocket PC) - Version 4.21 UA-color: color16 UA-pixels: 480x640 UA-CPU: Intel(R) PXA272 UA-Voice: FALSE Referer: http://www.caiman.us/scripts/fw/f1124.html UA-Language: JavaScript Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320) Host: www.usedcarsbelgium.be Connection: close As you can see, the right Referer header is sent out. The server's answer is as follows: HTTP/1.1 200 OK Content-Length: 12366113 Content-Type: application/x-zip-compressed Last-Modified: Tue, 13 Sep 2005 11:46:28 GMT Accept-Ranges: bytes ETag: "73f0dac658b8c51:465" Server: Microsoft-IIS/6.0 X-Powered-By: ASP.NET Date: Sun, 22 Jan 2006 08:09:44 GMT Connection: close <some tens of kilobytes of binary data> This is perfectly normal: the server is trying to send back the requested file. PIE, however, discards the contents of this response and, after the user has dismissed the dialog box, sends out a new request – for the same file, but now, without the Referer header: GET /ZZZ/dtm.zip HTTP/1.0 Accept: */* Accept-Language: en-us UA-Language: JavaScript Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320) Host: www.usedcarsbelgium.be Connection: close The server treats this request as a completely independent request from the earlier one (HTTP is a stateless protocol and there're no cookies here that would help tracking the originator of the request) and will just refuse to send back the requested resource - again, because of the lack of the Referer header - by just sending back an Internal Server Error message: HTTP/1.1 500 Internal Server Error Content-Length: 75 Content-Type: text/html Server: Microsoft-IIS/6.0 Date: Sun, 22 Jan 2006 08:09:48 GMT Connection: close Had the client sent the Referer header, the server would have sent back the file. How the Windows Mobile Programming Team can fix this problem? By either adding a Referer header to the second request or, even better, completely elminating the second request and automatically buffering all incoming responses/file transfers (and, possibly, refusing/cancelling them later, if the user chooses to cancel the file transfer). As with the Messaging bug I've discovered some days ago, in here, clearly, the two requests are generated by different, independent program code, which isn't the best practice in programming. The latter (eliminate sending out the second request and always buffer the incoming response using dynamically allocated memory so that the static memory consumption doesn't increase) is the best solution, because it also makes considerably less load on the HTTP server as no HTTP response will be ignored and file transfers autorestarted. What causes ftxPBrowser to fail? Unlike with PIE, there're no double requests. However, the requests sent out are either wrong (without the Referer header) or the correct response is ignored. Depending on the state of the "Open New Window" feature (see the above-linked ftxPBrowser blog entry on disabling/enabling it), there're four cases:

Mode:	Open New Window enabled	Open New Window disabled
Direct click on link	FAILED: No referer sent	Correct request sent; server sends the file in the response, but it's ignored
"Save Link As..."	FAILED: No referer sent and, just like in the 2nd request with PIE, only the UA-Language: JavaScript header is sent out.	FAILED: No referer sent and, just like in the 2nd request with PIE, only the UA-Language: JavaScript header is sent out.

Advanced EZDownload tips & tricks EZDownload severely lacks any kind of documentation or options settings and doesn't use the system-wide proxy setting. As its homepage does mention its being able to use proxies, I scrutinized the executable file with a file viewer and found the strings (around the file position 107A0) ProxyPort, ProxyServer and EnableProxy, which refer to registry value names. I've, therefore, manually created these values under [HKEY_CURRENT_USER\Software\Ez\EzDownload] to point to my HTTP proxy server: "ProxyPort"=dword:00001F90 "ProxyServer"="<proxy server's address, without any trailing http://>" "EnableProxy"=dword:00000001 You can also download the registry import file from here to avoid manual value creation. Remember to give "ProxyServer" a meaningful value before importing it; also, make sure you change "ProxyPort" if your proxy server listens to incoming requests at a different port than 8080. Incidentally, these are the HTTP request headers sent out by EZDownload. As you can see, it also (dynamically) contructs the Range header because it supports file download resuming operations like Opera and, unlike, say, PIE. GET http://www.usedcarsbelgium.be/ZZZ/dtm.zip HTTP/1.0 Host: www.usedcarsbelgium.be Range: bytes=0- Accept: */* Connection: close User-Agent: EzDownload/1.15 Referer: http://www.usedcarsbelgium.be/ZZZ/ Recommended reading In addition to the links above, I recommend the Bible of Pocket PC Web Browsers (alternatives: FL, iPAQ HQ, AximSite, PPC Magazine, BrightHand) for a detailed review of all the available Pocket PC Web browsers and everything related to accessing the Web on the Pocket PC. Much as it's linked from the previous roundup, I still find it useful to provide links directly to my "Downloading files with PIE" article too (alternatives: PPCT, AximSite, PPC Magazine, FirstLoox, BrightHand).