Found the explanation for/solution to probably the nastiest Pocket Inbox bug!

http://www.smartphonemag.com/forum/topic.asp?TOPIC_ID=20296

As I know quite a bit about TCP/IP protocols, including POP3 (I've, for example, (with some beans, "only" co-)authored several IBM AlphaWorks TCP/IP-related JavaBeans back in 2000-2001, including the POP3 bean), in my previous article on this problem (alternatives: iPAQ HQ, AximSite, PPCT, FirstLoox, BrightHand) and in other, related discussion threas (for example this one; I particularly recommend the Pocket PC Thoughts one), I've been promising to dive into probably the nastiest Pocket Inbox / Messaging bugs. Now that I had some free time to write a minimalistic POP3 server emulator to generate debugging-friendly mails for this purpose and could allocate some time for debugging, I'm proud to present the protocol-level description of and several solutions to the problem. What happens? What happens, in the first place? Pocket Inbox (a.k.a., starting with WM2003SE, Messaging; from now on, I only use the word "Messaging" to refer to it) is very dumb in that it assumes no messages have been deleted from the mail (the POP3) server when it auto-connects to auto-fetch a mail body when you choose Mark for Download for a given mail message. If there were messages deleted in the meantime, depending on whether the server-side serial number was higher or lower than that of the message-to-be-retrieved, Messaging may return the body of a completely unrelated mail. Note that I've emphasized the word 'auto'. It's only at during auto-fetching that Messaging is buggy. When you manually connect to fetch mail bodies, you won't run in any problems. But, then, you must guarantee no automatic connection can take place and/or no messages have been deleted from there since the last synchronization. What does exactly happen? (Please note that I'll also provide POP3-level debug scrutinization later. In here, I only give a less technical explanation of what is happening.) When you choose to download the body of a mail by choosing Edit/Mark for Download while you read the body of a mail (or, you choose Mark for Download from the context menu from the mail header list), Messaging will auto-connect to the POP3 server to fetch the mail as soon as possible. This is certainly a very nice feature while you're reading a mail – after all, this way you get the mail body as soon as possible, without having to exit from the mail reader screen. During this auto-fetch, however, Messaging doesn't request the so-called UID list from the server again to recompute the then-current mail serial number (it's only by UID's that a mail message in a POP3 server can be safely identified; all on-server post numbers must be computed based on them). Instead, it just passes the old post number to the POP3 server, which may point to an entirely different mail because of the following: on the POP3 server, when you delete a mail, the serial number of all mails that have followed it will "shift down" by one. That is, if you delete mail number 9, then, the previous mail number 10 will be referred to by the serial number 9. This is what causes all these problems because Messaging doesn't take the possible serial numer shifting into account when it auto-fetches mail bodies. It just blindly assumes all server-side mail bodies have exactly the same serial number than the last time it saw them. Solutions The best solution to this problem is the following:

• if you
  1. can afford it (because you aren't accessing your POP3 mailbox through time- or communication-wise expensive services) and
  2. don't want to access other POP3 mailboxes (you can only be connected to only one POP3 server at a time; that is, if you remain connected to a mailbox, you won't be able to fetch mail from the others. Then, the connection icons in other mailboxes will be rendered non-clickable, as in this screenshot)
  make sure the POP3 connection remains connected; that is, don't log out. To do this, just highlight the 'Network' icon next to the Accounts menu so that it becomes highlighted. In these cases, deletions will only be sent to the server when you actively click the refresh icon (this one). The body of messages that you request in the 'connected' state will be delivered without problems. This all means deletions may be committed waaaaaaaaaaay after you've finished requesting the mail bodies you're interested in. (Note that if the POP3 server forces a timeout (which may be true with public mailservers that need to serve thousands of POP3 client connections at the same time and, consequently, may not afford keeping up idle client connections for extended lengths of time) while you're browsing your mailbox, nothing wrong will happen. Messaging won't really notice the timeout, but after an unsuccessful message getting attempt, it will show that the connection wasn't successful: click for screenshot Then, as you have to manually reconnect to the server, no auto-fetching (our only enemy) will take place and, therefore, you'll be safe.)
• alternatively, if server-side mass-deletion of your mail is OK with you when you're absolutely sure you won't fetch any (more) mail bodies, make double sure you leave the "Empty deleted items" drop-down list at "Manually" in Accounts/Accounts.../Storage. Then, Messaging will only delete the mail when you go to the Deleted Items folder on your PDA and (mass-)delete the given message(s) from there: click for screenshot Please note that, in this case, when you delete something from the Inbox folder (by using, for example, the traditional message deletion icon or Delete from the context menu), these mails will never be deleted from the POP3 server unless you explicitly go to the Deleted items folder and manually initiate the deletion. Again: this means mail that you only delete from the Inbox folder will never be removed from the POP3 server. That will only happen when you set the "Empty deleted items" drop-down list to "Immediately" or "On connect/disconnect". click for screenshot
• if neither of the above solutions are OK with you because
  1. for some (pretty strange, I'd say; except for really-really strict security reasons) reason, you want to delete everything from the POP3 server as soon as possible without explicitly going to the Deleted Items folder (Manual deletion mode) or
  2. don't want to remain connected to the server because of, say, the high communication costs and want to choose the mail bodies to be downloaded while offline,
  don't let Messaging auto-fetch the messages you've chosen to fetch in their entirety. This means only choose "Mark for Download" when you're absolutely sure the PDA is not connected to the Internet and it can't connect it either (because, for example, you've disabled Bluetooth) even if it wanted. In this case, however, you'll need to get used to the usual "can't connect" bubbles. In this case, when you finally connect to fetch your mail, there won't be auto-fetching; manually initiated connections, as has already been emphasized, are perfectly safe.

Is this bug easy to fix, programming-wise? (This section is mostly meant for people that know how to program. It's not needed for being able to understand the ways the problem could be entirely avoided by the end user - that is, the contents of the previous section.) It is. All the developers in the Windows Mobile Team need to do is adding a state check to the full message autodownloader routine.

• If the state is connected (which is shown, as we've already seen, the network icon being highlighted), then, the current algorithm can be used.
• If the state isn't connected and a auto-reconnection must take place, however, the current algorithm must be abandoned and the 'manual' one should be used instead, which does download the UIDL and, based on the contents of UIDL, computes the parameter of the RETR command.

Geek stuff follows - protocol-level explanation of the problem For tech geeks that would like to know how I've done the POP3-level debugging: First, I've written a hand-made POP3 server emulator to generate debugging-friendly mails. The source of the server, as with all my Pocket PC-related software, can be found here. If you cast a glance at the source, you will see that it returns ten mail bodies with serial numbers starting from 0 and ending at 9 (the number of messages returned can be configured by modifying the NR_OF_MSG static field). These serial numbers are included in both the header and the body of the message so that messages that are 'off' can be easily caught and the serial number of the deleted/redownloaded messages can be easily taken trace of. This POP3 server emulator is tailored to serve the POP3 command sequence of VPOP3 Enterprise Version 2.3.0e, which is a nice Windows-based mail server. I've chosen this server because it has everything one can need for client debugging purposes: the capability to fully debug the client-server (that is, PDA - VPOP3) communication and to manually initiate the fetching of the mail from my POP3 server emulator, when new test mail generation is necessary. (Note that the two links in the previous sentence don't point to anything right now; they'll only work if you set up VPOP3 on your local computer to play with it.) It's VPOP3 that I made the Pocket PC-based Messaging point to. Let's see some POP3-level communication to see what's going on! 1. Messaging fetches the mail bodies from the VPOP3 server:

15/1/2006 15:42:13.218 - 00000032>+OK VPOP3 Server Ready <32.654.e676ff2>
15/1/2006 15:42:13.218 - 00000032<CAPA
15/1/2006 15:42:13.218 - 00000032>+OK Capability list follows
TOP
USER
UIDL
SASL CRAM-MD5
.
15/1/2006 15:42:13.234 - 00000032<AUTH
15/1/2006 15:42:13.234 - 00000032>-ERR Unrecognised AUTH type

Incidentally, here, as can be seen (and can also be read in the related RFC), this Messaging-issued AUTH command is faulty as AUTH must have the auth_type parameter. This bug doesn't result in any problem with a decent POP3 server, though.

15/1/2006 15:42:13.234 - 00000032<USER uus
15/1/2006 15:42:13.234 - 00000032>+OK User Accepted, PASSword required
15/1/2006 15:42:13.234 - 00000032<PASS uuspassu
15/1/2006 15:42:13.375 - 00000032>+OK uus has 10 message(s) (3110 octets)
15/1/2006 15:42:13.546 - 00000032<STAT
15/1/2006 15:42:14.046 - 00000032>+OK 10 3110
15/1/2006 15:42:14.062 - 00000032<UIDL
15/1/2006 15:42:14.062 - 00000032>+OK
15/1/2006 15:42:14.062 - 00000032>1 155
15/1/2006 15:42:14.062 - 00000032>2 157
15/1/2006 15:42:14.062 - 00000032>3 159
15/1/2006 15:42:14.062 - 00000032>4 161
15/1/2006 15:42:14.062 - 00000032>5 163
15/1/2006 15:42:14.062 - 00000032>6 165
15/1/2006 15:42:14.062 - 00000032>7 167
15/1/2006 15:42:14.062 - 00000032>8 169
15/1/2006 15:42:14.062 - 00000032>9 171
15/1/2006 15:42:14.062 - 00000032>10 173
15/1/2006 15:42:14.062 - 00000032>.

Now, the following body will be repeated ten times, with the 9 in the subject/body (and 10 in the TOP 10 25 command) gradually decreasing to zero (and, with TOP, to one):

15/1/2006 15:42:14.109 - 00000032<TOP 10 25
15/1/2006 15:42:14.109 - 00000032>+OK 311 octets
15/1/2006 15:42:14.109 - 00000032>Received: from 127.0.0.1 (UIDL=0:93944148259805504) by winmobiletech.com (VPOP3) with POP3; Sun, 15 Jan 2006 15:28:31 +0100
15/1/2006 15:42:14.109 - 00000032>From: 9
15/1/2006 15:42:14.109 - 00000032>To: Menneisyys <a@b>
15/1/2006 15:42:14.109 - 00000032> Date: Fri, 13 Jan 2006 00:13:07 (GMT)
15/1/2006 15:42:14.109 - 00000032>X-VPOP3-Spam: 12.5 - SuspiciousHeader(12)
15/1/2006 15:42:14.109 - 00000032>X-VPOP3-Spam2: added bayes ham
15/1/2006 15:42:14.109 - 00000032>X-VPOP3-SpamBayes: 0
15/1/2006 15:42:14.109 - 00000032>Subject: 9
15/1/2006 15:42:14.109 - 00000032>
15/1/2006 15:42:14.109 - 00000032>
15/1/2006 15:42:14.109 - 00000032>9
15/1/2006 15:42:14.109 - 00000032>.

15/1/2006 15:42:16.421 - 00000032<QUIT
15/1/2006 15:42:16.421 - 00000032>+OK VPOP3 Server Closing Connection

2. The user deletes three mails: as can clearly be seen (the UIDL is fetched by the client), there can be no problems in here because the deletion is UID-based, unlike auto-fetching:

15/1/2006 16:09:32.375 - 0000003b>+OK VPOP3 Server Ready <3b.1620.e8072e6>
15/1/2006 16:09:32.375 - 0000003b<CAPA
15/1/2006 16:09:32.375 - 0000003b>+OK Capability list follows
TOP
USER
UIDL
SASL CRAM-MD5
.
15/1/2006 16:09:32.390 - 0000003b<AUTH
15/1/2006 16:09:32.390 - 0000003b>-ERR Unrecognised AUTH type
15/1/2006 16:09:32.390 - 0000003b<USER uus
15/1/2006 16:09:32.390 - 0000003b>+OK User Accepted, PASSword required
15/1/2006 16:09:32.406 - 0000003b<PASS uuspassu
15/1/2006 16:09:32.406 - 0000003b>+OK uus has 10 message(s) (3100 octets)
15/1/2006 16:09:32.593 - 0000003b<STAT
15/1/2006 16:09:33.093 - 0000003b>+OK 10 3100
15/1/2006 16:09:33.203 - 0000003b<UIDL
15/1/2006 16:09:33.203 - 0000003b>+OK
15/1/2006 16:09:33.203 - 0000003b>1 175
15/1/2006 16:09:33.203 - 0000003b>2 177
15/1/2006 16:09:33.203 - 0000003b>3 179
15/1/2006 16:09:33.203 - 0000003b>4 181
15/1/2006 16:09:33.203 - 0000003b>5 183
15/1/2006 16:09:33.203 - 0000003b>6 185
15/1/2006 16:09:33.203 - 0000003b>7 187
15/1/2006 16:09:33.203 - 0000003b>8 189
15/1/2006 16:09:33.203 - 0000003b>9 191
15/1/2006 16:09:33.203 - 0000003b>10 193
15/1/2006 16:09:33.203 - 0000003b>.
15/1/2006 16:09:33.296 - 0000003b< DELE 8
15/1/2006 16:09:33.296 - 0000003b>+OK message 8 deleted
15/1/2006 16:09:33.312 - 0000003b< DELE 7
15/1/2006 16:09:33.312 - 0000003b>+OK message 7 deleted
15/1/2006 16:09:33.390 - 0000003b< DELE 4
15/1/2006 16:09:33.390 - 0000003b>+OK message 4 deleted
15/1/2006 16:09:59.109 - 0000003b<QUIT
15/1/2006 16:09:59.203 - 0000003b>+OK VPOP3 Server Closing Connection

3. The user chooses to download the last mail (serial number 10, with 9 in the subject/body); the auto-connect communication is as follows:

15/1/2006 16:10:09.562 - 0000003c>+OK VPOP3 Server Ready <3c.1080.e81042a>
15/1/2006 16:10:09.562 - 0000003c<CAPA
15/1/2006 16:10:09.562 - 0000003c>+OK Capability list follows
TOP
USER
UIDL
SASL CRAM-MD5
.
15/1/2006 16:10:09.578 - 0000003c<AUTH
15/1/2006 16:10:09.578 - 0000003c>-ERR Unrecognised AUTH type
15/1/2006 16:10:09.578 - 0000003c<USER uus
15/1/2006 16:10:09.578 - 0000003c>+OK User Accepted, PASSword required
15/1/2006 16:10:09.578 - 0000003c<PASS uuspassu
15/1/2006 16:10:09.593 - 0000003c>+OK uus has 7 message(s) (2170 octets)
15/1/2006 16:10:09.640 - 0000003c<RETR 10
15/1/2006 16:10:09.640 - 0000003c>-ERR Invalid Message Number
15/1/2006 16:10:09.640 - 0000003c<QUIT
15/1/2006 16:10:09.656 - 0000003c>+OK VPOP3 Server Closing Connection

Inbox shows the error that has been encountered by still keeping the mail in 'to-be-downloaded' state. This mail will be later downloaded upon manual connection, when the new UIDL is read and its new position is, accordingly, recomputed by Messaging. 4. the user downloads the body of a mail that has already been "shifted" down but of which the original serial number is still valid. That is, as message 4, 7 and 8 have been deleted, we choose the fifth mail (the one with '4' in the subject) to be downloaded. This mail has shifted by one position downwards because of the mail that has been deleted under it and, therefore, the server will return the body for the mail that has '5' in it, instead of the requested mail with '4':

15/1/2006 16:18:29.406 - 0000003d>+OK VPOP3 Server Ready <3d.13d4.e88a4ae>
15/1/2006 16:18:29.437 - 0000003d<CAPA
15/1/2006 16:18:29.437 - 0000003d>+OK Capability list follows
TOP
USER
UIDL
SASL CRAM-MD5
.
15/1/2006 16:18:29.437 - 0000003d<AUTH
15/1/2006 16:18:29.437 - 0000003d>-ERR Unrecognised AUTH type
15/1/2006 16:18:29.437 - 0000003d<USER uus
15/1/2006 16:18:29.453 - 0000003d>+OK User Accepted, PASSword required
15/1/2006 16:18:29.453 - 0000003d<PASS uuspassu
15/1/2006 16:18:29.468 - 0000003d>+OK uus has 7 message(s) (2170 octets)
15/1/2006 16:18:29.500 - 0000003d<RETR 5

As can be seen, the original serial number (the number originally used when the starting rows of the mail were first downloaded) was used to refer to the mail. This also means the server will return a mail that was later; in this case, the one that has '5' in the subject/body, instead of the one with '4':

15/1/2006 16:18:29.500 - 0000003d>+OK 310 octets
15/1/2006 16:18:29.500 - 0000003d>Received: from 127.0.0.1 (UIDL=0:1838254324449018) by winmobiletech.com (VPOP3) with POP3; Sun, 15 Jan 2006 16:07:44 +0100
15/1/2006 16:18:29.500 - 0000003d>From: 5
15/1/2006 16:18:29.500 - 0000003d>To: Menneisyys <a@b>
15/1/2006 16:18:29.500 - 0000003d> Date: Fri, 13 Jan 2006 00:13:07 (GMT)
15/1/2006 16:18:29.500 - 0000003d>X-VPOP3-Spam: 12.5 - SuspiciousHeader(12)
15/1/2006 16:18:29.500 - 0000003d>X-VPOP3-Spam2: added bayes ham
15/1/2006 16:18:29.500 - 0000003d>X-VPOP3-SpamBayes: 0
15/1/2006 16:18:29.500 - 0000003d>Subject: 5
15/1/2006 16:18:29.500 - 0000003d>
15/1/2006 16:18:29.500 - 0000003d>
15/1/2006 16:18:29.500 - 0000003d>5
15/1/2006 16:18:29.500 - 0000003d>.
15/1/2006 16:18:29.531 - 0000003d<QUIT
15/1/2006 16:18:29.546 - 0000003d>+OK VPOP3 Server Closing Connection