Is Qmail more efficient than Messaging, bandwidth usage-wise? (Technical, "geeky" stuff inside!)
I’ve just been asked the following question in my Mailer Roundup (also see this XDA-Dev thread for more info), which made me do some real experiments that may be of interest to some other readers too:
“Thanks for your roundup; it got me to try both npop and qmail. For your roundup update, I humbly suggest you look at data use. I was alerted to this issue when hit with GPRS roaming charges that were much higher than I experienced with Snappermail (a Palm only program).
I've done some research since, and found that the native WM5 messaging is simply very inefficient, especially if there are a lot (by which I mean 30 to 50, not 1000) of messages on the server. I started looking for a new program not because I was unsatisfied with WM5 messaging but because the data use was astronomical. Using SPB GPRS Monitor, I have found that nPOP is the most efficient, followed by QMail; both of these average less than a third of the data use of WM5 Messaging. Flexmail is no better than WM5 Messaging in this regard.
The big problems with nPOP from my perspective are the inability to search contracts for a name (one just gets a list and must scroll through them) and no one handed support -- it would be nice to pop up menus with the soft keys. That would be a useful thing.
Thanks again for your survey, which helped me tremendously. ”
So, here are the results of my looking into the problem (I’ve used the same tools as in my article Found the explanation for/solution to probably the nastiest Pocket Inbox bug!):
POP3 mail retrieving-wise,
Qmail is a bit more bandwidth-friendly in the case of you connecting to your mailbox containing a LOT of already-read mail and there weren’t any deletions since the last mail fetch. Then, Qmail only gets the so-called UID of the last (known) mail (referring to it with its serial numer), while Messaging gets the entire list of it.
Let’s assume you have 66 mails on your mailbox and you’ve already downloaded to your Messaging / Qmail 36 of them; that is, there are 30 new mails. In this case, a typical communication log will be the following:
Messaging (WM5 AKU2 - that is, the latest version):
19:37:02.125 - 0000000b>+OK VPOP3 Server Ready <b.6a8.1bd9584>
19:37:02.312 - 0000000b<CAPA
19:37:02.312 - 0000000b>+OK Capability list follows
TOP
USER
UIDL
SASL CRAM-MD5
.
19:37:02.328 - 0000000b<AUTH
19:37:02.328 - 0000000b>-ERR Unrecognised AUTH type
(BTW, AUTH is still undeeded here – particularly because the server didn’t understand it.)
19:37:02.328 - 0000000b<USER Menneisyys
19:37:02.343 - 0000000b>+OK User Accepted, PASSword required
19:37:02.343 - 0000000b<PASS passu
19:37:02.546 - 0000000b>+OK Menneisyys has 66 message(s) (68639 octets)
19:37:02.875 - 0000000b<STAT
19:37:03.375 - 0000000b>+OK 66 68639
Now, it’s known to Messaging that there’re 66 messages in the mailbox. It, nevertheless, gets the UID's of all mails:
19:37:03.578 - 0000000b<UIDL
Note that Messaging uses the parameter-less UIDL command; that is, it gets all the UID’s as can be seen in the response:
19:37:03.578 - 0000000b>+OK
19:37:03.578 - 0000000b>1 2
19:37:03.578 - 0000000b>2 3
19:37:03.593 - 0000000b>3 5
…
19:37:03.703 - 0000000b>65 88
19:37:03.703 - 0000000b>66 89
19:37:03.703 - 0000000b>.
Now, Messaging downloads the headers and the first 26 rows of the body of the last 30 messages, in decreasing order and, then, QUITs (note that Messaging in WM5 doesn't - optionally - use persistent connections any more, unlike with previous operating systems; that is, it always QUITs after POP3 sessions):
19:37:04.281 - 0000000b<LIST 66
19:37:04.281 - 0000000b>+OK 66 249
19:37:04.296 - 0000000b<TOP 66 26
19:37:04.296 - 0000000b>+OK 249 octets
…
19:37:18.046 - 0000000b<LIST 37
19:37:18.046 - 0000000b>+OK 37 842
19:37:18.062 - 0000000b<TOP 37 26
19:37:18.062 - 0000000b>+OK 842 octets
...
19:37:18.671 - 0000000b<QUIT
On the other hand, Qmail does the following (login):
19:38:27.546 - 0000000c>+OK VPOP3 Server Ready <c.f7c.1bee332>
19:38:27.750 - 0000000c<USER Menneisyys
19:38:27.750 - 0000000c>+OK User Accepted, PASSword required
19:38:27.890 - 0000000c<PASS passu
19:38:27.906 - 0000000c>+OK Menneisyys has 66 message(s) (68639 octets)
19:38:27.906 - 0000000c<STAT
19:38:28.406 - 0000000c>+OK 66 68639
19:38:28.531 - 0000000c<UIDL 36
19:38:28.531 - 0000000c>+OK 36 49
See the difference between Qmail and Messaging? Qmail assumes mail numbers haven’t changed in the meantime and tries to reduce communication costs by just getting the last-known mail UID to make sure this is the case (this is why it uses the parametrized UIDL command). This approach is, in a way (we’ll also see in what it’s worse) better than that of Messaging because if you DELEte a mail from a POP3 mailbox (let’s assume you delete mail that have the indexes 13…17 and are, therefore, before the mail with index 36) then, all the mails after the deleted mail will take a new number. That is, by simply checking the UID of the last-downloaded (last-known) index, you can safely know whether there were deletions. (Note that an once-assigned UID will never ever be given to any other mail.)
Now that Qmail knows there were no deletions, it can safely go on fetching mails starting with the 37th one:
19:38:28.546 - 0000000c<UIDL 37
19:38:28.546 - 0000000c>+OK 37 51
19:38:28.546 - 0000000c<LIST 37
19:38:28.546 - 0000000c>+OK 37 842
19:38:28.562 - 0000000c<TOP 37 0
19:38:28.562 - 0000000c>+OK 842 octets
…
19:38:38.546 - 0000000c<UIDL 66
19:38:38.546 - 0000000c>+OK 66 89
19:38:38.671 - 0000000c<LIST 66
19:38:38.671 - 0000000c>+OK 66 249
19:38:38.671 - 0000000c<TOP 66 0
19:38:38.671 - 0000000c>+OK 249 octets
…
19:38:39.218 - 0000000c<QUIT
Incidentally, as the UIDL POP3 command only has two versions (a parameter-less when it returns all the available mail UID’s and a parametrized when it returns the UID of just one – that is, there are no range requests like ‘return the UID’s of all mails starting with the 37th’), this approach only pays off in the case of having a LOT of old mails on your pop3 account AND frequently checking for (only few) new ones. When you always delete your mail (and you get many of them all the time), the additional bandwidth usage caused by Qmail’s approach can easily outweigh the savings it results in by using the parametrized UIDL when there were no deletions (if there were, it uses the parameter-less “UIDL” command to completely refresh its internal UIDL database). The reason for this is simple: it needs to get each mail message’s UIDL separately, with issuing a UIDL command for each of them.
By default, Qmail fetches only the mail headers, unlike Messaging
Therefore, the problem you’re facing may be related to the message retrieve settings. Are you downloading equally much data? Messaging, for example, downloads the first 26 rows (see the TOP <message number> 26 commands in the log) with the default, “retrieve 2 kbytes” setting. Qmail, by default, only retrieves the headers (see the TOP <message number> 0 command in the log); it’s only after you configure it to download message bodies that it will download any mail body. When you only allow Messaging to download mail headers, it’ll use the same TOP <message number> 0 command as Qmail. Are you sure you didn’t leave the two applications, when comparing them, using their default settings?
Printer-friendly version
In my test, I had WM5 Messaging set to download a maximum of 4KB, and QMail to download a maximum of 50 lines. So the increaased efficiency of QMail didn't result from the size of the downloaded mail being smaller.
However, I think the autofetch problem in WM5, described in your linked article, is precisely why I had a data use problem that doesn't arise with Qmail. I'm going to try turning autofetch off and see if that fixes the problem, even when I manually check frequently.
It is interesting that the programs are so different.
One more quick question: if I use skschema to manually check mail with Messaging, the fetching of unnecessary emails won't arise? If so, I can program skschema to check my email every half hour or so, and avoid the autofetch problem.
You can surely use SKSchema - but why? Messaging already has excellent auto-fetch capabilities.
If you do use auto-fetch, however, it'll fetch everything (at least the headers of new mails, depending on the actual setting).
I was thinking of using skschema to avoid the autofetch problem you described in the linked article, where the wrong message is downloaded. I regularly use two PCs as well as my MDA to check mail, and I was envisioning a solution for this.
BTW, if you fetch your mail very often (say, every 2-3 minutes), the above-explained behaviour (Messaging always gets the full UID list of all the messages, unlike Qmail, which only gets the last) can result in a considerable bandwidth consumption. For example, if you have some 30 mails in your mailbox, the UID list itself can easily take 400-500 bytes (a record takes 8-15 bytes, depending on how the POP3 server gives out UID's etc). If you have even more mails online, the situation will be even worse. (Of course, a lot depends on the packet size used in the communication too.)
I will speak to the Microsoft Messaging folks on, for example, adding a new checkbox to switch to a Qmail-like behavior. It'd RADICALLY reduce the consumed bandwidth in this (many mails in the mailbox and frequent connections to check whether there are new mails) case.
Mr. Ruotsalainen,
Thanks for the great blog, and all you do for the mobile community. I'm still having problems setting up Qmail, and I know that you once mentioned providing an instruction guide. Is there any way you could link to that when you have a moment? Thanks a lot!
NO tutorial as of yet, due to lack of time. However, exactly what do you have problems with? In the meantime, I've posted a lot of IMAP set-up tips and Modern Nomads have published some step-by-step tutorials to set up mailboxes.
Hi Werner,
How about a Gmail booklet, if not a bible. That is, when you can find some spare time (lol).