Subject: [sup-talk] Using DTrace to profile sup-sync

From: [email-masked] (Jjgod Jiang)
Date: Sat, 15 Dec 2007 23:05:04 +0800

Hi,

When I was trying to build index for my GMail account (which has 28525
mails in inbox)
with sup-sync, I found the estimated finish time is too long (24
hours) to accept. So I
decided to use DTrace to do some profiling work on sup-sync, and here
is the result.

The top n method spent most of the time is:

___ OVERLAP TIMES: ___                                    ______ ELAPSED ______
CLASS                    METHOD                   COUNT    AVG(us)      SUM(us)

Net::IMAP::ResponseParse next_token              570993        282    161345198
Net::IMAP::ResponseParse lookahead               770871        283    218826568
Class                    _parse                   28612       8051    230376494
Class                    parse                    28902       9105    263159874
Net::IMAP::ResponseParse match                   485344        563    273536106
Redwood::IMAP            make_id                  28534       9985    284928748
Net::IMAP::ResponseParse msg_att                  28534      11423    325954315
Array                    each                     86883       3819    331861324
Net::IMAP::ResponseParse numeric_response         28538      15239    434895984
Net::IMAP::ResponseParse response_untagged        28551      17003    485460410
Net::IMAP::ResponseParse response                 28566      19149    547032919
Net::IMAP::ResponseParse parse                    28566      19233    549418654
Net::IMAP                get_response             28551      19922    568811230

I can send the complete result if needed. HTH.

BTW: it only took mutt half an hour to fetch all the mail headers.

- Jiang

---

From: [email-masked] (William Morgan)
Date: Sat, 15 Dec 2007 11:25:45 -0800

Excerpts from Jjgod Jiang's message of Sat Dec 15 07:05:04 -0800 2007:
> mails in inbox) with sup-sync, I found the estimated finish time is
> too long (24 hours) to accept. So I decided to use DTrace to do some
> profiling work on sup-sync, and here is the result.

Thanks for the analysis! I am very interested in speeding up this sort
of thing. However, it looks like the culprit here is the Ruby IMAP
library, which presumably is so much slower than Mutt because it's Ruby.

So probably the only significant way to speed this up would be to use a
different IMAP library. If there's a C IMAP library with a Ruby bridge,
I'll consider hooking it in, but I don't know of one...

-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (William Morgan)
Date: Sat, 15 Dec 2007 11:38:43 -0800

Oh, the other difference between what sup-sync does and what Mutt does
is that sup-sync downloads the entire content of the message, parses it
for text regions, and stores those in the Ferret index. That's
significantly more work than just downloading the headers, though I
don't know how much that accounts for the difference in speed.

---

From: [email-masked] (William Morgan)
Date: Sat, 15 Dec 2007 13:04:10 -0800

Excerpts from Jjgod Jiang's message of Sat Dec 15 07:05:04 -0800 2007:
> too long (24 hours) to accept.

Actually, large IMAP folders will always present a significant problem
for Sup, because IMAP sucks balls. Short story: Sup has to *always*
download the headers of the entire mailbox in order to function
correctly. So every time you start up Sup you're going to be sitting
there, waiting for that to happen, before you can do anything with your
IMAP messages.

Long story: This is because IMAP provides no way of getting a
consistent, cross-session identifier for a single message, which is
what Sup needs to retrieve message content when you view a thread. You
can read more about why IMAP's UID is useless.

(Actually, what Sup really needs is a unique, cross-session identifier
for messages, which increments as messages get newer, so that it can
tell when new messages have appeared in the inbox. In case you were
wondering, the IMAP \Recent flag is useless for this.)

So, to get around this, Sup has to construct its own unique message id,
based on the size and the internal date of the message. It then maps
this internal id into the server's message id, and can finally both
retrieve any message in the index from the server, and can check for new
messages since the last time it connected to the server.

So, if you really have a large IMAP account that you want to index with
Sup, the solution is to convert it to mbox. That will solve all your
problems. Don't blame Sup, blame IMAP.

---

From: [email-masked] (William Morgan)
Date: Tue, 18 Dec 2007 20:38:33 -0800

Excerpts from William Morgan's message of Sat Dec 15 13:04:10 -0800 2007:
> there, waiting for that to happen, before you can do anything with
> your IMAP messages.

It might be possible for this not to be the case for well-behaved IMAP
clients, i.e. those that don't change UIDVALIDITY often. If I store the
UIDs directly, then no header pull would be necessary, but a change in
UIDVALIDITY would require a full rescan a la sup-sync --changed.