Subject: [sup-talk] header cache

From: [email-masked] (Giorgio Lando)
Date: Tue, 15 Jan 2008 13:54:44 +0100

Hi,
after ten days of successful use of sup. I would like to say that it
is very useful in many circumstances (I have tried maildir, mbox, IMAP
folders, and a mix of them).

When using it with gmail imap, the only thing I lack is the header
cache. In fact, I access directly the "All mail" folder (since
anything else in gmail is actually a duplicate of the mails in All
mail; moreover in this way any change by sup does not interfere with
the archiviation from the gmail web interface, and the archiviation
from the web interface does not prevent sup from seeing the message).

My "All mail" includes approximately 40000 messages so the fetching of
IMAP headers is highly time-consuming (also the initial sup-sync took
about 8 hours, but this happens only one time).

I see that in TODO the header cache is scheduled for the 0.4 release.
So am I correct to expect it quite soon?

Thanks in advance for any info
Giorgio Lando

---

From: [email-masked] (William Morgan)
Date: Tue, 15 Jan 2008 19:24:47 -0800

> So am I correct to expect it quite soon?

Yes, I have a stub implementation in the works that should be ready for
testing soon. I'll focus more attention on it now that I have a
customer. :)

-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Giorgio Lando)
Date: Wed, 16 Jan 2008 21:26:44 +0100

> testing soon. I'll focus more attention on it now that I have a

Fine, ready for eventual testing, although I have finally decided to
sync locally the 40000 messages with another tool, so to have them
indexed and opened by sup in a faster way. Anyway I guess that a cache
helps only relatively with such a monster (or at least also mutt's
header cache helps only a bit with 40000 messages in an IMAP folder), 
so I will be able to make tests with other, much smaller imap folders.
Giorgio

---

From: [email-masked] (William Morgan)
Date: Wed, 16 Jan 2008 12:48:47 -0800

Reformatted excerpts from Giorgio Lando's message of 2008-01-16:
> Anyway I guess that a cache helps only relatively with such a monster
> (or at least also mutt's header cache helps only a bit with 40000
> messages in an IMAP folder), so I will be able to make tests with
> other, much smaller imap folders.

If you look at the log messages, there's one at the emitted at the
beginning of loading the headers, and one at the end, so you can see
exactly how much time adding a cache will save you (minus however long
it takes to load things in from disk).

I'm curious what this number is for 40000 headers. It seems to be about
10 seconds for my 5000-message IMAP folder, so I can imagine it being
quite significant.

---

From: [email-masked] (Giorgio Lando)
Date: Wed, 16 Jan 2008 23:04:26 +0100

> 10 seconds for my 5000-message IMAP folder, so I can imagine it being

I guess you are referring to mutt: in this case I can confirm that the
cache (mutt is compiled with gdbm) saves about 80 secs.

What I meant is that it is nonetheless very slow, and that this is why
one SHOULD NOT ordinarily access an IMAP folder with 40000 headers. In fact I
do not do this in mutt, where on the contrary I enter INBOX and change folder only when
needed (so only seldom I change to All Mail). I can not do exactly the same 
with sup: either I pull mails only from the INBOX, but then everything already 
archived in gmail through the web interface will
not be seen by sup, or I go with All Mail, but then I have to load each
time 40000 headers, which is extra slow (and would be very slow also with an header cache).
Finally, there would not be any point to pull mails from all the other folders/labels, 
because they would include only duplicates of mails in All Mail and they
would not enrich sup index at all.

I have also the feeling that sup's philosophy is not very consistent
with IMAP's philosophy, which is to access the same structure of folders
from different interfaces, syncing immediately any operation on the server, 
so that folders and labels are exactly the same from any interface.
Sup follows its own philosophy: thus, it does not deal with folders at all, 
and (as you admit often) does not aim to play well with other ways to check emails.
Gmail's imap is not an exception to this conflict between sup's
philosophy and imap's philosophy, because gmail's labels are, when you
access gmail through IMAP, nothing else than folders!

So I have finally decided to go back to POP3, download all mails (including
mails sent from the web interface, in the case of gmail) locally and let sup
index and open them locally. The mails are anyway kept on the server, so
I can also access them through a traditional IMAP client with folders
(such as mutt) or through the web interface when I am around the world
without my laptop.
In this way, I can use sup in a fast way without any bad interplay with
other ways of checking emails. This is very important for me because I
travel a lot and it
is not possible for me to access emails exclusively from my own
computers with sup installed :) .

Thus I do not think that I will go back to access gmail through IMAP
with sup. Anyway I hope to be able to do some testing of the IMAP headers cache 
with other small IMAP servers I have access to.

Giorgio

---

From: [email-masked] (Alec Berryman)
Date: Wed, 16 Jan 2008 16:58:17 -0600

> (such as mutt) or through the web interface when I am around the world

Have you looked at offlineimap?  I started using it with mutt because
IMAP, even with header cachine, was just way too slow.

---

From: [email-masked] (Giorgio Lando)
Date: Thu, 17 Jan 2008 00:36:55 +0100

> IMAP, even with header cachine, was just way too slow.

Yes, I have used it and isync: you are right,they solve the problem of the slowness, 
since all the syncing happens in the background when using mutt or sup. 

But still I think that, when one does not care about folders and about
synchronization of different interfaces, the intrinsic advantages of
IMAP over POP3 tend to disappear. On the contrary there is the risk of
strange interplay (such as sup changing the flags in way that
offlineimap syncs to an IMAP server which does not understand or does
strange things); a solution is to use isync as a push-only synchronizer
(only sync the changes from the server to the local machine and not the
opposite), but really in this way you loose all the advantages of IMAP
and I prefer to use well-tested and powerful fetchers for POP3, such
as fdm and getmail.

Obviously if on the contrary for some reason you are forced to use IMAP,
then offlineimap and isync are the best solution

Giorgio

---

From: [email-masked] (Nicolas Pouillard)
Date: Thu, 17 Jan 2008 09:13:21 +0100

Excerpts from Giorgio Lando's message of Wed Jan 16 23:04:26 +0100 2008:

[...]

> (such as mutt) or through the web interface when I am around the world

I use this setup too, using mbox and it's quite fast.

-- 
Nicolas Pouillard aka Ertai

---

From: [email-masked] (Giorgio Lando)
Date: Thu, 17 Jan 2008 12:40:47 +0100

> I use this setup too, using mbox and it's quite fast.

Yes, it is fast also with maildir! Quite satisfied
Giorgio

-- 
Giorgio Lando <patroclo7 at gmail dot com>

---

From: [email-masked] (Nicolas Pouillard)
Date: Thu, 17 Jan 2008 15:56:19 +0100

> Yes, it is fast also with maildir! Quite satisfied

I've also tried maildir for some time, and found it slower.

That's because Sup needs to compute a hash from file names to message ids.

---

From: [email-masked] (William Morgan)
Date: Mon, 21 Jan 2008 18:36:48 -0800

> That's because Sup needs to compute a hash from file names to message

Like IMAP headers, this is a one-time cost upon startup and could easily
be cached. Maildir shouldn't be significantly slower than mbox except
for this.

---

From: [email-masked] (Nicolas Pouillard)
Date: Tue, 22 Jan 2008 09:40:05 +0100

> be cached. Maildir shouldn't be significantly slower than mbox except

Looking  at  the  code  'scan_mailbox' seems to be called quite often (but not
more than every 30 seconds).

I'm  wondering  if  peeking  a  file  in  a very large directory is as fast as
seeking to a particular offset in a large file?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 186 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/sup-talk/attachments/20080122/41112479/attachment.bin

---

From: [email-masked] (Gabriel Sean Farrell)
Date: Wed, 23 Jan 2008 15:33:07 -0500

> seeking to a particular offset in a large file?

If you're really interested in speed comparisons between mbox and
maildir, http://www.courier-mta.org/mbox-vs-maildir/ is a good read.

---

From: [email-masked] (William Morgan)
Date: Thu, 24 Jan 2008 19:28:07 -0800

[Sending to list. We accidentally fell off.]

Reformatted excerpts from nicolas.pouillard's message of 2008-01-23:
> Excerpts from William Morgan's message of Wed Jan 23 08:04:27 +0100 2008:
> > Reformatted excerpts from nicolas.pouillard's message of 2008-01-22:
> > > Looking  at  the  code  'scan_mailbox' seems to be called quite
> > > often (but not more than every 30 seconds).
> > 
> > Oops, you're exactly right.
> > 
> > > I'm  wondering  if  peeking  a  file  in  a very large directory
> > > is as fast as seeking to a particular offset in a large file?
> > 
> > Actually, we could short-circuit this check trivially by comparing
> > the directories' mtimes. Then this really shouldn't be a slowdown.
> 
> You will still scan it at when receiving new mails...

That's true. But some kind of scan is unavoidable. We need to know which
files are new, and one way or another that will involve pulling in the
whole list of files.

I think we can do a couple easy things to speed up Maildir dramatically.
Skip the scan when the mtime is unchanged, maybe increase the delay,
stop repeating make_id calls for messages we've already processed...
basically the current implementation is a little naive and I'm not ready
to give up on Maildir quite yet.

> What about having a custom mail box format for sup?
> 
> For  instance  I think the only drawback (in the context of using sup)
> of mbox is when you often backup you mails. Why not having a bunch of
> mbox using dates inbox-12-2008.mbox? Or a directory where file names
> are sup message ids?

What about mbox makes it difficult to do backups?

The amount of work for a custom format seems really high. (Look at all
the design considerations that went into Maildir!) And ultimately your
MTA has to be able to write that format, or there has to be a conversion
script, which has to be run every 30 seconds, and has to be fast, and
then we have the same set of problems we have now but in a different
place.

---

From: [email-masked] (Nicolas Pouillard)
Date: Fri, 25 Jan 2008 09:50:20 +0100

> to give up on Maildir quite yet.

What about having two directories? new and cur for instance :)
You always have to scan new and move them to cur when they are in the index.

> What about mbox makes it difficult to do backups?

Every  hour,  if  my  backup  drive is plugged in, my OS make a new tree of my
entire  disk  by  hard  linking  what's  haven't  change and by coping others.
However  this  behave particularly bad with files that just keep growing every
hour ~400MB at this day.

That's  why  splitting  out  my  big  mail  box in small ones, let's say every
month, would better behave w.r.t. to backups.

> then we have the same set of problems we have now but in a different

Your  MTA  call a MDA like maildrop, and making a specific maildrop in ruby is
pretty easy.

Best regards,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 186 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/sup-talk/attachments/20080125/cd24b771/attachment.bin