Subject: [sup-talk] Slow import of messages

From: [email-masked] (Nico Schottelius)
Date: Mon, 13 Jul 2009 11:15:16 +0200

Hello!

I'm looking at sup as a replacement for mutt.
Following the idea of sup, I took a snapshot of
my Mails from 2008-2009 (~56k mails) and run

   sup-sync --all-sources

Now the problem is that sup gets about 3 mails per
second into its index. That means it takes about

	56k/3 ~= 15k; 15000/60 = 250 minutes

That import is already from a local Maildir (synced with
offlineimap).

For me, this is incredibly slow. Using mutt it's way faster,
not even comparing the fact the speed improvement you get
when dividing into folders.

What's your impression? Am I totally wrong or is sup just
the wrong tool for handling a lot of messages?

Sincerly,

Nico

---

From: [email-masked] (Michael Stapelberg)
Date: Mon, 13 Jul 2009 11:26:14 +0200

Hi Nico,

> my Mails from 2008-2009 (~56k mails) and run
I have about 33k emails on my IMAP server in my LAN.

> second into its index. That means it takes about
I got around 10 MB/s performance using sup-sync. Maybe
your mails are all very large, I don?t know. Maybe maildir is
just a lot slower than IMAP in sup-sync.

> the wrong tool for handling a lot of messages?
My impression is that sup-sync can definitely be improved regarding
its speed. I?m not sure if it is actually doable or if it is some ruby
library (Ferret?) which is the limiting factor here. However, it was
not as bad for me as it is for you ;-).

Best regards,
Michael

---

From: [email-masked] (Jim Cheetham)
Date: Tue, 14 Jul 2009 09:22:04 +1200

On Mon, Jul 13, 2009 at 9:15 PM, Nico
> the wrong tool for handling a lot of messages?

No, it's just slow to index for the first time. Unless you receive
1000s of messages per hour, you should be OK ...

-jim

---

From: [email-masked] (William Morgan)
Date: Tue, 28 Jul 2009 12:10:57 -0700

Hi Nico,

Reformatted excerpts from Nico Schottelius's message of 2009-07-13:
> Now the problem is that sup gets about 3 mails per second into its
> index.

As others have pointed out, messages only have to be indexed once, so
this isn't ultimately that big of a deal. But 3m/s seems very slow for a
Maildir source.

With my mbox on my reasonably-fast laptop, I start with around 50m/s
using Ferret, and at 80m/s using Xapian. (These numbers degrade slightly
as the index gets fuller.) I would expect similar numbers from Maildir.

So something weird is happening for you. Do you have a notion of whether
it's disk or CPU that's the bottleneck?
-- 
William <wmorgan-sup at masanjin.net>