Subject: [sup-devel] [Heliotrope/Turnsole] How to use IMAP?

From: William Morgan <[email-masked]>
Date: Sun, 12 Feb 2012 22:16:33 -0800

Excerpts from Michael Stapelberg's message of 2012-01-08 14:48:18 -0800:
> What is the correct way to do this?

Please try the latest heliotrope master. You can do a full import with this:

  ruby -Ilib bin/heliotrope-import -a <maildir> -d <mailstore> -t <state file>

where <state file> is just a place for it to dump its state. This import need
only happen once. Then for subsequent additions you can do this:

  ruby -Ilib bin/heliotrope-add -a <maildir> -d <mailstore> -t <state file>

which will read to and write from that state file, and only add what it
hasn't yet seen.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Michael Stapelberg <[email-masked]>
Date: Thu, 23 Feb 2012 12:24:22 +0100

Hi William,

Excerpts from William Morgan's message of 2012-02-23 07:25:11 +0100:
> My understanding of the Maildir specification is that there is no way to
> determine the order of messages besides reading the date headers.
> Ordering messages correctly at import time is important because
> Heliotrope only serves messages in the reverse order to which they were
> imported. If there is a better solution, please feel free to educate me.
Well, I guess you are correct. So, when somebody sends me a spam email or a
malicious email with a faked date, what happens? I think that the code will
figure out it needs to re-add a lot of messages. Also, my index will need to be
reordered, right?

Also, what happens when somebody sends me a message with a faked date and I add
it using heliotrope-add? Do I need to reorder my index?

> Is there any problem with the current code besides the fact that you
> don't like the big scan?
That makes it sound like it’s just a matter of opinion. My concern is what I
stated above (huge processing load / disk IO caused by malicious messages) and
that it thrashes my poor disks.

Best regards,
Michael

---

From: William Morgan <[email-masked]>
Date: Fri, 24 Feb 2012 22:35:32 -0800

Excerpts from Michael Stapelberg's message of 2012-02-23 03:24:22 -0800:
> Well, I guess you are correct. So, when somebody sends me a spam email
> or a malicious email with a faked date, what happens? I think that the
> code will figure out it needs to re-add a lot of messages. Also, my
> index will need to be reordered, right?

You won't need to reorder. Importing is all incremental, at least if you
use --state-file: it will keep track of the last message imported, and
successive imports will only pick up new messages.

But you are making me realize that the current maildir implementation is
not right. I think the way to handle Maildir is to look only at the
ctime of the files and not the date headers. That will make incremental
importing possible (just save the filename of the last imported file,
and look for all files newer than that). It will speed things up
anyways.

And if you're in the funny situation where the ctime is not correlated
with the Date: headers in your files, which is perfectly possibly by the
Maildir spec, then you will have to reorder after that initial import.
But in reality that's a possibility with the other source types
too---it's just more likely with Maildir.

---

From: Michael Stapelberg <[email-masked]>
Date: Mon, 06 Feb 2012 20:45:56 +0000

Hi William,

Excerpts from William Morgan's message of 2012-01-10 00:40:56 +0000:
> I need to write some code to do this. For IMAP and GMail, heliotrope-add will
> keep a pointer to thelast message imported, by default. For mbox there is a
> trick you can use. But there's nothing for maildir right now. Please file
> an issue so that I don't forget this!
It’s been four weeks, and https://github.com/wmorgan/heliotrope/issues/32 is
still open. Just to let you know: This is what’s stopping me from using
heliotrope+turnsole and possibly participating in its development at the moment
:-/.

If it’s a task which is too big for your current spare time, could you please
describe how you would implement it, so that others can help?

Best regards,
Michael

---

From: William Morgan <[email-masked]>
Date: Wed, 22 Feb 2012 22:25:11 -0800

Excerpts from Michael Stapelberg's message of 2012-02-22 13:39:42 -0800:
> Thanks for adding code for this. I got the chance to test this today,
> and I’m afraid this doesn’t seem to be working as I would expect.
> 
> As far as I can tell from glancing at the source and the state file, this code
> examines the date stored in every mail, sorts that, then checks where it
> stopped before, and continues from there. I think this is a horrible way of
> doing things for two reasons:
> 
> 1) It involves opening every single email on disk.
> 2) Date specifications in emails are not to be trusted (think spam).

> Note that adding mails via stdin doesn’t work until you change the code like this:
> https://github.com/wmorgan/heliotrope/pull/34

Merged, thank you.

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Tue, 7 Feb 2012 22:54:08 +0100

Hi Michael,

On Tue, Feb 7, 2012 at 8:11 PM, Michael Stapelberg
<[email-masked]> wrote:
>  I’m not sure what you are trying to say.

Ok, I think I know what you mean.

You currently have a main IMAP server. You regularly sync it to your
notebook in maildir with offlineimap. You would like to use this
maildir directly with heliotrope. Right ?

So here are two options I can see for you:
* keep the maildir-backuping in a corner and sync your main IMAP
server directly with imaptrope ("heliotrope's" IMAP interface). This
would need two runs of offlineIMAP.
* use offlineimap to sync your maildir to imaptrope. This might create
problems, because the 2 syncings might interfere with each other.

-- 
Matthieu RAKOTOJAONA

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Tue, 7 Feb 2012 13:27:32 +0100

On Tue, Feb 7, 2012 at 12:46 AM, Michael Stapelberg
<[email-masked]> wrote:
> Uhm, that’s not what I am talking about. I want to fetch mails from an IMAP
> server and store them in heliotrope.

Yes, you can use offlineimap to sync 2 IMAP servers, although this is
still a work in progress. Or you can use 1-way copy tools, such as
imapcopy[0].

Oh, and heliotrope does tell you when it already has an email, based
on its Message-Id header.

[0] imapsync.lamiral.info/

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Tue, 7 Feb 2012 00:08:33 +0100

Hi,

> describe how you would implement it, so that others can help?

As far as I can tell, IMAP will not be implemented in heliotrope. I
have started some work on a IMAP frontend which you can fork from :

https://github.com/rakoo/imaptrope

You use it as a separate process. It will create a RESTClient
connected to heliotrope, and you just connect to it with classical
IMAP clients.

---

From: Michael Stapelberg <[email-masked]>
Date: Mon, 27 Feb 2012 10:16:43 +0100

Hi William,

Sorry for clinging to that topic, but it’s important for me to properly
understand it.

> successive imports will only pick up new messages.
Apparently, I didn’t properly describe the scenario I’m thinking about:

state-file contains an entry "/my/maildir/2012-02-22-foo-bar", which is an
email that contains 2012-02-22 as date.

Now, a spammer sends me an email with a faked date, let’s say 2001-01-01, let’s
call it "/my/maildir/2001-01-01-spam-mail". The next run of heliotrope-add has
two possibilities:

1) It will completely ignore it. I think this is what will happen, based on
   your description. This is horrible! Emails are absolutely not guaranteed to
   arrive in my system in the order they were sent. Is this the case? Do we
   ignore email because of date problems like this currently?

2) It will pick up this email because the date inside is older. Subsequent runs
   will discover all email from the last 10 years as new, and try to re-add it
   to the index. Lots of unnecessary overhead.

> and look for all files newer than that). It will speed things up
Right, that sounds good.

> too---it's just more likely with Maildir.
So, let’s stick to my above example of the spam email and let’s assume that you
changed the code to use ctimes. The spam mail arrives, its ctime is new, it
gets picked up into my index. To my understanding, I now need to reorder. I
have multiple questions:

1) What are the immediate consequences? Where will this email appear in my
   inbox, when my inbox contains 5 emails. Always at the top? Bottom? You
   mentioned I can see the effect when paging. Can you elaborate please?

2) How do I know when I have to reorder?

Best regards,
Michael

---

From: William Morgan <[email-masked]>
Date: Tue, 28 Feb 2012 17:11:56 -0800

Excerpts from Michael Stapelberg's message of 2012-02-27 01:16:43 -0800:
> 1) It will completely ignore it.

Correct. This is the problem that the ctime solution is meant to address.

> So, let’s stick to my above example of the spam email and let’s assume
> that you changed the code to use ctimes. The spam mail arrives, its
> ctime is new, it gets picked up into my index. To my understanding, I
> now need to reorder.  I have multiple questions:
> 

Heliotrope will return it as the first result, since it returns messages
in LIFO order.

Currently Turnsole currently re-sorts all messages returned by sort by
date, and so it will place it at the bottom. This is also where the
pagination issues may come from. Maybe it would be less confusing for it
to stick to search result order.

> 2) How do I know when I have to reorder?

You have to reorder whenever you want Heliotrope to return messages in
Date header order instead of in LIFO order. I think this will typically
happen when you import a couple batches of preexisting mail, and not
when you're adding new mail.

---

From: Michael Stapelberg <[email-masked]>
Date: Mon, 06 Feb 2012 23:46:17 +0000

Hi Matthieu,

Best regards,
Michael

---

From: Michael Stapelberg <[email-masked]>
Date: Tue, 07 Feb 2012 19:11:00 +0000

Hi Matthieu,

> still a work in progress. Or you can use 1-way copy tools, such as
I use offlineimap to sync my IMAP server at home with my notebook (maildir) and
that works fine.

> on its Message-Id header.
Yes, but heliotrope-add does a full scan on every run, which takes hours
(instead of seconds).

I explained this in my original email and William acknowledged that code for
this is just missing. I’m not sure what you are trying to say.

Best regards,
Michael

---

From: Michael Stapelberg <[email-masked]>
Date: Sun, 08 Jan 2012 22:48:18 +0000

Hi,

So, I’m unsure how to use IMAP or offlineimap + maildir to be precise. I’m
currently several hundred kilometers away from my IMAP server (which only has a
slow link, so completely re-syncing my mails would take days), but I have an
offlineimap copy of it which I keep updating. With sup, it would just scan
through the maildir and pick up new messages.

Now I want to keep using offlineimap (so that I can switch back to sup if
necessary) and use heliotrope/turnsole. heliotrope-add seems like the wrong
tool for this job, since it will do a full scan (!) of my 5 GB of mail (110k
messages) which takes several hours.

Best regards,
Michael

---

From: William Morgan <[email-masked]>
Date: Mon, 09 Jan 2012 16:40:56 -0800

---

From: Michael Stapelberg <[email-masked]>
Date: Wed, 22 Feb 2012 22:39:42 +0100

Hi William,

Thanks for adding code for this. I got the chance to test this today, and I’m
afraid this doesn’t seem to be working as I would expect.

For the time being, I let procmail save my messages in my IMAP server and in
heliotrope with a little script, which I include here for others to see:

#!/bin/bash
# Delivers mail via dovecot and heliotrope

# load RVM
. /home/michael/.rvm/scripts/rvm

TMPFILE=$(mktemp)

cat > $TMPFILE

cat $TMPFILE | sudo -u mail -- /usr/lib/dovecot/deliver -d michael
cat $TMPFILE | ruby -I/home/michael/heliotrope/lib /home/michael/heliotrope/bin/heliotrope-add >>/tmp/last-mail.stdout 2>>/tmp/last-mail.stderr
rm $TMPFILE

After ensuring that it works, you should get rid of the stdout/stderr redirects
or redirect them properly to a place with logfile rotation, of course.

Best regards,
Michael