* [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
@ 2012-01-08 22:48 Michael Stapelberg
2012-01-10 0:40 ` William Morgan
2012-02-13 6:16 ` William Morgan
0 siblings, 2 replies; 15+ messages in thread
From: Michael Stapelberg @ 2012-01-08 22:48 UTC (permalink / raw)
To: sup-devel
Hi,
So, I’m unsure how to use IMAP or offlineimap + maildir to be precise. I’m
currently several hundred kilometers away from my IMAP server (which only has a
slow link, so completely re-syncing my mails would take days), but I have an
offlineimap copy of it which I keep updating. With sup, it would just scan
through the maildir and pick up new messages.
Now I want to keep using offlineimap (so that I can switch back to sup if
necessary) and use heliotrope/turnsole. heliotrope-add seems like the wrong
tool for this job, since it will do a full scan (!) of my 5 GB of mail (110k
messages) which takes several hours.
What is the correct way to do this?
Best regards,
Michael
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
* [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
2012-01-08 22:48 [sup-devel] [Heliotrope/Turnsole] How to use IMAP? Michael Stapelberg
@ 2012-01-10 0:40 ` William Morgan
2012-02-06 20:45 ` Michael Stapelberg
2012-02-13 6:16 ` William Morgan
1 sibling, 1 reply; 15+ messages in thread
From: William Morgan @ 2012-01-10 0:40 UTC (permalink / raw)
To: sup-devel
Excerpts from Michael Stapelberg's message of 2012-01-08 14:48:18 -0800:
> What is the correct way to do this?
I need to write some code to do this. For IMAP and GMail, heliotrope-add will
keep a pointer to thelast message imported, by default. For mbox there is a
trick you can use. But there's nothing for maildir right now. Please file
an issue so that I don't forget this!
--
William <wmorgan@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
2012-01-10 0:40 ` William Morgan
@ 2012-02-06 20:45 ` Michael Stapelberg
2012-02-06 23:08 ` Matthieu Rakotojaona
0 siblings, 1 reply; 15+ messages in thread
From: Michael Stapelberg @ 2012-02-06 20:45 UTC (permalink / raw)
To: sup-devel
Hi William,
Excerpts from William Morgan's message of 2012-01-10 00:40:56 +0000:
> I need to write some code to do this. For IMAP and GMail, heliotrope-add will
> keep a pointer to thelast message imported, by default. For mbox there is a
> trick you can use. But there's nothing for maildir right now. Please file
> an issue so that I don't forget this!
It’s been four weeks, and https://github.com/wmorgan/heliotrope/issues/32 is
still open. Just to let you know: This is what’s stopping me from using
heliotrope+turnsole and possibly participating in its development at the moment
:-/.
If it’s a task which is too big for your current spare time, could you please
describe how you would implement it, so that others can help?
Best regards,
Michael
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
2012-02-06 20:45 ` Michael Stapelberg
@ 2012-02-06 23:08 ` Matthieu Rakotojaona
2012-02-06 23:46 ` Michael Stapelberg
0 siblings, 1 reply; 15+ messages in thread
From: Matthieu Rakotojaona @ 2012-02-06 23:08 UTC (permalink / raw)
To: Sup developer discussion
Hi,
On Mon, Feb 6, 2012 at 9:45 PM, Michael Stapelberg
<michael+sup@stapelberg.de> wrote:
> It’s been four weeks, and https://github.com/wmorgan/heliotrope/issues/32 is
> still open. Just to let you know: This is what’s stopping me from using
> heliotrope+turnsole and possibly participating in its development at the moment
> :-/.
>
> If it’s a task which is too big for your current spare time, could you please
> describe how you would implement it, so that others can help?
As far as I can tell, IMAP will not be implemented in heliotrope. I
have started some work on a IMAP frontend which you can fork from :
https://github.com/rakoo/imaptrope
You use it as a separate process. It will create a RESTClient
connected to heliotrope, and you just connect to it with classical
IMAP clients.
--
Matthieu RAKOTOJAONA
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
2012-02-06 23:08 ` Matthieu Rakotojaona
@ 2012-02-06 23:46 ` Michael Stapelberg
2012-02-07 12:27 ` Matthieu Rakotojaona
0 siblings, 1 reply; 15+ messages in thread
From: Michael Stapelberg @ 2012-02-06 23:46 UTC (permalink / raw)
To: sup-devel
Hi Matthieu,
Excerpts from Matthieu Rakotojaona's message of 2012-02-06 23:08:33 +0000:
> As far as I can tell, IMAP will not be implemented in heliotrope. I
> have started some work on a IMAP frontend which you can fork from :
Uhm, that’s not what I am talking about. I want to fetch mails from an IMAP
server and store them in heliotrope.
Best regards,
Michael
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
2012-02-06 23:46 ` Michael Stapelberg
@ 2012-02-07 12:27 ` Matthieu Rakotojaona
2012-02-07 19:11 ` Michael Stapelberg
0 siblings, 1 reply; 15+ messages in thread
From: Matthieu Rakotojaona @ 2012-02-07 12:27 UTC (permalink / raw)
To: Sup developer discussion
On Tue, Feb 7, 2012 at 12:46 AM, Michael Stapelberg
<michael+sup@stapelberg.de> wrote:
> Uhm, that’s not what I am talking about. I want to fetch mails from an IMAP
> server and store them in heliotrope.
Yes, you can use offlineimap to sync 2 IMAP servers, although this is
still a work in progress. Or you can use 1-way copy tools, such as
imapcopy[0].
Oh, and heliotrope does tell you when it already has an email, based
on its Message-Id header.
[0] imapsync.lamiral.info/
--
Matthieu RAKOTOJAONA
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
2012-02-07 12:27 ` Matthieu Rakotojaona
@ 2012-02-07 19:11 ` Michael Stapelberg
2012-02-07 21:54 ` Matthieu Rakotojaona
0 siblings, 1 reply; 15+ messages in thread
From: Michael Stapelberg @ 2012-02-07 19:11 UTC (permalink / raw)
To: sup-devel
Hi Matthieu,
Excerpts from Matthieu Rakotojaona's message of 2012-02-07 12:27:32 +0000:
> Yes, you can use offlineimap to sync 2 IMAP servers, although this is
> still a work in progress. Or you can use 1-way copy tools, such as
> imapcopy[0].
I use offlineimap to sync my IMAP server at home with my notebook (maildir) and
that works fine.
> Oh, and heliotrope does tell you when it already has an email, based
> on its Message-Id header.
Yes, but heliotrope-add does a full scan on every run, which takes hours
(instead of seconds).
I explained this in my original email and William acknowledged that code for
this is just missing. I’m not sure what you are trying to say.
Best regards,
Michael
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
2012-02-07 19:11 ` Michael Stapelberg
@ 2012-02-07 21:54 ` Matthieu Rakotojaona
0 siblings, 0 replies; 15+ messages in thread
From: Matthieu Rakotojaona @ 2012-02-07 21:54 UTC (permalink / raw)
To: Sup developer discussion
Hi Michael,
On Tue, Feb 7, 2012 at 8:11 PM, Michael Stapelberg
<michael+sup@stapelberg.de> wrote:
> I’m not sure what you are trying to say.
Ok, I think I know what you mean.
You currently have a main IMAP server. You regularly sync it to your
notebook in maildir with offlineimap. You would like to use this
maildir directly with heliotrope. Right ?
So here are two options I can see for you:
* keep the maildir-backuping in a corner and sync your main IMAP
server directly with imaptrope ("heliotrope's" IMAP interface). This
would need two runs of offlineIMAP.
* use offlineimap to sync your maildir to imaptrope. This might create
problems, because the 2 syncings might interfere with each other.
--
Matthieu RAKOTOJAONA
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
* [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
2012-01-08 22:48 [sup-devel] [Heliotrope/Turnsole] How to use IMAP? Michael Stapelberg
2012-01-10 0:40 ` William Morgan
@ 2012-02-13 6:16 ` William Morgan
2012-02-22 21:39 ` Michael Stapelberg
1 sibling, 1 reply; 15+ messages in thread
From: William Morgan @ 2012-02-13 6:16 UTC (permalink / raw)
To: sup-devel
Excerpts from Michael Stapelberg's message of 2012-01-08 14:48:18 -0800:
> What is the correct way to do this?
Please try the latest heliotrope master. You can do a full import with this:
ruby -Ilib bin/heliotrope-import -a <maildir> -d <mailstore> -t <state file>
where <state file> is just a place for it to dump its state. This import need
only happen once. Then for subsequent additions you can do this:
ruby -Ilib bin/heliotrope-add -a <maildir> -d <mailstore> -t <state file>
which will read to and write from that state file, and only add what it
hasn't yet seen.
--
William <wmorgan@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
2012-02-13 6:16 ` William Morgan
@ 2012-02-22 21:39 ` Michael Stapelberg
2012-02-23 6:25 ` William Morgan
0 siblings, 1 reply; 15+ messages in thread
From: Michael Stapelberg @ 2012-02-22 21:39 UTC (permalink / raw)
To: sup-devel
Hi William,
Thanks for adding code for this. I got the chance to test this today, and I’m
afraid this doesn’t seem to be working as I would expect.
As far as I can tell from glancing at the source and the state file, this code
examines the date stored in every mail, sorts that, then checks where it
stopped before, and continues from there. I think this is a horrible way of
doing things for two reasons:
1) It involves opening every single email on disk.
2) Date specifications in emails are not to be trusted (think spam).
For the time being, I let procmail save my messages in my IMAP server and in
heliotrope with a little script, which I include here for others to see:
#!/bin/bash
# Delivers mail via dovecot and heliotrope
# load RVM
. /home/michael/.rvm/scripts/rvm
TMPFILE=$(mktemp)
cat > $TMPFILE
cat $TMPFILE | sudo -u mail -- /usr/lib/dovecot/deliver -d michael
cat $TMPFILE | ruby -I/home/michael/heliotrope/lib /home/michael/heliotrope/bin/heliotrope-add >>/tmp/last-mail.stdout 2>>/tmp/last-mail.stderr
rm $TMPFILE
After ensuring that it works, you should get rid of the stdout/stderr redirects
or redirect them properly to a place with logfile rotation, of course.
Note that adding mails via stdin doesn’t work until you change the code like this:
https://github.com/wmorgan/heliotrope/pull/34
Best regards,
Michael
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
2012-02-22 21:39 ` Michael Stapelberg
@ 2012-02-23 6:25 ` William Morgan
2012-02-23 11:24 ` Michael Stapelberg
0 siblings, 1 reply; 15+ messages in thread
From: William Morgan @ 2012-02-23 6:25 UTC (permalink / raw)
To: sup-devel
Excerpts from Michael Stapelberg's message of 2012-02-22 13:39:42 -0800:
> Thanks for adding code for this. I got the chance to test this today,
> and I’m afraid this doesn’t seem to be working as I would expect.
>
> As far as I can tell from glancing at the source and the state file, this code
> examines the date stored in every mail, sorts that, then checks where it
> stopped before, and continues from there. I think this is a horrible way of
> doing things for two reasons:
>
> 1) It involves opening every single email on disk.
> 2) Date specifications in emails are not to be trusted (think spam).
My understanding of the Maildir specification is that there is no way to
determine the order of messages besides reading the date headers.
Ordering messages correctly at import time is important because
Heliotrope only serves messages in the reverse order to which they were
imported. If there is a better solution, please feel free to educate me.
Is there any problem with the current code besides the fact that you
don't like the big scan?
> Note that adding mails via stdin doesn’t work until you change the code like this:
> https://github.com/wmorgan/heliotrope/pull/34
Merged, thank you.
--
William <wmorgan@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
2012-02-23 6:25 ` William Morgan
@ 2012-02-23 11:24 ` Michael Stapelberg
2012-02-25 6:35 ` William Morgan
0 siblings, 1 reply; 15+ messages in thread
From: Michael Stapelberg @ 2012-02-23 11:24 UTC (permalink / raw)
To: sup-devel
Hi William,
Excerpts from William Morgan's message of 2012-02-23 07:25:11 +0100:
> My understanding of the Maildir specification is that there is no way to
> determine the order of messages besides reading the date headers.
> Ordering messages correctly at import time is important because
> Heliotrope only serves messages in the reverse order to which they were
> imported. If there is a better solution, please feel free to educate me.
Well, I guess you are correct. So, when somebody sends me a spam email or a
malicious email with a faked date, what happens? I think that the code will
figure out it needs to re-add a lot of messages. Also, my index will need to be
reordered, right?
Also, what happens when somebody sends me a message with a faked date and I add
it using heliotrope-add? Do I need to reorder my index?
> Is there any problem with the current code besides the fact that you
> don't like the big scan?
That makes it sound like it’s just a matter of opinion. My concern is what I
stated above (huge processing load / disk IO caused by malicious messages) and
that it thrashes my poor disks.
Best regards,
Michael
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
2012-02-23 11:24 ` Michael Stapelberg
@ 2012-02-25 6:35 ` William Morgan
2012-02-27 9:16 ` Michael Stapelberg
0 siblings, 1 reply; 15+ messages in thread
From: William Morgan @ 2012-02-25 6:35 UTC (permalink / raw)
To: sup-devel
Excerpts from Michael Stapelberg's message of 2012-02-23 03:24:22 -0800:
> Well, I guess you are correct. So, when somebody sends me a spam email
> or a malicious email with a faked date, what happens? I think that the
> code will figure out it needs to re-add a lot of messages. Also, my
> index will need to be reordered, right?
You won't need to reorder. Importing is all incremental, at least if you
use --state-file: it will keep track of the last message imported, and
successive imports will only pick up new messages.
But you are making me realize that the current maildir implementation is
not right. I think the way to handle Maildir is to look only at the
ctime of the files and not the date headers. That will make incremental
importing possible (just save the filename of the last imported file,
and look for all files newer than that). It will speed things up
anyways.
And if you're in the funny situation where the ctime is not correlated
with the Date: headers in your files, which is perfectly possibly by the
Maildir spec, then you will have to reorder after that initial import.
But in reality that's a possibility with the other source types
too---it's just more likely with Maildir.
--
William <wmorgan@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
2012-02-25 6:35 ` William Morgan
@ 2012-02-27 9:16 ` Michael Stapelberg
2012-02-29 1:11 ` William Morgan
0 siblings, 1 reply; 15+ messages in thread
From: Michael Stapelberg @ 2012-02-27 9:16 UTC (permalink / raw)
To: sup-devel
Hi William,
Sorry for clinging to that topic, but it’s important for me to properly
understand it.
Excerpts from William Morgan's message of 2012-02-25 07:35:32 +0100:
> Excerpts from Michael Stapelberg's message of 2012-02-23 03:24:22 -0800:
> > Well, I guess you are correct. So, when somebody sends me a spam email
> > or a malicious email with a faked date, what happens? I think that the
> > code will figure out it needs to re-add a lot of messages. Also, my
> > index will need to be reordered, right?
>
> You won't need to reorder. Importing is all incremental, at least if you
> use --state-file: it will keep track of the last message imported, and
> successive imports will only pick up new messages.
Apparently, I didn’t properly describe the scenario I’m thinking about:
state-file contains an entry "/my/maildir/2012-02-22-foo-bar", which is an
email that contains 2012-02-22 as date.
Now, a spammer sends me an email with a faked date, let’s say 2001-01-01, let’s
call it "/my/maildir/2001-01-01-spam-mail". The next run of heliotrope-add has
two possibilities:
1) It will completely ignore it. I think this is what will happen, based on
your description. This is horrible! Emails are absolutely not guaranteed to
arrive in my system in the order they were sent. Is this the case? Do we
ignore email because of date problems like this currently?
2) It will pick up this email because the date inside is older. Subsequent runs
will discover all email from the last 10 years as new, and try to re-add it
to the index. Lots of unnecessary overhead.
> But you are making me realize that the current maildir implementation is
> not right. I think the way to handle Maildir is to look only at the
> ctime of the files and not the date headers. That will make incremental
> importing possible (just save the filename of the last imported file,
> and look for all files newer than that). It will speed things up
> anyways.
Right, that sounds good.
> And if you're in the funny situation where the ctime is not correlated
> with the Date: headers in your files, which is perfectly possibly by the
> Maildir spec, then you will have to reorder after that initial import.
> But in reality that's a possibility with the other source types
> too---it's just more likely with Maildir.
So, let’s stick to my above example of the spam email and let’s assume that you
changed the code to use ctimes. The spam mail arrives, its ctime is new, it
gets picked up into my index. To my understanding, I now need to reorder. I
have multiple questions:
1) What are the immediate consequences? Where will this email appear in my
inbox, when my inbox contains 5 emails. Always at the top? Bottom? You
mentioned I can see the effect when paging. Can you elaborate please?
2) How do I know when I have to reorder?
Best regards,
Michael
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP?
2012-02-27 9:16 ` Michael Stapelberg
@ 2012-02-29 1:11 ` William Morgan
0 siblings, 0 replies; 15+ messages in thread
From: William Morgan @ 2012-02-29 1:11 UTC (permalink / raw)
To: sup-devel
Excerpts from Michael Stapelberg's message of 2012-02-27 01:16:43 -0800:
> 1) It will completely ignore it.
Correct. This is the problem that the ctime solution is meant to address.
> So, let’s stick to my above example of the spam email and let’s assume
> that you changed the code to use ctimes. The spam mail arrives, its
> ctime is new, it gets picked up into my index. To my understanding, I
> now need to reorder. I have multiple questions:
>
> 1) What are the immediate consequences? Where will this email appear in my
> inbox, when my inbox contains 5 emails. Always at the top? Bottom? You
> mentioned I can see the effect when paging. Can you elaborate please?
Heliotrope will return it as the first result, since it returns messages
in LIFO order.
Currently Turnsole currently re-sorts all messages returned by sort by
date, and so it will place it at the bottom. This is also where the
pagination issues may come from. Maybe it would be less confusing for it
to stick to search result order.
> 2) How do I know when I have to reorder?
You have to reorder whenever you want Heliotrope to return messages in
Date header order instead of in LIFO order. I think this will typically
happen when you import a couple batches of preexisting mail, and not
when you're adding new mail.
--
William <wmorgan@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2012-02-29 1:13 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-01-08 22:48 [sup-devel] [Heliotrope/Turnsole] How to use IMAP? Michael Stapelberg
2012-01-10 0:40 ` William Morgan
2012-02-06 20:45 ` Michael Stapelberg
2012-02-06 23:08 ` Matthieu Rakotojaona
2012-02-06 23:46 ` Michael Stapelberg
2012-02-07 12:27 ` Matthieu Rakotojaona
2012-02-07 19:11 ` Michael Stapelberg
2012-02-07 21:54 ` Matthieu Rakotojaona
2012-02-13 6:16 ` William Morgan
2012-02-22 21:39 ` Michael Stapelberg
2012-02-23 6:25 ` William Morgan
2012-02-23 11:24 ` Michael Stapelberg
2012-02-25 6:35 ` William Morgan
2012-02-27 9:16 ` Michael Stapelberg
2012-02-29 1:11 ` William Morgan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox