* [sup-devel] [Heliotrope/Turnsole] How to use IMAP? @ 2012-01-08 22:48 Michael Stapelberg 2012-01-10 0:40 ` William Morgan 2012-02-13 6:16 ` William Morgan 0 siblings, 2 replies; 15+ messages in thread From: Michael Stapelberg @ 2012-01-08 22:48 UTC (permalink / raw) To: sup-devel Hi, So, I’m unsure how to use IMAP or offlineimap + maildir to be precise. I’m currently several hundred kilometers away from my IMAP server (which only has a slow link, so completely re-syncing my mails would take days), but I have an offlineimap copy of it which I keep updating. With sup, it would just scan through the maildir and pick up new messages. Now I want to keep using offlineimap (so that I can switch back to sup if necessary) and use heliotrope/turnsole. heliotrope-add seems like the wrong tool for this job, since it will do a full scan (!) of my 5 GB of mail (110k messages) which takes several hours. What is the correct way to do this? Best regards, Michael _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* [sup-devel] [Heliotrope/Turnsole] How to use IMAP? 2012-01-08 22:48 [sup-devel] [Heliotrope/Turnsole] How to use IMAP? Michael Stapelberg @ 2012-01-10 0:40 ` William Morgan 2012-02-06 20:45 ` Michael Stapelberg 2012-02-13 6:16 ` William Morgan 1 sibling, 1 reply; 15+ messages in thread From: William Morgan @ 2012-01-10 0:40 UTC (permalink / raw) To: sup-devel Excerpts from Michael Stapelberg's message of 2012-01-08 14:48:18 -0800: > What is the correct way to do this? I need to write some code to do this. For IMAP and GMail, heliotrope-add will keep a pointer to thelast message imported, by default. For mbox there is a trick you can use. But there's nothing for maildir right now. Please file an issue so that I don't forget this! -- William <wmorgan@masanjin.net> _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP? 2012-01-10 0:40 ` William Morgan @ 2012-02-06 20:45 ` Michael Stapelberg 2012-02-06 23:08 ` Matthieu Rakotojaona 0 siblings, 1 reply; 15+ messages in thread From: Michael Stapelberg @ 2012-02-06 20:45 UTC (permalink / raw) To: sup-devel Hi William, Excerpts from William Morgan's message of 2012-01-10 00:40:56 +0000: > I need to write some code to do this. For IMAP and GMail, heliotrope-add will > keep a pointer to thelast message imported, by default. For mbox there is a > trick you can use. But there's nothing for maildir right now. Please file > an issue so that I don't forget this! It’s been four weeks, and https://github.com/wmorgan/heliotrope/issues/32 is still open. Just to let you know: This is what’s stopping me from using heliotrope+turnsole and possibly participating in its development at the moment :-/. If it’s a task which is too big for your current spare time, could you please describe how you would implement it, so that others can help? Best regards, Michael _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP? 2012-02-06 20:45 ` Michael Stapelberg @ 2012-02-06 23:08 ` Matthieu Rakotojaona 2012-02-06 23:46 ` Michael Stapelberg 0 siblings, 1 reply; 15+ messages in thread From: Matthieu Rakotojaona @ 2012-02-06 23:08 UTC (permalink / raw) To: Sup developer discussion Hi, On Mon, Feb 6, 2012 at 9:45 PM, Michael Stapelberg <michael+sup@stapelberg.de> wrote: > It’s been four weeks, and https://github.com/wmorgan/heliotrope/issues/32 is > still open. Just to let you know: This is what’s stopping me from using > heliotrope+turnsole and possibly participating in its development at the moment > :-/. > > If it’s a task which is too big for your current spare time, could you please > describe how you would implement it, so that others can help? As far as I can tell, IMAP will not be implemented in heliotrope. I have started some work on a IMAP frontend which you can fork from : https://github.com/rakoo/imaptrope You use it as a separate process. It will create a RESTClient connected to heliotrope, and you just connect to it with classical IMAP clients. -- Matthieu RAKOTOJAONA _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP? 2012-02-06 23:08 ` Matthieu Rakotojaona @ 2012-02-06 23:46 ` Michael Stapelberg 2012-02-07 12:27 ` Matthieu Rakotojaona 0 siblings, 1 reply; 15+ messages in thread From: Michael Stapelberg @ 2012-02-06 23:46 UTC (permalink / raw) To: sup-devel Hi Matthieu, Excerpts from Matthieu Rakotojaona's message of 2012-02-06 23:08:33 +0000: > As far as I can tell, IMAP will not be implemented in heliotrope. I > have started some work on a IMAP frontend which you can fork from : Uhm, that’s not what I am talking about. I want to fetch mails from an IMAP server and store them in heliotrope. Best regards, Michael _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP? 2012-02-06 23:46 ` Michael Stapelberg @ 2012-02-07 12:27 ` Matthieu Rakotojaona 2012-02-07 19:11 ` Michael Stapelberg 0 siblings, 1 reply; 15+ messages in thread From: Matthieu Rakotojaona @ 2012-02-07 12:27 UTC (permalink / raw) To: Sup developer discussion On Tue, Feb 7, 2012 at 12:46 AM, Michael Stapelberg <michael+sup@stapelberg.de> wrote: > Uhm, that’s not what I am talking about. I want to fetch mails from an IMAP > server and store them in heliotrope. Yes, you can use offlineimap to sync 2 IMAP servers, although this is still a work in progress. Or you can use 1-way copy tools, such as imapcopy[0]. Oh, and heliotrope does tell you when it already has an email, based on its Message-Id header. [0] imapsync.lamiral.info/ -- Matthieu RAKOTOJAONA _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP? 2012-02-07 12:27 ` Matthieu Rakotojaona @ 2012-02-07 19:11 ` Michael Stapelberg 2012-02-07 21:54 ` Matthieu Rakotojaona 0 siblings, 1 reply; 15+ messages in thread From: Michael Stapelberg @ 2012-02-07 19:11 UTC (permalink / raw) To: sup-devel Hi Matthieu, Excerpts from Matthieu Rakotojaona's message of 2012-02-07 12:27:32 +0000: > Yes, you can use offlineimap to sync 2 IMAP servers, although this is > still a work in progress. Or you can use 1-way copy tools, such as > imapcopy[0]. I use offlineimap to sync my IMAP server at home with my notebook (maildir) and that works fine. > Oh, and heliotrope does tell you when it already has an email, based > on its Message-Id header. Yes, but heliotrope-add does a full scan on every run, which takes hours (instead of seconds). I explained this in my original email and William acknowledged that code for this is just missing. I’m not sure what you are trying to say. Best regards, Michael _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP? 2012-02-07 19:11 ` Michael Stapelberg @ 2012-02-07 21:54 ` Matthieu Rakotojaona 0 siblings, 0 replies; 15+ messages in thread From: Matthieu Rakotojaona @ 2012-02-07 21:54 UTC (permalink / raw) To: Sup developer discussion Hi Michael, On Tue, Feb 7, 2012 at 8:11 PM, Michael Stapelberg <michael+sup@stapelberg.de> wrote: > I’m not sure what you are trying to say. Ok, I think I know what you mean. You currently have a main IMAP server. You regularly sync it to your notebook in maildir with offlineimap. You would like to use this maildir directly with heliotrope. Right ? So here are two options I can see for you: * keep the maildir-backuping in a corner and sync your main IMAP server directly with imaptrope ("heliotrope's" IMAP interface). This would need two runs of offlineIMAP. * use offlineimap to sync your maildir to imaptrope. This might create problems, because the 2 syncings might interfere with each other. -- Matthieu RAKOTOJAONA _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* [sup-devel] [Heliotrope/Turnsole] How to use IMAP? 2012-01-08 22:48 [sup-devel] [Heliotrope/Turnsole] How to use IMAP? Michael Stapelberg 2012-01-10 0:40 ` William Morgan @ 2012-02-13 6:16 ` William Morgan 2012-02-22 21:39 ` Michael Stapelberg 1 sibling, 1 reply; 15+ messages in thread From: William Morgan @ 2012-02-13 6:16 UTC (permalink / raw) To: sup-devel Excerpts from Michael Stapelberg's message of 2012-01-08 14:48:18 -0800: > What is the correct way to do this? Please try the latest heliotrope master. You can do a full import with this: ruby -Ilib bin/heliotrope-import -a <maildir> -d <mailstore> -t <state file> where <state file> is just a place for it to dump its state. This import need only happen once. Then for subsequent additions you can do this: ruby -Ilib bin/heliotrope-add -a <maildir> -d <mailstore> -t <state file> which will read to and write from that state file, and only add what it hasn't yet seen. -- William <wmorgan@masanjin.net> _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP? 2012-02-13 6:16 ` William Morgan @ 2012-02-22 21:39 ` Michael Stapelberg 2012-02-23 6:25 ` William Morgan 0 siblings, 1 reply; 15+ messages in thread From: Michael Stapelberg @ 2012-02-22 21:39 UTC (permalink / raw) To: sup-devel Hi William, Thanks for adding code for this. I got the chance to test this today, and I’m afraid this doesn’t seem to be working as I would expect. As far as I can tell from glancing at the source and the state file, this code examines the date stored in every mail, sorts that, then checks where it stopped before, and continues from there. I think this is a horrible way of doing things for two reasons: 1) It involves opening every single email on disk. 2) Date specifications in emails are not to be trusted (think spam). For the time being, I let procmail save my messages in my IMAP server and in heliotrope with a little script, which I include here for others to see: #!/bin/bash # Delivers mail via dovecot and heliotrope # load RVM . /home/michael/.rvm/scripts/rvm TMPFILE=$(mktemp) cat > $TMPFILE cat $TMPFILE | sudo -u mail -- /usr/lib/dovecot/deliver -d michael cat $TMPFILE | ruby -I/home/michael/heliotrope/lib /home/michael/heliotrope/bin/heliotrope-add >>/tmp/last-mail.stdout 2>>/tmp/last-mail.stderr rm $TMPFILE After ensuring that it works, you should get rid of the stdout/stderr redirects or redirect them properly to a place with logfile rotation, of course. Note that adding mails via stdin doesn’t work until you change the code like this: https://github.com/wmorgan/heliotrope/pull/34 Best regards, Michael _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP? 2012-02-22 21:39 ` Michael Stapelberg @ 2012-02-23 6:25 ` William Morgan 2012-02-23 11:24 ` Michael Stapelberg 0 siblings, 1 reply; 15+ messages in thread From: William Morgan @ 2012-02-23 6:25 UTC (permalink / raw) To: sup-devel Excerpts from Michael Stapelberg's message of 2012-02-22 13:39:42 -0800: > Thanks for adding code for this. I got the chance to test this today, > and I’m afraid this doesn’t seem to be working as I would expect. > > As far as I can tell from glancing at the source and the state file, this code > examines the date stored in every mail, sorts that, then checks where it > stopped before, and continues from there. I think this is a horrible way of > doing things for two reasons: > > 1) It involves opening every single email on disk. > 2) Date specifications in emails are not to be trusted (think spam). My understanding of the Maildir specification is that there is no way to determine the order of messages besides reading the date headers. Ordering messages correctly at import time is important because Heliotrope only serves messages in the reverse order to which they were imported. If there is a better solution, please feel free to educate me. Is there any problem with the current code besides the fact that you don't like the big scan? > Note that adding mails via stdin doesn’t work until you change the code like this: > https://github.com/wmorgan/heliotrope/pull/34 Merged, thank you. -- William <wmorgan@masanjin.net> _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP? 2012-02-23 6:25 ` William Morgan @ 2012-02-23 11:24 ` Michael Stapelberg 2012-02-25 6:35 ` William Morgan 0 siblings, 1 reply; 15+ messages in thread From: Michael Stapelberg @ 2012-02-23 11:24 UTC (permalink / raw) To: sup-devel Hi William, Excerpts from William Morgan's message of 2012-02-23 07:25:11 +0100: > My understanding of the Maildir specification is that there is no way to > determine the order of messages besides reading the date headers. > Ordering messages correctly at import time is important because > Heliotrope only serves messages in the reverse order to which they were > imported. If there is a better solution, please feel free to educate me. Well, I guess you are correct. So, when somebody sends me a spam email or a malicious email with a faked date, what happens? I think that the code will figure out it needs to re-add a lot of messages. Also, my index will need to be reordered, right? Also, what happens when somebody sends me a message with a faked date and I add it using heliotrope-add? Do I need to reorder my index? > Is there any problem with the current code besides the fact that you > don't like the big scan? That makes it sound like it’s just a matter of opinion. My concern is what I stated above (huge processing load / disk IO caused by malicious messages) and that it thrashes my poor disks. Best regards, Michael _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP? 2012-02-23 11:24 ` Michael Stapelberg @ 2012-02-25 6:35 ` William Morgan 2012-02-27 9:16 ` Michael Stapelberg 0 siblings, 1 reply; 15+ messages in thread From: William Morgan @ 2012-02-25 6:35 UTC (permalink / raw) To: sup-devel Excerpts from Michael Stapelberg's message of 2012-02-23 03:24:22 -0800: > Well, I guess you are correct. So, when somebody sends me a spam email > or a malicious email with a faked date, what happens? I think that the > code will figure out it needs to re-add a lot of messages. Also, my > index will need to be reordered, right? You won't need to reorder. Importing is all incremental, at least if you use --state-file: it will keep track of the last message imported, and successive imports will only pick up new messages. But you are making me realize that the current maildir implementation is not right. I think the way to handle Maildir is to look only at the ctime of the files and not the date headers. That will make incremental importing possible (just save the filename of the last imported file, and look for all files newer than that). It will speed things up anyways. And if you're in the funny situation where the ctime is not correlated with the Date: headers in your files, which is perfectly possibly by the Maildir spec, then you will have to reorder after that initial import. But in reality that's a possibility with the other source types too---it's just more likely with Maildir. -- William <wmorgan@masanjin.net> _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP? 2012-02-25 6:35 ` William Morgan @ 2012-02-27 9:16 ` Michael Stapelberg 2012-02-29 1:11 ` William Morgan 0 siblings, 1 reply; 15+ messages in thread From: Michael Stapelberg @ 2012-02-27 9:16 UTC (permalink / raw) To: sup-devel Hi William, Sorry for clinging to that topic, but it’s important for me to properly understand it. Excerpts from William Morgan's message of 2012-02-25 07:35:32 +0100: > Excerpts from Michael Stapelberg's message of 2012-02-23 03:24:22 -0800: > > Well, I guess you are correct. So, when somebody sends me a spam email > > or a malicious email with a faked date, what happens? I think that the > > code will figure out it needs to re-add a lot of messages. Also, my > > index will need to be reordered, right? > > You won't need to reorder. Importing is all incremental, at least if you > use --state-file: it will keep track of the last message imported, and > successive imports will only pick up new messages. Apparently, I didn’t properly describe the scenario I’m thinking about: state-file contains an entry "/my/maildir/2012-02-22-foo-bar", which is an email that contains 2012-02-22 as date. Now, a spammer sends me an email with a faked date, let’s say 2001-01-01, let’s call it "/my/maildir/2001-01-01-spam-mail". The next run of heliotrope-add has two possibilities: 1) It will completely ignore it. I think this is what will happen, based on your description. This is horrible! Emails are absolutely not guaranteed to arrive in my system in the order they were sent. Is this the case? Do we ignore email because of date problems like this currently? 2) It will pick up this email because the date inside is older. Subsequent runs will discover all email from the last 10 years as new, and try to re-add it to the index. Lots of unnecessary overhead. > But you are making me realize that the current maildir implementation is > not right. I think the way to handle Maildir is to look only at the > ctime of the files and not the date headers. That will make incremental > importing possible (just save the filename of the last imported file, > and look for all files newer than that). It will speed things up > anyways. Right, that sounds good. > And if you're in the funny situation where the ctime is not correlated > with the Date: headers in your files, which is perfectly possibly by the > Maildir spec, then you will have to reorder after that initial import. > But in reality that's a possibility with the other source types > too---it's just more likely with Maildir. So, let’s stick to my above example of the spam email and let’s assume that you changed the code to use ctimes. The spam mail arrives, its ctime is new, it gets picked up into my index. To my understanding, I now need to reorder. I have multiple questions: 1) What are the immediate consequences? Where will this email appear in my inbox, when my inbox contains 5 emails. Always at the top? Bottom? You mentioned I can see the effect when paging. Can you elaborate please? 2) How do I know when I have to reorder? Best regards, Michael _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [sup-devel] [Heliotrope/Turnsole] How to use IMAP? 2012-02-27 9:16 ` Michael Stapelberg @ 2012-02-29 1:11 ` William Morgan 0 siblings, 0 replies; 15+ messages in thread From: William Morgan @ 2012-02-29 1:11 UTC (permalink / raw) To: sup-devel Excerpts from Michael Stapelberg's message of 2012-02-27 01:16:43 -0800: > 1) It will completely ignore it. Correct. This is the problem that the ctime solution is meant to address. > So, let’s stick to my above example of the spam email and let’s assume > that you changed the code to use ctimes. The spam mail arrives, its > ctime is new, it gets picked up into my index. To my understanding, I > now need to reorder. I have multiple questions: > > 1) What are the immediate consequences? Where will this email appear in my > inbox, when my inbox contains 5 emails. Always at the top? Bottom? You > mentioned I can see the effect when paging. Can you elaborate please? Heliotrope will return it as the first result, since it returns messages in LIFO order. Currently Turnsole currently re-sorts all messages returned by sort by date, and so it will place it at the bottom. This is also where the pagination issues may come from. Maybe it would be less confusing for it to stick to search result order. > 2) How do I know when I have to reorder? You have to reorder whenever you want Heliotrope to return messages in Date header order instead of in LIFO order. I think this will typically happen when you import a couple batches of preexisting mail, and not when you're adding new mail. -- William <wmorgan@masanjin.net> _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2012-02-29 1:13 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-01-08 22:48 [sup-devel] [Heliotrope/Turnsole] How to use IMAP? Michael Stapelberg 2012-01-10 0:40 ` William Morgan 2012-02-06 20:45 ` Michael Stapelberg 2012-02-06 23:08 ` Matthieu Rakotojaona 2012-02-06 23:46 ` Michael Stapelberg 2012-02-07 12:27 ` Matthieu Rakotojaona 2012-02-07 19:11 ` Michael Stapelberg 2012-02-07 21:54 ` Matthieu Rakotojaona 2012-02-13 6:16 ` William Morgan 2012-02-22 21:39 ` Michael Stapelberg 2012-02-23 6:25 ` William Morgan 2012-02-23 11:24 ` Michael Stapelberg 2012-02-25 6:35 ` William Morgan 2012-02-27 9:16 ` Michael Stapelberg 2012-02-29 1:11 ` William Morgan
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox