From hsanson@gmail.com Sun Jun 2 02:45:47 2013 From: hsanson@gmail.com (Horacio Sanson) Date: Sun, 2 Jun 2013 11:45:47 +0900 Subject: [sup-devel] Experimental Gmail Source In-Reply-To: <1369172802-sup-2003@kpad> References: <518E1A2B.2080903@gaute.vetsj.com> <1369172802-sup-2003@kpad> Message-ID: Thanks for checking the source and sorry for the late response... I can only look into this on rare free weekends. On Wed, May 22, 2013 at 6:47 AM, Matthieu Rakotojaona < matthieu.rakotojaona at gmail.com> wrote: > Hey Horacio, > > I took a stab at your gmail_source branch, and made a few > fixes/improvements [0]: > > - Add configuration option in sup-add > - Dump the LevelDB path in the sources.yaml > - Add a load_from_yaml method for a source to initialize its working > values (for instance, the @db cannot be serialized, it needs to be > reconstructed) > - Fixed the msg_att monkey-patch for imap.rb > > Great, I will add these changes to my branch.... > All in all, the gmail source seems to work. I tested it on my usual > gmail account, I haven't tried to download it all, but I did download a > few dozens of emails without a problem. I'd like to warn users about > LevelDB though: it's sad to say, but as other wmorgan's stuff, it looks > abandoned. There are at least 2 bugs you will encounter if you try it: a > pb in configuration (fixed in [1]) and you need the `snappy` gem to make > it work if your db is more than 4MB large [2]. There are some up-to-date > forks, though. > > I see LevelDB is used mostly for storing messages and mailboxes > uid{validity/last}, but if we are to use gmail (it's the only IMAP > provider that makes sense for sup), I believe we would stick to the All > Mail label, right ? So, no need for storing this in db, rather in the > sources.yaml file. Also, if leveldb-ruby is unreliable (I did encounter > some issues way back about something with glibc...), and we want to use > it for caching messages, I think we can salvage heliotrope's zmbox [3] > because it's so simple to use yet far better than simple mbox. Using zmbox, mbox, maildir or any other mail storage (mix?) means I need to keep track of three indexes to allow two way sync between the Gmail source and the Sup index. I would need the Sup index id, the store id (e.g. zmbox file index) and the Gmail X-GM-MSGID. That complicates things a lot. Using key/value stores like LevelDB allows me to directly store the messages and associate them directly with the Gmail X-GM-MSGID. Also LevelDB comes with high compression for text data, perfect for emails, and high performance [1]. The issues you mention seem to be on the ruby library rather than LevelDB itself and they are fixable. If there are no bigger issues (e.g. data corruption/loss) I will stick with LevelDB. Regarding your ids questions, if you want to access the sup's messages > from the gmail source, you could use the mail's Message-ID header and > apply the same logic as in Message.sanitize_message_id. Caution, > however: I've already encountered the case where multiple messages in > GMail (i.e multiple X-GM-MSGID) have the same Message-ID, so they would > be considered the same in sup/heliotrope... yeah, that's annoying as > hell, and I don't know how we can solve this in the case of multiple > sources. > > Thanks, this comment put me on track and I found a way to get the emails from the index using the message id provided by the source. All I need to do is call Message.build_from_source(source, info) where info is the message id provided by the source. In my case this would be the X-GM-MSGID string. If you want to sync-back, maybe sup can call a source-level "sync_back" > method with the current known state ? Speaking of which, for general > synchronization we could reuse the elegant offlineimap's sync algorithm > [4]. The idea is basic: have each source class store a snapshot of the > state. When a message is modified on the source, diff the change with > the known status and propagate to sup; when a message is modified in > sup, diff with the known status and propagate to the source. > > Interesting and simple algorithm. Let me study it a little more and see how it is applicable to Sup. [1] http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html regards, Horacio Just a brain dump. > > [0] https://github.com/rakoo/sup/tree/gmail_source > [1] https://github.com/wmorgan/leveldb-ruby/pull/27 > [2] https://github.com/wmorgan/leveldb-ruby/issues/23 > [3] > https://github.com/sup-heliotrope/heliotrope/blob/64d4b50d5649ec616a311a4cf6955137fdaeb13d/lib/heliotrope/zmbox.rb > [4] http://offlineimap.org/howitworks.html > > Regards, > > -- > Matthieu Rakotojaona > > _______________________________________________ > Sup-devel mailing list > Sup-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/sup-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eg@gaute.vetsj.com Thu Jun 13 20:33:28 2013 From: eg@gaute.vetsj.com (Gaute Hope) Date: Thu, 13 Jun 2013 22:33:28 +0200 Subject: [sup-devel] going for 0.13.1 and possibly 0.14? Message-ID: <1371155525-sup-5029@qwerzila> Greetings, as briefly discussed in #75 [1] some adjustments to the original plan should be made. We were thinking: - Release another release in line-0.13 compatible with ruby 1.8 since we are starting to diverge in develop now. any efforts in fixing and maintaining ruby 1.8 support should be backported to this branch. - Work towards a 0.14 release with ruby 1.9 support on 'develop' and not necessarily with ruby 2.0 support yet. Already a several fixes have been done that have break ruby 1.8 which would be nice to get released in not too long. Any thoughts? Ruby 2.0: I am currently writing this from the tip of #61, rmail-to-mail-inital [2]. Having successfully indexed >30k (except 1..) of highly heterogeneous mails in all kinds of encodings. Tests are now also working, as well as sending email obviously. Still lots of parts are incomplete (e.g. no attachments yet..). The effort to get to a sup with ruby 2.0 support (with maildir-syncback, Mail - not RMail) has not reached a stable level. But I hope it could get to a state where it makes sense to include it in a separate branch in 'origin' so that it gets more available and easier to work against for everybody. Regards, Gaute [1] https://github.com/sup-heliotrope/sup/pull/75 [2] https://github.com/sup-heliotrope/sup/pull/61 From eg@gaute.vetsj.com Fri Jun 21 12:07:19 2013 From: eg@gaute.vetsj.com (Gaute Hope) Date: Fri, 21 Jun 2013 14:07:19 +0200 Subject: [sup-devel] Release 0.13.1 Message-ID: <1371816327-sup-6968@qwerzila> Greetings, I have just pushed version 0.13.1 of Sup to rubygems. This release holds should still be Ruby 1.8 compatible and holds the backported patches since the last release. No big changes, but a number of fixes has been applied. Regards, Gaute From eg@gaute.vetsj.com Sat Jun 22 10:48:13 2013 From: eg@gaute.vetsj.com (Gaute Hope) Date: Sat, 22 Jun 2013 12:48:13 +0200 Subject: [sup-devel] [sup-talk] Release 0.13.1 In-Reply-To: References: <1371816327-sup-6968@qwerzila> Message-ID: <1371897680-sup-4189@qwerzila> Excerpts from James Pearson's message of 2013-06-22 04:10:29 +0200: > I was going to ask why the tags on the Gitorious repo[0] hadn't been > updated, when I visited the (new!) website and saw that development has > moved onto Github[1] (and restarted!). I'm apparently very behind the > times - and perhaps now I should dig out those changes I made and submit > them... > > Glad to see things moving along again. > - P > > [0]: http://gitorious.org/sup/mainline > [1]: https://github.com/sup-heliotrope/sup Yup! Dig 'em up. We generally prefer pull requests [0] now a days. Also, we've collected contribution/development resources in the new wiki [1]. I have asked William if he would agree to redirect to/mention the new setup in the old gitorious repo. He gave me access to the sup-team, but I am unfortunately not allowed to make any admin changes to the repository. Regards, Gaute [0] https://github.com/sup-heliotrope/sup/pulls [1] https://github.com/sup-heliotrope/sup/wiki/Development From eg@gaute.vetsj.com Wed Jun 26 12:38:00 2013 From: eg@gaute.vetsj.com (Gaute Hope) Date: Wed, 26 Jun 2013 14:38:00 +0200 Subject: [sup-devel] Release Sup 0.13.2 and ncursesw-sup 1.3.1.3 Message-ID: <1372250076-sup-9668@qwerzila> Greetings, another release in the 0.13-line. Together with a new release of ncursesw-sup with help from Steven Lawrance Sup should now work on FreeBSD 10 (a wiki page with more info is in the works). $ gem install ncursesw-sup $ gem install sup Thanks to all contributors! Cheers, Gaute From dmishd@gmail.com Sun Jun 30 21:22:34 2013 From: dmishd@gmail.com (Hamish D) Date: Sun, 30 Jun 2013 22:22:34 +0100 Subject: [sup-devel] [sup-talk] Release 0.13.1 In-Reply-To: <1371897680-sup-4189@qwerzila> References: <1371816327-sup-6968@qwerzila> <1371897680-sup-4189@qwerzila> Message-ID: > I have asked William if he would agree to redirect to/mention the new setup > in the old gitorious repo. He gave me access to the sup-team, but I am > unfortunately not allowed to make any admin changes to the repository. > In lieu of anything better, I've updated the readme.txt in the gitorious copy, but I can't edit the description either. Hamish > > > Regards, Gaute > > > [0] https://github.com/sup-heliotrope/sup/pulls > [1] https://github.com/sup-heliotrope/sup/wiki/Development > > _______________________________________________ > Sup-devel mailing list > Sup-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/sup-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: