sup

community/pipermail-archives/sup-devel/2013-06.txt (10521B) - raw

 From hsanson@gmail.com  Sun Jun  2 02:45:47 2013
 From: hsanson@gmail.com (Horacio Sanson)
 Date: Sun, 2 Jun 2013 11:45:47 +0900
 Subject: [sup-devel] Experimental Gmail Source
 In-Reply-To: <1369172802-sup-2003@kpad>
 References: <CAHWBo_Y6jEWCH8O+m1iw5_p4qd8ghxgvCsa1+15xR7fh+6S0nA@mail.gmail.com>
 	<518E1A2B.2080903@gaute.vetsj.com>
 	<CAHWBo_ZMce1GV=ChkLrSNbz9Lf-DtPb4W479kChXmUFMQf5qpw@mail.gmail.com>
 	<CAHWBo_ZY7bWTxfqbQ1H3hu-q6mzKZm-YYe-+t0b-0M7xwUDfYQ@mail.gmail.com>
 	<1369172802-sup-2003@kpad>
 Message-ID: <CAHWBo_acPoWYtBrcd+NTkOit7L7Vx_=Tkxequ-iQimPugWjsJQ@mail.gmail.com>
 
 Thanks for checking the source and sorry for the late response... I can
 only look into this on rare free weekends.
 
 
 On Wed, May 22, 2013 at 6:47 AM, Matthieu Rakotojaona <
 matthieu.rakotojaona at gmail.com> wrote:
 
 > Hey Horacio,
 >
 > I took a stab at your gmail_source branch, and made a few
 > fixes/improvements [0]:
 >
 > - Add configuration option in sup-add
 > - Dump the LevelDB path in the sources.yaml
 > - Add a load_from_yaml method for a source to initialize its working
 >   values (for instance, the @db cannot be serialized, it needs to be
 >   reconstructed)
 > - Fixed the msg_att monkey-patch for imap.rb
 >
 >
 Great, I will add these changes to my branch....
 
 
 > All in all, the gmail source seems to work. I tested it on my usual
 > gmail account, I haven't tried to download it all, but I did download a
 > few dozens of emails without a problem. I'd like to warn users about
 > LevelDB though: it's sad to say, but as other wmorgan's stuff, it looks
 > abandoned. There are at least 2 bugs you will encounter if you try it: a
 > pb in configuration (fixed in [1]) and you need the `snappy` gem to make
 > it work if your db is more than 4MB large [2]. There are some up-to-date
 > forks, though.
 
 
 
 >
 >
 I see LevelDB is used mostly for storing messages and mailboxes
 > uid{validity/last}, but if we are to use gmail (it's the only IMAP
 > provider that makes sense for sup), I believe we would stick to the All
 > Mail label, right ? So, no need for storing this in db, rather in the
 > sources.yaml file. Also, if leveldb-ruby is unreliable (I did encounter
 > some issues way back about something with glibc...), and we want to use
 > it for caching messages, I think we can salvage heliotrope's zmbox [3]
 > because it's so simple to use yet far better than simple mbox.
 
 
 Using zmbox, mbox, maildir or any other mail storage (mix?) means I need to
 keep track of three indexes to allow two way sync between the Gmail source
 and the Sup index. I would need the Sup index id, the store id (e.g. zmbox
 file index) and the Gmail X-GM-MSGID. That complicates things a lot.
 
 Using key/value stores like LevelDB allows me to directly store the
 messages and associate them directly with the Gmail X-GM-MSGID. Also
 LevelDB comes with high compression for text data, perfect for emails, and
 high performance [1]. The issues you mention seem to be on the ruby library
 rather than LevelDB itself and they are fixable. If there are no bigger
 issues (e.g. data corruption/loss) I will stick with LevelDB.
 
 Regarding your ids questions, if you want to access the sup's messages
 > from the gmail source, you could use the mail's Message-ID header and
 > apply the same logic as in Message.sanitize_message_id. Caution,
 > however: I've already encountered the case where multiple messages in
 > GMail (i.e multiple X-GM-MSGID) have the same Message-ID, so they would
 > be considered the same in sup/heliotrope... yeah, that's annoying as
 > hell, and I don't know how we can solve this in the case of multiple
 > sources.
 >
 >
 Thanks, this comment put me on track and I found a way to get the emails
 from the index using the message id provided by the source. All I need to
 do is call Message.build_from_source(source, info) where info is the
 message id provided by the source. In my case this would be the X-GM-MSGID
 string.
 
 If you want to sync-back, maybe sup can call a source-level "sync_back"
 > method with the current known state ? Speaking of which, for general
 > synchronization we could reuse the elegant offlineimap's sync algorithm
 > [4]. The idea is basic: have each source class store a snapshot of the
 > state.  When a message is modified on the source, diff the change with
 > the known status and propagate to sup; when a message is modified in
 > sup, diff with the known status and propagate to the source.
 >
 >
 Interesting and simple algorithm. Let me study it a little more and see how
 it is applicable to Sup.
 
 [1]  http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html
 
 regards,
 Horacio
 
 Just a brain dump.
 >
 > [0] https://github.com/rakoo/sup/tree/gmail_source
 > [1] https://github.com/wmorgan/leveldb-ruby/pull/27
 > [2] https://github.com/wmorgan/leveldb-ruby/issues/23
 > [3]
 > https://github.com/sup-heliotrope/heliotrope/blob/64d4b50d5649ec616a311a4cf6955137fdaeb13d/lib/heliotrope/zmbox.rb
 > [4] http://offlineimap.org/howitworks.html
 >
 > Regards,
 >
 > --
 > Matthieu Rakotojaona
 >
 > _______________________________________________
 > Sup-devel mailing list
 > Sup-devel at rubyforge.org
 > http://rubyforge.org/mailman/listinfo/sup-devel
 >
 >
 -------------- next part --------------
 An HTML attachment was scrubbed...
 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20130602/acec77c1/attachment.html>
 
 From eg@gaute.vetsj.com  Thu Jun 13 20:33:28 2013
 From: eg@gaute.vetsj.com (Gaute Hope)
 Date: Thu, 13 Jun 2013 22:33:28 +0200
 Subject: [sup-devel] going for 0.13.1 and possibly 0.14?
 Message-ID: <1371155525-sup-5029@qwerzila>
 
 Greetings,
 
 as briefly discussed in #75 [1] some adjustments to the original plan should
 be made. We were thinking:
 
   - Release another release in line-0.13 compatible with ruby 1.8 since
     we are starting to diverge in develop now. any efforts in fixing and
     maintaining ruby 1.8 support should be backported to this branch.
 
   - Work towards a 0.14 release with ruby 1.9 support on 'develop' and not
     necessarily with ruby 2.0 support yet. Already a several fixes have
     been done that have break ruby 1.8 which would be nice to get
     released in not too long.
 
 
 Any thoughts?
 
 
 Ruby 2.0: I am currently writing this from the tip of #61,
 rmail-to-mail-inital [2]. Having successfully indexed >30k (except 1..)
 of highly heterogeneous mails in all kinds of encodings. Tests are now
 also working, as well as sending email obviously. Still lots of parts are
 incomplete (e.g. no attachments yet..).
 
 The effort to get to a sup with ruby 2.0 support (with maildir-syncback,
 Mail - not RMail) has not reached a stable level. But I hope it could
 get to a state where it makes sense to include it in a separate branch
 in 'origin' so that it gets more available and easier to work against
 for everybody.
 
 Regards, Gaute
 
 [1] https://github.com/sup-heliotrope/sup/pull/75
 [2] https://github.com/sup-heliotrope/sup/pull/61
 
 
 From eg@gaute.vetsj.com  Fri Jun 21 12:07:19 2013
 From: eg@gaute.vetsj.com (Gaute Hope)
 Date: Fri, 21 Jun 2013 14:07:19 +0200
 Subject: [sup-devel] Release 0.13.1
 Message-ID: <1371816327-sup-6968@qwerzila>
 
 Greetings,
 
 I have just pushed version 0.13.1 of Sup to rubygems. This release holds
 should still be Ruby 1.8 compatible and holds the backported patches
 since the last release. 
 
 No big changes, but a number of fixes has been applied.
 
 Regards, Gaute
 
 
 From eg@gaute.vetsj.com  Sat Jun 22 10:48:13 2013
 From: eg@gaute.vetsj.com (Gaute Hope)
 Date: Sat, 22 Jun 2013 12:48:13 +0200
 Subject: [sup-devel] [sup-talk] Release 0.13.1
 In-Reply-To: <CA+ds9gJCfauTgS1LvrsOF0QREAKh9pKy=a550nj=1ZNi7wBvzA@mail.gmail.com>
 References: <1371816327-sup-6968@qwerzila>
 	<CA+ds9gJCfauTgS1LvrsOF0QREAKh9pKy=a550nj=1ZNi7wBvzA@mail.gmail.com>
 Message-ID: <1371897680-sup-4189@qwerzila>
 
 Excerpts from James Pearson's message of 2013-06-22 04:10:29 +0200:
 > I was going to ask why the tags on the Gitorious repo[0] hadn't been
 > updated, when I visited the (new!) website and saw that development has
 > moved onto Github[1] (and restarted!).  I'm apparently very behind the
 > times - and perhaps now I should dig out those changes I made and submit
 > them...
 > 
 > Glad to see things moving along again.
 >  - P
 > 
 > [0]: http://gitorious.org/sup/mainline
 > [1]: https://github.com/sup-heliotrope/sup
 
 Yup! Dig 'em up. We generally prefer pull requests [0] now a days. Also,
 we've collected contribution/development resources in the new wiki [1].
 
 
 I have asked William if he would agree to redirect to/mention the new setup
 in the old gitorious repo. He gave me access to the sup-team, but I am
 unfortunately not allowed to make any admin changes to the repository.
 
 
 Regards, Gaute
 
 
 [0] https://github.com/sup-heliotrope/sup/pulls
 [1] https://github.com/sup-heliotrope/sup/wiki/Development
 
 
 From eg@gaute.vetsj.com  Wed Jun 26 12:38:00 2013
 From: eg@gaute.vetsj.com (Gaute Hope)
 Date: Wed, 26 Jun 2013 14:38:00 +0200
 Subject: [sup-devel] Release Sup 0.13.2 and ncursesw-sup 1.3.1.3
 Message-ID: <1372250076-sup-9668@qwerzila>
 
 Greetings,
 
 another release in the 0.13-line. Together with a new release of
 ncursesw-sup with help from Steven Lawrance Sup should now work on
 FreeBSD 10 (a wiki page with more info is in the works).
 
 $ gem install ncursesw-sup
 $ gem install sup
 
 
 Thanks to all contributors!
 
 
 Cheers,
 Gaute
 
 
 From dmishd@gmail.com  Sun Jun 30 21:22:34 2013
 From: dmishd@gmail.com (Hamish D)
 Date: Sun, 30 Jun 2013 22:22:34 +0100
 Subject: [sup-devel] [sup-talk] Release 0.13.1
 In-Reply-To: <1371897680-sup-4189@qwerzila>
 References: <1371816327-sup-6968@qwerzila>
 	<CA+ds9gJCfauTgS1LvrsOF0QREAKh9pKy=a550nj=1ZNi7wBvzA@mail.gmail.com>
 	<1371897680-sup-4189@qwerzila>
 Message-ID: <CAOxvSbdps4S0g=iEANYpjEDvN6xy78+j_-ifvXaXU8TarN6BVg@mail.gmail.com>
 
 > I have asked William if he would agree to redirect to/mention the new setup
 > in the old gitorious repo. He gave me access to the sup-team, but I am
 > unfortunately not allowed to make any admin changes to the repository.
 >
 
 In lieu of anything better, I've updated the readme.txt in the gitorious
 copy, but I can't edit the description either.
 
 Hamish
 
 
 >
 >
 > Regards, Gaute
 >
 >
 > [0] https://github.com/sup-heliotrope/sup/pulls
 > [1] https://github.com/sup-heliotrope/sup/wiki/Development
 >
 > _______________________________________________
 > Sup-devel mailing list
 > Sup-devel at rubyforge.org
 > http://rubyforge.org/mailman/listinfo/sup-devel
 >
 -------------- next part --------------
 An HTML attachment was scrubbed...
 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20130630/f7daad95/attachment.html>