Re: [sup-devel] Experimental Gmail Source

From: Horacio Sanson <hsanson@gmail.com>
To: Sup developer discussion <sup-devel@rubyforge.org>
Subject: Re: [sup-devel] Experimental Gmail Source
Date: Sun, 2 Jun 2013 11:45:47 +0900	[thread overview]
Message-ID: <CAHWBo_acPoWYtBrcd+NTkOit7L7Vx_=Tkxequ-iQimPugWjsJQ@mail.gmail.com> (raw)
In-Reply-To: <1369172802-sup-2003@kpad>

[-- Attachment #1.1: Type: text/plain, Size: 4542 bytes --]

Thanks for checking the source and sorry for the late response... I can
only look into this on rare free weekends.

On Wed, May 22, 2013 at 6:47 AM, Matthieu Rakotojaona <
matthieu.rakotojaona@gmail.com> wrote:

> Hey Horacio,
>
> I took a stab at your gmail_source branch, and made a few
> fixes/improvements [0]:
>
> - Add configuration option in sup-add
> - Dump the LevelDB path in the sources.yaml
> - Add a load_from_yaml method for a source to initialize its working
>   values (for instance, the @db cannot be serialized, it needs to be
>   reconstructed)
> - Fixed the msg_att monkey-patch for imap.rb
>
>
Great, I will add these changes to my branch....

> All in all, the gmail source seems to work. I tested it on my usual
> gmail account, I haven't tried to download it all, but I did download a
> few dozens of emails without a problem. I'd like to warn users about
> LevelDB though: it's sad to say, but as other wmorgan's stuff, it looks
> abandoned. There are at least 2 bugs you will encounter if you try it: a
> pb in configuration (fixed in [1]) and you need the `snappy` gem to make
> it work if your db is more than 4MB large [2]. There are some up-to-date
> forks, though.

>
>
I see LevelDB is used mostly for storing messages and mailboxes
> uid{validity/last}, but if we are to use gmail (it's the only IMAP
> provider that makes sense for sup), I believe we would stick to the All
> Mail label, right ? So, no need for storing this in db, rather in the
> sources.yaml file. Also, if leveldb-ruby is unreliable (I did encounter
> some issues way back about something with glibc...), and we want to use
> it for caching messages, I think we can salvage heliotrope's zmbox [3]
> because it's so simple to use yet far better than simple mbox.

Using zmbox, mbox, maildir or any other mail storage (mix?) means I need to
keep track of three indexes to allow two way sync between the Gmail source
and the Sup index. I would need the Sup index id, the store id (e.g. zmbox
file index) and the Gmail X-GM-MSGID. That complicates things a lot.

Using key/value stores like LevelDB allows me to directly store the
messages and associate them directly with the Gmail X-GM-MSGID. Also
LevelDB comes with high compression for text data, perfect for emails, and
high performance [1]. The issues you mention seem to be on the ruby library
rather than LevelDB itself and they are fixable. If there are no bigger
issues (e.g. data corruption/loss) I will stick with LevelDB.

Regarding your ids questions, if you want to access the sup's messages
> from the gmail source, you could use the mail's Message-ID header and
> apply the same logic as in Message.sanitize_message_id. Caution,
> however: I've already encountered the case where multiple messages in
> GMail (i.e multiple X-GM-MSGID) have the same Message-ID, so they would
> be considered the same in sup/heliotrope... yeah, that's annoying as
> hell, and I don't know how we can solve this in the case of multiple
> sources.
>
>
Thanks, this comment put me on track and I found a way to get the emails
from the index using the message id provided by the source. All I need to
do is call Message.build_from_source(source, info) where info is the
message id provided by the source. In my case this would be the X-GM-MSGID
string.

If you want to sync-back, maybe sup can call a source-level "sync_back"
> method with the current known state ? Speaking of which, for general
> synchronization we could reuse the elegant offlineimap's sync algorithm
> [4]. The idea is basic: have each source class store a snapshot of the
> state.  When a message is modified on the source, diff the change with
> the known status and propagate to sup; when a message is modified in
> sup, diff with the known status and propagate to the source.
>
>
Interesting and simple algorithm. Let me study it a little more and see how
it is applicable to Sup.

[1]  http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html

regards,
Horacio

Just a brain dump.
>
> [0] https://github.com/rakoo/sup/tree/gmail_source
> [1] https://github.com/wmorgan/leveldb-ruby/pull/27
> [2] https://github.com/wmorgan/leveldb-ruby/issues/23
> [3]
> https://github.com/sup-heliotrope/heliotrope/blob/64d4b50d5649ec616a311a4cf6955137fdaeb13d/lib/heliotrope/zmbox.rb
> [4] http://offlineimap.org/howitworks.html
>
> Regards,
>
> --
> Matthieu Rakotojaona
>
> _______________________________________________
> Sup-devel mailing list
> Sup-devel@rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-devel
>
>

[-- Attachment #1.2: Type: text/html, Size: 7223 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel