Subject: Re: [sup-devel] Experimental Gmail Source

From: Horacio Sanson <[email-masked]>
Date: Sun, 2 Jun 2013 11:45:47 +0900

Thanks for checking the source and sorry for the late response... I can
only look into this on rare free weekends.

On Wed, May 22, 2013 at 6:47 AM, Matthieu Rakotojaona <
[email-masked]> wrote:

> Hey Horacio,
>
> I took a stab at your gmail_source branch, and made a few
> fixes/improvements [0]:
>
> - Add configuration option in sup-add
> - Dump the LevelDB path in the sources.yaml
> - Add a load_from_yaml method for a source to initialize its working
>   values (for instance, the @db cannot be serialized, it needs to be
>   reconstructed)
> - Fixed the msg_att monkey-patch for imap.rb
>
>
Great, I will add these changes to my branch....

> All in all, the gmail source seems to work. I tested it on my usual
> gmail account, I haven't tried to download it all, but I did download a
> few dozens of emails without a problem. I'd like to warn users about
> LevelDB though: it's sad to say, but as other wmorgan's stuff, it looks
> abandoned. There are at least 2 bugs you will encounter if you try it: a
> pb in configuration (fixed in [1]) and you need the `snappy` gem to make
> it work if your db is more than 4MB large [2]. There are some up-to-date
> forks, though.

I see LevelDB is used mostly for storing messages and mailboxes
> uid{validity/last}, but if we are to use gmail (it's the only IMAP
> provider that makes sense for sup), I believe we would stick to the All
> Mail label, right ? So, no need for storing this in db, rather in the
> sources.yaml file. Also, if leveldb-ruby is unreliable (I did encounter
> some issues way back about something with glibc...), and we want to use
> it for caching messages, I think we can salvage heliotrope's zmbox [3]
> because it's so simple to use yet far better than simple mbox.

Using zmbox, mbox, maildir or any other mail storage (mix?) means I need to
keep track of three indexes to allow two way sync between the Gmail source
and the Sup index. I would need the Sup index id, the store id (e.g. zmbox
file index) and the Gmail X-GM-MSGID. That complicates things a lot.

Using key/value stores like LevelDB allows me to directly store the
messages and associate them directly with the Gmail X-GM-MSGID. Also
LevelDB comes with high compression for text data, perfect for emails, and
high performance [1]. The issues you mention seem to be on the ruby library
rather than LevelDB itself and they are fixable. If there are no bigger
issues (e.g. data corruption/loss) I will stick with LevelDB.

Regarding your ids questions, if you want to access the sup's messages
> from the gmail source, you could use the mail's Message-ID header and
> apply the same logic as in Message.sanitize_message_id. Caution,
> however: I've already encountered the case where multiple messages in
> GMail (i.e multiple X-GM-MSGID) have the same Message-ID, so they would
> be considered the same in sup/heliotrope... yeah, that's annoying as
> hell, and I don't know how we can solve this in the case of multiple
> sources.
>
>
Thanks, this comment put me on track and I found a way to get the emails
from the index using the message id provided by the source. All I need to
do is call Message.build_from_source(source, info) where info is the
message id provided by the source. In my case this would be the X-GM-MSGID
string.

If you want to sync-back, maybe sup can call a source-level "sync_back"
> method with the current known state ? Speaking of which, for general
> synchronization we could reuse the elegant offlineimap's sync algorithm
> [4]. The idea is basic: have each source class store a snapshot of the
> state.  When a message is modified on the source, diff the change with
> the known status and propagate to sup; when a message is modified in
> sup, diff with the known status and propagate to the source.
>
>
Interesting and simple algorithm. Let me study it a little more and see how
it is applicable to Sup.

[1]  http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html

regards,
Horacio

Just a brain dump.
>
> [0] https://github.com/rakoo/sup/tree/gmail_source
> [1] https://github.com/wmorgan/leveldb-ruby/pull/27
> [2] https://github.com/wmorgan/leveldb-ruby/issues/23
> [3]
> https://github.com/sup-heliotrope/heliotrope/blob/64d4b50d5649ec616a311a4cf6955137fdaeb13d/lib/heliotrope/zmbox.rb
> [4] http://offlineimap.org/howitworks.html
>
> Regards,
>
> --
> Matthieu Rakotojaona
>
> _______________________________________________
> Sup-devel mailing list
> [email-masked]
> http://rubyforge.org/mailman/listinfo/sup-devel
>
>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Gaute Hope <[email-masked]>
Date: Sat, 11 May 2013 12:15:07 +0200

On 09. mai 2013 11:28, Horacio Sanson wrote:
> I am trying to implement a new source for Gmail accounts. This is 
> copied from my efforts to do the same in Heliotrope.
> 
> Here is an experimental implementation that can read the email from 
> Gmail and add it to the Sup index:
> 
> https://github.com/hsanson/sup/tree/gmail_source
> 
> To use:
> 
> - Install leveldb gem "sudo gem install leveldb-ruby" - Add a gmail 
> source:   sup-add gmail://[email-masked] - Start sup and see how 
> it syncs your emails.
> 
> Warnings:
> 
> - This is experimental - This always syncs only the All mailbox so 
> make sure to use an account with not too many emails for testing. - 
> All email data and headers are stored in a LevelDB database at: 
> ~/.sup/gmail/account
> 

> - For some reason I get duplicate "Inbox" and "Sent" labels in the 
> list of labels and I am not sure why. 
>  - I still have no clue on how  to handle sync-back. That is how to
propagate
> changes made in sup
> back to Gmail. Any tips on how the maildir source does it would be 
> appreciated.

Hi Horacio,

nice work. Working directly with GMail labels is probably a good idea
(the other option is to move messages between IMAP folders). I have a
design question though:

Should remote sources be part of regular sup? Or should rather the
fetching and syncing be put in a separate script which creates a LevelDB
setup like you have it with a Gmail source in sup working directly on it?

Recall that the IMAP source was removed in 52e29ba [1] (discussion
probably on the mailinglist somewhere).

I like this approach for GMail, but I would like to see it for regular
IMAP sources as well with folders as labels.. I briefly experimented
with a maildir-root folder approach [2] which treats all underlying
maildirs as sources which correspond to a label (do not use,
incomplete). It of course presents a plethora of questions on how to
sync messages between labels, but implementing it is is probably
relatively straight forward.

> Help:
> 
> - How do I stop the source poll when I quit sup? If I have a large 
> amount of emails when polling is running and I quit sup the process
> hangs there. 

This normally runs in a separate thread, I don't think maildir really
stops the polling - so I sometimes get an error if I quit sup while the
polling is running and various stuff just disappears underneath the poller.

I don't have the chance to get into your other questions at the moment.

Regards, Gaute

[1] https://github.com/sup-heliotrope/sup/commit/52e29ba
[2] https://github.com/gauteh/sup/tree/maildir-root

---

From: Horacio Sanson <[email-masked]>
Date: Wed, 15 May 2013 11:10:22 +0900

Hello all,

I am still trying to implement sync-back functionality to my GMail source
but not going anywhere.
The problem I have is that the Sup index keeps the messages ids provided by
RMail (e.g. [email-masked]) and not the id I get
from the source that is the X-MSG-ID provided by Gmail
(e.g. 1434541737393941768). I need a way to query the index using the Gmail
provided id and get a message back or at least it's labels so I can compare
and update them on the server side if required.

This is what I have found so far:

  - There is no way to query the index for a message using the source
(Gmail) provided id.
  - The only places I can see where the source message ids are stored is in
a locations array
     that keeps the source own id and the source message id.
  - Sup index provides methods to query messages by id (build_message,
contains_id?, etc)
    but the id they accept is the message id provided my RMail, not the id
of the source.

Questions:

  - What is the purpose of the locations array? To allow a single message
to exists on
    multiple sources?
  - Is there an easy way to query the index to return a message/messages
using the
    source provided id?
  - Is a good idea to add a term to the index that keeps the source id
along with the message id?

regards,
Horacio

On Sun, May 12, 2013 at 3:18 AM, Horacio Sanson <[email-masked]> wrote:

>
>
>
> On Sat, May 11, 2013 at 7:15 PM, Gaute Hope <[email-masked]> wrote:
>
>>
>>
> I don't like the current two step sync setup that Sup uses now (IMAP ->
> Maildir -> Sup). It requires external programs (offlineimap) and
> synchronization is one way only. I know about the sync-back branch but
> AFAIK it is still limited to flags only. Also Maildir is an old storage
> format that doesn't work well with current email workflows. For example I
> have tons of duplicate emails on several folders due to the use of
> mailboxes rather than labels.
>
> I would prefer Sup to take care of the mail storage/indexing (as
> heliotrope does) and the sources be in charge of syncing the Sup
> storage/index with the remote servers. I understand that this can be
> difficult due to the difference in paradigm between IMAP/POP and Sup but
> GMail offers extensions that map directly to Sup workflow. GMail gives each
> email a unique 64bit ascending indentifier to all mail messages and adds
> labels them. I don't even need the LevelDB database as I could easily
> implement the source so it fetches the mail headers and body from the Gmail
> server directly when requested. I only added the LevelDB storage as a cache
> to speed up the message lookups and for offline use.
>
>
> Yes I know this and reading the commits and source code of this source I
> can tell that William hated the IMAP protocol. This is understandable as I
> myself have dealt with this protocol and know first hand how broken it is.
> But again with the extensions supported by GMail servers the implementation
> is far easier to do.
>
>
> IMAP is a horrible protocol and implementing it requires herculean effort.
> Still once the Gmail source is finished it can become a starting point for
> a more complete IMAP source.
>
>
>> > Help:
>> >
> I see that sup simply kills the threads... would be better if each source
> had a stop method that Sup could invoke to stop the polling. I will try to
> look into this issue as it is problematic for remote sources with large
> amounts of emails.
>
> Thanks for answering my inquiries. If you have a chance I would really
> like to know how to get the labels for a specific email from the index. I
> need this to implement the sync-back part of the Gmail source.
>
> regards,
> Horacio
>
>
>> Regards, Gaute
>>

---

From: Horacio Sanson <[email-masked]>
Date: Thu, 9 May 2013 18:28:10 +0900

I am trying to implement a new source for Gmail accounts. This is copied
from my efforts to do the same in Heliotrope.

Here is an experimental implementation that can read the email from Gmail
and add it to the Sup index:

To use:

  - Install leveldb gem "sudo gem install leveldb-ruby"
  - Add a gmail source:   sup-add gmail://[email-masked]
  - Start sup and see how it syncs your emails.

Warnings:

  - This is experimental
  - This always syncs only the All mailbox so make sure to use an
     account with not too many emails for testing.
  - All email data and headers are stored in a LevelDB database at:
       ~/.sup/gmail/account

Help:

  - How do I stop the source poll when I quit sup? If I have a large amount
    of emails when polling is running and I quit sup the process hangs
there.
    Does sup has a mechanism to stop the polling in progress?
  - For some reason I get duplicate "Inbox" and "Sent" labels in the list of
    labels and I am not sure why.
  - I still have no clue on how to handle sync-back. That is how to
propagate
    changes made in sup back to Gmail. Any tips on how the maildir source
    does it would be appreciated.

regards
Horacio Sanson

---

From: Horacio Sanson <[email-masked]>
Date: Sun, 12 May 2013 03:18:46 +0900

I would prefer Sup to take care of the mail storage/indexing (as heliotrope
does) and the sources be in charge of syncing the Sup storage/index with
the remote servers. I understand that this can be difficult due to the
difference in paradigm between IMAP/POP and Sup but GMail offers extensions
that map directly to Sup workflow. GMail gives each email a unique 64bit
ascending indentifier to all mail messages and adds labels them. I don't
even need the LevelDB database as I could easily implement the source so it
fetches the mail headers and body from the Gmail server directly when
requested. I only added the LevelDB storage as a cache to speed up the
message lookups and for offline use.

Thanks for answering my inquiries. If you have a chance I would really like
to know how to get the labels for a specific email from the index. I need
this to implement the sync-back part of the Gmail source.

regards,
Horacio

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Tue, 21 May 2013 23:47:16 +0200

Hey Horacio,

Just a brain dump.

[3] https://github.com/sup-heliotrope/heliotrope/blob/64d4b50d5649ec616a311a4cf6955137fdaeb13d/lib/heliotrope/zmbox.rb

Regards,

---

From: Eric Weikl <[email-masked]>
Date: Thu, 09 May 2013 21:13:12 +0200

Hi Horacio,

Excerpts from Horacio Sanson's message of 2013-05-09 11:28:10 +0200:
>     propagate changes made in sup back to Gmail. Any tips on how the
>     maildir source does it would be appreciated.

The maildir branch currently only syncs the flags supported by the
maildir spec (by manipulating the file name, see "What can I put in
info?"[1]). These flags can be synced back to GMail or other mail
providers using offlineimap.

To sync arbitrary labels, we would probably have to modify the original
mail headers, like GMail seems to do with the X-GM-LABELS label (I only
looked at your code, I don't have a GMail account).

Gaute and Matthieu started gathering some ideas in the Wiki[2].

Cheers,
Eric

Links:
[1] http://cr.yp.to/proto/maildir.html
[2] https://github.com/sup-heliotrope/sup/wiki/Development%3A-Maildir-syncback
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel