Archive of RubyForge sup-devel mailing list
 help / color / mirror / Atom feed

* [sup-devel] Experimental Gmail Source
@ 2013-05-09  9:28 Horacio Sanson
  2013-05-09 19:13 ` Eric Weikl
  2013-05-11 10:15 ` Gaute Hope
  0 siblings, 2 replies; 7+ messages in thread
From: Horacio Sanson @ 2013-05-09  9:28 UTC (permalink / raw)
  To: Sup developer discussion


[-- Attachment #1.1: Type: text/plain, Size: 1243 bytes --]

I am trying to implement a new source for Gmail accounts. This is copied
from my efforts to do the same in Heliotrope.

Here is an experimental implementation that can read the email from Gmail
and add it to the Sup index:

https://github.com/hsanson/sup/tree/gmail_source

To use:

  - Install leveldb gem "sudo gem install leveldb-ruby"
  - Add a gmail source:   sup-add gmail://username@gmail.com
  - Start sup and see how it syncs your emails.

Warnings:

  - This is experimental
  - This always syncs only the All mailbox so make sure to use an
     account with not too many emails for testing.
  - All email data and headers are stored in a LevelDB database at:
       ~/.sup/gmail/account

Help:

  - How do I stop the source poll when I quit sup? If I have a large amount
    of emails when polling is running and I quit sup the process hangs
there.
    Does sup has a mechanism to stop the polling in progress?
  - For some reason I get duplicate "Inbox" and "Sent" labels in the list of
    labels and I am not sure why.
  - I still have no clue on how to handle sync-back. That is how to
propagate
    changes made in sup back to Gmail. Any tips on how the maildir source
    does it would be appreciated.

regards
Horacio Sanson

[-- Attachment #1.2: Type: text/html, Size: 1958 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel

^ permalink raw reply	[flat|nested] 7+ messages in thread

• * Re: [sup-devel] Experimental Gmail Source
    2013-05-09  9:28 [sup-devel] Experimental Gmail Source Horacio Sanson
  @ 2013-05-09 19:13 ` Eric Weikl
    2013-05-11 10:15 ` Gaute Hope
    1 sibling, 0 replies; 7+ messages in thread
  From: Eric Weikl @ 2013-05-09 19:13 UTC (permalink / raw)
    To: Horacio Sanson; +Cc: Sup developer discussion
  
  Hi Horacio,
  
  Excerpts from Horacio Sanson's message of 2013-05-09 11:28:10 +0200:
  >   - I still have no clue on how to handle sync-back. That is how to
  >     propagate changes made in sup back to Gmail. Any tips on how the
  >     maildir source does it would be appreciated.
  
  The maildir branch currently only syncs the flags supported by the
  maildir spec (by manipulating the file name, see "What can I put in
  info?"[1]). These flags can be synced back to GMail or other mail
  providers using offlineimap.
  
  To sync arbitrary labels, we would probably have to modify the original
  mail headers, like GMail seems to do with the X-GM-LABELS label (I only
  looked at your code, I don't have a GMail account).
  
  Gaute and Matthieu started gathering some ideas in the Wiki[2].
  
  Cheers,
  Eric
  
  Links:
  [1] http://cr.yp.to/proto/maildir.html
  [2] https://github.com/sup-heliotrope/sup/wiki/Development%3A-Maildir-syncback
  _______________________________________________
  Sup-devel mailing list
  Sup-devel@rubyforge.org
  http://rubyforge.org/mailman/listinfo/sup-devel
  
  
  ^ permalink raw reply	[flat|nested] 7+ messages in thread

• * Re: [sup-devel] Experimental Gmail Source
    2013-05-09  9:28 [sup-devel] Experimental Gmail Source Horacio Sanson
    2013-05-09 19:13 ` Eric Weikl
  @ 2013-05-11 10:15 ` Gaute Hope
    2013-05-11 18:18   ` Horacio Sanson
    1 sibling, 1 reply; 7+ messages in thread
  From: Gaute Hope @ 2013-05-11 10:15 UTC (permalink / raw)
    To: Sup developer discussion; +Cc: Horacio Sanson
  
  
  
  On 09. mai 2013 11:28, Horacio Sanson wrote:
  > I am trying to implement a new source for Gmail accounts. This is 
  > copied from my efforts to do the same in Heliotrope.
  > 
  > Here is an experimental implementation that can read the email from 
  > Gmail and add it to the Sup index:
  > 
  > https://github.com/hsanson/sup/tree/gmail_source
  > 
  > To use:
  > 
  > - Install leveldb gem "sudo gem install leveldb-ruby" - Add a gmail 
  > source:   sup-add gmail://username@gmail.com - Start sup and see how 
  > it syncs your emails.
  > 
  > Warnings:
  > 
  > - This is experimental - This always syncs only the All mailbox so 
  > make sure to use an account with not too many emails for testing. - 
  > All email data and headers are stored in a LevelDB database at: 
  > ~/.sup/gmail/account
  > 
  
  > - For some reason I get duplicate "Inbox" and "Sent" labels in the 
  > list of labels and I am not sure why. 
  >  - I still have no clue on how  to handle sync-back. That is how to
  propagate
  > changes made in sup
  > back to Gmail. Any tips on how the maildir source does it would be 
  > appreciated.
  
  Hi Horacio,
  
  nice work. Working directly with GMail labels is probably a good idea
  (the other option is to move messages between IMAP folders). I have a
  design question though:
  
  Should remote sources be part of regular sup? Or should rather the
  fetching and syncing be put in a separate script which creates a LevelDB
  setup like you have it with a Gmail source in sup working directly on it?
  
  Recall that the IMAP source was removed in 52e29ba [1] (discussion
  probably on the mailinglist somewhere).
  
  I like this approach for GMail, but I would like to see it for regular
  IMAP sources as well with folders as labels.. I briefly experimented
  with a maildir-root folder approach [2] which treats all underlying
  maildirs as sources which correspond to a label (do not use,
  incomplete). It of course presents a plethora of questions on how to
  sync messages between labels, but implementing it is is probably
  relatively straight forward.
  
  > Help:
  > 
  > - How do I stop the source poll when I quit sup? If I have a large 
  > amount of emails when polling is running and I quit sup the process
  > hangs there. 
  
  This normally runs in a separate thread, I don't think maildir really
  stops the polling - so I sometimes get an error if I quit sup while the
  polling is running and various stuff just disappears underneath the poller.
  
  I don't have the chance to get into your other questions at the moment.
  
  Regards, Gaute
  
  [1] https://github.com/sup-heliotrope/sup/commit/52e29ba
  [2] https://github.com/gauteh/sup/tree/maildir-root
  
  
  ^ permalink raw reply	[flat|nested] 7+ messages in thread

• • * Re: [sup-devel] Experimental Gmail Source
      2013-05-11 10:15 ` Gaute Hope
    @ 2013-05-11 18:18   ` Horacio Sanson
      2013-05-15  2:10     ` Horacio Sanson
      0 siblings, 1 reply; 7+ messages in thread
    From: Horacio Sanson @ 2013-05-11 18:18 UTC (permalink / raw)
      To: Gaute Hope; +Cc: Sup developer discussion
    
    [-- Attachment #1: Type: text/plain, Size: 4900 bytes --]
    
    On Sat, May 11, 2013 at 7:15 PM, Gaute Hope <eg@gaute.vetsj.com> wrote:
    
    >
    >
    > On 09. mai 2013 11:28, Horacio Sanson wrote:
    > > I am trying to implement a new source for Gmail accounts. This is
    > > copied from my efforts to do the same in Heliotrope.
    > >
    > > Here is an experimental implementation that can read the email from
    > > Gmail and add it to the Sup index:
    > >
    > > https://github.com/hsanson/sup/tree/gmail_source
    > >
    > > To use:
    > >
    > > - Install leveldb gem "sudo gem install leveldb-ruby" - Add a gmail
    > > source:   sup-add gmail://username@gmail.com - Start sup and see how
    > > it syncs your emails.
    > >
    > > Warnings:
    > >
    > > - This is experimental - This always syncs only the All mailbox so
    > > make sure to use an account with not too many emails for testing. -
    > > All email data and headers are stored in a LevelDB database at:
    > > ~/.sup/gmail/account
    > >
    >
    > > - For some reason I get duplicate "Inbox" and "Sent" labels in the
    > > list of labels and I am not sure why.
    > >  - I still have no clue on how  to handle sync-back. That is how to
    > propagate
    > > changes made in sup
    > > back to Gmail. Any tips on how the maildir source does it would be
    > > appreciated.
    >
    > Hi Horacio,
    >
    > nice work. Working directly with GMail labels is probably a good idea
    > (the other option is to move messages between IMAP folders). I have a
    > design question though:
    >
    > Should remote sources be part of regular sup? Or should rather the
    > fetching and syncing be put in a separate script which creates a LevelDB
    > setup like you have it with a Gmail source in sup working directly on it?
    >
    
    
    I don't like the current two step sync setup that Sup uses now (IMAP ->
    Maildir -> Sup). It requires external programs (offlineimap) and
    synchronization is one way only. I know about the sync-back branch but
    AFAIK it is still limited to flags only. Also Maildir is an old storage
    format that doesn't work well with current email workflows. For example I
    have tons of duplicate emails on several folders due to the use of
    mailboxes rather than labels.
    
    I would prefer Sup to take care of the mail storage/indexing (as heliotrope
    does) and the sources be in charge of syncing the Sup storage/index with
    the remote servers. I understand that this can be difficult due to the
    difference in paradigm between IMAP/POP and Sup but GMail offers extensions
    that map directly to Sup workflow. GMail gives each email a unique 64bit
    ascending indentifier to all mail messages and adds labels them. I don't
    even need the LevelDB database as I could easily implement the source so it
    fetches the mail headers and body from the Gmail server directly when
    requested. I only added the LevelDB storage as a cache to speed up the
    message lookups and for offline use.
    
    
    > Recall that the IMAP source was removed in 52e29ba [1] (discussion
    > probably on the mailinglist somewhere).
    >
    >
    Yes I know this and reading the commits and source code of this source I
    can tell that William hated the IMAP protocol. This is understandable as I
    myself have dealt with this protocol and know first hand how broken it is.
    But again with the extensions supported by GMail servers the implementation
    is far easier to do.
    
    
    > I like this approach for GMail, but I would like to see it for regular
    > IMAP sources as well with folders as labels.. I briefly experimented
    > with a maildir-root folder approach [2] which treats all underlying
    > maildirs as sources which correspond to a label (do not use,
    > incomplete). It of course presents a plethora of questions on how to
    > sync messages between labels, but implementing it is is probably
    > relatively straight forward.
    >
    >
    IMAP is a horrible protocol and implementing it requires herculean effort.
    Still once the Gmail source is finished it can become a starting point for
    a more complete IMAP source.
    
    
    > > Help:
    > >
    > > - How do I stop the source poll when I quit sup? If I have a large
    > > amount of emails when polling is running and I quit sup the process
    > > hangs there.
    >
    > This normally runs in a separate thread, I don't think maildir really
    > stops the polling - so I sometimes get an error if I quit sup while the
    > polling is running and various stuff just disappears underneath the poller.
    >
    >
    I see that sup simply kills the threads... would be better if each source
    had a stop method that Sup could invoke to stop the polling. I will try to
    look into this issue as it is problematic for remote sources with large
    amounts of emails.
    
    I don't have the chance to get into your other questions at the moment.
    >
    >
    Thanks for answering my inquiries. If you have a chance I would really like
    to know how to get the labels for a specific email from the index. I need
    this to implement the sync-back part of the Gmail source.
    
    regards,
    Horacio
    
    
    > Regards, Gaute
    >
    > [1] https://github.com/sup-heliotrope/sup/commit/52e29ba
    > [2] https://github.com/gauteh/sup/tree/maildir-root
    
    [-- Attachment #2: Type: text/html, Size: 6822 bytes --]
    
    ^ permalink raw reply	[flat|nested] 7+ messages in thread

  • • * Re: [sup-devel] Experimental Gmail Source
        2013-05-11 18:18   ` Horacio Sanson
      @ 2013-05-15  2:10     ` Horacio Sanson
        2013-05-21 21:47       ` Matthieu Rakotojaona
        0 siblings, 1 reply; 7+ messages in thread
      From: Horacio Sanson @ 2013-05-15  2:10 UTC (permalink / raw)
        To: Gaute Hope; +Cc: Sup developer discussion
      
      [-- Attachment #1: Type: text/plain, Size: 6474 bytes --]
      
      Hello all,
      
      I am still trying to implement sync-back functionality to my GMail source
      but not going anywhere.
      The problem I have is that the Sup index keeps the messages ids provided by
      RMail (e.g. 20cf301af801a2aa8b04dc6e9931@google.com) and not the id I get
      from the source that is the X-MSG-ID provided by Gmail
      (e.g. 1434541737393941768). I need a way to query the index using the Gmail
      provided id and get a message back or at least it's labels so I can compare
      and update them on the server side if required.
      
      This is what I have found so far:
      
        - There is no way to query the index for a message using the source
      (Gmail) provided id.
        - The only places I can see where the source message ids are stored is in
      a locations array
           that keeps the source own id and the source message id.
        - Sup index provides methods to query messages by id (build_message,
      contains_id?, etc)
          but the id they accept is the message id provided my RMail, not the id
      of the source.
      
      Questions:
      
        - What is the purpose of the locations array? To allow a single message
      to exists on
          multiple sources?
        - Is there an easy way to query the index to return a message/messages
      using the
          source provided id?
        - Is a good idea to add a term to the index that keeps the source id
      along with the message id?
      
      regards,
      Horacio
      
      
      
      
      On Sun, May 12, 2013 at 3:18 AM, Horacio Sanson <hsanson@gmail.com> wrote:
      
      >
      >
      >
      > On Sat, May 11, 2013 at 7:15 PM, Gaute Hope <eg@gaute.vetsj.com> wrote:
      >
      >>
      >>
      >> On 09. mai 2013 11:28, Horacio Sanson wrote:
      >> > I am trying to implement a new source for Gmail accounts. This is
      >> > copied from my efforts to do the same in Heliotrope.
      >> >
      >> > Here is an experimental implementation that can read the email from
      >> > Gmail and add it to the Sup index:
      >> >
      >> > https://github.com/hsanson/sup/tree/gmail_source
      >> >
      >> > To use:
      >> >
      >> > - Install leveldb gem "sudo gem install leveldb-ruby" - Add a gmail
      >> > source:   sup-add gmail://username@gmail.com - Start sup and see how
      >> > it syncs your emails.
      >> >
      >> > Warnings:
      >> >
      >> > - This is experimental - This always syncs only the All mailbox so
      >> > make sure to use an account with not too many emails for testing. -
      >> > All email data and headers are stored in a LevelDB database at:
      >> > ~/.sup/gmail/account
      >> >
      >>
      >> > - For some reason I get duplicate "Inbox" and "Sent" labels in the
      >> > list of labels and I am not sure why.
      >> >  - I still have no clue on how  to handle sync-back. That is how to
      >> propagate
      >> > changes made in sup
      >> > back to Gmail. Any tips on how the maildir source does it would be
      >> > appreciated.
      >>
      >> Hi Horacio,
      >>
      >> nice work. Working directly with GMail labels is probably a good idea
      >> (the other option is to move messages between IMAP folders). I have a
      >> design question though:
      >>
      >> Should remote sources be part of regular sup? Or should rather the
      >> fetching and syncing be put in a separate script which creates a LevelDB
      >> setup like you have it with a Gmail source in sup working directly on it?
      >>
      >
      >
      > I don't like the current two step sync setup that Sup uses now (IMAP ->
      > Maildir -> Sup). It requires external programs (offlineimap) and
      > synchronization is one way only. I know about the sync-back branch but
      > AFAIK it is still limited to flags only. Also Maildir is an old storage
      > format that doesn't work well with current email workflows. For example I
      > have tons of duplicate emails on several folders due to the use of
      > mailboxes rather than labels.
      >
      > I would prefer Sup to take care of the mail storage/indexing (as
      > heliotrope does) and the sources be in charge of syncing the Sup
      > storage/index with the remote servers. I understand that this can be
      > difficult due to the difference in paradigm between IMAP/POP and Sup but
      > GMail offers extensions that map directly to Sup workflow. GMail gives each
      > email a unique 64bit ascending indentifier to all mail messages and adds
      > labels them. I don't even need the LevelDB database as I could easily
      > implement the source so it fetches the mail headers and body from the Gmail
      > server directly when requested. I only added the LevelDB storage as a cache
      > to speed up the message lookups and for offline use.
      >
      >
      >> Recall that the IMAP source was removed in 52e29ba [1] (discussion
      >> probably on the mailinglist somewhere).
      >>
      >>
      > Yes I know this and reading the commits and source code of this source I
      > can tell that William hated the IMAP protocol. This is understandable as I
      > myself have dealt with this protocol and know first hand how broken it is.
      > But again with the extensions supported by GMail servers the implementation
      > is far easier to do.
      >
      >
      >> I like this approach for GMail, but I would like to see it for regular
      >> IMAP sources as well with folders as labels.. I briefly experimented
      >> with a maildir-root folder approach [2] which treats all underlying
      >> maildirs as sources which correspond to a label (do not use,
      >> incomplete). It of course presents a plethora of questions on how to
      >> sync messages between labels, but implementing it is is probably
      >> relatively straight forward.
      >>
      >>
      > IMAP is a horrible protocol and implementing it requires herculean effort.
      > Still once the Gmail source is finished it can become a starting point for
      > a more complete IMAP source.
      >
      >
      >> > Help:
      >> >
      >> > - How do I stop the source poll when I quit sup? If I have a large
      >> > amount of emails when polling is running and I quit sup the process
      >> > hangs there.
      >>
      >> This normally runs in a separate thread, I don't think maildir really
      >> stops the polling - so I sometimes get an error if I quit sup while the
      >> polling is running and various stuff just disappears underneath the
      >> poller.
      >>
      >>
      > I see that sup simply kills the threads... would be better if each source
      > had a stop method that Sup could invoke to stop the polling. I will try to
      > look into this issue as it is problematic for remote sources with large
      > amounts of emails.
      >
      > I don't have the chance to get into your other questions at the moment.
      >>
      >>
      > Thanks for answering my inquiries. If you have a chance I would really
      > like to know how to get the labels for a specific email from the index. I
      > need this to implement the sync-back part of the Gmail source.
      >
      > regards,
      > Horacio
      >
      >
      >> Regards, Gaute
      >>
      >> [1] https://github.com/sup-heliotrope/sup/commit/52e29ba
      >> [2] https://github.com/gauteh/sup/tree/maildir-root
      >
      >
      
      [-- Attachment #2: Type: text/html, Size: 9078 bytes --]
      
      ^ permalink raw reply	[flat|nested] 7+ messages in thread

    • • * Re: [sup-devel] Experimental Gmail Source
          2013-05-15  2:10     ` Horacio Sanson
        @ 2013-05-21 21:47       ` Matthieu Rakotojaona
          2013-06-02  2:45         ` Horacio Sanson
          0 siblings, 1 reply; 7+ messages in thread
        From: Matthieu Rakotojaona @ 2013-05-21 21:47 UTC (permalink / raw)
          To: sup-devel
        
        
        [-- Attachment #1.1: Type: text/plain, Size: 2864 bytes --]
        
        Hey Horacio,
        
        I took a stab at your gmail_source branch, and made a few
        fixes/improvements [0]:
        
        - Add configuration option in sup-add
        - Dump the LevelDB path in the sources.yaml
        - Add a load_from_yaml method for a source to initialize its working
          values (for instance, the @db cannot be serialized, it needs to be
          reconstructed)
        - Fixed the msg_att monkey-patch for imap.rb
        
        All in all, the gmail source seems to work. I tested it on my usual
        gmail account, I haven't tried to download it all, but I did download a
        few dozens of emails without a problem. I'd like to warn users about
        LevelDB though: it's sad to say, but as other wmorgan's stuff, it looks
        abandoned. There are at least 2 bugs you will encounter if you try it: a
        pb in configuration (fixed in [1]) and you need the `snappy` gem to make
        it work if your db is more than 4MB large [2]. There are some up-to-date
        forks, though.
        
        I see LevelDB is used mostly for storing messages and mailboxes
        uid{validity/last}, but if we are to use gmail (it's the only IMAP
        provider that makes sense for sup), I believe we would stick to the All
        Mail label, right ? So, no need for storing this in db, rather in the
        sources.yaml file. Also, if leveldb-ruby is unreliable (I did encounter
        some issues way back about something with glibc...), and we want to use
        it for caching messages, I think we can salvage heliotrope's zmbox [3]
        because it's so simple to use yet far better than simple mbox.
        
        Regarding your ids questions, if you want to access the sup's messages
        from the gmail source, you could use the mail's Message-ID header and
        apply the same logic as in Message.sanitize_message_id. Caution,
        however: I've already encountered the case where multiple messages in
        GMail (i.e multiple X-GM-MSGID) have the same Message-ID, so they would
        be considered the same in sup/heliotrope... yeah, that's annoying as
        hell, and I don't know how we can solve this in the case of multiple
        sources.
        
        If you want to sync-back, maybe sup can call a source-level "sync_back"
        method with the current known state ? Speaking of which, for general
        synchronization we could reuse the elegant offlineimap's sync algorithm
        [4]. The idea is basic: have each source class store a snapshot of the
        state.  When a message is modified on the source, diff the change with
        the known status and propagate to sup; when a message is modified in
        sup, diff with the known status and propagate to the source.
        
        Just a brain dump.
        
        [0] https://github.com/rakoo/sup/tree/gmail_source
        [1] https://github.com/wmorgan/leveldb-ruby/pull/27
        [2] https://github.com/wmorgan/leveldb-ruby/issues/23
        [3] https://github.com/sup-heliotrope/heliotrope/blob/64d4b50d5649ec616a311a4cf6955137fdaeb13d/lib/heliotrope/zmbox.rb
        [4] http://offlineimap.org/howitworks.html
        
        Regards,
        
        -- 
        Matthieu Rakotojaona
        
        [-- Attachment #1.2: signature.asc --]
        [-- Type: application/pgp-signature, Size: 836 bytes --]
        
        [-- Attachment #2: Type: text/plain, Size: 143 bytes --]
        
        _______________________________________________
        Sup-devel mailing list
        Sup-devel@rubyforge.org
        http://rubyforge.org/mailman/listinfo/sup-devel
        
        ^ permalink raw reply	[flat|nested] 7+ messages in thread

      • • * Re: [sup-devel] Experimental Gmail Source
            2013-05-21 21:47       ` Matthieu Rakotojaona
          @ 2013-06-02  2:45         ` Horacio Sanson
            0 siblings, 0 replies; 7+ messages in thread
          From: Horacio Sanson @ 2013-06-02  2:45 UTC (permalink / raw)
            To: Sup developer discussion
          
          
          [-- Attachment #1.1: Type: text/plain, Size: 4542 bytes --]
          
          Thanks for checking the source and sorry for the late response... I can
          only look into this on rare free weekends.
          
          
          On Wed, May 22, 2013 at 6:47 AM, Matthieu Rakotojaona <
          matthieu.rakotojaona@gmail.com> wrote:
          
          > Hey Horacio,
          >
          > I took a stab at your gmail_source branch, and made a few
          > fixes/improvements [0]:
          >
          > - Add configuration option in sup-add
          > - Dump the LevelDB path in the sources.yaml
          > - Add a load_from_yaml method for a source to initialize its working
          >   values (for instance, the @db cannot be serialized, it needs to be
          >   reconstructed)
          > - Fixed the msg_att monkey-patch for imap.rb
          >
          >
          Great, I will add these changes to my branch....
          
          
          > All in all, the gmail source seems to work. I tested it on my usual
          > gmail account, I haven't tried to download it all, but I did download a
          > few dozens of emails without a problem. I'd like to warn users about
          > LevelDB though: it's sad to say, but as other wmorgan's stuff, it looks
          > abandoned. There are at least 2 bugs you will encounter if you try it: a
          > pb in configuration (fixed in [1]) and you need the `snappy` gem to make
          > it work if your db is more than 4MB large [2]. There are some up-to-date
          > forks, though.
          
          
          
          >
          >
          I see LevelDB is used mostly for storing messages and mailboxes
          > uid{validity/last}, but if we are to use gmail (it's the only IMAP
          > provider that makes sense for sup), I believe we would stick to the All
          > Mail label, right ? So, no need for storing this in db, rather in the
          > sources.yaml file. Also, if leveldb-ruby is unreliable (I did encounter
          > some issues way back about something with glibc...), and we want to use
          > it for caching messages, I think we can salvage heliotrope's zmbox [3]
          > because it's so simple to use yet far better than simple mbox.
          
          
          Using zmbox, mbox, maildir or any other mail storage (mix?) means I need to
          keep track of three indexes to allow two way sync between the Gmail source
          and the Sup index. I would need the Sup index id, the store id (e.g. zmbox
          file index) and the Gmail X-GM-MSGID. That complicates things a lot.
          
          Using key/value stores like LevelDB allows me to directly store the
          messages and associate them directly with the Gmail X-GM-MSGID. Also
          LevelDB comes with high compression for text data, perfect for emails, and
          high performance [1]. The issues you mention seem to be on the ruby library
          rather than LevelDB itself and they are fixable. If there are no bigger
          issues (e.g. data corruption/loss) I will stick with LevelDB.
          
          Regarding your ids questions, if you want to access the sup's messages
          > from the gmail source, you could use the mail's Message-ID header and
          > apply the same logic as in Message.sanitize_message_id. Caution,
          > however: I've already encountered the case where multiple messages in
          > GMail (i.e multiple X-GM-MSGID) have the same Message-ID, so they would
          > be considered the same in sup/heliotrope... yeah, that's annoying as
          > hell, and I don't know how we can solve this in the case of multiple
          > sources.
          >
          >
          Thanks, this comment put me on track and I found a way to get the emails
          from the index using the message id provided by the source. All I need to
          do is call Message.build_from_source(source, info) where info is the
          message id provided by the source. In my case this would be the X-GM-MSGID
          string.
          
          If you want to sync-back, maybe sup can call a source-level "sync_back"
          > method with the current known state ? Speaking of which, for general
          > synchronization we could reuse the elegant offlineimap's sync algorithm
          > [4]. The idea is basic: have each source class store a snapshot of the
          > state.  When a message is modified on the source, diff the change with
          > the known status and propagate to sup; when a message is modified in
          > sup, diff with the known status and propagate to the source.
          >
          >
          Interesting and simple algorithm. Let me study it a little more and see how
          it is applicable to Sup.
          
          [1]  http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html
          
          regards,
          Horacio
          
          Just a brain dump.
          >
          > [0] https://github.com/rakoo/sup/tree/gmail_source
          > [1] https://github.com/wmorgan/leveldb-ruby/pull/27
          > [2] https://github.com/wmorgan/leveldb-ruby/issues/23
          > [3]
          > https://github.com/sup-heliotrope/heliotrope/blob/64d4b50d5649ec616a311a4cf6955137fdaeb13d/lib/heliotrope/zmbox.rb
          > [4] http://offlineimap.org/howitworks.html
          >
          > Regards,
          >
          > --
          > Matthieu Rakotojaona
          >
          > _______________________________________________
          > Sup-devel mailing list
          > Sup-devel@rubyforge.org
          > http://rubyforge.org/mailman/listinfo/sup-devel
          >
          >
          
          [-- Attachment #1.2: Type: text/html, Size: 7223 bytes --]
          
          [-- Attachment #2: Type: text/plain, Size: 143 bytes --]
          
          _______________________________________________
          Sup-devel mailing list
          Sup-devel@rubyforge.org
          http://rubyforge.org/mailman/listinfo/sup-devel
          
          ^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2013-06-02  2:51 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-05-09  9:28 [sup-devel] Experimental Gmail Source Horacio Sanson
2013-05-09 19:13 ` Eric Weikl
2013-05-11 10:15 ` Gaute Hope
2013-05-11 18:18   ` Horacio Sanson
2013-05-15  2:10     ` Horacio Sanson
2013-05-21 21:47       ` Matthieu Rakotojaona
2013-06-02  2:45         ` Horacio Sanson

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox