From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by 10.52.177.71 with SMTP id co7csp136978vdc; Tue, 21 May 2013 14:55:53 -0700 (PDT) X-Received: by 10.182.118.42 with SMTP id kj10mr2773450obb.99.1369173353086; Tue, 21 May 2013 14:55:53 -0700 (PDT) Return-Path: Received: from rubyforge.org ([50.56.192.79]) by mx.google.com with ESMTP id yw5si2745853obb.164.2013.05.21.14.55.52 for ; Tue, 21 May 2013 14:55:53 -0700 (PDT) Received-SPF: pass (google.com: domain of sup-devel-bounces@rubyforge.org designates 50.56.192.79 as permitted sender) client-ip=50.56.192.79; Authentication-Results: mx.google.com; spf=pass (google.com: domain of sup-devel-bounces@rubyforge.org designates 50.56.192.79 as permitted sender) smtp.mail=sup-devel-bounces@rubyforge.org; dkim=neutral (bad format) header.i=@gmail.com Received: from localhost.localdomain (localhost [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id D9C822E14C; Tue, 21 May 2013 21:55:53 +0000 (UTC) Received: from mail-wg0-f44.google.com (mail-wg0-f44.google.com [74.125.82.44]) by rubyforge.org (Postfix) with ESMTP id CFFFF2E146 for ; Tue, 21 May 2013 21:55:39 +0000 (UTC) Received: by mail-wg0-f44.google.com with SMTP id a12so633701wgh.23 for ; Tue, 21 May 2013 14:55:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:subject:in-reply-to:references:x-pgp-key:date:message-id :user-agent:content-transfer-encoding:mime-version:content-type; bh=jaSbBxANeIR6thaTdS0IJpHwbMSDTjt4qWwoRvUPLEI=; b=xVJlNGI8o5uus60AAqm86+6v33QgdEm94ceYQvw02uSRcwn2XVy9FQr9mEcs10EfCJ 8sZQcM0808BFqS5/QDwATk6I2XSNo+ySF71ggWVU7lB8e6DUZXJuW/YsH8TWJZ6YVi0x QxWydU6c7MsWF0XarE4NgN5CfzNsG7qPMJ+JpTCa8yXYPyjWIzRGpLQ3OYrpnSqBXqtv lCqwH6/ycgpUrc3+CqWFTacAFXenPPaIjS6F6YlqMfjlzpchB8HZwEbiavZfF4PrBEJP F5MCQyIbL9AjJdddl+sgiSm2gB0UqMGwG0dlNkxVxNaU5clMJ8OWEwiWdF5LCQ7AgBDX Oo3Q== X-Received: by 10.180.36.205 with SMTP id s13mr27466985wij.31.1369172846909; Tue, 21 May 2013 14:47:26 -0700 (PDT) Received: from kpad (155-29-190-109.dsl.ovh.fr. [109.190.29.155]) by mx.google.com with ESMTPSA id e5sm6851846wiy.5.2013.05.21.14.47.25 for (version=TLSv1.2 cipher=AES128-SHA bits=128/128); Tue, 21 May 2013 14:47:25 -0700 (PDT) Received: from localhost (1000@localhost [local]); by kpad (OpenSMTPD) with ESMTP id 077a4276; for ; Tue, 21 May 2013 21:47:23 +0000 (UTC) From: Matthieu Rakotojaona To: sup-devel In-reply-to: References: <518E1A2B.2080903@gaute.vetsj.com> X-pgp-key: http://otokar.looc2011.eu/static/matthieu.rakotojaona.asc Date: Tue, 21 May 2013 23:47:16 +0200 Message-Id: <1369172802-sup-2003@kpad> User-Agent: Sup/git MIME-Version: 1.0 Subject: Re: [sup-devel] Experimental Gmail Source X-BeenThere: sup-devel@rubyforge.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Sup developer discussion List-Id: Sup developer discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============7484492807840341504==" Sender: sup-devel-bounces@rubyforge.org Errors-To: sup-devel-bounces@rubyforge.org --===============7484492807840341504== Content-Transfer-Encoding: 8bit Content-Type: multipart/signed; protocol="application/pgp-signature"; boundary="=-1369172843-32089-23321-7430-1-=" --=-1369172843-32089-23321-7430-1-= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hey Horacio, I took a stab at your gmail_source branch, and made a few fixes/improvements [0]: - Add configuration option in sup-add - Dump the LevelDB path in the sources.yaml - Add a load_from_yaml method for a source to initialize its working values (for instance, the @db cannot be serialized, it needs to be reconstructed) - Fixed the msg_att monkey-patch for imap.rb All in all, the gmail source seems to work. I tested it on my usual gmail account, I haven't tried to download it all, but I did download a few dozens of emails without a problem. I'd like to warn users about LevelDB though: it's sad to say, but as other wmorgan's stuff, it looks abandoned. There are at least 2 bugs you will encounter if you try it: a pb in configuration (fixed in [1]) and you need the `snappy` gem to make it work if your db is more than 4MB large [2]. There are some up-to-date forks, though. I see LevelDB is used mostly for storing messages and mailboxes uid{validity/last}, but if we are to use gmail (it's the only IMAP provider that makes sense for sup), I believe we would stick to the All Mail label, right ? So, no need for storing this in db, rather in the sources.yaml file. Also, if leveldb-ruby is unreliable (I did encounter some issues way back about something with glibc...), and we want to use it for caching messages, I think we can salvage heliotrope's zmbox [3] because it's so simple to use yet far better than simple mbox. Regarding your ids questions, if you want to access the sup's messages from the gmail source, you could use the mail's Message-ID header and apply the same logic as in Message.sanitize_message_id. Caution, however: I've already encountered the case where multiple messages in GMail (i.e multiple X-GM-MSGID) have the same Message-ID, so they would be considered the same in sup/heliotrope... yeah, that's annoying as hell, and I don't know how we can solve this in the case of multiple sources. If you want to sync-back, maybe sup can call a source-level "sync_back" method with the current known state ? Speaking of which, for general synchronization we could reuse the elegant offlineimap's sync algorithm [4]. The idea is basic: have each source class store a snapshot of the state. When a message is modified on the source, diff the change with the known status and propagate to sup; when a message is modified in sup, diff with the known status and propagate to the source. Just a brain dump. [0] https://github.com/rakoo/sup/tree/gmail_source [1] https://github.com/wmorgan/leveldb-ruby/pull/27 [2] https://github.com/wmorgan/leveldb-ruby/issues/23 [3] https://github.com/sup-heliotrope/heliotrope/blob/64d4b50d5649ec616a3= 11a4cf6955137fdaeb13d/lib/heliotrope/zmbox.rb [4] http://offlineimap.org/howitworks.html Regards, -- = Matthieu Rakotojaona --=-1369172843-32089-23321-7430-1-= Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) iQIcBAEBAgAGBQJRm+tqAAoJEDp04ZRcPZciU/wQAJ8bva+VhjOxa4qeYlqJYl1f y7z7XMF/5kMSrwCTqsN7ioo2ZF4JJx54dcQtEsJq6f7q+1GYrEf4AD66n7sGtEQi aqkARLPDtr6UcewKOllkBD55JhF2oz4pFwFW8jqAuzNPsk63eNLXwRBTm4nt/5rR 121Rq4v7058worOwrNtdKaMwbuEDqnFO1Q0xNEReIC2KU0QhPfA33mflOegYrD8U zAzLNTAmq4M9aUpDLm+GPWCL8P/ss2h0Jg/UHT4fl1yGWe/gs++CjwZauw/N6odg gE3DTKmfS64RAZjrA0J+CVM98ZPERk+4UfAAzzoYmx6axpkIP4rEhe/uNGY+CGyo ykYpWNthGxIpjigBEzElQ9VFL4bXZTFqAutsUcPqJ/F/dFWYVWKFRggUggmGWDFD Rj6bcukHLRyaYxOvU4aN4UydO4ERTqWRGf4bdyKNMhQBRJ7bu2r5lDfQ26NQpBlY fpecdPLbS4IvV8WQG7I4hCrUgksO41XcvRRWxOPkQok/alAhHKKgyUKnf09yQ1TI nU5eLp9bBlRC1HwwU/p5ZL7tPAWQcR8KuTJBrRdi9KHItx84w8VgHUVyq/WXIycv PUD2hv8dbty7lvLz+iuoY9HN9RoAA3YqMSNB1kNKzEKG5hgHrBIVHD6+2jz2fbf0 dro9B1ll2RYhuF9O8yc9 =urTa -----END PGP SIGNATURE----- --=-1369172843-32089-23321-7430-1-=-- --===============7484492807840341504== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel --===============7484492807840341504==--