sup

community/pipermail-archives/sup-devel/2013-07.txt (17386B) - raw

 From hsanson@gmail.com  Wed Jul  3 09:44:49 2013
 From: hsanson@gmail.com (Horacio Sanson)
 Date: Wed, 3 Jul 2013 18:44:49 +0900
 Subject: [sup-devel] Gmail source sync back.
 Message-ID: <CAHWBo_bRwKArh7nLQ8EMWdWvBV_kRGT=NqDU=Z69pTY+fNpMUQ@mail.gmail.com>
 
 I finally got the sync back feature of the Gmail source working. The sync
 code uses the offlineimap sync algorithm the Matthieu mentioned some emails
 ago.
 
 So far it works ok with my small tests but this code should be considered
 alpha so do not test it with your work emails.
 
 https://github.com/hsanson/sup/tree/gmail_source
 
 I also added some fixes and the config option provided by Matthieu.
 
 Testing and feedback is greatly appreciated.
 
 Issues to solve:
 
 - Sync back flags: Not sure if this is needed or how it may work. All
 messages I get from Gmail have the \Seen flag no matter if I have seen or
 not the message on Gmail web interface. The \Flagged flag can be changed
 using the \Starred label instead. The only flag that I really need to care
 about is the \Deleted flag.
 
 - How to refresh the index after polling finishes? When new labels are
 synced from Gmail to Sup the index is not updated with the new labels until
 I reload sup or press "@" to refresh the index. Is there a callback to
 refresh the index when the polling finishes?
 
 - Sync back iterates over all messages in the Gmail account checking for
 differences between message labels. This can take a long time and may not
 finish before the next polling starts. How to do this efficiently?
 
 regards,
 Horacio Sanson
 -------------- next part --------------
 An HTML attachment was scrubbed...
 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20130703/e235f993/attachment.html>
 
 From matthieu.rakotojaona@gmail.com  Thu Jul  4 20:38:30 2013
 From: matthieu.rakotojaona@gmail.com (Matthieu Rakotojaona)
 Date: Thu, 04 Jul 2013 22:38:30 +0200
 Subject: [sup-devel] Gmail source sync back.
 In-Reply-To: <CAHWBo_bRwKArh7nLQ8EMWdWvBV_kRGT=NqDU=Z69pTY+fNpMUQ@mail.gmail.com>
 References: <CAHWBo_bRwKArh7nLQ8EMWdWvBV_kRGT=NqDU=Z69pTY+fNpMUQ@mail.gmail.com>
 Message-ID: <1372966209-sup-7024@kpad>
 
 Hello,
 
 Excerpts from Horacio Sanson's message of 2013-07-03 11:44:49 +0200:
 > I finally got the sync back feature of the Gmail source working. 
 
 Awesome, thank you! I've been using your gmail source branch with
 success for a while now. There are still a few quirks with the Inbox
 tag, but overall I'm really pleased with it. Will try your branch.
 
 > Issues to solve:
 >
 > [...]
 > 
 > - Sync back iterates over all messages in the Gmail account checking for
 > differences between message labels. This can take a long time and may not
 > finish before the next polling starts. How to do this efficiently?
 
 Ah, that sounds bad. Reading the always-helpful Stack Overflow [0] says
 that there is nothing for this, unfortunately. There is a IMAP RFC
 extension, 4551, that is supposed to help such situations, but it is not
 implemented by Gmail, although there is a beta-test for this, if you're
 interested :
 
 http://mailman2.u.washington.edu/pipermail/imap-protocol/2013-January/001962.html
 
 Another way could be to use the search extension specific to gmail [1].
 I don't think you would gain much more than bandwidth, though.
 
 
 [0] http://stackoverflow.com/questions/10076690/ruby-imap-changes-since-last-check
 [1] https://developers.google.com/gmail/imap_extensions#extension_of_the_search_command_x-gm-raw
 
 > 
 > regards,
 > Horacio Sanson
 
 -- 
 Matthieu Rakotojaona
 -------------- next part --------------
 A non-text attachment was scrubbed...
 Name: signature.asc
 Type: application/pgp-signature
 Size: 836 bytes
 Desc: not available
 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20130704/f5993774/attachment.bin>
 
 From ruthard.baudach@web.de  Fri Jul  5 10:32:48 2013
 From: ruthard.baudach@web.de (Ruthard Baudach)
 Date: Fri, 05 Jul 2013 12:32:48 +0200
 Subject: [sup-devel] reading html mails
 Message-ID: <1373019929-sup-8525@prxbdc.dyndns.org>
 
 In the last years I'm receiving more and more emails containing only
 html-formatted text and no plain text.
 
 sup displays these emails as empty mails with an html attachment.
 
 It would be great if sup was able to distinguish between actual html
 attachments and such deficient mails, and would extract and display
 the text part of these mails.
 
 I'm not able to hack this myself at the moment, but perhaps someone
 picks up the idea?
 
 It's written a lot of times, but let me repeat it:
 
 Good that sup is alive again!
 
 
 Greetings to all
 
 Ruthard
 
 
 From m.klinik@gmx.de  Fri Jul  5 13:15:48 2013
 From: m.klinik@gmx.de (Markus Klinik)
 Date: Fri, 05 Jul 2013 15:15:48 +0200
 Subject: [sup-devel] reading html mails
 In-Reply-To: <1373019929-sup-8525@prxbdc.dyndns.org>
 References: <1373019929-sup-8525@prxbdc.dyndns.org>
 Message-ID: <1373030020-sup-8455@x41>
 
 Excerpts from Ruthard Baudach's message of 2013-07-05 12:32:48 +0200:
 > In the last years I'm receiving more and more emails containing only
 > html-formatted text and no plain text.
 > 
 > sup displays these emails as empty mails with an html attachment.
 > 
 > It would be great if sup was able to distinguish between actual html
 > attachments and such deficient mails, and would extract and display
 > the text part of these mails.
 
 Hi,
 
 I have a mime-decode hook for exactly this problem. It unfolds html
 attachments if there is no plaintext part in an email.
 
     $ cat mime-decode.rb 
     unless sibling_types.member? "text/plain"
       case content_type
       when "text/html"
        `/usr/bin/elinks -dump '#{filename}'`
       end
     end
 
 Best regards,
 Markus
 
 From v@asch.cz  Fri Jul  5 12:02:43 2013
 From: v@asch.cz (Vojtech Aschenbrenner)
 Date: Fri, 05 Jul 2013 14:02:43 +0200
 Subject: [sup-devel] reading html mails
 In-Reply-To: <1373019929-sup-8525@prxbdc.dyndns.org>
 References: <1373019929-sup-8525@prxbdc.dyndns.org>
 Message-ID: <1373025677-sup-3594@x220>
 
 Excerpts from Ruthard Baudach's message of 2013-07-05 12:32:48 +0200:
 > sup displays these emails as empty mails with an html attachment.
 > 
 > It would be great if sup was able to distinguish between actual html
 > attachments and such deficient mails, and would extract and display
 > the text part of these mails.
 
 This can be solved with mime-decode hook [1].
 
 [1] https://github.com/sup-heliotrope/sup/wiki/Viewing-Attachments
 -------------- next part --------------
 A non-text attachment was scrubbed...
 Name: signature.asc
 Type: application/pgp-signature
 Size: 836 bytes
 Desc: not available
 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20130705/da72b1b1/attachment.bin>
 
 From 5longluna@gmail.com  Mon Jul  8 18:42:34 2013
 From: 5longluna@gmail.com (Whyme Lyu)
 Date: Tue, 09 Jul 2013 02:42:34 +0800
 Subject: [sup-devel] reading html mails
 In-Reply-To: <1373019929-sup-8525@prxbdc.dyndns.org>
 References: <1373019929-sup-8525@prxbdc.dyndns.org>
 Message-ID: <1373307811-sup-1164@whymeDesk>
 
 Excerpts from Ruthard Baudach's message of 2013-07-05 18:32:48 +0800:
 > In the last years I'm receiving more and more emails containing only
 > html-formatted text and no plain text.
 
 I'm feeling the same pain here mainly due to unpolished in-house services
 sending out html-only mails. More annoyingly, sup doesn't index these mails
 since HTML is treated as attachments.
 
 I've created an issue[1] on Github to talk about it further.
 
 [1]: https://github.com/sup-heliotrope/sup/issues/101
 
 -- 
 -whyme http://randombu.gs/
 
 From ruthard.baudach@web.de  Tue Jul  9 12:13:49 2013
 From: ruthard.baudach@web.de (Ruthard Baudach)
 Date: Tue, 09 Jul 2013 14:13:49 +0200
 Subject: [sup-devel] reading html mails
 In-Reply-To: <1373307811-sup-1164@whymeDesk>
 References: <1373019929-sup-8525@prxbdc.dyndns.org>
 	<1373307811-sup-1164@whymeDesk>
 Message-ID: <1373371906-sup-3425@prxbdc.dyndns.org>
 
 >== Ausz?ge aus der Nachricht von  Whyme Lyu vom 2013-07-08 20:42:
 > Excerpts from Ruthard Baudach's message of 2013-07-05 18:32:48 +0800:
 > > In the last years I'm receiving more and more emails containing only
 > > html-formatted text and no plain text.
 > 
 > I'm feeling the same pain here mainly due to unpolished in-house services
 > sending out html-only mails. More annoyingly, sup doesn't index these mails
 > since HTML is treated as attachments.
 What do you mean by "sup doesn't index" these mails -- I have no
 problems with them except the annoying need of opening them manually,
 that is solved with the mime-decode hook?
 
 Greetings,
 
 Ruthard
 
 From marka@pobox.com  Wed Jul 10 00:38:58 2013
 From: marka@pobox.com (Mark Alexander)
 Date: Wed, 10 Jul 2013 00:38:58 -0000
 Subject: [sup-devel] reading html mails
 In-Reply-To: <1373371906-sup-3425@prxbdc.dyndns.org>
 References: <1373019929-sup-8525@prxbdc.dyndns.org>
 	<1373307811-sup-1164@whymeDesk>
 	<1373371906-sup-3425@prxbdc.dyndns.org>
 Message-ID: <1373403508-sup-4420@x200s>
 
 Excerpts from Ruthard Baudach's message of 2013-07-09 08:13:49 -0400:
 > >== Ausz?ge aus der Nachricht von  Whyme Lyu vom 2013-07-08 20:42:
 > > I'm feeling the same pain here mainly due to unpolished in-house services
 > > sending out html-only mails. More annoyingly, sup doesn't index these mails
 > > since HTML is treated as attachments.
 > What do you mean by "sup doesn't index" these mails -- I have no
 > problems with them except the annoying need of opening them manually,
 > that is solved with the mime-decode hook?
 
 I can verify the problem on my installation of sup.  It doesn't index
 the HTML-only emails, i.e., it doesn't read the contents of the HTML
 attachment to put those contents in the xapian index.  The result is
 that you can't search for words that appear in an HTML-only email and
 expect that email to appear in the search results.
 
 From eg@gaute.vetsj.com  Sat Jul 20 21:51:54 2013
 From: eg@gaute.vetsj.com (Gaute Hope)
 Date: Sat, 20 Jul 2013 23:51:54 +0200
 Subject: [sup-devel] We should get ready for sup 0.14
 Message-ID: <51EB067A.600@gaute.vetsj.com>
 
 Hi all,
 
 think it is time to collect the latest stuff in 'develop' and release
 sup 0.14. This release would support ruby 1.9.3 and up. There is an
 issue tracking this release [1], please add any comments there.
 
 Regards, Gaute
 
 [1] https://github.com/sup-heliotrope/sup/issues/105
 
 From eg@gaute.vetsj.com  Mon Jul 29 07:54:01 2013
 From: eg@gaute.vetsj.com (Gaute Hope)
 Date: Mon, 29 Jul 2013 09:54:01 +0200
 Subject: [sup-devel] GMail,
  was: Re: [sup] Please review the documentation in class Source for
  implementing new Sources (#113)
 In-Reply-To: <sup-heliotrope/sup/issues/113/21703944@github.com>
 References: <sup-heliotrope/sup/issues/113@github.com>
 	<sup-heliotrope/sup/issues/113/21703944@github.com>
 Message-ID: <51F61F99.8030202@gaute.vetsj.com>
 
 Hi Petr,
 
 Take a look at Horacio's work, he has gotten pretty far doing just that!
 But some of the label stuff can be very slow for large volumes atm.
 
 His gmail.rb is well-written and relatively easy to understand:
 https://github.com/hsanson/sup/tree/gmail_source
 
 His last message to sup-devel:
 http://rubyforge.org/pipermail/sup-devel/2013-July/001415.html
 
 Also, feel free to sign up to the devel mailing list:
 http://rubyforge.org/mailman/listinfo/sup-devel
 
 or the users list, sup-talk: http://rubyforge.org/mailman/listinfo/sup-talk
 
 Regards, Gaute
 
 On 29. juli 2013 09:44, Petr Pudl?k wrote:
 > Thanks for help. To clarify, my aim is to use sup with GMail on 
 > multiple computers with locally cached messages. For this to work 
 > correctly I need to synchronize labels back to GMail. Since this 
 > seems problematic with maildir, my idea is to:
 > 
 > - Store emails in a SQLite database. This way I'll get ACID 
 > properties for free, as well as the possibility to add additional 
 > information to messages, such as labels. Another bonus are simple 
 > backups - everything is in one file. - Write a simple GMail-IMAP 
 > synchronization tool that only downloads messages from _All mail_ to 
 > the database and synchronizes labels back using [GMail 
 > extensions](https://developers.google.com/gmail/imap_extensions#special-use_extension_of_the_list_command).
 >
 > 
 (There could be some time limit like "download only messages newer
 > than 1 year or so.) - Modify sup so that it can use this SQLite
 > store and store (or copy) labels into it.
 > 
 > This also solves the awkward intermediate step when using
 > OfflineIMAP that labels are first converted into IMAP folders,
 > downloaded locally to multiple folders and then again converted to
 > labels for sup (and sup needs to check multiple folders). Using
 > instead one big folder that understand labels seems much cleaner.
 > 
 > (Perhaps it would be worth to store GMail thread ID too?)
 > 
 > --- Reply to this email directly or view it on GitHub: 
 > https://github.com/sup-heliotrope/sup/issues/113#issuecomment-21703944
 >
 
 From eg@gaute.vetsj.com  Mon Jul 29 19:37:30 2013
 From: eg@gaute.vetsj.com (Gaute Hope)
 Date: Mon, 29 Jul 2013 21:37:30 +0200
 Subject: [sup-devel] Gmail source sync back.
 In-Reply-To: <CAHWBo_bRwKArh7nLQ8EMWdWvBV_kRGT=NqDU=Z69pTY+fNpMUQ@mail.gmail.com>
 References: <CAHWBo_bRwKArh7nLQ8EMWdWvBV_kRGT=NqDU=Z69pTY+fNpMUQ@mail.gmail.com>
 Message-ID: <51F6C47A.3010004@gaute.vetsj.com>
 
 On 03. juli 2013 11:44, Horacio Sanson wrote:
 > - How to refresh the index after polling finishes? When new labels are
 > synced from Gmail to Sup the index is not updated with the new labels until
 > I reload sup or press "@" to refresh the index. Is there a callback to
 > refresh the index when the polling finishes?
 
 Check out pull request #2 towards your github repo, yielding the message
 on update + some other modifications now updates the message on my setup.
 
 > - Sync back iterates over all messages in the Gmail account checking for
 > differences between message labels. This can take a long time and may not
 > finish before the next polling starts. How to do this efficiently?
 
 If this turns out not be solvable (e.g. being able to get a list of
 changed messages since a given date), an option could be to allow the
 user to configure remote-to-local syncing to only be manual.
 Local-to-remote you can store locally whether it has been synced or not.
 
 There's also a pull request #1 which allows arbitrary domains to be
 entered for the gmail source (for google apps setups).
 
 Regards, Gaute
 
 From eg@gaute.vetsj.com  Mon Jul 29 19:48:14 2013
 From: eg@gaute.vetsj.com (Gaute Hope)
 Date: Mon, 29 Jul 2013 21:48:14 +0200
 Subject: [sup-devel] Gmail source sync back.
 In-Reply-To: <CAHWBo_bRwKArh7nLQ8EMWdWvBV_kRGT=NqDU=Z69pTY+fNpMUQ@mail.gmail.com>
 References: <CAHWBo_bRwKArh7nLQ8EMWdWvBV_kRGT=NqDU=Z69pTY+fNpMUQ@mail.gmail.com>
 Message-ID: <51F6C6FE.1090604@gaute.vetsj.com>
 
 On 03. juli 2013 11:44, Horacio Sanson wrote:
 > - Sync back iterates over all messages in the Gmail account checking for
 > differences between message labels. This can take a long time and may not
 > finish before the next polling starts. How to do this efficiently?
 
 By the way, with the source poll locking recently added no second poll
 should be able to be initiated when there is one running already.
 
 - gaute
 
 From eg@gaute.vetsj.com  Mon Jul 29 20:01:15 2013
 From: eg@gaute.vetsj.com (Gaute Hope)
 Date: Mon, 29 Jul 2013 22:01:15 +0200
 Subject: [sup-devel] Gmail source sync back.
 In-Reply-To: <CAHWBo_bRwKArh7nLQ8EMWdWvBV_kRGT=NqDU=Z69pTY+fNpMUQ@mail.gmail.com>
 References: <CAHWBo_bRwKArh7nLQ8EMWdWvBV_kRGT=NqDU=Z69pTY+fNpMUQ@mail.gmail.com>
 Message-ID: <51F6CA0B.7090406@gaute.vetsj.com>
 
 
 
 On 03. juli 2013 11:44, Horacio Sanson wrote:
 > I finally got the sync back feature of the Gmail source working. The sync
 > code uses the offlineimap sync algorithm the Matthieu mentioned some emails
 > ago.
 
 
 > Testing and feedback is greatly appreciated.
 
 Further testing; if I check a message to be important after it has
 already been added I get an IMAP Invalid label error: Important.
 
 Regards, gaute
 
 From hsanson@gmail.com  Tue Jul 30 12:41:11 2013
 From: hsanson@gmail.com (Horacio Sanson)
 Date: Tue, 30 Jul 2013 21:41:11 +0900
 Subject: [sup-devel] Xapian 1.2.15 dependency causes issues on Ubuntu 13.04
 Message-ID: <CAHWBo_ZX8HjuEGWL-Xx_KK+QkyZ-=xqnOhHzQECkPHXFMG7Z+g@mail.gmail.com>
 
 Ubuntu 13.04 installs the distribution xapian dev and lib packages that are
 version 1.2.12. This causes sup to complain that the xapian 1.2.15 or later
 is required and bails out.
 
 I understand xapian-ruby installs its own xapian lib but sup still tries to
 load the system one.
 
 I changed the index.rb file to allow version 1.2.12 and had not encounter
 any issues so far. Could it be possible to set this version as the min
 required version?
 
 regards,
 Horacio
 -------------- next part --------------
 An HTML attachment was scrubbed...
 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20130730/84356302/attachment.html>
 
 From hsanson@gmail.com  Tue Jul 30 13:18:27 2013
 From: hsanson@gmail.com (Horacio Sanson)
 Date: Tue, 30 Jul 2013 22:18:27 +0900
 Subject: [sup-devel] How do I force flush the log file in sup?
 Message-ID: <CAHWBo_bM_WmaUUHbER+1V8=o_McxibJCCipnxKnepsviV8Rrhw@mail.gmail.com>
 
 Is there a way to force sup to flush the log file after each write? It is
 difficult to debug my gmail source as I cannot see real time what is going
 on. Maybe set sup to always flush when SUP_LOG_LEVEL is set to debug?
 
 regards,
 Horacio
 -------------- next part --------------
 An HTML attachment was scrubbed...
 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20130730/dfb9c00c/attachment.html>