From hsanson@gmail.com Wed Jul 3 09:44:49 2013 From: hsanson@gmail.com (Horacio Sanson) Date: Wed, 3 Jul 2013 18:44:49 +0900 Subject: [sup-devel] Gmail source sync back. Message-ID: I finally got the sync back feature of the Gmail source working. The sync code uses the offlineimap sync algorithm the Matthieu mentioned some emails ago. So far it works ok with my small tests but this code should be considered alpha so do not test it with your work emails. https://github.com/hsanson/sup/tree/gmail_source I also added some fixes and the config option provided by Matthieu. Testing and feedback is greatly appreciated. Issues to solve: - Sync back flags: Not sure if this is needed or how it may work. All messages I get from Gmail have the \Seen flag no matter if I have seen or not the message on Gmail web interface. The \Flagged flag can be changed using the \Starred label instead. The only flag that I really need to care about is the \Deleted flag. - How to refresh the index after polling finishes? When new labels are synced from Gmail to Sup the index is not updated with the new labels until I reload sup or press "@" to refresh the index. Is there a callback to refresh the index when the polling finishes? - Sync back iterates over all messages in the Gmail account checking for differences between message labels. This can take a long time and may not finish before the next polling starts. How to do this efficiently? regards, Horacio Sanson -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.rakotojaona@gmail.com Thu Jul 4 20:38:30 2013 From: matthieu.rakotojaona@gmail.com (Matthieu Rakotojaona) Date: Thu, 04 Jul 2013 22:38:30 +0200 Subject: [sup-devel] Gmail source sync back. In-Reply-To: References: Message-ID: <1372966209-sup-7024@kpad> Hello, Excerpts from Horacio Sanson's message of 2013-07-03 11:44:49 +0200: > I finally got the sync back feature of the Gmail source working. Awesome, thank you! I've been using your gmail source branch with success for a while now. There are still a few quirks with the Inbox tag, but overall I'm really pleased with it. Will try your branch. > Issues to solve: > > [...] > > - Sync back iterates over all messages in the Gmail account checking for > differences between message labels. This can take a long time and may not > finish before the next polling starts. How to do this efficiently? Ah, that sounds bad. Reading the always-helpful Stack Overflow [0] says that there is nothing for this, unfortunately. There is a IMAP RFC extension, 4551, that is supposed to help such situations, but it is not implemented by Gmail, although there is a beta-test for this, if you're interested : http://mailman2.u.washington.edu/pipermail/imap-protocol/2013-January/001962.html Another way could be to use the search extension specific to gmail [1]. I don't think you would gain much more than bandwidth, though. [0] http://stackoverflow.com/questions/10076690/ruby-imap-changes-since-last-check [1] https://developers.google.com/gmail/imap_extensions#extension_of_the_search_command_x-gm-raw > > regards, > Horacio Sanson -- Matthieu Rakotojaona -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ruthard.baudach@web.de Fri Jul 5 10:32:48 2013 From: ruthard.baudach@web.de (Ruthard Baudach) Date: Fri, 05 Jul 2013 12:32:48 +0200 Subject: [sup-devel] reading html mails Message-ID: <1373019929-sup-8525@prxbdc.dyndns.org> In the last years I'm receiving more and more emails containing only html-formatted text and no plain text. sup displays these emails as empty mails with an html attachment. It would be great if sup was able to distinguish between actual html attachments and such deficient mails, and would extract and display the text part of these mails. I'm not able to hack this myself at the moment, but perhaps someone picks up the idea? It's written a lot of times, but let me repeat it: Good that sup is alive again! Greetings to all Ruthard From m.klinik@gmx.de Fri Jul 5 13:15:48 2013 From: m.klinik@gmx.de (Markus Klinik) Date: Fri, 05 Jul 2013 15:15:48 +0200 Subject: [sup-devel] reading html mails In-Reply-To: <1373019929-sup-8525@prxbdc.dyndns.org> References: <1373019929-sup-8525@prxbdc.dyndns.org> Message-ID: <1373030020-sup-8455@x41> Excerpts from Ruthard Baudach's message of 2013-07-05 12:32:48 +0200: > In the last years I'm receiving more and more emails containing only > html-formatted text and no plain text. > > sup displays these emails as empty mails with an html attachment. > > It would be great if sup was able to distinguish between actual html > attachments and such deficient mails, and would extract and display > the text part of these mails. Hi, I have a mime-decode hook for exactly this problem. It unfolds html attachments if there is no plaintext part in an email. $ cat mime-decode.rb unless sibling_types.member? "text/plain" case content_type when "text/html" `/usr/bin/elinks -dump '#{filename}'` end end Best regards, Markus From v@asch.cz Fri Jul 5 12:02:43 2013 From: v@asch.cz (Vojtech Aschenbrenner) Date: Fri, 05 Jul 2013 14:02:43 +0200 Subject: [sup-devel] reading html mails In-Reply-To: <1373019929-sup-8525@prxbdc.dyndns.org> References: <1373019929-sup-8525@prxbdc.dyndns.org> Message-ID: <1373025677-sup-3594@x220> Excerpts from Ruthard Baudach's message of 2013-07-05 12:32:48 +0200: > sup displays these emails as empty mails with an html attachment. > > It would be great if sup was able to distinguish between actual html > attachments and such deficient mails, and would extract and display > the text part of these mails. This can be solved with mime-decode hook [1]. [1] https://github.com/sup-heliotrope/sup/wiki/Viewing-Attachments -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From 5longluna@gmail.com Mon Jul 8 18:42:34 2013 From: 5longluna@gmail.com (Whyme Lyu) Date: Tue, 09 Jul 2013 02:42:34 +0800 Subject: [sup-devel] reading html mails In-Reply-To: <1373019929-sup-8525@prxbdc.dyndns.org> References: <1373019929-sup-8525@prxbdc.dyndns.org> Message-ID: <1373307811-sup-1164@whymeDesk> Excerpts from Ruthard Baudach's message of 2013-07-05 18:32:48 +0800: > In the last years I'm receiving more and more emails containing only > html-formatted text and no plain text. I'm feeling the same pain here mainly due to unpolished in-house services sending out html-only mails. More annoyingly, sup doesn't index these mails since HTML is treated as attachments. I've created an issue[1] on Github to talk about it further. [1]: https://github.com/sup-heliotrope/sup/issues/101 -- -whyme http://randombu.gs/ From ruthard.baudach@web.de Tue Jul 9 12:13:49 2013 From: ruthard.baudach@web.de (Ruthard Baudach) Date: Tue, 09 Jul 2013 14:13:49 +0200 Subject: [sup-devel] reading html mails In-Reply-To: <1373307811-sup-1164@whymeDesk> References: <1373019929-sup-8525@prxbdc.dyndns.org> <1373307811-sup-1164@whymeDesk> Message-ID: <1373371906-sup-3425@prxbdc.dyndns.org> >== Ausz?ge aus der Nachricht von Whyme Lyu vom 2013-07-08 20:42: > Excerpts from Ruthard Baudach's message of 2013-07-05 18:32:48 +0800: > > In the last years I'm receiving more and more emails containing only > > html-formatted text and no plain text. > > I'm feeling the same pain here mainly due to unpolished in-house services > sending out html-only mails. More annoyingly, sup doesn't index these mails > since HTML is treated as attachments. What do you mean by "sup doesn't index" these mails -- I have no problems with them except the annoying need of opening them manually, that is solved with the mime-decode hook? Greetings, Ruthard From marka@pobox.com Wed Jul 10 00:38:58 2013 From: marka@pobox.com (Mark Alexander) Date: Wed, 10 Jul 2013 00:38:58 -0000 Subject: [sup-devel] reading html mails In-Reply-To: <1373371906-sup-3425@prxbdc.dyndns.org> References: <1373019929-sup-8525@prxbdc.dyndns.org> <1373307811-sup-1164@whymeDesk> <1373371906-sup-3425@prxbdc.dyndns.org> Message-ID: <1373403508-sup-4420@x200s> Excerpts from Ruthard Baudach's message of 2013-07-09 08:13:49 -0400: > >== Ausz?ge aus der Nachricht von Whyme Lyu vom 2013-07-08 20:42: > > I'm feeling the same pain here mainly due to unpolished in-house services > > sending out html-only mails. More annoyingly, sup doesn't index these mails > > since HTML is treated as attachments. > What do you mean by "sup doesn't index" these mails -- I have no > problems with them except the annoying need of opening them manually, > that is solved with the mime-decode hook? I can verify the problem on my installation of sup. It doesn't index the HTML-only emails, i.e., it doesn't read the contents of the HTML attachment to put those contents in the xapian index. The result is that you can't search for words that appear in an HTML-only email and expect that email to appear in the search results. From eg@gaute.vetsj.com Sat Jul 20 21:51:54 2013 From: eg@gaute.vetsj.com (Gaute Hope) Date: Sat, 20 Jul 2013 23:51:54 +0200 Subject: [sup-devel] We should get ready for sup 0.14 Message-ID: <51EB067A.600@gaute.vetsj.com> Hi all, think it is time to collect the latest stuff in 'develop' and release sup 0.14. This release would support ruby 1.9.3 and up. There is an issue tracking this release [1], please add any comments there. Regards, Gaute [1] https://github.com/sup-heliotrope/sup/issues/105 From eg@gaute.vetsj.com Mon Jul 29 07:54:01 2013 From: eg@gaute.vetsj.com (Gaute Hope) Date: Mon, 29 Jul 2013 09:54:01 +0200 Subject: [sup-devel] GMail, was: Re: [sup] Please review the documentation in class Source for implementing new Sources (#113) In-Reply-To: References: Message-ID: <51F61F99.8030202@gaute.vetsj.com> Hi Petr, Take a look at Horacio's work, he has gotten pretty far doing just that! But some of the label stuff can be very slow for large volumes atm. His gmail.rb is well-written and relatively easy to understand: https://github.com/hsanson/sup/tree/gmail_source His last message to sup-devel: http://rubyforge.org/pipermail/sup-devel/2013-July/001415.html Also, feel free to sign up to the devel mailing list: http://rubyforge.org/mailman/listinfo/sup-devel or the users list, sup-talk: http://rubyforge.org/mailman/listinfo/sup-talk Regards, Gaute On 29. juli 2013 09:44, Petr Pudl?k wrote: > Thanks for help. To clarify, my aim is to use sup with GMail on > multiple computers with locally cached messages. For this to work > correctly I need to synchronize labels back to GMail. Since this > seems problematic with maildir, my idea is to: > > - Store emails in a SQLite database. This way I'll get ACID > properties for free, as well as the possibility to add additional > information to messages, such as labels. Another bonus are simple > backups - everything is in one file. - Write a simple GMail-IMAP > synchronization tool that only downloads messages from _All mail_ to > the database and synchronizes labels back using [GMail > extensions](https://developers.google.com/gmail/imap_extensions#special-use_extension_of_the_list_command). > > (There could be some time limit like "download only messages newer > than 1 year or so.) - Modify sup so that it can use this SQLite > store and store (or copy) labels into it. > > This also solves the awkward intermediate step when using > OfflineIMAP that labels are first converted into IMAP folders, > downloaded locally to multiple folders and then again converted to > labels for sup (and sup needs to check multiple folders). Using > instead one big folder that understand labels seems much cleaner. > > (Perhaps it would be worth to store GMail thread ID too?) > > --- Reply to this email directly or view it on GitHub: > https://github.com/sup-heliotrope/sup/issues/113#issuecomment-21703944 > From eg@gaute.vetsj.com Mon Jul 29 19:37:30 2013 From: eg@gaute.vetsj.com (Gaute Hope) Date: Mon, 29 Jul 2013 21:37:30 +0200 Subject: [sup-devel] Gmail source sync back. In-Reply-To: References: Message-ID: <51F6C47A.3010004@gaute.vetsj.com> On 03. juli 2013 11:44, Horacio Sanson wrote: > - How to refresh the index after polling finishes? When new labels are > synced from Gmail to Sup the index is not updated with the new labels until > I reload sup or press "@" to refresh the index. Is there a callback to > refresh the index when the polling finishes? Check out pull request #2 towards your github repo, yielding the message on update + some other modifications now updates the message on my setup. > - Sync back iterates over all messages in the Gmail account checking for > differences between message labels. This can take a long time and may not > finish before the next polling starts. How to do this efficiently? If this turns out not be solvable (e.g. being able to get a list of changed messages since a given date), an option could be to allow the user to configure remote-to-local syncing to only be manual. Local-to-remote you can store locally whether it has been synced or not. There's also a pull request #1 which allows arbitrary domains to be entered for the gmail source (for google apps setups). Regards, Gaute From eg@gaute.vetsj.com Mon Jul 29 19:48:14 2013 From: eg@gaute.vetsj.com (Gaute Hope) Date: Mon, 29 Jul 2013 21:48:14 +0200 Subject: [sup-devel] Gmail source sync back. In-Reply-To: References: Message-ID: <51F6C6FE.1090604@gaute.vetsj.com> On 03. juli 2013 11:44, Horacio Sanson wrote: > - Sync back iterates over all messages in the Gmail account checking for > differences between message labels. This can take a long time and may not > finish before the next polling starts. How to do this efficiently? By the way, with the source poll locking recently added no second poll should be able to be initiated when there is one running already. - gaute From eg@gaute.vetsj.com Mon Jul 29 20:01:15 2013 From: eg@gaute.vetsj.com (Gaute Hope) Date: Mon, 29 Jul 2013 22:01:15 +0200 Subject: [sup-devel] Gmail source sync back. In-Reply-To: References: Message-ID: <51F6CA0B.7090406@gaute.vetsj.com> On 03. juli 2013 11:44, Horacio Sanson wrote: > I finally got the sync back feature of the Gmail source working. The sync > code uses the offlineimap sync algorithm the Matthieu mentioned some emails > ago. > Testing and feedback is greatly appreciated. Further testing; if I check a message to be important after it has already been added I get an IMAP Invalid label error: Important. Regards, gaute From hsanson@gmail.com Tue Jul 30 12:41:11 2013 From: hsanson@gmail.com (Horacio Sanson) Date: Tue, 30 Jul 2013 21:41:11 +0900 Subject: [sup-devel] Xapian 1.2.15 dependency causes issues on Ubuntu 13.04 Message-ID: Ubuntu 13.04 installs the distribution xapian dev and lib packages that are version 1.2.12. This causes sup to complain that the xapian 1.2.15 or later is required and bails out. I understand xapian-ruby installs its own xapian lib but sup still tries to load the system one. I changed the index.rb file to allow version 1.2.12 and had not encounter any issues so far. Could it be possible to set this version as the min required version? regards, Horacio -------------- next part -------------- An HTML attachment was scrubbed... URL: From hsanson@gmail.com Tue Jul 30 13:18:27 2013 From: hsanson@gmail.com (Horacio Sanson) Date: Tue, 30 Jul 2013 22:18:27 +0900 Subject: [sup-devel] How do I force flush the log file in sup? Message-ID: Is there a way to force sup to flush the log file after each write? It is difficult to debug my gmail source as I cannot see real time what is going on. Maybe set sup to always flush when SUP_LOG_LEVEL is set to debug? regards, Horacio -------------- next part -------------- An HTML attachment was scrubbed... URL: