Subject: Re: [sup-devel] Adding backward synchronization for Maildir sources

From: Damien Leone <[email-masked]>
Date: Wed, 21 Jul 2010 10:54:18 +0200

Excerpts from Damien Leone's message of Wed Jul 21 10:52:00 +0200 2010:
> What is your setup? IMAP + offlineimap?

My bad, you mentioned offlineimap a few mails ago.

--
Damien Leone <[email-masked]>

Web: http://dleone.fensalir.fr/
GPG: 0x82EB4DDF
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Ben Walton <[email-masked]>
Date: Sat, 17 Jul 2010 13:05:44 -0400

Excerpts from Damien Leone's message of Sat Jul 17 08:55:56 -0400 2010:

> So, any reviewer? I thought backward synchronization was a wanted
> feature.

I believe it is (not by me personally as I'm not affected).  How does
it play with the recent maildir branch merge from Rich?

Thanks
-Ben
--
Ben Walton
Systems Programmer - CHASS
University of Toronto
C:416.407.5610 | W:416.978.4302

---

From: "Edward Z. Yang" <[email-masked]>
Date: Sun, 18 Jul 2010 18:57:31 -0400

Here is one trace that I managed to scrounge up

undefined method `content_width' for nil:NilClass
/home/ezyang/Dev/sup/lib/sup/modes/thread-index-mode.rb:953:in `from_width'
/home/ezyang/Dev/sup/lib/sup/modes/thread-index-mode.rb:870:in `text_for_thread_at'
/home/ezyang/Dev/sup/lib/sup/hook.rb:55:in `each_with_index'
/home/ezyang/Dev/sup/lib/sup/modes/thread-index-mode.rb:869:in `each'
/home/ezyang/Dev/sup/lib/sup/modes/thread-index-mode.rb:869:in `each_with_index'
/home/ezyang/Dev/sup/lib/sup/modes/thread-index-mode.rb:869:in `text_for_thread_at'
/home/ezyang/Dev/sup/lib/sup/modes/thread-index-mode.rb:811:in `regen_text'
/home/ezyang/Dev/sup/lib/sup/util.rb:423:in `map_with_index'
/home/ezyang/Dev/sup/lib/sup/hook.rb:55:in `each_with_index'
/home/ezyang/Dev/sup/lib/sup/util.rb:423:in `each'
/home/ezyang/Dev/sup/lib/sup/util.rb:423:in `each_with_index'
/home/ezyang/Dev/sup/lib/sup/util.rb:423:in `map_with_index'
/home/ezyang/Dev/sup/lib/sup/modes/thread-index-mode.rb:811:in `regen_text'
/home/ezyang/Dev/sup/lib/sup/modes/thread-index-mode.rb:262:in `update'
/home/ezyang/Dev/sup/lib/sup/modes/thread-index-mode.rb:217:in `handle_location_deleted_update'
/home/ezyang/Dev/sup/lib/sup/update.rb:26:in `send'
/home/ezyang/Dev/sup/lib/sup/update.rb:26:in `relay'
/home/ezyang/Dev/sup/lib/sup/update.rb:26:in `each'
/home/ezyang/Dev/sup/lib/sup/update.rb:26:in `relay'
/home/ezyang/Dev/sup/lib/sup/util.rb:588:in `send'
/home/ezyang/Dev/sup/lib/sup/util.rb:588:in `method_missing'
/home/ezyang/Dev/sup/lib/sup/poll.rb:197:in `poll_from'
/home/ezyang/Dev/sup/lib/sup/index.rb:258:in `each_message'
/home/ezyang/Dev/sup/lib/sup/index.rb:245:in `each_id'
/home/ezyang/Dev/sup/lib/sup/index.rb:245:in `each'
/home/ezyang/Dev/sup/lib/sup/index.rb:245:in `each_id'
/home/ezyang/Dev/sup/lib/sup/index.rb:257:in `each_message'
/home/ezyang/Dev/sup/lib/sup/util.rb:588:in `send'
/home/ezyang/Dev/sup/lib/sup/util.rb:588:in `method_missing'
[snip]

Cheers,
Edward

---

From: Tero Tilus <[email-masked]>
Date: Sun, 18 Jul 2010 14:19:55 +0300

Damien Leone, 2010-07-17 15:55:
> So, any reviewer?

I haven't had the time yet.  I definitely will once the temperature
here in Finland drops below 20 degrees Celcius. ;)  For now I'm almost
totally off the net and on the beach.

> I thought backward synchronization was a wanted
> feature.

And you were _very_ much right!

> I often check my gmail web interface and everything is synched.

My setup (once I get there) will be mainly sup + occasional mutt (on
maildirs) and evolution (via dovecot).

-- 
Tero Tilus ## 050 3635 235 ## http://tero.tilus.net/
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: "Edward Z. Yang" <[email-masked]>
Date: Tue, 20 Jul 2010 20:44:38 -0400

Excerpts from Damien Leone's message of Tue Jul 20 05:39:41 -0400 2010:
> Excerpts from Edward Z. Yang's message of Mon Jul 19 00:46:33 +0200 2010:
> > It makes Sup very crashy and there's some strange behavior.  This is probably
> > the sixth time I'm composing this message (yeah, haven't learned my
> > lesson.) In more detail...
> 
> I'm impressed, I never experienced such crashes, it is quite stable
> here but I use only very basic hooks.

In retrospect, I'm fairly certain the majority of crashes were from user hooks.

> > * Backwards syncing appears to make old deleted mail show up again; I'm not
> >   sure if it's because touching the Maildir causes Sup to pick up on deleted
> >   tags again or something, but I had to go back and delete lots of deleted mail.
> 
> Hmm, strange, do you use the latest commits of the branch? I pushed a
> fix for deleted mails about 10 days ago.
> 
> The mails that show up again don't have the deleted flag at all?

My Sup is running off of f150276f62e263d146fa81ede645b5e90e06a8a1.  They don't
appear to have the deleted flag, but I didn't look too closely.  It's *possible*
that a regular synchronization would have picked them up, if they accidentally
got dropped from my Sup index but were still inside my Maildir.  I suggest not
worrying too much about this unless someone else reproduces.

> I did not test the code with hooks, what do you suggest for this one?

I mentioned this in the id and Nil message; basically, when iterating over all
messages messages might disappear, and some hooks need to check if a message is
Nil before doing anything to it.

> > * In a similar vein, while messages are going away, '@' will occasionally return
> >   no messages (pressing again usually fixes it).  Messages that are labelled inbox
> >   will show up in 'U', but not in the inbox view without a Sup restart.
> 
> I never use this keybinding but indeed I noticed that pressing '@' two
> times in a row was not showing anything.

It is possible that this is a bug in vanilla Sup.

Cheers,
Edward

---

From: Damien Leone <[email-masked]>
Date: Tue, 20 Jul 2010 11:45:41 +0200

Hi,

Excerpts from Edward Z. Yang's message of Mon Jul 19 00:50:39 +0200 2010:
> I would recommend having some sort of flag like 'sync-back'd' programatically
> generated by sup-sync-back-maildir so that Sup will refuse to start if
> you turned on sync-backs without running this command.

Good idea, that might prevent some bad suprises.

> It seems to me that merging the Maildir flags and the Sup flags would
> let you preserve this, but for heavy Sup users this isn't a big deal.

True, and that could be done by sup-sync-back-maildir.

> I think this will be a pretty killer feature to have, because it means
> we can then assign "primary" tags and use those to have sync-backs move
> messages between Maildir folders (on the same account, I imagine), which
> would translate into a folder-sorted webmail view, which is about the
> best you can do without first-class label support in Maildir.  I know of
> at least one friend who this would be a deal-maker.

And that doesn't seem hard to implement, we should definitely consider
this feature, maybe after the primary code has been stabilized.

Cheers,

---

From: "Edward Z. Yang" <[email-masked]>
Date: Sun, 18 Jul 2010 18:46:33 -0400

Sup Damien,

I cut over to your branch, plus one patch of my own making.

THE GOOD NEWS

It works!  Whereby, messages are marked read in my webmail client and
deleted messages go away.

THE BAD NEWS

* You want the following OfflineImap patch, because otherwise the massive
  changes to labels that will surely be induced by backwards syncing will
  cause OfflineImap to CRAWL.

diff --git a/offlineimap/folder/LocalStatus.py b/offlineimap/folder/LocalStatus.py
index 157989d..7f4dc5b 100644
--- a/offlineimap/folder/LocalStatus.py
+++ b/offlineimap/folder/LocalStatus.py
@@ -33,7 +33,7 @@ class LocalStatusFolder(BaseFolder):
         self.messagelist = None
         self.repository = repository
         self.savelock = threading.Lock()
-        self.doautosave = 1
+        self.doautosave = 0
         self.accountname = accountname
         BaseFolder.__init__(self)

  Though, you probably want to undo this patch when you're done.  Autosaving
  means partial progress is saved.

* When Sup is deleting messages, it is /extremely/ unsafe to iterate over
  a list of all messages in the index and query them for info: the threading means
  that any given message might not exist anymore.  I had a hook that kept triggering
  this bug, and earlier on in the process drawing functions would also frequently
  Nil error.  This is probably under-tested codepaths in Sup that we should track
  down; unfortunately I didn't have the presence of mind to record traces until
  later (and the ones I got all look like my hooks fault).

I haven't done a code review yet.

Cheers,
Edward

---

From: Justin Mazzi <[email-masked]>
Date: Thu, 27 Oct 2011 12:34:35 +0000 (UTC)

Damien Leone <damien.leone <at> fensalir.fr> writes:

> 
> Sup guys,
> 
> Sorry for having been so long.
> 
> What's new:
> 
> - I added an option in sup-sync-back-maildir to merge the :replied and
> :forwarded labels/flags so the first time you run the script you
> should not lose any information on the IMAP side;
> 
> - A way to detect if sup-sync-back-to-maildir should be executed in
> order to synchronize your index back to the Maildir source in order to
> prevent any loss of information. A magic file is written in the
> configuration directory everytime the script is executed and it is
> removed when the sync_back_to_maildir option is set from false to
> true;
> 
> - I fixed a case when messages were not properly updated when they
> were remotely modified.
> 
> I have pushed the last commits to the maildir-sync branch [0], it is
> rebased from master. I also have a branch rebased from next on my own
> repository if you'd rather a more up to date branch [1].
> 
> Please test and review! For more informations and a howto, read the
> message quoted below this mail.
> 
> [0] http://gitorious.org/sup/mainline/commits/maildir-sync
> [1] http://git.fensalir.fr/?p=dleone/sup.git

Hey,

The syncing is working pretty well for me except for one thing (with Gmail). 

When archiving a message in sup it doesn't remove the Inbox label 
from the message. Removing the Inbox label from the message is 
what makes the email disappear from other clients, like a 
smartphone.

- Justin Mazzi

---

From: Damien Leone <[email-masked]>
Date: Sat, 17 Jul 2010 14:55:56 +0200

For information I have been testing it with my real account for more
than a week now without noticing any serious issue, I often check my
gmail web interface and everything is synched.

I had once a location trouble after having sent an email but I could
not reproduce and it has been resolved after a sup restart.

---

From: Damien Leone <[email-masked]>
Date: Tue, 20 Jul 2010 11:39:41 +0200

Sup Edward,

Thank you for your feedback!

I will try to track down these bugs.

Thanks again!
Cheers,

---

From: "Edward Z. Yang" <[email-masked]>
Date: Fri, 09 Jul 2010 12:56:31 -0400

Hey Damien,

This is great!  A few quick questions for you:

1. Suppose we decide to test your branch, and run into some show-stopper
   bugs.  What’s the procedure for backing out changes?

2. What precisely are the semantics for handling Sup labels?

Thanks,
Edward

Excerpts from Damien Leone's message of Fri Jul 09 12:44:01 -0400 2010:
> Sup guys,
> 
> I have been working on backward synchronization for Maildir sources
> last week and I need your help to test and review the code.
> 
> I pushed my commits (based on the 'next' branch) to my repo which you
> can find here [0] for the web interface and here [1] for the git
> address.
> 
> The following is related to Maildir sources ONLY.
> 
> So what does work and what does not (if you don't want to read this,
> scroll to the quick howto below):
> 
> - Sup will now stay synchronized with your remote sources by detecting
> if a message has been remotely updated (ie: you change a flag from
> another client), these updates will be applied to your local index and
> your thread-view-mode buffers should be properly refreshed;
> 
> - When a message is remotely deleted, it should now disappear from
> your index;
> 
> - There is a new configuration option called "sync_back_to_maildir"
> which is false by default. When true, this option updates your actual
> Maildir files when you change a label, this is done in real time
> before being saved to xapian. So for instance if you use offlineimap
> the changes should be synched back to your IMAP server by the next
> poll in Sup (assuming that your before-poll hook runs offlineimap);
> 
> - I wrote a "sup-sync-back-maildir" script that will synchronize all
> messages from your sources at once. This should be executed the first
> time BEFORE any polling, otherwise your IMAP server will be synched to
> your Sup index and if you did not used another client to mark your
> emails as read etc. (like me) and to update your IMAP server then you
> will lose all what you did in Sup (labels, etc.);
> 
> - I added two new hidden labels in Sup: 'replied' and 'forwarded',
> they are automatically added when replying, forwarding or bouncing a
> message, this is to bring a better Maildir support to Sup, this should
> be invisible for users;
> 
> - However you will probably lose all your remote 'replied' and
> 'forwarded' flags after your first backward synchronization to Maildir
> since Sup dropped these labels when it added your messages to xapian.
> 
> - Moving a message from a Maildir source to another is not (yet?)
> supported.
> 
> So, you should now be able to use multiple clients to handle your
> emails, for instance I can use gmail to mark messages as starred or
> so, it will be reflected in Sup. In the same way, if I mark a message
> as read in Sup it will appear as read in the gmail web interface.
> 
> I tested this code with my main account (2 GB and like 15 Maildir
> sources) it works fine so far.
> 
> Be warned that it is highly experimental, but if you use offlineimap
> it has a realdelete option so you *should* not lose any email, in the
> worst case it might mess your flags/labels up.
> 
> I tested it only with offlineimap working with a gmail account.
> 
> Quick howto:
> 
> 1. Close sup
> 2. Backup your emails and your xapian index or use another Sup session
> 3. Clone the branch [1]
> 4. IMPORTANT: run "bin/sup-sync-back-maildir" to synchronize the
>    Maildirs you wish, check the help
> 5. Add ":sync_back_to_maildir: true" to your config.yaml
> 6. Run and use sup
> 
> Please test it and make reviews! :)
> 
> NOTE: if you want this to work, get sure that you are not calling
> offlineimap with the '-q' option (as suggested in the wiki) otherwise
> it will ignore the flag updates on IMAP server, so check your
> before-poll hook.
> 
> [0] http://git.fensalir.fr/?p=dleone/sup.git
> [1] $ git clone git://fensalir.fr/dleone/sup.git -b maildir-sync
>

---

From: Damien Leone <[email-masked]>
Date: Sun, 13 Mar 2011 17:41:04 +0100

Sup guys,

What's new:

---

From: "Edward Z. Yang" <[email-masked]>
Date: Fri, 09 Jul 2010 15:46:08 -0400

Excerpts from Damien Leone's message of Fri Jul 09 13:14:58 -0400 2010:
> You should backup your Maildir directories and your xapian index, this
> is what I did and if something goes wrong I can still restore my
> backups and run offlineimap to resynch everything as it was before on
> the IMAP server.
> 
> But for my initial tests I used a test Maildir containing nothing
> important so maybe this is what you should do. ;)

I doubt a test Maildir will give you the sort of test pressure that you’re
looking for. Go go real datasets.

> I followed the Maildir documentation [0]. When synching a message, Sup
> builds the flags string and relink the file to the disk, moving it
> from new/ to cur/ or vice versa if needed.
> 
> Related labels are: unread, replied, forwarded, deleted, draft (but it
> won't work for now because draft messages in Sup are handled
> separately), flagged (aka starred).

Great, thanks!  I’ll give it a spin this weekend.

Edward

---

From: Damien Leone <[email-masked]>
Date: Fri, 09 Jul 2010 19:14:58 +0200

Hello Edward,

Every other labels are ignored because not supported.

[0] http://cr.yp.to/proto/maildir.html

---

From: "Edward Z. Yang" <[email-masked]>
Date: Sun, 25 Jul 2010 18:25:26 -0400

Excerpts from Rich Lane's message of Sat Jul 24 17:30:16 -0400 2010:
> I've rebased this against master and merged it to branch maildir-sync in
> the official repo. This is a big change, so help with testing and code
> review would be much appreciated.

Hi Rich,

I've moved my Sup checkout onto this branch. *test test test test*
After a week of usage, I'd say the changes are pretty stable.

Cheers,
Edward

---

From: Rich Lane <[email-masked]>
Date: Sat, 24 Jul 2010 17:30:16 -0400

---

From: Damien Leone <[email-masked]>
Date: Wed, 21 Jul 2010 10:52:00 +0200

Sup Edward,

> In retrospect, I'm fairly certain the majority of crashes were from user hooks.

Could you paste me these hooks please?

> worrying too much about this unless someone else reproduces.

What is your setup? IMAP + offlineimap? Do deleted mails work fine
now?

Cheers,

---

From: "Edward Z. Yang" <[email-masked]>
Date: Sun, 18 Jul 2010 18:50:39 -0400

Here are some suggestions...

Cheers,
Edward

---

From: Damien Leone <[email-masked]>
Date: Sat, 17 Jul 2010 19:26:16 +0200

Hi Ben,

> it play with the recent maildir branch merge from Rich?

It should play well since I based my branch on the 'next' branch which
already had the 'maildir' merge.

---

From: Damien Leone <[email-masked]>
Date: Fri, 09 Jul 2010 18:44:01 +0200

Sup guys,

Quick howto:

1. Close sup