Subject: Re: [sup-devel] A maildir root source: syncing labels with maildirs (imap folders)

From: Gaute Hope <[email-masked]>
Date: Wed, 27 Nov 2013 14:24:15 +0100

Excerpts from Gaute Hope's message of 2013-11-27 14:08:36 +0100:
> Excerpts from Steven Schmeiser's message of 2013-11-27 13:42:33 +0100:
> > > I was thinking about changing one thing: currently messages are not
> > > copied into 'archive' before it has been _no_ other labels. On GMail
> > > you can still find it on the 'All mail' page on the web interface,
> > > but for regular IMAP accounts that doesn't work. Should the message
> > > always be copied to 'archive' when it is removed from the inbox?
> > >
> > > - gaute
> >
> > I don't have much personal preference between these two options.  The
> > second option is closer to Gmail's "All Mail" folder that a lot of
> > people are familiar with, so maybe that is the way to go.  Especially
> > if the "copy" is a hard link that doesn't take up disk space.
>
> I noticed Gmail does it that way in their IMAP interface (archive on no
> other labels), thats why I did it. I guess it doesn't matter all that
> much as long as we are all using Sup. Archive becomes a place for the
> uncategorized. I don't delete an email from archive if it is already in
> there, so if it at some point has been copied in there - it stays.
>
> Copy is hard-link, but I doubt offlineimap is able to identify remote
> identical emails. So hard-linking is only on the machine that the
> link happened on.

Another nice feature is that I now have the possibility to have my
filters on the server copy/move the message to a IMAP folder and it will be
automatically tagged in Sup as well.

Some servers automatically delete stuff in Trash / Deleted Items or
email marked as Trash (T), these will also be removed from Sup when they
are deleted (might not work when it is in several folders and only
marked with T).

- gaute

---

From: Gaute Hope <[email-masked]>
Date: Fri, 29 Nov 2013 10:22:44 +0100

Excerpts from Gaute Hope's message of 2013-11-27 22:10:38 +0100:
> Excerpts from Matthieu Rakotojaona's message of 2013-11-27 19:32:02 +0100:
[..]

> It also
> currently does not check new messages (since non-gmail sources will not
> have a copy in archive), we should do a check upon :add in
> poll.rb#do_poll to the source that checks if existing_sources has
> @archive, if not add and return to poll with updated locations (no
> sync_back needed and no index sync needed as poll will do that
> afterwards).

As of commit db5e75f we check in poll if the message is in archive,
otherwise add (archive is the first subdir to be polled so it should
always be present).

Another issue is messages that get labels added automatically, these are
not synced back before the message is opened or somehow otherwise
activated. Could possibly do a full sync_back after :add, :delete or
:update in poll.

- gaute

---

From: Steven Schmeiser <[email-masked]>
Date: Mon, 25 Nov 2013 13:23:24 -0500

Hi,
I'm very interested in trying out this branch.

I don't know the sup internals very well, so I'm having a hard time figuring this out on my own, but what config options need to be changed/set to use maildir-root?  I'm currently using marildir-sync.

Thanks for all your work on sup!

Steve
 
> Hi fellow suppers,
> 
> for a while I have been using a 'maildir-root' source that I created.
> It is based on the maildir-syncback code, but goes further: you only
> specify a root folder with maildirs to the source, then each sub folder
> (maildirsub) is mapped to a label. Some of the labels are special: like
> deleted, inbox and archive (the special-label to maildirsub mapping can
> be specified in the source configuration).
> 
> This means that a sup label corresponds to maildir inside the
> maildir-root and the behavior works like this:
> 
>   - (remote to local) if a message is found in a maildirsub that label is added to it.
>   - (remote to local) if a message is removed from a maildirsub that
>     label is removed (unless it has just been moved).
>   - (local to remote) if you add a label the message is written to that
>     maildirsub (of all the maildir-roots the message belongs to)
>   - (local to remote) if you remove a label from a message the message is deleted
>     from that maildirsub. if it is deleted from all maildirsubs it is
>     copied to the archive maildirsub.
>   - (local to remote) if you archive a message it is removed from the
>     inbox maildirsub, if it doesn't belong to any other
>     labels/maildirsubs it is copied to archive.
>   - (local to remote / remote to local) the same flag features as
>     maildir-syncback (done on all copies in all maildirsubs)
>   - (local to remote) if you add a label to a message and the message
>     belongs to a maildir-root where no corresponding maildirsub exists a
>     maildir will be created for it.
> 
> I have been using it for 4-5 months now and it is getting fairly stable.
> It works fairly well on gmail (label sync), but I am also using it on
> regular IMAP servers (here it might, in the future, be useful to always keep a copy in
> archive).
> 
> If you are interested in trying this please have a look at:
> https://github.com/gauteh/sup/blob/maildir-root/lib/sup/maildirroot.rb
> 
> or my branch: https://github.com/gauteh/sup/tree/maildir-root, but please
> be careful.
> 
> I am using this with offlineimap and utf-7-imap to utf-8 conversion. It
> is probably necessary to do a full sync every time to make sure things
> stay in sync. Remember that it is technically possible to still do
> changes to the maildir while offlineimap is syncing, I haven't had any
> problems with this (except read flag), but it is potentially a little
> careless.
> 
> 
> Cheers, Gaute
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Gaute Hope <[email-masked]>
Date: Wed, 27 Nov 2013 07:23:52 +0100

Excerpts from Gaute Hope's message of 2013-11-26 17:55:16 +0100:
> I haven't implemented :updated in sup-sync yet (see first, or possibly
> second email). To update: start Sup normally, let the Sup poll _finish_
> before you try to open the problematic message. This happens now and
> then, when you label a message (copy to a label dir) it gets a generic
> name, upon offlinimap sync it is renamed (by offlineimap) and untill a
> new poll has been run the source is out of sync. Since the poll has to
> finish to fix the error it is easy to trigger it again when if you start
> sup again.

This, by the way, is also a problem in current Sup (maildir-syncback) -
:updated is not handled in sup-sync. Only in lib/sup/poll.rb.

- gaute

---

From: Gaute Hope <[email-masked]>
Date: Wed, 27 Nov 2013 14:08:36 +0100

- gaute

---

From: Steven Schmeiser <[email-masked]>
Date: Tue, 26 Nov 2013 09:43:49 -0500

I've installed the branch and am trying it out.  So far it looks good, but there have been a few oddities...

I'm running with a standard IMAP account (not Gmail).  The root looks like

[steve@indy ~]$ ls -l .mail
total 0
drwx------  5 steve  staff  170 Nov 19 13:44 Archive
drwx------  5 steve  staff  170 Nov 22 12:31 INBOX
drwx------  5 steve  staff  170 Nov 19 13:44 drafts
drwx------  5 steve  staff  170 Nov 19 13:44 econpapers
drwx------  5 steve  staff  170 Nov 19 13:44 sent
drwx------  5 steve  staff  170 Nov 19 13:44 spam
drwx------  5 steve  staff  170 Nov 25 19:25 trash

and my sources are configured as follows

[steve@indy ~]$ more .sup/sources.yaml
---
- !supmua.org,2006-10-01/Redwood/MaildirRoot
  uri: maildirroot:///Users/steve/.mail
  usual: true
  archived: false
  id: 1
  labels: []
  sync_back: true
  confirm_enable_experimental: true
  maildir_creation_allowed: true
  inbox_folder: INBOX
  sent_folder: sent
  drafts_folder: drafts
  spam_folder: spam
  trash_folder: trash
  archive_folder: Archive
- !supmua.org,2006-10-01/Redwood/SentLoader{}

I had around 40k messages in the Archive folder and ~20 in the INBOX.  After sup-sync, around 10k of the messages in Archive were moved into sup's inbox with labels 'INBOX' and 'Inbox'.  They were also (incorrectly) marked as unread.

I'll keep experimenting.  Thanks for your effort -- I think this is a great new feature and would really like to see it merged into the mainline someday.

Steve

> Hi,
> 
> I have attached my relevant configs (and hopefully managed to anonymize
> them enough). As you can see I use the 'utf7.py' module (placed in
> ~/.offlineimap/) to decode/encode between imap4-utf-7 and utf-8 on
> system (if your filesystem uses a different encoding you might have
> issues). Please don't use the configs verbatim, but rather as
> inspiration: I might have cut out important parts. Some of this is
> in the wiki.
> 
> Some of the folders are translated (bi-directional of course) in my
> offlineimaprc, but if you check out my sources.yaml you will see that I
> have specified which folders are related to special (deleted, starred,
> inbox, etc) for some of the sources.
> 
> My translation setup in offlineimaprc for my first GMail account
> corresponds to the default values the maildirroot source suggests. The
> only relevant change in config.yaml is to set sync_back_to_maildir:
> true.
> 
> The way I set up sup with maildirroot is:
> $ sup-config
> 
> fill in, don't add any sources. If you have a working config.yaml, you
> can use it.
> 
> add sources:
> $ sup-add maildirroot:///home/gaute/.mail/imap_account
> 
> I edit ~/.sup/sources.yaml and enable:
> - confirm_enable_experimental: true
> - sync_back: true
> - maildir_creation_allowed: true
> 
> and tweak the folder names if necessary.
> 
> Then if your index is empty (you _cannot_ merge an existing maildir
> source by using the same source id) - I run sup-sync:
> $ sup-sync -v
> 
> Note that sup-sync does not support the :updated flag (when a message
> has moved on disk), for that you have to run regular Sup and do a poll.
> sup-sync scans through and adds labels; you can also just start up sup
> and see what happens, but that takes a longer time: you probably do not
> need to enable the stuff in sources.yaml before after sup-sync if you
> are nervous (I would be!). 
> 
> Some annoying issues:
> - deleting a maildir is hard:
>   * if you delete it remotely it will be recreated by offlineimap (i
>     think)
>   * you have to first remove all emails remotely, sync sup, to make sure
>     labels are removed.
>   * if you remove all messages from a label the maildir will be empty,
>     to delete the maildir: sync, delete the maildir locally + delete it
>     remotely; then re-sync.
> 
>     this is the behaviour of offlineimap, but I think it is hard to do
>     it differently. this means that if you accidentally add a bogus
>     label, it is a bit of a pain to get rid of the maildir.
> 
> 
> - if a message is deleted from the inbox remotely (archived) it is not
>   removed from the inbox view (or any thread index view it is visible
>   in) before sup is restarted.
> 
> To test maildirroot:
> checkout my branch; make the gem: $ rake gem and install it: $ gem
> install pkg/sup-999.gem. If develop has changed you can try to merge it in
> first.
> 
> Again; please make a backup copy of your .mail so that you can restore
> it if it turns out maildirroot has been eating or multiplying emails :)
> 
> I probably don't have the time to difficult bugs, but I would be happy
> to accept patches and I could also put this branch on official Sup if it
> turns out to be worth something.
> 
> 
> Cheers, Gaute

---

From: Steven Schmeiser <[email-masked]>
Date: Wed, 27 Nov 2013 07:42:33 -0500

> Excerpts from Steven Schmeiser's message of 2013-11-27 01:48:20 +0100:
> >
> > > Excerpts from Steven Schmeiser's message of 2013-11-26 17:42:48 +0100:
> > > >
> > > > On Nov 26, 2013, at 10:14, Gaute Hope <[email-masked]> wrote:
> > > >
> > > > > Excerpts from Steven Schmeiser's message of 2013-11-26 15:43:49 +0100:
> > > > [2013-11-26 11:34:21 -0500] maildirsub set up, type: deleted, label: deleted
> > > > [2013-11-26 11:34:21 -0500] setting up generic folders..
> > > > [2013-11-26 11:34:21 -0500] maildirsub set up, type: generic, label: econpapers
> > > > [2013-11-26 11:34:21 -0500] no draft source, auto-adding...
> > > > [2013-11-26 11:34:21 -0500] starting curses
> > > > [2013-11-26 11:34:21 -0500] loading user colors from /Users/steve/.sup/colors.yaml
> > > > [2013-11-26 11:34:21 -0500] initializing log buffer
> > > > [2013-11-26 11:34:21 -0500] Welcome to Sup! Log level is set to debug.
> > > > [2013-11-26 11:34:21 -0500] initializing inbox buffer
> > > > [2013-11-26 11:34:21 -0500] ready for interaction!
> > >
> > > Looks good: Only econpapers is identified as a generic label, the other
> > > folders are mapped to one of the specials.
> >
> > It turned out to be a before-add-message hook that was causing all the problems.  Working great now.
> 
> Cool! Let me know how it works in the longer run!
> 
> copied into 'archive' before it has been _no_ other labels. On GMail you
> can still find it on the 'All mail' page on the web interface, but for
> regular IMAP accounts that doesn't work. Should the message always be
> copied to 'archive' when it is removed from the inbox?
> 
> - gaute

I don't have much personal preference between these two options.  The second option is closer to Gmail's "All Mail" folder that a lot of people are familiar with, so maybe that is the way to go.  Especially if the "copy" is a hard link that doesn't take up disk space. 

Steve

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Wed, 27 Nov 2013 19:32:02 +0100

Excerpts from Gaute Hope's message of 2013-11-27 14:08:36 +0100:
> other labels), thats why I did it.

_Any_ email in gmail is also in the All Mail [0], except when it is
manually moved to the [Gmail]/Trash folder, in which case it disappears
forever (well, at least for you :). I think this is the best solution,
because it is the most understandable for all MUAs out there: the fact
that "a maildir is a label" is something pretty specific to
gmail-pioneered MUAs, so at least when I backup the [Gmail]/All Mail I
know I won't have lost any of them (even if I have lost the labels).

Regarding hardlinking, I'm afraid of the cross-platform issues. I know
some people here use Mac OSX, is it safe to use hardlinking on this
platform ?

(Oh and by the way I found a script [1] that basically hardlinks every
copy of an email, this could be especially useful for gmail-based
installations)

[0] https://support.google.com/mail/answer/78755?hl=en
[1] http://jak-linux.org/projects/hardlink/

-- 
Matthieu Rakotojaona

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Mon, 25 Nov 2013 22:08:48 +0100

Wow, that's amazing stuff. Would you care to share your .offlineimaprc ?
I myself use the gmail source
(https://github.com/rakoo/sup/tree/gmail_source), but it's not very
smart at syncing old labels, and it's mostly targeted towards gmail, so
I'd like to switch to something else (plus having a local tree of
maildirs is always helpful)

---

From: Steven Schmeiser <[email-masked]>
Date: Tue, 26 Nov 2013 19:48:20 -0500

Steve

---

From: Steven Schmeiser <[email-masked]>
Date: Tue, 26 Nov 2013 11:42:48 -0500

On Nov 26, 2013, at 10:14, Gaute Hope <[email-masked]> wrote:

> Ok, there are likely to be a few more of the oddities - you are brave :)
> 
> Please set the environment variable: SUP_LOG_LEVEL=debug and try to
> start sup, it would be interesting to see how the maildirsubs are
> initialized (the startup section from ~/.sup/log).
> 
> Were any of the messages in Archive also somewhere else (duplicates with
> the same id)? Also, if sup discovers a message that is part of a thread
> it will join them and (possibly) apply all the labels to all the
> messages in that thread. If any of those are unread it might be
> affected.
> 
> The labels 'INBOX' and 'Inbox' are both not the same as the special
> label 'inbox', so apparently it is not recognized as the special label
> but a custom label.
> 
> It could be that you have more luck with starting with a clean index and
> just starting regular sup (not sup-sync). You can follow the log by
> doing: tail -f ~/.sup/log.
> 
> - gaute

Startup log follows...

[2013-11-26 11:34:20 -0500] using character set encoding "UTF-8"
[2013-11-26 11:34:21 -0500] dynamically loading setlocale() from libc.dylib
[2013-11-26 11:34:21 -0500] setting locale...
[2013-11-26 11:34:21 -0500] locking /Users/steve/.sup/lock...
[2013-11-26 11:34:21 -0500] maildirroot:///Users/steve/.mail: setting up maildirroot..
[2013-11-26 11:34:21 -0500] setting up maildir subs..
[2013-11-26 11:34:21 -0500] maildirsub set up, type: archive, label: archive
[2013-11-26 11:34:21 -0500] maildirsub set up, type: inbox, label: inbox
[2013-11-26 11:34:21 -0500] maildirsub set up, type: sent, label: sent
[2013-11-26 11:34:21 -0500] maildirsub set up, type: draft, label: draft
[2013-11-26 11:34:21 -0500] maildirsub set up, type: spam, label: spam
[2013-11-26 11:34:21 -0500] maildirsub set up, type: deleted, label: deleted
[2013-11-26 11:34:21 -0500] setting up generic folders..
[2013-11-26 11:34:21 -0500] maildirsub set up, type: generic, label: econpapers
[2013-11-26 11:34:21 -0500] no draft source, auto-adding...
[2013-11-26 11:34:21 -0500] starting curses
[2013-11-26 11:34:21 -0500] loading user colors from /Users/steve/.sup/colors.yaml
[2013-11-26 11:34:21 -0500] initializing log buffer
[2013-11-26 11:34:21 -0500] Welcome to Sup! Log level is set to debug.
[2013-11-26 11:34:21 -0500] initializing inbox buffer
[2013-11-26 11:34:21 -0500] ready for interaction!

I'm now not able to run sup-sync or open some of the threads in my inbox without crashing with the following error:

/usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/bin/sup-sync:180:in `block (2 levels) in <top (required)>': NotImplementedError (NotImplementedError)
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/poll.rb:292:in `block (2 levels) in poll_from'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/index.rb:298:in `block in each_message'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/index.rb:285:in `block in each_id'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/index.rb:285:in `each'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/index.rb:285:in `each_id'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/index.rb:297:in `each_message'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/util.rb:649:in `method_missing'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/poll.rb:251:in `block in poll_from'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/maildirroot.rb:512:in `block (2 levels) in poll'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/maildirroot.rb:511:in `each'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/maildirroot.rb:511:in `block in poll'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/util.rb:261:in `call'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/util.rb:261:in `block in benchmark'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/1.9.1/benchmark.rb:280:in `measure'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/util.rb:261:in `benchmark'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/maildirroot.rb:474:in `poll'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/poll.rb:201:in `poll_from'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/lib/sup/util.rb:649:in `method_missing'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/bin/sup-sync:125:in `block in <top (required)>'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/bin/sup-sync:120:in `each'
        from /usr/local/Cellar/ruby193/1.9.3-p448/lib/ruby/gems/1.9.1/gems/sup-999/bin/sup-sync:120:in `<top (required)>'
        from /usr/local/opt/ruby193/bin/sup-sync:23:in `load'
        from /usr/local/opt/ruby193/bin/sup-sync:23:in `<main>'

I'll try starting with a clean index.

Steve

---

From: Gaute Hope <[email-masked]>
Date: Wed, 27 Nov 2013 07:21:41 +0100

- gaute

---

From: Gaute Hope <[email-masked]>
Date: Wed, 27 Nov 2013 22:10:38 +0100

> know I won't have lost any of them (even if I have lost the labels).

Hm ok. I just figured it out by trail-and-error, I probably missed
something. I have changed sync_back to always make sure the message is
present in the archive. This does not have any effect on existing
messages before you do some edit that triggers sync_back. It also
currently does not check new messages (since non-gmail sources will not
have a copy in archive), we should do a check upon :add in
poll.rb#do_poll to the source that checks if existing_sources has
@archive, if not add and return to poll with updated locations (no
sync_back needed and no index sync needed as poll will do that
afterwards).

About deletion, current behavior:
- apply :deleted label; this means:
- the message is moved to the trash folder
- the T flag is added to all the labels it is part of

this might not work exactly as it should on GMail, but it is the only
way to preserve labels (in case of restore).

> some people here use Mac OSX, is it safe to use hardlinking on this

I am using the util function I created for maildir-sync (in develop)
which falls back to copying the file if hardlinking fails, that function
works on Mac OS X. Hardlinking doesn't work for me since I am using
encfs to store mail.

> [1] http://jak-linux.org/projects/hardlink/

That could be very useful and run as a cronjob now and then.

- gaute

---

From: Gaute Hope <[email-masked]>
Date: Tue, 26 Nov 2013 17:55:16 +0100

> I'll try starting with a clean index.

Always a good idea :)

- gaute

---

From: Gaute Hope <[email-masked]>
Date: Mon, 25 Nov 2013 22:51:09 +0100

Hi,

add sources:

Some annoying issues:

To test maildirroot:

Cheers, Gaute

---

From: Gaute Hope <[email-masked]>
Date: Mon, 25 Nov 2013 23:28:32 +0100

By the way:

sub dirs show up with dots between them so maildir:
dev/sup becomes label dev.sup (can't remember if this is standard
offlineimap behavior or something I did).

- gaute

---

From: Gaute Hope <[email-masked]>
Date: Tue, 26 Nov 2013 16:14:25 +0100

- gaute

---

From: Gaute Hope <[email-masked]>
Date: Mon, 25 Nov 2013 16:24:58 +0100

Hi fellow suppers,

Cheers, Gaute