Subject: Re: [sup-devel] [PATCH] Implement moving message between Maildir sources based on label.

From: "Edward Z. Yang" <[email-masked]>
Date: Wed, 22 Aug 2012 01:54:15 -0400

Usage instructions:

In config.yaml, you need to add a new option :maildir_labels:

    :maildir_labels:
      :stanford: [[:inbox, 4], [null, 6]]

Maildir labels is a dictionary of "accounts" to lists of precedences.
Read it as follows:

    For messages in source 4 or source 6 (consult sources.yaml),
    if the message has the :inbox tag, move it to source 4, otherwise
    move it to source 6.

So in this case, 6 would be some sort of Archive folder, and 4 would be INBOX.
If you want "export-only" folders, just tack them on after the null entry;
the labels are checked *in order*.  Multiple accounts are supported, but
these should be disjoint sets of sources.

This will automatically start working for any new mail you change the labels of.
In order to apply this to old mail, you need to run sup-sync-back-maildir.
OfflineIMAP will shit its pants [1] if you move too much mail, so I recommend
holding on until I implement the companion patch for OfflineIMAP if you have
a lot of mail.

Edward

[1] Namely, it will reupload every single article of mail, and if you get
unlucky and "Archive" is sorted before "INBOX", it will probably run you out
of quota too.  If you arrange to delete everything from INBOX first, and then
sync Archive, it will probably just spend a lot of time uploading.

Excerpts from Edward Z. Yang's message of Wed Aug 22 01:45:31 -0400 2012:
> From: "Edward Z. Yang" <[email-masked]>
> 
> Signed-off-by: Edward Z. Yang <[email-masked]>
> ---
>  lib/sup/maildir.rb | 28 ++++++++++++++++++++++------
>  lib/sup/message.rb |  9 +++++++--
>  2 files changed, 29 insertions(+), 8 deletions(-)
> 
> diff --git a/lib/sup/maildir.rb b/lib/sup/maildir.rb
> index 95305c2..ff8da23 100644
> --- a/lib/sup/maildir.rb
> +++ b/lib/sup/maildir.rb
> @@ -77,8 +77,17 @@ class Maildir < Source
>    end
>  
>    def sync_back id, labels
> +    new_source = @id
> +    $config[:maildir_labels].each do |k,v|
> +      v.each do |lbl,i|
> +        if lbl.nil? or labels.member? lbl
> +          new_source = i
> +          break
> +        end
> +      end if v.any? { |lbl,i| i == @id }
> +    end if $config[:maildir_labels]
>      flags = maildir_reconcile_flags id, labels
> -    maildir_mark_file id, flags
> +    maildir_move_file id, new_source, flags
>    end
>  
>    def raw_header id
> @@ -221,24 +230,31 @@ private
>        new_flags.to_a.sort.join
>    end
>  
> -  def maildir_mark_file orig_path, flags
> +  def maildir_move_file orig_path, new_source_id, flags
>      @mutex.synchronize do
>        new_base = (flags.include?("S")) ? "cur" : "new"
>        md_base, md_ver, md_flags = maildir_data orig_path
>  
> -      return if md_flags == flags
> +      return if md_flags == flags and new_source_id == @id
> +
> +      new_source = SourceManager[new_source_id]
>  
>        new_loc = File.join new_base, "#{md_base}:#{md_ver},#{flags}"
>        orig_path = File.join @dir, orig_path
> -      new_path  = File.join @dir, new_loc
> +      new_path  = File.join new_source.file_path, new_loc
>        tmp_path  = File.join @dir, "tmp", "#{md_base}:#{md_ver},#{flags}"
>  
>        File.link orig_path, tmp_path
>        File.unlink orig_path
> -      File.link tmp_path, new_path
> +      begin
> +        File.link tmp_path, new_path
> +      rescue SystemCallError
> +        File.unlink new_path # XXX kinda unsafe eh
> +        File.link tmp_path, new_path
> +      end
>        File.unlink tmp_path
>  
> -      new_loc
> +      [new_source, new_loc]
>      end
>    end
>  end
> diff --git a/lib/sup/message.rb b/lib/sup/message.rb
> index 3eeea66..d6016df 100644
> --- a/lib/sup/message.rb
> +++ b/lib/sup/message.rb
> @@ -726,8 +726,13 @@ class Location
>    end
>  
>    def sync_back labels
> -    new_info = source.sync_back(@info, labels) if source.respond_to? :sync_back
> -    @info = new_info if new_info
> +    pair = source.sync_back(@info, labels) if source.respond_to? :sync_back
> +    if pair
> +      new_source, new_info = pair
> +      @source = new_source if new_source
> +      @info = new_info if new_info
> +    end
> +    pair
>    end
>  
>    ## much faster than raw_message
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: "Edward Z. Yang" <[email-masked]>
Date: Thu, 23 Aug 2012 10:40:22 -0400

It looks like the current patchset will *destroy* all labels you have
on messages, so be careful! (This was masked for me since most of my
labels are from before-add-message.rb, and that hook gets reapplied)

Edward

Excerpts from Edward Z. Yang's message of Wed Aug 22 16:09:40 -0400 2012:
> Here is an improved version that handles the thread/message distinction
> better; previously, if you archived something, it's possible only some of
> the messages got moved if you hadn't run sup-sync-back-maildir.
> 
> commit a25c345c15c8859a041cc9dc13090d7b8feb3d58
> Author: Edward Z. Yang <[email-masked]>
> Date:   Tue Aug 21 23:40:03 2012 -0400
> 
>     Implement moving message between Maildir sources based on label.
>     
> index 95305c2..a7bf567 100644
> --- a/lib/sup/maildir.rb
> +++ b/lib/sup/maildir.rb
> @@ -76,9 +76,23 @@ class Maildir < Source
>      with_file_for(id) { |f| RMail::Parser.read f }
>    end
>  
> +  def wrong_source? labels
> +    new_source = nil
> +    new_source
> +    new_source = wrong_source? labels || @id
> @@ -221,24 +235,31 @@ private
>        new_flags.to_a.sort.join
>    end
>  
> index 3eeea66..4139f3e 100644
> --- a/lib/sup/message.rb
> +++ b/lib/sup/message.rb
> @@ -285,6 +285,10 @@ EOS
>      location.each_raw_message_line &b
>    end
>  
> +  def wrong_source?
> +    $config[:sync_back_to_maildir] and @locations.any? { |l| l.valid? and l.source.is_a? Maildir and l.wrong_source? @labels}
> +  end
> +
>    def sync_back
>      if @locations.map { |l|
>        l.sync_back @labels if l.valid? and $config[:sync_back_to_maildir] and l.source.is_a? Maildir
> @@ -726,8 +730,17 @@ class Location
>    end
>  
>    def sync_back labels
> +  def wrong_source? labels
> +    source.wrong_source? labels if source.respond_to? :wrong_source
>    end
>  
> diff --git a/lib/sup/thread.rb b/lib/sup/thread.rb
> index b08bae2..c1e4b40 100644
> --- a/lib/sup/thread.rb
> +++ b/lib/sup/thread.rb
> @@ -113,7 +113,7 @@ class Thread
>  
>    def set_labels l; each { |m, *o| m && m.labels = l }; end
>    def has_label? t; any? { |m, *o| m && m.has_label?(t) }; end
> -  def each_dirty_message; each { |m, *o| m && m.dirty? && yield(m) }; end
> +  def each_dirty_message; each { |m, *o| m && (m.dirty? || m.wrong_source?) && yield(m) }; end
>  
>    def direct_participants
>      map { |m, *o| [m.from] + m.to if m }.flatten.compact.uniq

---

From: "Edward Z. Yang" <[email-masked]>
Date: Thu, 23 Aug 2012 11:28:22 -0400

Yes. The current plan is to snoop the message ID of newly added messages
in maildirs, and do moved message detection based on that.  If you have a
better idea I'm all ears. (Another possibility is to add sha1sum to the
index and compare that.)

Also note that it won't destroy the labels *unless you run OfflineIMAP*.
The trouble is OfflineIMAP reuploads your message and then reassigns
the UID based on the IMAP server, but doesn't communicate this back to Sup.

Edward

Excerpts from Alvaro Herrera's message of Thu Aug 23 10:52:21 -0400 2012:
> Excerpts from Edward Z. Yang's message of jue ago 23 10:40:22 -0400 2012:
> Thanks for the notice.  Most of my labels come from that hook too, but a
> few do not, so I could easily lose some!  I was planning on trying out
> your patch, but with this caveat I'm likely to refrain.  Are you
> planning on submitting a fixed version?
>

---

From: "Edward Z. Yang" <[email-masked]>
Date: Tue, 21 Aug 2012 10:35:45 -0400

I think it would be nice if we at least support the :inbox label,
so that we can keep our inboxes tidy and our OfflineIMAP sync
times down.

Proposed implementation strategy:

    - Define a label/source mapping, as well as a default source for
      unrecognized labels and an ordering on labels if there is
      a conflict.  (Each Maildir folder is its own source.)

    - Implement moving messages between Maildir sources in Sup.
      Probably the easiest way is to do this:

        1. Copy message form old Maildir to new Maildir
        2. Add new Maildir copy to index (with all other tags
           and metadata added on)
        3. Delete old Maildir copy from index

      OfflineIMAP will then DTRT as long as SEARCH is supported,
      I think.

    - Hook in this behavior on the right places (label change,
      after message add)

    - Implement a script to go through old mail and move it.

Edward

---

From: Alvaro Herrera <[email-masked]>
Date: Thu, 23 Aug 2012 10:52:21 -0400

-- 
Álvaro Herrera <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: "Edward Z. Yang" <[email-masked]>
Date: Thu, 23 Aug 2012 14:20:18 -0400

With pleasure, I report that Sup is actually doing this!  So solving
the "hooks run too many times" is as easy as this:

commit ec3afe7ac01741bc67f68009193d2d2e73f05529
Author: Edward Z. Yang <[email-masked]>
Date:   Thu Aug 23 14:19:24 2012 -0400

    Don't run before-add-message hook for messages already in index.

diff --git a/lib/sup/poll.rb b/lib/sup/poll.rb
index 1b64098..7d5aaa2 100644
--- a/lib/sup/poll.rb
+++ b/lib/sup/poll.rb
@@ -184,7 +184,7 @@ EOS
           m.labels.each { |l| LabelManager << l }
           m.labels = old_m.labels + (m.labels - [:unread, :inbox]) if old_m
           m.locations = old_m.locations + m.locations if old_m
-          HookManager.run "before-add-message", :message => m
+          HookManager.run "before-add-message", :message => m if not old_m
           yield :add, m, old_m, args[:progress] if block_given?
           Index.sync_message m, true

---

From: "Edward Z. Yang" <[email-masked]>
Date: Wed, 22 Aug 2012 16:09:40 -0400

def raw_header id

 

-      new_loc
+      [new_source, new_loc]
     end
   end
 end
 
+  def wrong_source?
 
   def sync_back labels

 
   def direct_participants

---

From: "Edward Z. Yang" <[email-masked]>
Date: Wed, 22 Aug 2012 01:45:31 -0400

def sync_back id, labels
+    new_source = @id
 
   def raw_header id

 

-      new_loc
+      [new_source, new_loc]
     end
   end
 end
 
   def sync_back labels
 
1.7.11.3