* [sup-devel] Maildir synchronizing other labels
@ 2012-08-21 14:35 Edward Z. Yang
2012-08-22 5:45 ` [sup-devel] [PATCH] Implement moving message between Maildir sources based on label Edward Z. Yang
0 siblings, 1 reply; 8+ messages in thread
From: Edward Z. Yang @ 2012-08-21 14:35 UTC (permalink / raw)
To: Damien Leone; +Cc: sup-devel
I think it would be nice if we at least support the :inbox label,
so that we can keep our inboxes tidy and our OfflineIMAP sync
times down.
Proposed implementation strategy:
- Define a label/source mapping, as well as a default source for
unrecognized labels and an ordering on labels if there is
a conflict. (Each Maildir folder is its own source.)
- Implement moving messages between Maildir sources in Sup.
Probably the easiest way is to do this:
1. Copy message form old Maildir to new Maildir
2. Add new Maildir copy to index (with all other tags
and metadata added on)
3. Delete old Maildir copy from index
OfflineIMAP will then DTRT as long as SEARCH is supported,
I think.
- Hook in this behavior on the right places (label change,
after message add)
- Implement a script to go through old mail and move it.
Edward
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 8+ messages in thread* [sup-devel] [PATCH] Implement moving message between Maildir sources based on label. 2012-08-21 14:35 [sup-devel] Maildir synchronizing other labels Edward Z. Yang @ 2012-08-22 5:45 ` Edward Z. Yang 2012-08-22 5:54 ` Edward Z. Yang 0 siblings, 1 reply; 8+ messages in thread From: Edward Z. Yang @ 2012-08-22 5:45 UTC (permalink / raw) To: sup-devel, damien.leone From: "Edward Z. Yang" <ezyang@mit.edu> Signed-off-by: Edward Z. Yang <ezyang@mit.edu> --- lib/sup/maildir.rb | 28 ++++++++++++++++++++++------ lib/sup/message.rb | 9 +++++++-- 2 files changed, 29 insertions(+), 8 deletions(-) diff --git a/lib/sup/maildir.rb b/lib/sup/maildir.rb index 95305c2..ff8da23 100644 --- a/lib/sup/maildir.rb +++ b/lib/sup/maildir.rb @@ -77,8 +77,17 @@ class Maildir < Source end def sync_back id, labels + new_source = @id + $config[:maildir_labels].each do |k,v| + v.each do |lbl,i| + if lbl.nil? or labels.member? lbl + new_source = i + break + end + end if v.any? { |lbl,i| i == @id } + end if $config[:maildir_labels] flags = maildir_reconcile_flags id, labels - maildir_mark_file id, flags + maildir_move_file id, new_source, flags end def raw_header id @@ -221,24 +230,31 @@ private new_flags.to_a.sort.join end - def maildir_mark_file orig_path, flags + def maildir_move_file orig_path, new_source_id, flags @mutex.synchronize do new_base = (flags.include?("S")) ? "cur" : "new" md_base, md_ver, md_flags = maildir_data orig_path - return if md_flags == flags + return if md_flags == flags and new_source_id == @id + + new_source = SourceManager[new_source_id] new_loc = File.join new_base, "#{md_base}:#{md_ver},#{flags}" orig_path = File.join @dir, orig_path - new_path = File.join @dir, new_loc + new_path = File.join new_source.file_path, new_loc tmp_path = File.join @dir, "tmp", "#{md_base}:#{md_ver},#{flags}" File.link orig_path, tmp_path File.unlink orig_path - File.link tmp_path, new_path + begin + File.link tmp_path, new_path + rescue SystemCallError + File.unlink new_path # XXX kinda unsafe eh + File.link tmp_path, new_path + end File.unlink tmp_path - new_loc + [new_source, new_loc] end end end diff --git a/lib/sup/message.rb b/lib/sup/message.rb index 3eeea66..d6016df 100644 --- a/lib/sup/message.rb +++ b/lib/sup/message.rb @@ -726,8 +726,13 @@ class Location end def sync_back labels - new_info = source.sync_back(@info, labels) if source.respond_to? :sync_back - @info = new_info if new_info + pair = source.sync_back(@info, labels) if source.respond_to? :sync_back + if pair + new_source, new_info = pair + @source = new_source if new_source + @info = new_info if new_info + end + pair end ## much faster than raw_message -- 1.7.11.3 _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [sup-devel] [PATCH] Implement moving message between Maildir sources based on label. 2012-08-22 5:45 ` [sup-devel] [PATCH] Implement moving message between Maildir sources based on label Edward Z. Yang @ 2012-08-22 5:54 ` Edward Z. Yang 2012-08-22 20:09 ` Edward Z. Yang 0 siblings, 1 reply; 8+ messages in thread From: Edward Z. Yang @ 2012-08-22 5:54 UTC (permalink / raw) To: sup-devel, damien.leone Usage instructions: In config.yaml, you need to add a new option :maildir_labels: :maildir_labels: :stanford: [[:inbox, 4], [null, 6]] Maildir labels is a dictionary of "accounts" to lists of precedences. Read it as follows: For messages in source 4 or source 6 (consult sources.yaml), if the message has the :inbox tag, move it to source 4, otherwise move it to source 6. So in this case, 6 would be some sort of Archive folder, and 4 would be INBOX. If you want "export-only" folders, just tack them on after the null entry; the labels are checked *in order*. Multiple accounts are supported, but these should be disjoint sets of sources. This will automatically start working for any new mail you change the labels of. In order to apply this to old mail, you need to run sup-sync-back-maildir. OfflineIMAP will shit its pants [1] if you move too much mail, so I recommend holding on until I implement the companion patch for OfflineIMAP if you have a lot of mail. Edward [1] Namely, it will reupload every single article of mail, and if you get unlucky and "Archive" is sorted before "INBOX", it will probably run you out of quota too. If you arrange to delete everything from INBOX first, and then sync Archive, it will probably just spend a lot of time uploading. Excerpts from Edward Z. Yang's message of Wed Aug 22 01:45:31 -0400 2012: > From: "Edward Z. Yang" <ezyang@mit.edu> > > Signed-off-by: Edward Z. Yang <ezyang@mit.edu> > --- > lib/sup/maildir.rb | 28 ++++++++++++++++++++++------ > lib/sup/message.rb | 9 +++++++-- > 2 files changed, 29 insertions(+), 8 deletions(-) > > diff --git a/lib/sup/maildir.rb b/lib/sup/maildir.rb > index 95305c2..ff8da23 100644 > --- a/lib/sup/maildir.rb > +++ b/lib/sup/maildir.rb > @@ -77,8 +77,17 @@ class Maildir < Source > end > > def sync_back id, labels > + new_source = @id > + $config[:maildir_labels].each do |k,v| > + v.each do |lbl,i| > + if lbl.nil? or labels.member? lbl > + new_source = i > + break > + end > + end if v.any? { |lbl,i| i == @id } > + end if $config[:maildir_labels] > flags = maildir_reconcile_flags id, labels > - maildir_mark_file id, flags > + maildir_move_file id, new_source, flags > end > > def raw_header id > @@ -221,24 +230,31 @@ private > new_flags.to_a.sort.join > end > > - def maildir_mark_file orig_path, flags > + def maildir_move_file orig_path, new_source_id, flags > @mutex.synchronize do > new_base = (flags.include?("S")) ? "cur" : "new" > md_base, md_ver, md_flags = maildir_data orig_path > > - return if md_flags == flags > + return if md_flags == flags and new_source_id == @id > + > + new_source = SourceManager[new_source_id] > > new_loc = File.join new_base, "#{md_base}:#{md_ver},#{flags}" > orig_path = File.join @dir, orig_path > - new_path = File.join @dir, new_loc > + new_path = File.join new_source.file_path, new_loc > tmp_path = File.join @dir, "tmp", "#{md_base}:#{md_ver},#{flags}" > > File.link orig_path, tmp_path > File.unlink orig_path > - File.link tmp_path, new_path > + begin > + File.link tmp_path, new_path > + rescue SystemCallError > + File.unlink new_path # XXX kinda unsafe eh > + File.link tmp_path, new_path > + end > File.unlink tmp_path > > - new_loc > + [new_source, new_loc] > end > end > end > diff --git a/lib/sup/message.rb b/lib/sup/message.rb > index 3eeea66..d6016df 100644 > --- a/lib/sup/message.rb > +++ b/lib/sup/message.rb > @@ -726,8 +726,13 @@ class Location > end > > def sync_back labels > - new_info = source.sync_back(@info, labels) if source.respond_to? :sync_back > - @info = new_info if new_info > + pair = source.sync_back(@info, labels) if source.respond_to? :sync_back > + if pair > + new_source, new_info = pair > + @source = new_source if new_source > + @info = new_info if new_info > + end > + pair > end > > ## much faster than raw_message _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [sup-devel] [PATCH] Implement moving message between Maildir sources based on label. 2012-08-22 5:54 ` Edward Z. Yang @ 2012-08-22 20:09 ` Edward Z. Yang 2012-08-23 14:40 ` Edward Z. Yang 0 siblings, 1 reply; 8+ messages in thread From: Edward Z. Yang @ 2012-08-22 20:09 UTC (permalink / raw) To: sup-devel, damien.leone Here is an improved version that handles the thread/message distinction better; previously, if you archived something, it's possible only some of the messages got moved if you hadn't run sup-sync-back-maildir. commit a25c345c15c8859a041cc9dc13090d7b8feb3d58 Author: Edward Z. Yang <ezyang@mit.edu> Date: Tue Aug 21 23:40:03 2012 -0400 Implement moving message between Maildir sources based on label. Signed-off-by: Edward Z. Yang <ezyang@mit.edu> diff --git a/lib/sup/maildir.rb b/lib/sup/maildir.rb index 95305c2..a7bf567 100644 --- a/lib/sup/maildir.rb +++ b/lib/sup/maildir.rb @@ -76,9 +76,23 @@ class Maildir < Source with_file_for(id) { |f| RMail::Parser.read f } end + def wrong_source? labels + new_source = nil + $config[:maildir_labels].each do |k,v| + v.each do |lbl,i| + if lbl.nil? or labels.member? lbl + new_source = i + break + end + end if v.any? { |lbl,i| i == @id } + end if $config[:maildir_labels] + new_source + end + def sync_back id, labels + new_source = wrong_source? labels || @id flags = maildir_reconcile_flags id, labels - maildir_mark_file id, flags + maildir_move_file id, new_source, flags end def raw_header id @@ -221,24 +235,31 @@ private new_flags.to_a.sort.join end - def maildir_mark_file orig_path, flags + def maildir_move_file orig_path, new_source_id, flags @mutex.synchronize do new_base = (flags.include?("S")) ? "cur" : "new" md_base, md_ver, md_flags = maildir_data orig_path - return if md_flags == flags + return if md_flags == flags and new_source_id == @id + + new_source = SourceManager[new_source_id] new_loc = File.join new_base, "#{md_base}:#{md_ver},#{flags}" orig_path = File.join @dir, orig_path - new_path = File.join @dir, new_loc + new_path = File.join new_source.file_path, new_loc tmp_path = File.join @dir, "tmp", "#{md_base}:#{md_ver},#{flags}" File.link orig_path, tmp_path File.unlink orig_path - File.link tmp_path, new_path + begin + File.link tmp_path, new_path + rescue SystemCallError + File.unlink new_path # XXX kinda unsafe eh + File.link tmp_path, new_path + end File.unlink tmp_path - new_loc + [new_source, new_loc] end end end diff --git a/lib/sup/message.rb b/lib/sup/message.rb index 3eeea66..4139f3e 100644 --- a/lib/sup/message.rb +++ b/lib/sup/message.rb @@ -285,6 +285,10 @@ EOS location.each_raw_message_line &b end + def wrong_source? + $config[:sync_back_to_maildir] and @locations.any? { |l| l.valid? and l.source.is_a? Maildir and l.wrong_source? @labels} + end + def sync_back if @locations.map { |l| l.sync_back @labels if l.valid? and $config[:sync_back_to_maildir] and l.source.is_a? Maildir @@ -726,8 +730,17 @@ class Location end def sync_back labels - new_info = source.sync_back(@info, labels) if source.respond_to? :sync_back - @info = new_info if new_info + pair = source.sync_back(@info, labels) if source.respond_to? :sync_back + if pair + new_source, new_info = pair + @source = new_source if new_source + @info = new_info if new_info + end + pair + end + + def wrong_source? labels + source.wrong_source? labels if source.respond_to? :wrong_source end ## much faster than raw_message diff --git a/lib/sup/thread.rb b/lib/sup/thread.rb index b08bae2..c1e4b40 100644 --- a/lib/sup/thread.rb +++ b/lib/sup/thread.rb @@ -113,7 +113,7 @@ class Thread def set_labels l; each { |m, *o| m && m.labels = l }; end def has_label? t; any? { |m, *o| m && m.has_label?(t) }; end - def each_dirty_message; each { |m, *o| m && m.dirty? && yield(m) }; end + def each_dirty_message; each { |m, *o| m && (m.dirty? || m.wrong_source?) && yield(m) }; end def direct_participants map { |m, *o| [m.from] + m.to if m }.flatten.compact.uniq _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [sup-devel] [PATCH] Implement moving message between Maildir sources based on label. 2012-08-22 20:09 ` Edward Z. Yang @ 2012-08-23 14:40 ` Edward Z. Yang 2012-08-23 14:52 ` Alvaro Herrera 0 siblings, 1 reply; 8+ messages in thread From: Edward Z. Yang @ 2012-08-23 14:40 UTC (permalink / raw) To: sup-devel, damien.leone It looks like the current patchset will *destroy* all labels you have on messages, so be careful! (This was masked for me since most of my labels are from before-add-message.rb, and that hook gets reapplied) Edward Excerpts from Edward Z. Yang's message of Wed Aug 22 16:09:40 -0400 2012: > Here is an improved version that handles the thread/message distinction > better; previously, if you archived something, it's possible only some of > the messages got moved if you hadn't run sup-sync-back-maildir. > > commit a25c345c15c8859a041cc9dc13090d7b8feb3d58 > Author: Edward Z. Yang <ezyang@mit.edu> > Date: Tue Aug 21 23:40:03 2012 -0400 > > Implement moving message between Maildir sources based on label. > > Signed-off-by: Edward Z. Yang <ezyang@mit.edu> > > diff --git a/lib/sup/maildir.rb b/lib/sup/maildir.rb > index 95305c2..a7bf567 100644 > --- a/lib/sup/maildir.rb > +++ b/lib/sup/maildir.rb > @@ -76,9 +76,23 @@ class Maildir < Source > with_file_for(id) { |f| RMail::Parser.read f } > end > > + def wrong_source? labels > + new_source = nil > + $config[:maildir_labels].each do |k,v| > + v.each do |lbl,i| > + if lbl.nil? or labels.member? lbl > + new_source = i > + break > + end > + end if v.any? { |lbl,i| i == @id } > + end if $config[:maildir_labels] > + new_source > + end > + > def sync_back id, labels > + new_source = wrong_source? labels || @id > flags = maildir_reconcile_flags id, labels > - maildir_mark_file id, flags > + maildir_move_file id, new_source, flags > end > > def raw_header id > @@ -221,24 +235,31 @@ private > new_flags.to_a.sort.join > end > > - def maildir_mark_file orig_path, flags > + def maildir_move_file orig_path, new_source_id, flags > @mutex.synchronize do > new_base = (flags.include?("S")) ? "cur" : "new" > md_base, md_ver, md_flags = maildir_data orig_path > > - return if md_flags == flags > + return if md_flags == flags and new_source_id == @id > + > + new_source = SourceManager[new_source_id] > > new_loc = File.join new_base, "#{md_base}:#{md_ver},#{flags}" > orig_path = File.join @dir, orig_path > - new_path = File.join @dir, new_loc > + new_path = File.join new_source.file_path, new_loc > tmp_path = File.join @dir, "tmp", "#{md_base}:#{md_ver},#{flags}" > > File.link orig_path, tmp_path > File.unlink orig_path > - File.link tmp_path, new_path > + begin > + File.link tmp_path, new_path > + rescue SystemCallError > + File.unlink new_path # XXX kinda unsafe eh > + File.link tmp_path, new_path > + end > File.unlink tmp_path > > - new_loc > + [new_source, new_loc] > end > end > end > diff --git a/lib/sup/message.rb b/lib/sup/message.rb > index 3eeea66..4139f3e 100644 > --- a/lib/sup/message.rb > +++ b/lib/sup/message.rb > @@ -285,6 +285,10 @@ EOS > location.each_raw_message_line &b > end > > + def wrong_source? > + $config[:sync_back_to_maildir] and @locations.any? { |l| l.valid? and l.source.is_a? Maildir and l.wrong_source? @labels} > + end > + > def sync_back > if @locations.map { |l| > l.sync_back @labels if l.valid? and $config[:sync_back_to_maildir] and l.source.is_a? Maildir > @@ -726,8 +730,17 @@ class Location > end > > def sync_back labels > - new_info = source.sync_back(@info, labels) if source.respond_to? :sync_back > - @info = new_info if new_info > + pair = source.sync_back(@info, labels) if source.respond_to? :sync_back > + if pair > + new_source, new_info = pair > + @source = new_source if new_source > + @info = new_info if new_info > + end > + pair > + end > + > + def wrong_source? labels > + source.wrong_source? labels if source.respond_to? :wrong_source > end > > ## much faster than raw_message > diff --git a/lib/sup/thread.rb b/lib/sup/thread.rb > index b08bae2..c1e4b40 100644 > --- a/lib/sup/thread.rb > +++ b/lib/sup/thread.rb > @@ -113,7 +113,7 @@ class Thread > > def set_labels l; each { |m, *o| m && m.labels = l }; end > def has_label? t; any? { |m, *o| m && m.has_label?(t) }; end > - def each_dirty_message; each { |m, *o| m && m.dirty? && yield(m) }; end > + def each_dirty_message; each { |m, *o| m && (m.dirty? || m.wrong_source?) && yield(m) }; end > > def direct_participants > map { |m, *o| [m.from] + m.to if m }.flatten.compact.uniq _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [sup-devel] [PATCH] Implement moving message between Maildir sources based on label. 2012-08-23 14:40 ` Edward Z. Yang @ 2012-08-23 14:52 ` Alvaro Herrera 2012-08-23 15:28 ` Edward Z. Yang 2012-08-23 18:20 ` Edward Z. Yang 0 siblings, 2 replies; 8+ messages in thread From: Alvaro Herrera @ 2012-08-23 14:52 UTC (permalink / raw) To: Sup developer discussion Excerpts from Edward Z. Yang's message of jue ago 23 10:40:22 -0400 2012: > It looks like the current patchset will *destroy* all labels you have > on messages, so be careful! (This was masked for me since most of my > labels are from before-add-message.rb, and that hook gets reapplied) Thanks for the notice. Most of my labels come from that hook too, but a few do not, so I could easily lose some! I was planning on trying out your patch, but with this caveat I'm likely to refrain. Are you planning on submitting a fixed version? -- Álvaro Herrera <alvherre@alvh.no-ip.org> _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [sup-devel] [PATCH] Implement moving message between Maildir sources based on label. 2012-08-23 14:52 ` Alvaro Herrera @ 2012-08-23 15:28 ` Edward Z. Yang 2012-08-23 18:20 ` Edward Z. Yang 1 sibling, 0 replies; 8+ messages in thread From: Edward Z. Yang @ 2012-08-23 15:28 UTC (permalink / raw) To: Alvaro Herrera; +Cc: Sup developer discussion Yes. The current plan is to snoop the message ID of newly added messages in maildirs, and do moved message detection based on that. If you have a better idea I'm all ears. (Another possibility is to add sha1sum to the index and compare that.) Also note that it won't destroy the labels *unless you run OfflineIMAP*. The trouble is OfflineIMAP reuploads your message and then reassigns the UID based on the IMAP server, but doesn't communicate this back to Sup. Edward Excerpts from Alvaro Herrera's message of Thu Aug 23 10:52:21 -0400 2012: > Excerpts from Edward Z. Yang's message of jue ago 23 10:40:22 -0400 2012: > > It looks like the current patchset will *destroy* all labels you have > > on messages, so be careful! (This was masked for me since most of my > > labels are from before-add-message.rb, and that hook gets reapplied) > > Thanks for the notice. Most of my labels come from that hook too, but a > few do not, so I could easily lose some! I was planning on trying out > your patch, but with this caveat I'm likely to refrain. Are you > planning on submitting a fixed version? > _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [sup-devel] [PATCH] Implement moving message between Maildir sources based on label. 2012-08-23 14:52 ` Alvaro Herrera 2012-08-23 15:28 ` Edward Z. Yang @ 2012-08-23 18:20 ` Edward Z. Yang 1 sibling, 0 replies; 8+ messages in thread From: Edward Z. Yang @ 2012-08-23 18:20 UTC (permalink / raw) To: Alvaro Herrera; +Cc: Sup developer discussion With pleasure, I report that Sup is actually doing this! So solving the "hooks run too many times" is as easy as this: commit ec3afe7ac01741bc67f68009193d2d2e73f05529 Author: Edward Z. Yang <ezyang@mit.edu> Date: Thu Aug 23 14:19:24 2012 -0400 Don't run before-add-message hook for messages already in index. Signed-off-by: Edward Z. Yang <ezyang@mit.edu> diff --git a/lib/sup/poll.rb b/lib/sup/poll.rb index 1b64098..7d5aaa2 100644 --- a/lib/sup/poll.rb +++ b/lib/sup/poll.rb @@ -184,7 +184,7 @@ EOS m.labels.each { |l| LabelManager << l } m.labels = old_m.labels + (m.labels - [:unread, :inbox]) if old_m m.locations = old_m.locations + m.locations if old_m - HookManager.run "before-add-message", :message => m + HookManager.run "before-add-message", :message => m if not old_m yield :add, m, old_m, args[:progress] if block_given? Index.sync_message m, true Excerpts from Alvaro Herrera's message of Thu Aug 23 10:52:21 -0400 2012: > Excerpts from Edward Z. Yang's message of jue ago 23 10:40:22 -0400 2012: > > It looks like the current patchset will *destroy* all labels you have > > on messages, so be careful! (This was masked for me since most of my > > labels are from before-add-message.rb, and that hook gets reapplied) > > Thanks for the notice. Most of my labels come from that hook too, but a > few do not, so I could easily lose some! I was planning on trying out > your patch, but with this caveat I'm likely to refrain. Are you > planning on submitting a fixed version? > _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2012-08-23 18:52 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-08-21 14:35 [sup-devel] Maildir synchronizing other labels Edward Z. Yang 2012-08-22 5:45 ` [sup-devel] [PATCH] Implement moving message between Maildir sources based on label Edward Z. Yang 2012-08-22 5:54 ` Edward Z. Yang 2012-08-22 20:09 ` Edward Z. Yang 2012-08-23 14:40 ` Edward Z. Yang 2012-08-23 14:52 ` Alvaro Herrera 2012-08-23 15:28 ` Edward Z. Yang 2012-08-23 18:20 ` Edward Z. Yang
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox