From ezyang@MIT.EDU Mon Sep 3 04:59:31 2012 From: ezyang@MIT.EDU (Edward Z. Yang) Date: Mon, 3 Sep 2012 00:59:31 -0400 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT) In-Reply-To: <1345564795-sup-3898@alvh.no-ip.org> References: <1345564795-sup-3898@alvh.no-ip.org> Message-ID: <1346648371-12305-1-git-send-email-ezyang@mit.edu> From: "Edward Z. Yang" Signed-off-by: Edward Z. Yang --- lib/sup/maildir.rb | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++-- lib/sup/poll.rb | 33 ++++++++++++++++++++++++++------- lib/sup/source.rb | 4 ++++ 3 files changed, 80 insertions(+), 9 deletions(-) diff --git a/lib/sup/maildir.rb b/lib/sup/maildir.rb index 2a91f05..743156d 100644 --- a/lib/sup/maildir.rb +++ b/lib/sup/maildir.rb @@ -1,5 +1,6 @@ require 'uri' require 'set' +require 'inotify' module Redwood @@ -184,6 +185,45 @@ class Maildir < Source nil end + def continuous_poll poll_mutex + i = Inotify.new + watches = {} + @ctimes.each do |d,prev_ctime| + subdir = File.join @dir, d + wd = i.add_watch(subdir, Inotify::CREATE | Inotify::DELETE | Inotify::MOVE) + watches[wd] = d + end + i.each_event do |ev| + poll_mutex.synchronize do + @mutex.synchronize do + begin + ::Thread.current[@dir] = true + id = File.join watches[ev.wd], ev.name + # check if inotify is stale + # since we have @mutex, there is no race (except for + # an external program fucking us over) + next unless File.exists? File.join(@dir, id) + x = Enumerator.new(Index.instance, :each_source_info, self.id, "#{id}").to_a + if ev.mask & Inotify::CREATE or ev.mask & Inotify::MOVE_TO + next unless x.empty? + yield :add, + :info => id, + :labels => @labels + maildir_labels(id) + [:inbox], + :progress => 0 + elsif ev.mask & Inotify::DELETE or ev.mask & Inotify::MOVE_FROM + next unless !x.empty? + yield :delete, + :info => id, + :progress => 0 + end + ensure + ::Thread.current[@dir] = nil + end + end + end + end + end + def labels? id maildir_labels id end @@ -248,7 +288,16 @@ private end def maildir_move_file orig_path, new_source_id, flags - @mutex.synchronize do + if ::Thread.current[@dir] + _maildir_move_file orig_path, new_source_id, flags + else + @mutex.synchronize do + _maildir_move_file orig_path, new_source_id, flags + end + end + end + + def _maildir_move_file orig_path, new_source_id, flags new_base = (flags.include?("S")) ? "cur" : "new" md_base, md_ver, md_flags = maildir_data orig_path @@ -292,7 +341,6 @@ private end [new_source, new_loc] - end end end diff --git a/lib/sup/poll.rb b/lib/sup/poll.rb index dbd351f..51e0afa 100644 --- a/lib/sup/poll.rb +++ b/lib/sup/poll.rb @@ -94,11 +94,27 @@ EOS poll if @last_poll.nil? || (Time.now - @last_poll) >= @delay end end + # XXX dup dup + SourceManager.usual_sources.each do |source| + Redwood::reporting_thread("inotify poll for #{source}") do + source.continuous_poll @mutex do |sym, args| + poll_handler source, sym, args + end + end + end + SourceManager.unusual_sources.each do |source| + Redwood::reporting_thread("inotify poll for #{source}") do + source.continuous_poll @mutex do |sym, args| + poll_handler source, sym, args + end + end + end end def stop @thread.kill if @thread @thread = nil + # handle inotify polls end def do_poll @@ -172,7 +188,16 @@ EOS ## from the index after being yielded. def poll_from source, opts={} begin - source.poll do |sym, args| + source.poll do |sym,args| + poll_handler source, sym, args + end + source.go_idle + rescue SourceError => e + warn "problem getting messages from #{source}: #{e.message}" + end + end + + def poll_handler source, sym, args case sym when :add m = Message.build_from_source source, args[:info] @@ -224,12 +249,6 @@ EOS UpdateManager.relay self, :updated, m end end - end - - source.go_idle - rescue SourceError => e - warn "problem getting messages from #{source}: #{e.message}" - end end def handle_idle_update sender, idle_since; @should_clear_running_totals = false; end diff --git a/lib/sup/source.rb b/lib/sup/source.rb index 06b6e6b..073a10a 100644 --- a/lib/sup/source.rb +++ b/lib/sup/source.rb @@ -102,6 +102,10 @@ class Source unimplemented end + ## Like poll, but never returns (it is continuous, and uses something + ## like inotify. Will always be run in another thread.) + def continuous_poll poll_mutex; [] end + def valid? info true end -- 1.7.11.3 From ezyang@MIT.EDU Mon Sep 3 05:00:59 2012 From: ezyang@MIT.EDU (Edward Z. Yang) Date: Mon, 03 Sep 2012 01:00:59 -0400 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT) In-Reply-To: <1346648371-12305-1-git-send-email-ezyang@mit.edu> References: <1345564795-sup-3898@alvh.no-ip.org> <1346648371-12305-1-git-send-email-ezyang@mit.edu> Message-ID: <1346648395-sup-4546@javelin> The locking is a downright crime (where's the STM when you need it), and it's still racy, but it should work OK. Excerpts from Edward Z. Yang's message of Mon Sep 03 00:59:31 -0400 2012: > From: "Edward Z. Yang" > > Signed-off-by: Edward Z. Yang > --- > lib/sup/maildir.rb | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++-- > lib/sup/poll.rb | 33 ++++++++++++++++++++++++++------- > lib/sup/source.rb | 4 ++++ > 3 files changed, 80 insertions(+), 9 deletions(-) > > diff --git a/lib/sup/maildir.rb b/lib/sup/maildir.rb > index 2a91f05..743156d 100644 > --- a/lib/sup/maildir.rb > +++ b/lib/sup/maildir.rb > @@ -1,5 +1,6 @@ > require 'uri' > require 'set' > +require 'inotify' > > module Redwood > > @@ -184,6 +185,45 @@ class Maildir < Source > nil > end > > + def continuous_poll poll_mutex > + i = Inotify.new > + watches = {} > + @ctimes.each do |d,prev_ctime| > + subdir = File.join @dir, d > + wd = i.add_watch(subdir, Inotify::CREATE | Inotify::DELETE | Inotify::MOVE) > + watches[wd] = d > + end > + i.each_event do |ev| > + poll_mutex.synchronize do > + @mutex.synchronize do > + begin > + ::Thread.current[@dir] = true > + id = File.join watches[ev.wd], ev.name > + # check if inotify is stale > + # since we have @mutex, there is no race (except for > + # an external program fucking us over) > + next unless File.exists? File.join(@dir, id) > + x = Enumerator.new(Index.instance, :each_source_info, self.id, "#{id}").to_a > + if ev.mask & Inotify::CREATE or ev.mask & Inotify::MOVE_TO > + next unless x.empty? > + yield :add, > + :info => id, > + :labels => @labels + maildir_labels(id) + [:inbox], > + :progress => 0 > + elsif ev.mask & Inotify::DELETE or ev.mask & Inotify::MOVE_FROM > + next unless !x.empty? > + yield :delete, > + :info => id, > + :progress => 0 > + end > + ensure > + ::Thread.current[@dir] = nil > + end > + end > + end > + end > + end > + > def labels? id > maildir_labels id > end > @@ -248,7 +288,16 @@ private > end > > def maildir_move_file orig_path, new_source_id, flags > - @mutex.synchronize do > + if ::Thread.current[@dir] > + _maildir_move_file orig_path, new_source_id, flags > + else > + @mutex.synchronize do > + _maildir_move_file orig_path, new_source_id, flags > + end > + end > + end > + > + def _maildir_move_file orig_path, new_source_id, flags > new_base = (flags.include?("S")) ? "cur" : "new" > md_base, md_ver, md_flags = maildir_data orig_path > > @@ -292,7 +341,6 @@ private > end > > [new_source, new_loc] > - end > end > end > > diff --git a/lib/sup/poll.rb b/lib/sup/poll.rb > index dbd351f..51e0afa 100644 > --- a/lib/sup/poll.rb > +++ b/lib/sup/poll.rb > @@ -94,11 +94,27 @@ EOS > poll if @last_poll.nil? || (Time.now - @last_poll) >= @delay > end > end > + # XXX dup dup > + SourceManager.usual_sources.each do |source| > + Redwood::reporting_thread("inotify poll for #{source}") do > + source.continuous_poll @mutex do |sym, args| > + poll_handler source, sym, args > + end > + end > + end > + SourceManager.unusual_sources.each do |source| > + Redwood::reporting_thread("inotify poll for #{source}") do > + source.continuous_poll @mutex do |sym, args| > + poll_handler source, sym, args > + end > + end > + end > end > > def stop > @thread.kill if @thread > @thread = nil > + # handle inotify polls > end > > def do_poll > @@ -172,7 +188,16 @@ EOS > ## from the index after being yielded. > def poll_from source, opts={} > begin > - source.poll do |sym, args| > + source.poll do |sym,args| > + poll_handler source, sym, args > + end > + source.go_idle > + rescue SourceError => e > + warn "problem getting messages from #{source}: #{e.message}" > + end > + end > + > + def poll_handler source, sym, args > case sym > when :add > m = Message.build_from_source source, args[:info] > @@ -224,12 +249,6 @@ EOS > UpdateManager.relay self, :updated, m > end > end > - end > - > - source.go_idle > - rescue SourceError => e > - warn "problem getting messages from #{source}: #{e.message}" > - end > end > > def handle_idle_update sender, idle_since; @should_clear_running_totals = false; end > diff --git a/lib/sup/source.rb b/lib/sup/source.rb > index 06b6e6b..073a10a 100644 > --- a/lib/sup/source.rb > +++ b/lib/sup/source.rb > @@ -102,6 +102,10 @@ class Source > unimplemented > end > > + ## Like poll, but never returns (it is continuous, and uses something > + ## like inotify. Will always be run in another thread.) > + def continuous_poll poll_mutex; [] end > + > def valid? info > true > end From alvherre@alvh.no-ip.org Mon Sep 3 16:02:26 2012 From: alvherre@alvh.no-ip.org (Alvaro Herrera) Date: Mon, 03 Sep 2012 13:02:26 -0300 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT) In-Reply-To: <1346648395-sup-4546@javelin> References: <1345564795-sup-3898@alvh.no-ip.org> <1346648371-12305-1-git-send-email-ezyang@mit.edu> <1346648395-sup-4546@javelin> Message-ID: <1346688000-sup-4643@alvh.no-ip.org> Excerpts from Edward Z. Yang's message of lun sep 03 02:00:59 -0300 2012: > The locking is a downright crime (where's the STM when you need it), > and it's still racy, but it should work OK. Hm. I tried this but ran into trouble: I currently run branch "next", and your patch doesn't apply there; so I tried your ~ezyang fork and branch maildir-sync there, but I find that when in that branch (with or without this patch), Sup seems to eat 100% of a CPU core doing clock_gettime() and select() continuously. Not sure what's happening. I assume you don't see that behavior. -- ?lvaro Herrera From alvherre@alvh.no-ip.org Mon Sep 3 16:07:28 2012 From: alvherre@alvh.no-ip.org (Alvaro Herrera) Date: Mon, 03 Sep 2012 13:07:28 -0300 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT) In-Reply-To: <1346688000-sup-4643@alvh.no-ip.org> References: <1345564795-sup-3898@alvh.no-ip.org> <1346648371-12305-1-git-send-email-ezyang@mit.edu> <1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org> Message-ID: <1346688387-sup-5906@alvh.no-ip.org> Excerpts from Alvaro Herrera's message of lun sep 03 13:02:26 -0300 2012: > Excerpts from Edward Z. Yang's message of lun sep 03 02:00:59 -0300 2012: > > The locking is a downright crime (where's the STM when you need it), > > and it's still racy, but it should work OK. > > Hm. I tried this but ran into trouble: I currently run branch "next", > and your patch doesn't apply there; so I tried your ~ezyang fork and > branch maildir-sync there, but I find that when in that branch (with or > without this patch), Sup seems to eat 100% of a CPU core doing > clock_gettime() and select() continuously. Not sure what's happening. > I assume you don't see that behavior. Oh, I see what's going on: it's deleting all my deleted email! :-) Nevermind. I'll just wait for it to finish before trying out your patch. -- ?lvaro Herrera From ezyang@MIT.EDU Mon Sep 3 16:10:13 2012 From: ezyang@MIT.EDU (ezyang) Date: Mon, 03 Sep 2012 12:10:13 -0400 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT) In-Reply-To: <1346688387-sup-5906@alvh.no-ip.org> References: <1345564795-sup-3898@alvh.no-ip.org> <1346648371-12305-1-git-send-email-ezyang@mit.edu> <1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org> <1346688387-sup-5906@alvh.no-ip.org> Message-ID: <20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu> Quoting Alvaro Herrera : > Excerpts from Alvaro Herrera's message of lun sep 03 13:02:26 -0300 2012: >> Excerpts from Edward Z. Yang's message of lun sep 03 02:00:59 -0300 2012: >> > The locking is a downright crime (where's the STM when you need it), >> > and it's still racy, but it should work OK. >> >> Hm. I tried this but ran into trouble: I currently run branch "next", >> and your patch doesn't apply there; so I tried your ~ezyang fork and >> branch maildir-sync there, but I find that when in that branch (with or >> without this patch), Sup seems to eat 100% of a CPU core doing >> clock_gettime() and select() continuously. Not sure what's happening. >> I assume you don't see that behavior. > > Oh, I see what's going on: it's deleting all my deleted email! :-) > Nevermind. I'll just wait for it to finish before trying out your patch. There is a bug in the patch, where it notices changes that Sup makes from inotify, and as a result undoes any flag changes you make. So there needs to be a fix here but I'm not sure what it is yet. Edward From ezyang@MIT.EDU Mon Sep 3 18:09:57 2012 From: ezyang@MIT.EDU (Edward Z. Yang) Date: Mon, 03 Sep 2012 14:09:57 -0400 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT) In-Reply-To: <20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu> References: <1345564795-sup-3898@alvh.no-ip.org> <1346648371-12305-1-git-send-email-ezyang@mit.edu> <1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org> <1346688387-sup-5906@alvh.no-ip.org> <20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu> Message-ID: <1346695742-sup-7650@javelin> OK, cracked a fix; you need this extra patch: commit f9ea07f3c4982ab46d8171fdba8eabc3af00c840 Author: Edward Z. Yang Date: Mon Sep 3 14:09:34 2012 -0400 sync_back after writing to index, not before. Signed-off-by: Edward Z. Yang diff --git a/lib/sup/index.rb b/lib/sup/index.rb index 13798d6..4b533a7 100644 --- a/lib/sup/index.rb +++ b/lib/sup/index.rb @@ -657,10 +657,6 @@ EOS end def sync_message m, overwrite - ## TODO: we should not save the message if the sync_back failed - ## since it would overwrite the location field - m.sync_back - doc = synchronize { find_doc(m.id) } existed = doc != nil doc ||= Xapian::Document.new @@ -703,6 +699,10 @@ EOS @xapian.replace_document docid, doc end + # sync_back must be after label update, so that inotify gets + # fresh data from the index + m.sync_back + m.labels.each { |l| LabelManager << l } true end From ezyang@MIT.EDU Mon Sep 3 18:29:43 2012 From: ezyang@MIT.EDU (Edward Z. Yang) Date: Mon, 03 Sep 2012 14:29:43 -0400 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT) In-Reply-To: <1346695742-sup-7650@javelin> References: <1345564795-sup-3898@alvh.no-ip.org> <1346648371-12305-1-git-send-email-ezyang@mit.edu> <1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org> <1346688387-sup-5906@alvh.no-ip.org> <20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu> <1346695742-sup-7650@javelin> Message-ID: <1346696946-sup-4602@javelin> I think I may have seen Sup peg IO R/W in the way you saw, but when I restarted Sup with debugging, it went away. Do let me know if you see it again; I think there's an infinite loop somewhere. Edward Excerpts from Edward Z. Yang's message of Mon Sep 03 14:09:57 -0400 2012: > OK, cracked a fix; you need this extra patch: > > commit f9ea07f3c4982ab46d8171fdba8eabc3af00c840 > Author: Edward Z. Yang > Date: Mon Sep 3 14:09:34 2012 -0400 > > sync_back after writing to index, not before. > > Signed-off-by: Edward Z. Yang > > diff --git a/lib/sup/index.rb b/lib/sup/index.rb > index 13798d6..4b533a7 100644 > --- a/lib/sup/index.rb > +++ b/lib/sup/index.rb > @@ -657,10 +657,6 @@ EOS > end > > def sync_message m, overwrite > - ## TODO: we should not save the message if the sync_back failed > - ## since it would overwrite the location field > - m.sync_back > - > doc = synchronize { find_doc(m.id) } > existed = doc != nil > doc ||= Xapian::Document.new > @@ -703,6 +699,10 @@ EOS > @xapian.replace_document docid, doc > end > > + # sync_back must be after label update, so that inotify gets > + # fresh data from the index > + m.sync_back > + > m.labels.each { |l| LabelManager << l } > true > end From alvherre@alvh.no-ip.org Mon Sep 3 18:42:55 2012 From: alvherre@alvh.no-ip.org (Alvaro Herrera) Date: Mon, 03 Sep 2012 15:42:55 -0300 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT) In-Reply-To: <1346696946-sup-4602@javelin> References: <1345564795-sup-3898@alvh.no-ip.org> <1346648371-12305-1-git-send-email-ezyang@mit.edu> <1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org> <1346688387-sup-5906@alvh.no-ip.org> <20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu> <1346695742-sup-7650@javelin> <1346696946-sup-4602@javelin> Message-ID: <1346697582-sup-8965@alvh.no-ip.org> Excerpts from Edward Z. Yang's message of lun sep 03 15:29:43 -0300 2012: > I think I may have seen Sup peg IO R/W in the way you saw, but when I > restarted Sup with debugging, it went away. Do let me know if you see it > again; I think there's an infinite loop somewhere. Sure. For the record, I let it run for a while and after that the poll log had a couple thousand lines saying "Deleting ". After that, so far it behaves normally. I don't know what prompted those particular messages to be deleted; I certainly have lots of messages "is:deleted" yet. Another curious thing with your branch is that the poll log gets these four lines each time a poll takes place: Message at 0 has changed its source location. Updating labels from draft,personal => draft,personal Message at 1 has changed its source location. Updating labels from draft,inbox,personal => draft,inbox,personal Message at 2 has changed its source location. Updating labels from draft,personal => draft,personal Message at 3 has changed its source location. Updating labels from draft,personal => draft,personal -- ?lvaro Herrera From ezyang@MIT.EDU Mon Sep 3 18:49:42 2012 From: ezyang@MIT.EDU (Edward Z. Yang) Date: Mon, 03 Sep 2012 14:49:42 -0400 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT) In-Reply-To: <1346697582-sup-8965@alvh.no-ip.org> References: <1345564795-sup-3898@alvh.no-ip.org> <1346648371-12305-1-git-send-email-ezyang@mit.edu> <1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org> <1346688387-sup-5906@alvh.no-ip.org> <20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu> <1346695742-sup-7650@javelin> <1346696946-sup-4602@javelin> <1346697582-sup-8965@alvh.no-ip.org> Message-ID: <1346697811-sup-463@javelin> Excerpts from Alvaro Herrera's message of Mon Sep 03 14:42:55 -0400 2012: > Message at 0 has changed its source location. Updating labels from draft,personal => draft,personal > Message at 1 has changed its source location. Updating labels from draft,inbox,personal => draft,inbox,personal > Message at 2 has changed its source location. Updating labels from draft,personal => draft,personal > Message at 3 has changed its source location. Updating labels from draft,personal => draft,personal That's odd, I guess Enumerator.new(Index.instance, :each_source_info, self.id).to_a returning a null set so we always emit :add? If you could add a debug call on that enumerator and get me the contents of old_ids and new_ids whereabouts draft.rb line 70 that would be great. Edward From ezyang@MIT.EDU Mon Sep 3 19:27:13 2012 From: ezyang@MIT.EDU (Edward Z. Yang) Date: Mon, 03 Sep 2012 15:27:13 -0400 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT) In-Reply-To: <1346697582-sup-8965@alvh.no-ip.org> References: <1345564795-sup-3898@alvh.no-ip.org> <1346648371-12305-1-git-send-email-ezyang@mit.edu> <1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org> <1346688387-sup-5906@alvh.no-ip.org> <20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu> <1346695742-sup-7650@javelin> <1346696946-sup-4602@javelin> <1346697582-sup-8965@alvh.no-ip.org> Message-ID: <1346700222-sup-1486@javelin> Excerpts from Alvaro Herrera's message of Mon Sep 03 14:42:55 -0400 2012: > For the record, I let it run for a while and after that the poll log had > a couple thousand lines saying "Deleting ". > After that, so far it behaves normally. I don't know what prompted > those particular messages to be deleted; I certainly have lots of > messages "is:deleted" yet. Confirmed it's Xapian taking it's sweet time to delete the mail. You might do better with a higher XAPIAN_FLUSH_THRESHOLD value in bin/sup, but the process seems fairly CPU bound. Edward From ezyang@MIT.EDU Mon Sep 3 23:31:28 2012 From: ezyang@MIT.EDU (Edward Z. Yang) Date: Mon, 03 Sep 2012 19:31:28 -0400 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT) In-Reply-To: <1346695742-sup-7650@javelin> References: <1345564795-sup-3898@alvh.no-ip.org> <1346648371-12305-1-git-send-email-ezyang@mit.edu> <1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org> <1346688387-sup-5906@alvh.no-ip.org> <20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu> <1346695742-sup-7650@javelin> Message-ID: <1346714661-sup-2638@javelin> Extra note: the ruby-inotify plugin appears to be braindead on at least Ruby 1.8.7 for Ubuntu Precise, and will spin-loop. If sup is chomping 15% CPU on idle, try this patch on ruby-inotify Reported here: https://github.com/ruby-building-blocks/ruby-inotify/issues/8 diff --git a/lib/inotify/inotify_native.rb b/lib/inotify/inotify_native.rb index 66db2b4..27d1043 100644 --- a/lib/inotify/inotify_native.rb +++ b/lib/inotify/inotify_native.rb @@ -115,7 +115,7 @@ require 'ffi' # each_event() provides an easy way to loop over all events as they occur def each_event loop do - ready = IO.select([@io], nil, nil, nil) + ready = IO.select([@io]) event = self.read_event yield event end Excerpts from Edward Z. Yang's message of Mon Sep 03 14:09:57 -0400 2012: > OK, cracked a fix; you need this extra patch: > > commit f9ea07f3c4982ab46d8171fdba8eabc3af00c840 > Author: Edward Z. Yang > Date: Mon Sep 3 14:09:34 2012 -0400 > > sync_back after writing to index, not before. > > Signed-off-by: Edward Z. Yang > > diff --git a/lib/sup/index.rb b/lib/sup/index.rb > index 13798d6..4b533a7 100644 > --- a/lib/sup/index.rb > +++ b/lib/sup/index.rb > @@ -657,10 +657,6 @@ EOS > end > > def sync_message m, overwrite > - ## TODO: we should not save the message if the sync_back failed > - ## since it would overwrite the location field > - m.sync_back > - > doc = synchronize { find_doc(m.id) } > existed = doc != nil > doc ||= Xapian::Document.new > @@ -703,6 +699,10 @@ EOS > @xapian.replace_document docid, doc > end > > + # sync_back must be after label update, so that inotify gets > + # fresh data from the index > + m.sync_back > + > m.labels.each { |l| LabelManager << l } > true > end From ezyang@MIT.EDU Tue Sep 4 16:06:07 2012 From: ezyang@MIT.EDU (Edward Z. Yang) Date: Tue, 04 Sep 2012 12:06:07 -0400 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT) In-Reply-To: <1346714661-sup-2638@javelin> References: <1345564795-sup-3898@alvh.no-ip.org> <1346648371-12305-1-git-send-email-ezyang@mit.edu> <1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org> <1346688387-sup-5906@alvh.no-ip.org> <20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu> <1346695742-sup-7650@javelin> <1346714661-sup-2638@javelin> Message-ID: <1346774717-sup-3579@javelin> I've pushed a new revision of the branch: http://gitorious.org/~ezyang/sup/ezyang/commits/maildir-sync which solves some bad interactions with sync-back. Edward From alvherre@alvh.no-ip.org Tue Sep 4 18:09:54 2012 From: alvherre@alvh.no-ip.org (Alvaro Herrera) Date: Tue, 04 Sep 2012 15:09:54 -0300 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT) In-Reply-To: <1346774717-sup-3579@javelin> References: <1345564795-sup-3898@alvh.no-ip.org> <1346648371-12305-1-git-send-email-ezyang@mit.edu> <1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org> <1346688387-sup-5906@alvh.no-ip.org> <20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu> <1346695742-sup-7650@javelin> <1346714661-sup-2638@javelin> <1346774717-sup-3579@javelin> Message-ID: <1346781893-sup-7790@alvh.no-ip.org> Excerpts from Edward Z. Yang's message of mar sep 04 13:06:07 -0300 2012: > I've pushed a new revision of the branch: http://gitorious.org/~ezyang/sup/ezyang/commits/maildir-sync > which solves some bad interactions with sync-back. Okay, I'm running this now. Having the emails show up instantly works great ... hooray for continuous, permanent distraction ;-) I haven't debugged the other issue yet. I'll let you know. -- ?lvaro Herrera From matthieu.rakotojaona@gmail.com Mon Sep 24 22:13:57 2012 From: matthieu.rakotojaona@gmail.com (Matthieu Rakotojaona) Date: Tue, 25 Sep 2012 00:13:57 +0200 Subject: [sup-devel] Update in heliotrope ! Message-ID: Hello everyone, I have updated my version of heliotrope on my imap branch (https://github.com/rakoo/heliotrope/tree/imap). You just have to clone it and checkout the branch. You may now synchronize your mails between heliotrope and a maildir with offlineimap. I tested most of the situations that can happen when using it (Adding/removing a label on heliotrope side, adding/removing a mail on maildir side) A few notes : * It is not "multi-write-safe", in the sense that you should stop all other activity with heliotrope while you sync with offlinimap. Concurrent access will have unknown outcome. * To continue on this point, use offlineimap with the `-1` flag, which will turn off multithreading. It should work without it though, because AFAIK offlineimap threads do not share mailboxes. But I'm not sure. * There are a few hardcoded labels/dir that you should find in your maildir, such as INBOX/inbox and "All Mail", which has no equivalent in heliotrope. Fear not, this one is taken care of. * Syncing with another IMAP server should also work,because it is transparent to offlineimap (at least from the outside) Have fun ! -- Matthieu RAKOTOJAONA From sdothum@gmail.com Tue Sep 25 19:15:01 2012 From: sdothum@gmail.com (Steven Hum) Date: Tue, 25 Sep 2012 15:15:01 -0400 Subject: [sup-devel] Update in heliotrope ! In-Reply-To: References: Message-ID: <1348599178-sup-12@luna> Hello Matthieu, I am able to initialize the heliotrope mailstore directory easy enough with ruby -Ilib bin/heliotrope-server -d (stuff gets created :-) but when I try to import with ruby -Ilib bin/heliotrope-import -a -d being the dir that contains the cur/new/tmp directories that contain the actual emails downloaded by offlineimap, the import process hangs after the "Adding mail..." message and doesn't load any emails (I just terminate the process with a ctrl-c). What am I missing? I tried using heliotrope-add with the server running, but it hung at the same point (not surpisingly). Secondly, your note "stop all other activity with heliotrope while you sync with offlinimap". Does that mean heliotrope-server should not be running if offlineimap is (updating the maildirs)? Thanks, Steven > Excerpts from Matthieu Rakotojaona's message of 2012-09-24 18:13:57 -0400: > Hello everyone, > > I have updated my version of heliotrope on my imap branch > (https://github.com/rakoo/heliotrope/tree/imap). You just have to > clone it and checkout the branch. > You may now synchronize your mails between heliotrope and a maildir > with offlineimap. I tested most of the situations that can happen > when using it (Adding/removing a label on heliotrope side, > adding/removing a mail on maildir side) > > A few notes : > > * It is not "multi-write-safe", in the sense that you should stop all > other activity with heliotrope while you sync with offlinimap. > Concurrent access will have unknown outcome. > * To continue on this point, use offlineimap with the `-1` flag, which > will turn off multithreading. It should work without it though, > because AFAIK offlineimap threads do not share mailboxes. But I'm not > sure. > * There are a few hardcoded labels/dir that you should find in your > maildir, such as INBOX/inbox and "All Mail", which has no equivalent > in heliotrope. Fear not, this one is taken care of. > * Syncing with another IMAP server should also work,because it is > transparent to offlineimap (at least from the outside) > > Have fun ! > -- "Truth or die." Steven Hum 5 - 28 Gilmour St Ottawa, ON K2P 0N3 email sdothum at gmail.com tel 613.237.9058 From matthieu.rakotojaona@gmail.com Tue Sep 25 19:44:49 2012 From: matthieu.rakotojaona@gmail.com (Matthieu Rakotojaona) Date: Tue, 25 Sep 2012 21:44:49 +0200 Subject: [sup-devel] Update in heliotrope ! In-Reply-To: <1348599178-sup-12@luna> References: <1348599178-sup-12@luna> Message-ID: On Tue, Sep 25, 2012 at 9:15 PM, Steven Hum wrote: > Hello Matthieu, Hello, > I am able to initialize the heliotrope mailstore directory easy enough > with > > ruby -Ilib bin/heliotrope-server -d > > (stuff gets created :-) but when I try to import with > > ruby -Ilib bin/heliotrope-import -a -d Did you even read the README ? =] It clearly says : "To bulk import mail, use heltrope-import. You must stop your server first." The thing is, both heliotrope-server and heliotrope-import will try to access the same resources, which are not designed for multi-concurrency. So, you have to shut heliotrope-server down before using heliotrope-import. > I tried using heliotrope-add with the server running, but it > hung at the same point (not surpisingly). This is more problematic, because it should work. Can you share some of your logs ? > Secondly, your note "stop all other activity with heliotrope while you > sync with offlinimap". Does that mean heliotrope-server should not be > running if offlineimap is (updating the maildirs)? No, on the contrary, it means that while heliotrope-server is running, you should either access it with a traditional http client, or with an IMAP client, but you should never do both the same time. But you will need heliotrope-server running for accessing IMAP. -- Matthieu RAKOTOJAONA From sdothum@gmail.com Tue Sep 25 21:01:45 2012 From: sdothum@gmail.com (Steven Hum) Date: Tue, 25 Sep 2012 17:01:45 -0400 Subject: [sup-devel] Update in heliotrope ! In-Reply-To: References: <1348599178-sup-12@luna> Message-ID: <1348605639-sup-455@luna> Yes! :-) (though I did try the import with the server running after the persistent hanging without.. just in case the documentation wasn't correct!). As nothing appeared to be working, I purged the heliotrope mailstore and performed 2 actions: 1) the initial server startup to initialize the mailstore and associated files, and 2) an import of a small maildir (after stopping the server). I presume the logs you are referring to is the LOG file in the /store dir. LOG from step 1) (or LOG.old after completing both steps) 2012/09/25-16:42:13.770269 7ff2345bc700 Delete type=3 #1 LOG from attempted import, step 2) 2012/09/25-16:43:12.450537 7f0c45ff8700 Recovering log #3 2012/09/25-16:43:12.450585 7f0c45ff8700 Level-0 table #5: started 2012/09/25-16:43:12.458919 7f0c45ff8700 Level-0 table #5: 344 bytes OK 2012/09/25-16:43:12.460026 7f0c45ff8700 Delete type=0 #3 2012/09/25-16:43:12.460050 7f0c45ff8700 Delete type=3 #2 Oh, the second thing you should know is that I run on debian (test / wheezy) and have been using rvm 1.9.3. I know that the debian environment doesn'd always make it easy for ruby installs at times. In addition to the gems requirements listed in the README, I had to install the mail and thin gems to get heliotrope to run. Regards, Steven Excerpts from Matthieu Rakotojaona's message of 2012-09-25 15:44:49 -0400: > On Tue, Sep 25, 2012 at 9:15 PM, Steven Hum wrote: > > Hello Matthieu, > > Hello, > > > I am able to initialize the heliotrope mailstore directory easy enough > > with > > > > ruby -Ilib bin/heliotrope-server -d > > > > (stuff gets created :-) but when I try to import with > > > > ruby -Ilib bin/heliotrope-import -a -d > > Did you even read the README ? =] > > It clearly says : "To bulk import mail, use heltrope-import. You must > stop your server first." > > The thing is, both heliotrope-server and heliotrope-import will try to > access the same resources, which are not designed for > multi-concurrency. So, you have to shut heliotrope-server down before > using heliotrope-import. > > > I tried using heliotrope-add with the server running, but it > > hung at the same point (not surpisingly). > > This is more problematic, because it should work. Can you share some > of your logs ? > > > Secondly, your note "stop all other activity with heliotrope while you > > sync with offlinimap". Does that mean heliotrope-server should not be > > running if offlineimap is (updating the maildirs)? > > No, on the contrary, it means that while heliotrope-server is running, > you should either access it with a traditional http client, or with an > IMAP client, but you should never do both the same time. > But you will need heliotrope-server running for accessing IMAP. > -- "Truth or die." Steven Hum 5 - 28 Gilmour St Ottawa, ON K2P 0N3 email sdothum at gmail.com tel 613.237.9058 From matthieu.rakotojaona@gmail.com Thu Sep 27 15:50:40 2012 From: matthieu.rakotojaona@gmail.com (Matthieu Rakotojaona) Date: Thu, 27 Sep 2012 17:50:40 +0200 Subject: [sup-devel] Update in heliotrope ! In-Reply-To: <1348605639-sup-455@luna> References: <1348599178-sup-12@luna> <1348605639-sup-455@luna> Message-ID: Sorry for the delay. On Tue, Sep 25, 2012 at 11:01 PM, Steven Hum wrote: > I presume the logs you are referring to is > the LOG file in the /store dir. Hmm no, I would like to see the heliotrope-server logs, or the heliotrope-import logs. These could help solve the problem. > Oh, the second thing you should know is that I run on debian (test > / wheezy) and have been using rvm 1.9.3. I know that the debian > environment doesn'd always make it easy for ruby installs at times. In > addition to the gems requirements listed in the README, I had to install > the mail and thin gems to get heliotrope to run. Sorry, this is totally undocumented, but yes, you have to install the mail gem. I decided to part from rmail, since it isn't maintained anymore. Thin was just there to improve performance and is optional. I just found the HTTP part stable enough fore my needs, so I changed to it (And effectively saw some improvements) -- Matthieu RAKOTOJAONA From sdothum@gmail.com Thu Sep 27 16:46:01 2012 From: sdothum@gmail.com (Steven Hum) Date: Thu, 27 Sep 2012 12:46:01 -0400 Subject: [sup-devel] Update in heliotrope ! In-Reply-To: References: <1348599178-sup-12@luna> <1348605639-sup-455@luna> Message-ID: <1348763216-sup-299@luna> I couldn't find any physical log files anywhere (at least any text files with an obvious name...). The only other file in the heliotrope mailstore directory I found was a 32 byte binary file named "000022.log" Where should these be located? Looks like there could be something fundamentally broken on my system if these log files aren't even being written! Regards, Steven Excerpts from Matthieu Rakotojaona's message of 2012-09-27 11:50:40 -0400: > Sorry for the delay. > > On Tue, Sep 25, 2012 at 11:01 PM, Steven Hum wrote: > > I presume the logs you are referring to is > > the LOG file in the /store dir. > > Hmm no, I would like to see the heliotrope-server logs, or the > heliotrope-import logs. These could help solve the problem. > > > Oh, the second thing you should know is that I run on debian (test > > / wheezy) and have been using rvm 1.9.3. I know that the debian > > environment doesn'd always make it easy for ruby installs at times. In > > addition to the gems requirements listed in the README, I had to install > > the mail and thin gems to get heliotrope to run. > > Sorry, this is totally undocumented, but yes, you have to install the > mail gem. I decided to part from rmail, since it isn't maintained > anymore. Thin was just there to improve performance and is optional. I > just found the HTTP part stable enough fore my needs, so I changed to > it (And effectively saw some improvements) > -- "Truth or die." Steven Hum 5 - 28 Gilmour St Ottawa, ON K2P 0N3 email sdothum at gmail.com tel 613.237.9058