From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by 10.58.247.3 with SMTP id ya3csp488982vec; Sun, 2 Sep 2012 22:05:22 -0700 (PDT) Received: by 10.182.174.9 with SMTP id bo9mr1483717obc.19.1346648722369; Sun, 02 Sep 2012 22:05:22 -0700 (PDT) Return-Path: Received: from rubyforge.org (50-56-192-79.static.cloud-ips.com. [50.56.192.79]) by mx.google.com with ESMTP id r4si6071081obz.176.2012.09.02.22.05.22; Sun, 02 Sep 2012 22:05:22 -0700 (PDT) Received-SPF: pass (google.com: domain of sup-devel-bounces@rubyforge.org designates 50.56.192.79 as permitted sender) client-ip=50.56.192.79; Authentication-Results: mx.google.com; spf=pass (google.com: domain of sup-devel-bounces@rubyforge.org designates 50.56.192.79 as permitted sender) smtp.mail=sup-devel-bounces@rubyforge.org Received: from localhost.localdomain (localhost [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id BDFE82E06E; Mon, 3 Sep 2012 05:05:21 +0000 (UTC) Received: from dmz-mailsec-scanner-1.mit.edu (DMZ-MAILSEC-SCANNER-1.MIT.EDU [18.9.25.12]) by rubyforge.org (Postfix) with ESMTP id 3198A2E066 for ; Mon, 3 Sep 2012 05:01:00 +0000 (UTC) X-AuditID: 1209190c-b7fd26d0000008d9-c4-5044398c4cad Received: from mailhub-auth-4.mit.edu ( [18.7.62.39]) by dmz-mailsec-scanner-1.mit.edu (Symantec Messaging Gateway) with SMTP id DB.48.02265.C8934405; Mon, 3 Sep 2012 01:01:00 -0400 (EDT) Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103]) by mailhub-auth-4.mit.edu (8.13.8/8.9.2) with ESMTP id q83510L2020775; Mon, 3 Sep 2012 01:01:00 -0400 Received: from localhost ([18.111.73.140]) (authenticated bits=0) (User authenticated as ezyang@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id q8350xEY025706; Mon, 3 Sep 2012 01:00:59 -0400 (EDT) From: "Edward Z. Yang" In-reply-to: <1346648371-12305-1-git-send-email-ezyang@mit.edu> To: sup-devel , alvherre References: <1345564795-sup-3898@alvh.no-ip.org> <1346648371-12305-1-git-send-email-ezyang@mit.edu> Date: Mon, 03 Sep 2012 01:00:59 -0400 Message-Id: <1346648395-sup-4546@javelin> User-Agent: Sup/git X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrMIsWRmVeSWpSXmKPExsUixG6nrttj6RJg8Pa4usWFyX/YLR5P38Pu wORx5ddEdo99+yYyBzBFcdmkpOZklqUW6dslcGWsOTudvWCXVsWyx79ZGhg7ZLsYOTkkBEwk vm69zwJhi0lcuLeerYuRi0NIYB+jxOWHc6Gc9YwSq+62skI4Txkl1k5+zdzFyMHBLKAusX6e EEg3G5D56NhTVhCbU8BeYvnTRiYQW0TAW2LisXmMILaQQLLE8b0vwLYJCzhIrPr7lhnEZhFQ lWi/uAashldAU+Lvnc/sILaogLDEkyPNYDXMAvISzVtnM09g5J+FsHkWkswCRqZVjLIpuVW6 uYmZOcWpybrFyYl5ealFuoZ6uZkleqkppZsYQWHHKcmzg/HNQaVDjAIcjEo8vAHvnAOEWBPL iitzDzFKcjApifLuNnYJEOJLyk+pzEgszogvKs1JLT7EKMHBrCTCe/cSUDlvSmJlVWpRPkxK moNFSZz3cspNfyGB9MSS1OzU1ILUIpisDAeHkgSvmwXQUMGi1PTUirTMnBKENBMHJ8hwHqDh /SA1vMUFibnFmekQ+VOMilLivNEgCQGQREZpHlwvLC28YhQHekWYtwSkigeYUuC6XwENZgIa fEwQ5OrikkSElFQDI8Mnzqs8078teVpmcTX26aLEfSbnLWccOPro3p5LpvPvFNcuOr2H46gg y0qpJx9q44X/7bz2szNzX2NRIr9g5r+uC+sb3+Q/sbjhrrlDflmRGQ8L28Epeda/rfz8ipeK HdV+nV/NwZXnV7T3p94p3oS9hQ+POrO4C7mvWssb754+4Z6S6RXLD0osxRmJhlrMRcWJAFeB iMzmAgAA Subject: Re: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT) X-BeenThere: sup-devel@rubyforge.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Sup developer discussion List-Id: Sup developer discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: sup-devel-bounces@rubyforge.org Errors-To: sup-devel-bounces@rubyforge.org The locking is a downright crime (where's the STM when you need it), and it's still racy, but it should work OK. Excerpts from Edward Z. Yang's message of Mon Sep 03 00:59:31 -0400 2012: > From: "Edward Z. Yang" > > Signed-off-by: Edward Z. Yang > --- > lib/sup/maildir.rb | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++-- > lib/sup/poll.rb | 33 ++++++++++++++++++++++++++------- > lib/sup/source.rb | 4 ++++ > 3 files changed, 80 insertions(+), 9 deletions(-) > > diff --git a/lib/sup/maildir.rb b/lib/sup/maildir.rb > index 2a91f05..743156d 100644 > --- a/lib/sup/maildir.rb > +++ b/lib/sup/maildir.rb > @@ -1,5 +1,6 @@ > require 'uri' > require 'set' > +require 'inotify' > > module Redwood > > @@ -184,6 +185,45 @@ class Maildir < Source > nil > end > > + def continuous_poll poll_mutex > + i = Inotify.new > + watches = {} > + @ctimes.each do |d,prev_ctime| > + subdir = File.join @dir, d > + wd = i.add_watch(subdir, Inotify::CREATE | Inotify::DELETE | Inotify::MOVE) > + watches[wd] = d > + end > + i.each_event do |ev| > + poll_mutex.synchronize do > + @mutex.synchronize do > + begin > + ::Thread.current[@dir] = true > + id = File.join watches[ev.wd], ev.name > + # check if inotify is stale > + # since we have @mutex, there is no race (except for > + # an external program fucking us over) > + next unless File.exists? File.join(@dir, id) > + x = Enumerator.new(Index.instance, :each_source_info, self.id, "#{id}").to_a > + if ev.mask & Inotify::CREATE or ev.mask & Inotify::MOVE_TO > + next unless x.empty? > + yield :add, > + :info => id, > + :labels => @labels + maildir_labels(id) + [:inbox], > + :progress => 0 > + elsif ev.mask & Inotify::DELETE or ev.mask & Inotify::MOVE_FROM > + next unless !x.empty? > + yield :delete, > + :info => id, > + :progress => 0 > + end > + ensure > + ::Thread.current[@dir] = nil > + end > + end > + end > + end > + end > + > def labels? id > maildir_labels id > end > @@ -248,7 +288,16 @@ private > end > > def maildir_move_file orig_path, new_source_id, flags > - @mutex.synchronize do > + if ::Thread.current[@dir] > + _maildir_move_file orig_path, new_source_id, flags > + else > + @mutex.synchronize do > + _maildir_move_file orig_path, new_source_id, flags > + end > + end > + end > + > + def _maildir_move_file orig_path, new_source_id, flags > new_base = (flags.include?("S")) ? "cur" : "new" > md_base, md_ver, md_flags = maildir_data orig_path > > @@ -292,7 +341,6 @@ private > end > > [new_source, new_loc] > - end > end > end > > diff --git a/lib/sup/poll.rb b/lib/sup/poll.rb > index dbd351f..51e0afa 100644 > --- a/lib/sup/poll.rb > +++ b/lib/sup/poll.rb > @@ -94,11 +94,27 @@ EOS > poll if @last_poll.nil? || (Time.now - @last_poll) >= @delay > end > end > + # XXX dup dup > + SourceManager.usual_sources.each do |source| > + Redwood::reporting_thread("inotify poll for #{source}") do > + source.continuous_poll @mutex do |sym, args| > + poll_handler source, sym, args > + end > + end > + end > + SourceManager.unusual_sources.each do |source| > + Redwood::reporting_thread("inotify poll for #{source}") do > + source.continuous_poll @mutex do |sym, args| > + poll_handler source, sym, args > + end > + end > + end > end > > def stop > @thread.kill if @thread > @thread = nil > + # handle inotify polls > end > > def do_poll > @@ -172,7 +188,16 @@ EOS > ## from the index after being yielded. > def poll_from source, opts={} > begin > - source.poll do |sym, args| > + source.poll do |sym,args| > + poll_handler source, sym, args > + end > + source.go_idle > + rescue SourceError => e > + warn "problem getting messages from #{source}: #{e.message}" > + end > + end > + > + def poll_handler source, sym, args > case sym > when :add > m = Message.build_from_source source, args[:info] > @@ -224,12 +249,6 @@ EOS > UpdateManager.relay self, :updated, m > end > end > - end > - > - source.go_idle > - rescue SourceError => e > - warn "problem getting messages from #{source}: #{e.message}" > - end > end > > def handle_idle_update sender, idle_since; @should_clear_running_totals = false; end > diff --git a/lib/sup/source.rb b/lib/sup/source.rb > index 06b6e6b..073a10a 100644 > --- a/lib/sup/source.rb > +++ b/lib/sup/source.rb > @@ -102,6 +102,10 @@ class Source > unimplemented > end > > + ## Like poll, but never returns (it is continuous, and uses something > + ## like inotify. Will always be run in another thread.) > + def continuous_poll poll_mutex; [] end > + > def valid? info > true > end _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel