sup

A curses threads-with-tags style email client

sup-website.git

git clone https://supmua.dev/git/sup-website/

community/pipermail-archives/sup-devel/2012-09.txt (39969B) - raw

      1 From ezyang@MIT.EDU  Mon Sep  3 04:59:31 2012
      2 From: ezyang@MIT.EDU (Edward Z. Yang)
      3 Date: Mon,  3 Sep 2012 00:59:31 -0400
      4 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT)
      5 In-Reply-To: <1345564795-sup-3898@alvh.no-ip.org>
      6 References: <1345564795-sup-3898@alvh.no-ip.org>
      7 Message-ID: <1346648371-12305-1-git-send-email-ezyang@mit.edu>
      8 
      9 From: "Edward Z. Yang" <ezyang at mit.edu>
     10 
     11 Signed-off-by: Edward Z. Yang <ezyang at mit.edu>
     12 ---
     13  lib/sup/maildir.rb | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++--
     14  lib/sup/poll.rb    | 33 ++++++++++++++++++++++++++-------
     15  lib/sup/source.rb  |  4 ++++
     16  3 files changed, 80 insertions(+), 9 deletions(-)
     17 
     18 diff --git a/lib/sup/maildir.rb b/lib/sup/maildir.rb
     19 index 2a91f05..743156d 100644
     20 --- a/lib/sup/maildir.rb
     21 +++ b/lib/sup/maildir.rb
     22 @@ -1,5 +1,6 @@
     23  require 'uri'
     24  require 'set'
     25 +require 'inotify'
     26  
     27  module Redwood
     28  
     29 @@ -184,6 +185,45 @@ class Maildir < Source
     30      nil
     31    end
     32  
     33 +  def continuous_poll poll_mutex
     34 +    i = Inotify.new
     35 +    watches = {}
     36 +    @ctimes.each do |d,prev_ctime|
     37 +      subdir = File.join @dir, d
     38 +      wd = i.add_watch(subdir, Inotify::CREATE | Inotify::DELETE | Inotify::MOVE)
     39 +      watches[wd] = d
     40 +    end
     41 +    i.each_event do |ev|
     42 +      poll_mutex.synchronize do
     43 +        @mutex.synchronize do
     44 +          begin
     45 +            ::Thread.current[@dir] = true
     46 +            id = File.join watches[ev.wd], ev.name
     47 +            # check if inotify is stale
     48 +            # since we have @mutex, there is no race (except for
     49 +            # an external program fucking us over)
     50 +            next unless File.exists? File.join(@dir, id)
     51 +            x = Enumerator.new(Index.instance, :each_source_info, self.id, "#{id}").to_a
     52 +            if ev.mask & Inotify::CREATE or ev.mask & Inotify::MOVE_TO
     53 +              next unless x.empty?
     54 +              yield :add,
     55 +                :info => id,
     56 +                :labels => @labels + maildir_labels(id) + [:inbox],
     57 +                :progress => 0
     58 +            elsif ev.mask & Inotify::DELETE or ev.mask & Inotify::MOVE_FROM
     59 +              next unless !x.empty?
     60 +              yield :delete,
     61 +                :info => id,
     62 +                :progress => 0
     63 +            end
     64 +          ensure
     65 +            ::Thread.current[@dir] = nil
     66 +          end
     67 +        end
     68 +      end
     69 +    end
     70 +  end
     71 +
     72    def labels? id
     73      maildir_labels id
     74    end
     75 @@ -248,7 +288,16 @@ private
     76    end
     77  
     78    def maildir_move_file orig_path, new_source_id, flags
     79 -    @mutex.synchronize do
     80 +    if ::Thread.current[@dir]
     81 +      _maildir_move_file orig_path, new_source_id, flags
     82 +    else
     83 +      @mutex.synchronize do
     84 +        _maildir_move_file orig_path, new_source_id, flags
     85 +      end
     86 +    end
     87 +  end
     88 +
     89 +  def _maildir_move_file orig_path, new_source_id, flags
     90        new_base = (flags.include?("S")) ? "cur" : "new"
     91        md_base, md_ver, md_flags = maildir_data orig_path
     92  
     93 @@ -292,7 +341,6 @@ private
     94        end
     95  
     96        [new_source, new_loc]
     97 -    end
     98    end
     99  end
    100  
    101 diff --git a/lib/sup/poll.rb b/lib/sup/poll.rb
    102 index dbd351f..51e0afa 100644
    103 --- a/lib/sup/poll.rb
    104 +++ b/lib/sup/poll.rb
    105 @@ -94,11 +94,27 @@ EOS
    106          poll if @last_poll.nil? || (Time.now - @last_poll) >= @delay
    107        end
    108      end
    109 +    # XXX dup dup
    110 +    SourceManager.usual_sources.each do |source|
    111 +      Redwood::reporting_thread("inotify poll for #{source}") do
    112 +        source.continuous_poll @mutex do |sym, args|
    113 +          poll_handler source, sym, args
    114 +        end
    115 +      end
    116 +    end
    117 +    SourceManager.unusual_sources.each do |source|
    118 +      Redwood::reporting_thread("inotify poll for #{source}") do
    119 +        source.continuous_poll @mutex do |sym, args|
    120 +          poll_handler source, sym, args
    121 +        end
    122 +      end
    123 +    end
    124    end
    125  
    126    def stop
    127      @thread.kill if @thread
    128      @thread = nil
    129 +    # handle inotify polls
    130    end
    131  
    132    def do_poll
    133 @@ -172,7 +188,16 @@ EOS
    134    ## from the index after being yielded.
    135    def poll_from source, opts={}
    136      begin
    137 -      source.poll do |sym, args|
    138 +      source.poll do |sym,args|
    139 +        poll_handler source, sym, args
    140 +      end
    141 +      source.go_idle
    142 +    rescue SourceError => e
    143 +      warn "problem getting messages from #{source}: #{e.message}"
    144 +    end
    145 +  end
    146 +
    147 +  def poll_handler source, sym, args
    148          case sym
    149          when :add
    150            m = Message.build_from_source source, args[:info]
    151 @@ -224,12 +249,6 @@ EOS
    152              UpdateManager.relay self, :updated, m
    153            end
    154          end
    155 -      end
    156 -
    157 -      source.go_idle
    158 -    rescue SourceError => e
    159 -      warn "problem getting messages from #{source}: #{e.message}"
    160 -    end
    161    end
    162  
    163    def handle_idle_update sender, idle_since; @should_clear_running_totals = false; end
    164 diff --git a/lib/sup/source.rb b/lib/sup/source.rb
    165 index 06b6e6b..073a10a 100644
    166 --- a/lib/sup/source.rb
    167 +++ b/lib/sup/source.rb
    168 @@ -102,6 +102,10 @@ class Source
    169      unimplemented
    170    end
    171  
    172 +  ## Like poll, but never returns (it is continuous, and uses something
    173 +  ## like inotify. Will always be run in another thread.)
    174 +  def continuous_poll poll_mutex; [] end
    175 +
    176    def valid? info
    177      true
    178    end
    179 -- 
    180 1.7.11.3
    181 
    182 
    183 From ezyang@MIT.EDU  Mon Sep  3 05:00:59 2012
    184 From: ezyang@MIT.EDU (Edward Z. Yang)
    185 Date: Mon, 03 Sep 2012 01:00:59 -0400
    186 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT)
    187 In-Reply-To: <1346648371-12305-1-git-send-email-ezyang@mit.edu>
    188 References: <1345564795-sup-3898@alvh.no-ip.org>
    189 	<1346648371-12305-1-git-send-email-ezyang@mit.edu>
    190 Message-ID: <1346648395-sup-4546@javelin>
    191 
    192 The locking is a downright crime (where's the STM when you need it),
    193 and it's still racy, but it should work OK.
    194 
    195 Excerpts from Edward Z. Yang's message of Mon Sep 03 00:59:31 -0400 2012:
    196 > From: "Edward Z. Yang" <ezyang at mit.edu>
    197 > 
    198 > Signed-off-by: Edward Z. Yang <ezyang at mit.edu>
    199 > ---
    200 >  lib/sup/maildir.rb | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++--
    201 >  lib/sup/poll.rb    | 33 ++++++++++++++++++++++++++-------
    202 >  lib/sup/source.rb  |  4 ++++
    203 >  3 files changed, 80 insertions(+), 9 deletions(-)
    204 > 
    205 > diff --git a/lib/sup/maildir.rb b/lib/sup/maildir.rb
    206 > index 2a91f05..743156d 100644
    207 > --- a/lib/sup/maildir.rb
    208 > +++ b/lib/sup/maildir.rb
    209 > @@ -1,5 +1,6 @@
    210 >  require 'uri'
    211 >  require 'set'
    212 > +require 'inotify'
    213 >  
    214 >  module Redwood
    215 >  
    216 > @@ -184,6 +185,45 @@ class Maildir < Source
    217 >      nil
    218 >    end
    219 >  
    220 > +  def continuous_poll poll_mutex
    221 > +    i = Inotify.new
    222 > +    watches = {}
    223 > +    @ctimes.each do |d,prev_ctime|
    224 > +      subdir = File.join @dir, d
    225 > +      wd = i.add_watch(subdir, Inotify::CREATE | Inotify::DELETE | Inotify::MOVE)
    226 > +      watches[wd] = d
    227 > +    end
    228 > +    i.each_event do |ev|
    229 > +      poll_mutex.synchronize do
    230 > +        @mutex.synchronize do
    231 > +          begin
    232 > +            ::Thread.current[@dir] = true
    233 > +            id = File.join watches[ev.wd], ev.name
    234 > +            # check if inotify is stale
    235 > +            # since we have @mutex, there is no race (except for
    236 > +            # an external program fucking us over)
    237 > +            next unless File.exists? File.join(@dir, id)
    238 > +            x = Enumerator.new(Index.instance, :each_source_info, self.id, "#{id}").to_a
    239 > +            if ev.mask & Inotify::CREATE or ev.mask & Inotify::MOVE_TO
    240 > +              next unless x.empty?
    241 > +              yield :add,
    242 > +                :info => id,
    243 > +                :labels => @labels + maildir_labels(id) + [:inbox],
    244 > +                :progress => 0
    245 > +            elsif ev.mask & Inotify::DELETE or ev.mask & Inotify::MOVE_FROM
    246 > +              next unless !x.empty?
    247 > +              yield :delete,
    248 > +                :info => id,
    249 > +                :progress => 0
    250 > +            end
    251 > +          ensure
    252 > +            ::Thread.current[@dir] = nil
    253 > +          end
    254 > +        end
    255 > +      end
    256 > +    end
    257 > +  end
    258 > +
    259 >    def labels? id
    260 >      maildir_labels id
    261 >    end
    262 > @@ -248,7 +288,16 @@ private
    263 >    end
    264 >  
    265 >    def maildir_move_file orig_path, new_source_id, flags
    266 > -    @mutex.synchronize do
    267 > +    if ::Thread.current[@dir]
    268 > +      _maildir_move_file orig_path, new_source_id, flags
    269 > +    else
    270 > +      @mutex.synchronize do
    271 > +        _maildir_move_file orig_path, new_source_id, flags
    272 > +      end
    273 > +    end
    274 > +  end
    275 > +
    276 > +  def _maildir_move_file orig_path, new_source_id, flags
    277 >        new_base = (flags.include?("S")) ? "cur" : "new"
    278 >        md_base, md_ver, md_flags = maildir_data orig_path
    279 >  
    280 > @@ -292,7 +341,6 @@ private
    281 >        end
    282 >  
    283 >        [new_source, new_loc]
    284 > -    end
    285 >    end
    286 >  end
    287 >  
    288 > diff --git a/lib/sup/poll.rb b/lib/sup/poll.rb
    289 > index dbd351f..51e0afa 100644
    290 > --- a/lib/sup/poll.rb
    291 > +++ b/lib/sup/poll.rb
    292 > @@ -94,11 +94,27 @@ EOS
    293 >          poll if @last_poll.nil? || (Time.now - @last_poll) >= @delay
    294 >        end
    295 >      end
    296 > +    # XXX dup dup
    297 > +    SourceManager.usual_sources.each do |source|
    298 > +      Redwood::reporting_thread("inotify poll for #{source}") do
    299 > +        source.continuous_poll @mutex do |sym, args|
    300 > +          poll_handler source, sym, args
    301 > +        end
    302 > +      end
    303 > +    end
    304 > +    SourceManager.unusual_sources.each do |source|
    305 > +      Redwood::reporting_thread("inotify poll for #{source}") do
    306 > +        source.continuous_poll @mutex do |sym, args|
    307 > +          poll_handler source, sym, args
    308 > +        end
    309 > +      end
    310 > +    end
    311 >    end
    312 >  
    313 >    def stop
    314 >      @thread.kill if @thread
    315 >      @thread = nil
    316 > +    # handle inotify polls
    317 >    end
    318 >  
    319 >    def do_poll
    320 > @@ -172,7 +188,16 @@ EOS
    321 >    ## from the index after being yielded.
    322 >    def poll_from source, opts={}
    323 >      begin
    324 > -      source.poll do |sym, args|
    325 > +      source.poll do |sym,args|
    326 > +        poll_handler source, sym, args
    327 > +      end
    328 > +      source.go_idle
    329 > +    rescue SourceError => e
    330 > +      warn "problem getting messages from #{source}: #{e.message}"
    331 > +    end
    332 > +  end
    333 > +
    334 > +  def poll_handler source, sym, args
    335 >          case sym
    336 >          when :add
    337 >            m = Message.build_from_source source, args[:info]
    338 > @@ -224,12 +249,6 @@ EOS
    339 >              UpdateManager.relay self, :updated, m
    340 >            end
    341 >          end
    342 > -      end
    343 > -
    344 > -      source.go_idle
    345 > -    rescue SourceError => e
    346 > -      warn "problem getting messages from #{source}: #{e.message}"
    347 > -    end
    348 >    end
    349 >  
    350 >    def handle_idle_update sender, idle_since; @should_clear_running_totals = false; end
    351 > diff --git a/lib/sup/source.rb b/lib/sup/source.rb
    352 > index 06b6e6b..073a10a 100644
    353 > --- a/lib/sup/source.rb
    354 > +++ b/lib/sup/source.rb
    355 > @@ -102,6 +102,10 @@ class Source
    356 >      unimplemented
    357 >    end
    358 >  
    359 > +  ## Like poll, but never returns (it is continuous, and uses something
    360 > +  ## like inotify. Will always be run in another thread.)
    361 > +  def continuous_poll poll_mutex; [] end
    362 > +
    363 >    def valid? info
    364 >      true
    365 >    end
    366 
    367 From alvherre@alvh.no-ip.org  Mon Sep  3 16:02:26 2012
    368 From: alvherre@alvh.no-ip.org (Alvaro Herrera)
    369 Date: Mon, 03 Sep 2012 13:02:26 -0300
    370 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT)
    371 In-Reply-To: <1346648395-sup-4546@javelin>
    372 References: <1345564795-sup-3898@alvh.no-ip.org>
    373 	<1346648371-12305-1-git-send-email-ezyang@mit.edu>
    374 	<1346648395-sup-4546@javelin>
    375 Message-ID: <1346688000-sup-4643@alvh.no-ip.org>
    376 
    377 Excerpts from Edward Z. Yang's message of lun sep 03 02:00:59 -0300 2012:
    378 > The locking is a downright crime (where's the STM when you need it),
    379 > and it's still racy, but it should work OK.
    380 
    381 Hm.  I tried this but ran into trouble: I currently run branch "next",
    382 and your patch doesn't apply there; so I tried your ~ezyang fork and
    383 branch maildir-sync there, but I find that when in that branch (with or
    384 without this patch), Sup seems to eat 100% of a CPU core doing
    385 clock_gettime() and select() continuously.  Not sure what's happening.
    386 I assume you don't see that behavior.
    387 
    388 -- 
    389 ?lvaro Herrera <alvherre at alvh.no-ip.org>
    390 
    391 From alvherre@alvh.no-ip.org  Mon Sep  3 16:07:28 2012
    392 From: alvherre@alvh.no-ip.org (Alvaro Herrera)
    393 Date: Mon, 03 Sep 2012 13:07:28 -0300
    394 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT)
    395 In-Reply-To: <1346688000-sup-4643@alvh.no-ip.org>
    396 References: <1345564795-sup-3898@alvh.no-ip.org>
    397 	<1346648371-12305-1-git-send-email-ezyang@mit.edu>
    398 	<1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org>
    399 Message-ID: <1346688387-sup-5906@alvh.no-ip.org>
    400 
    401 Excerpts from Alvaro Herrera's message of lun sep 03 13:02:26 -0300 2012:
    402 > Excerpts from Edward Z. Yang's message of lun sep 03 02:00:59 -0300 2012:
    403 > > The locking is a downright crime (where's the STM when you need it),
    404 > > and it's still racy, but it should work OK.
    405 > 
    406 > Hm.  I tried this but ran into trouble: I currently run branch "next",
    407 > and your patch doesn't apply there; so I tried your ~ezyang fork and
    408 > branch maildir-sync there, but I find that when in that branch (with or
    409 > without this patch), Sup seems to eat 100% of a CPU core doing
    410 > clock_gettime() and select() continuously.  Not sure what's happening.
    411 > I assume you don't see that behavior.
    412 
    413 Oh, I see what's going on: it's deleting all my deleted email!  :-)
    414 Nevermind.  I'll just wait for it to finish before trying out your patch.
    415 
    416 -- 
    417 ?lvaro Herrera <alvherre at alvh.no-ip.org>
    418 
    419 From ezyang@MIT.EDU  Mon Sep  3 16:10:13 2012
    420 From: ezyang@MIT.EDU (ezyang)
    421 Date: Mon, 03 Sep 2012 12:10:13 -0400
    422 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT)
    423 In-Reply-To: <1346688387-sup-5906@alvh.no-ip.org>
    424 References: <1345564795-sup-3898@alvh.no-ip.org>
    425 	<1346648371-12305-1-git-send-email-ezyang@mit.edu>
    426 	<1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org>
    427 	<1346688387-sup-5906@alvh.no-ip.org>
    428 Message-ID: <20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu>
    429 
    430 Quoting Alvaro Herrera <alvherre at alvh.no-ip.org>:
    431 
    432 > Excerpts from Alvaro Herrera's message of lun sep 03 13:02:26 -0300 2012:
    433 >> Excerpts from Edward Z. Yang's message of lun sep 03 02:00:59 -0300 2012:
    434 >> > The locking is a downright crime (where's the STM when you need it),
    435 >> > and it's still racy, but it should work OK.
    436 >>
    437 >> Hm.  I tried this but ran into trouble: I currently run branch "next",
    438 >> and your patch doesn't apply there; so I tried your ~ezyang fork and
    439 >> branch maildir-sync there, but I find that when in that branch (with or
    440 >> without this patch), Sup seems to eat 100% of a CPU core doing
    441 >> clock_gettime() and select() continuously.  Not sure what's happening.
    442 >> I assume you don't see that behavior.
    443 >
    444 > Oh, I see what's going on: it's deleting all my deleted email!  :-)
    445 > Nevermind.  I'll just wait for it to finish before trying out your patch.
    446 
    447 There is a bug in the patch, where it notices changes that Sup makes from
    448 inotify, and as a result undoes any flag changes you make. So there 
    449 needs to be
    450 a fix here but I'm not sure what it is yet.
    451 
    452 Edward
    453 
    454 From ezyang@MIT.EDU  Mon Sep  3 18:09:57 2012
    455 From: ezyang@MIT.EDU (Edward Z. Yang)
    456 Date: Mon, 03 Sep 2012 14:09:57 -0400
    457 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT)
    458 In-Reply-To: <20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu>
    459 References: <1345564795-sup-3898@alvh.no-ip.org>
    460 	<1346648371-12305-1-git-send-email-ezyang@mit.edu>
    461 	<1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org>
    462 	<1346688387-sup-5906@alvh.no-ip.org>
    463 	<20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu>
    464 Message-ID: <1346695742-sup-7650@javelin>
    465 
    466 OK, cracked a fix; you need this extra patch:
    467 
    468 commit f9ea07f3c4982ab46d8171fdba8eabc3af00c840
    469 Author: Edward Z. Yang <ezyang at mit.edu>
    470 Date:   Mon Sep 3 14:09:34 2012 -0400
    471 
    472     sync_back after writing to index, not before.
    473     
    474     Signed-off-by: Edward Z. Yang <ezyang at mit.edu>
    475 
    476 diff --git a/lib/sup/index.rb b/lib/sup/index.rb
    477 index 13798d6..4b533a7 100644
    478 --- a/lib/sup/index.rb
    479 +++ b/lib/sup/index.rb
    480 @@ -657,10 +657,6 @@ EOS
    481    end
    482  
    483    def sync_message m, overwrite
    484 -    ## TODO: we should not save the message if the sync_back failed
    485 -    ## since it would overwrite the location field
    486 -    m.sync_back
    487 -
    488      doc = synchronize { find_doc(m.id) }
    489      existed = doc != nil
    490      doc ||= Xapian::Document.new
    491 @@ -703,6 +699,10 @@ EOS
    492        @xapian.replace_document docid, doc
    493      end
    494  
    495 +    # sync_back must be after label update, so that inotify gets
    496 +    # fresh data from the index
    497 +    m.sync_back
    498 +
    499      m.labels.each { |l| LabelManager << l }
    500      true
    501    end
    502 
    503 From ezyang@MIT.EDU  Mon Sep  3 18:29:43 2012
    504 From: ezyang@MIT.EDU (Edward Z. Yang)
    505 Date: Mon, 03 Sep 2012 14:29:43 -0400
    506 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT)
    507 In-Reply-To: <1346695742-sup-7650@javelin>
    508 References: <1345564795-sup-3898@alvh.no-ip.org>
    509 	<1346648371-12305-1-git-send-email-ezyang@mit.edu>
    510 	<1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org>
    511 	<1346688387-sup-5906@alvh.no-ip.org>
    512 	<20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu>
    513 	<1346695742-sup-7650@javelin>
    514 Message-ID: <1346696946-sup-4602@javelin>
    515 
    516 I think I may have seen Sup peg IO R/W in the way you saw, but when I
    517 restarted Sup with debugging, it went away.  Do let me know if you see it
    518 again; I think there's an infinite loop somewhere.
    519 
    520 Edward
    521 
    522 Excerpts from Edward Z. Yang's message of Mon Sep 03 14:09:57 -0400 2012:
    523 > OK, cracked a fix; you need this extra patch:
    524 > 
    525 > commit f9ea07f3c4982ab46d8171fdba8eabc3af00c840
    526 > Author: Edward Z. Yang <ezyang at mit.edu>
    527 > Date:   Mon Sep 3 14:09:34 2012 -0400
    528 > 
    529 >     sync_back after writing to index, not before.
    530 >     
    531 >     Signed-off-by: Edward Z. Yang <ezyang at mit.edu>
    532 > 
    533 > diff --git a/lib/sup/index.rb b/lib/sup/index.rb
    534 > index 13798d6..4b533a7 100644
    535 > --- a/lib/sup/index.rb
    536 > +++ b/lib/sup/index.rb
    537 > @@ -657,10 +657,6 @@ EOS
    538 >    end
    539 >  
    540 >    def sync_message m, overwrite
    541 > -    ## TODO: we should not save the message if the sync_back failed
    542 > -    ## since it would overwrite the location field
    543 > -    m.sync_back
    544 > -
    545 >      doc = synchronize { find_doc(m.id) }
    546 >      existed = doc != nil
    547 >      doc ||= Xapian::Document.new
    548 > @@ -703,6 +699,10 @@ EOS
    549 >        @xapian.replace_document docid, doc
    550 >      end
    551 >  
    552 > +    # sync_back must be after label update, so that inotify gets
    553 > +    # fresh data from the index
    554 > +    m.sync_back
    555 > +
    556 >      m.labels.each { |l| LabelManager << l }
    557 >      true
    558 >    end
    559 
    560 From alvherre@alvh.no-ip.org  Mon Sep  3 18:42:55 2012
    561 From: alvherre@alvh.no-ip.org (Alvaro Herrera)
    562 Date: Mon, 03 Sep 2012 15:42:55 -0300
    563 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT)
    564 In-Reply-To: <1346696946-sup-4602@javelin>
    565 References: <1345564795-sup-3898@alvh.no-ip.org>
    566 	<1346648371-12305-1-git-send-email-ezyang@mit.edu>
    567 	<1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org>
    568 	<1346688387-sup-5906@alvh.no-ip.org>
    569 	<20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu>
    570 	<1346695742-sup-7650@javelin> <1346696946-sup-4602@javelin>
    571 Message-ID: <1346697582-sup-8965@alvh.no-ip.org>
    572 
    573 Excerpts from Edward Z. Yang's message of lun sep 03 15:29:43 -0300 2012:
    574 > I think I may have seen Sup peg IO R/W in the way you saw, but when I
    575 > restarted Sup with debugging, it went away.  Do let me know if you see it
    576 > again; I think there's an infinite loop somewhere.
    577 
    578 Sure.
    579 
    580 For the record, I let it run for a while and after that the poll log had
    581 a couple thousand lines saying "Deleting <some-msg-id> <its subject>".
    582 After that, so far it behaves normally.  I don't know what prompted
    583 those particular messages to be deleted; I certainly have lots of
    584 messages "is:deleted" yet.
    585 
    586 Another curious thing with your branch is that the poll log gets these
    587 four lines each time a poll takes place:
    588 
    589 Message at 0 has changed its source location. Updating labels from draft,personal => draft,personal
    590 Message at 1 has changed its source location. Updating labels from draft,inbox,personal => draft,inbox,personal
    591 Message at 2 has changed its source location. Updating labels from draft,personal => draft,personal
    592 Message at 3 has changed its source location. Updating labels from draft,personal => draft,personal
    593 
    594 -- 
    595 ?lvaro Herrera <alvherre at alvh.no-ip.org>
    596 
    597 From ezyang@MIT.EDU  Mon Sep  3 18:49:42 2012
    598 From: ezyang@MIT.EDU (Edward Z. Yang)
    599 Date: Mon, 03 Sep 2012 14:49:42 -0400
    600 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT)
    601 In-Reply-To: <1346697582-sup-8965@alvh.no-ip.org>
    602 References: <1345564795-sup-3898@alvh.no-ip.org>
    603 	<1346648371-12305-1-git-send-email-ezyang@mit.edu>
    604 	<1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org>
    605 	<1346688387-sup-5906@alvh.no-ip.org>
    606 	<20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu>
    607 	<1346695742-sup-7650@javelin> <1346696946-sup-4602@javelin>
    608 	<1346697582-sup-8965@alvh.no-ip.org>
    609 Message-ID: <1346697811-sup-463@javelin>
    610 
    611 Excerpts from Alvaro Herrera's message of Mon Sep 03 14:42:55 -0400 2012:
    612 > Message at 0 has changed its source location. Updating labels from draft,personal => draft,personal
    613 > Message at 1 has changed its source location. Updating labels from draft,inbox,personal => draft,inbox,personal
    614 > Message at 2 has changed its source location. Updating labels from draft,personal => draft,personal
    615 > Message at 3 has changed its source location. Updating labels from draft,personal => draft,personal
    616 
    617 That's odd, I guess Enumerator.new(Index.instance, :each_source_info, self.id).to_a
    618 returning a null set so we always emit :add?  If you could add a debug call on
    619 that enumerator and get me the contents of old_ids and new_ids whereabouts draft.rb
    620 line 70 that would be great.
    621 
    622 Edward
    623 
    624 From ezyang@MIT.EDU  Mon Sep  3 19:27:13 2012
    625 From: ezyang@MIT.EDU (Edward Z. Yang)
    626 Date: Mon, 03 Sep 2012 15:27:13 -0400
    627 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT)
    628 In-Reply-To: <1346697582-sup-8965@alvh.no-ip.org>
    629 References: <1345564795-sup-3898@alvh.no-ip.org>
    630 	<1346648371-12305-1-git-send-email-ezyang@mit.edu>
    631 	<1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org>
    632 	<1346688387-sup-5906@alvh.no-ip.org>
    633 	<20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu>
    634 	<1346695742-sup-7650@javelin> <1346696946-sup-4602@javelin>
    635 	<1346697582-sup-8965@alvh.no-ip.org>
    636 Message-ID: <1346700222-sup-1486@javelin>
    637 
    638 Excerpts from Alvaro Herrera's message of Mon Sep 03 14:42:55 -0400 2012:
    639 > For the record, I let it run for a while and after that the poll log had
    640 > a couple thousand lines saying "Deleting <some-msg-id> <its subject>".
    641 > After that, so far it behaves normally.  I don't know what prompted
    642 > those particular messages to be deleted; I certainly have lots of
    643 > messages "is:deleted" yet.
    644 
    645 Confirmed it's Xapian taking it's sweet time to delete the mail.
    646 You might do better with a higher XAPIAN_FLUSH_THRESHOLD value in bin/sup,
    647 but the process seems fairly CPU bound.
    648 
    649 Edward
    650 
    651 From ezyang@MIT.EDU  Mon Sep  3 23:31:28 2012
    652 From: ezyang@MIT.EDU (Edward Z. Yang)
    653 Date: Mon, 03 Sep 2012 19:31:28 -0400
    654 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT)
    655 In-Reply-To: <1346695742-sup-7650@javelin>
    656 References: <1345564795-sup-3898@alvh.no-ip.org>
    657 	<1346648371-12305-1-git-send-email-ezyang@mit.edu>
    658 	<1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org>
    659 	<1346688387-sup-5906@alvh.no-ip.org>
    660 	<20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu>
    661 	<1346695742-sup-7650@javelin>
    662 Message-ID: <1346714661-sup-2638@javelin>
    663 
    664 Extra note: the ruby-inotify plugin appears to be braindead
    665 on at least Ruby 1.8.7 for Ubuntu Precise, and will spin-loop.  If
    666 sup is chomping 15% CPU on idle, try this patch on ruby-inotify
    667 
    668 Reported here: https://github.com/ruby-building-blocks/ruby-inotify/issues/8
    669 
    670 diff --git a/lib/inotify/inotify_native.rb b/lib/inotify/inotify_native.rb
    671 index 66db2b4..27d1043 100644
    672 --- a/lib/inotify/inotify_native.rb
    673 +++ b/lib/inotify/inotify_native.rb
    674 @@ -115,7 +115,7 @@ require 'ffi'
    675      # each_event() provides an easy way to loop over all events as they occur
    676      def each_event
    677        loop do
    678 -        ready = IO.select([@io], nil, nil, nil)
    679 +        ready = IO.select([@io])
    680          event = self.read_event
    681          yield event
    682        end
    683 
    684 
    685 Excerpts from Edward Z. Yang's message of Mon Sep 03 14:09:57 -0400 2012:
    686 > OK, cracked a fix; you need this extra patch:
    687 > 
    688 > commit f9ea07f3c4982ab46d8171fdba8eabc3af00c840
    689 > Author: Edward Z. Yang <ezyang at mit.edu>
    690 > Date:   Mon Sep 3 14:09:34 2012 -0400
    691 > 
    692 >     sync_back after writing to index, not before.
    693 >     
    694 >     Signed-off-by: Edward Z. Yang <ezyang at mit.edu>
    695 > 
    696 > diff --git a/lib/sup/index.rb b/lib/sup/index.rb
    697 > index 13798d6..4b533a7 100644
    698 > --- a/lib/sup/index.rb
    699 > +++ b/lib/sup/index.rb
    700 > @@ -657,10 +657,6 @@ EOS
    701 >    end
    702 >  
    703 >    def sync_message m, overwrite
    704 > -    ## TODO: we should not save the message if the sync_back failed
    705 > -    ## since it would overwrite the location field
    706 > -    m.sync_back
    707 > -
    708 >      doc = synchronize { find_doc(m.id) }
    709 >      existed = doc != nil
    710 >      doc ||= Xapian::Document.new
    711 > @@ -703,6 +699,10 @@ EOS
    712 >        @xapian.replace_document docid, doc
    713 >      end
    714 >  
    715 > +    # sync_back must be after label update, so that inotify gets
    716 > +    # fresh data from the index
    717 > +    m.sync_back
    718 > +
    719 >      m.labels.each { |l| LabelManager << l }
    720 >      true
    721 >    end
    722 
    723 From ezyang@MIT.EDU  Tue Sep  4 16:06:07 2012
    724 From: ezyang@MIT.EDU (Edward Z. Yang)
    725 Date: Tue, 04 Sep 2012 12:06:07 -0400
    726 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT)
    727 In-Reply-To: <1346714661-sup-2638@javelin>
    728 References: <1345564795-sup-3898@alvh.no-ip.org>
    729 	<1346648371-12305-1-git-send-email-ezyang@mit.edu>
    730 	<1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org>
    731 	<1346688387-sup-5906@alvh.no-ip.org>
    732 	<20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu>
    733 	<1346695742-sup-7650@javelin> <1346714661-sup-2638@javelin>
    734 Message-ID: <1346774717-sup-3579@javelin>
    735 
    736 I've pushed a new revision of the branch: http://gitorious.org/~ezyang/sup/ezyang/commits/maildir-sync
    737 which solves some bad interactions with sync-back.
    738 
    739 Edward
    740 
    741 From alvherre@alvh.no-ip.org  Tue Sep  4 18:09:54 2012
    742 From: alvherre@alvh.no-ip.org (Alvaro Herrera)
    743 Date: Tue, 04 Sep 2012 15:09:54 -0300
    744 Subject: [sup-devel] [PATCH] Inotify support for Maildirs. (FIRST DRAFT)
    745 In-Reply-To: <1346774717-sup-3579@javelin>
    746 References: <1345564795-sup-3898@alvh.no-ip.org>
    747 	<1346648371-12305-1-git-send-email-ezyang@mit.edu>
    748 	<1346648395-sup-4546@javelin> <1346688000-sup-4643@alvh.no-ip.org>
    749 	<1346688387-sup-5906@alvh.no-ip.org>
    750 	<20120903121013.8cpe3jr5wg0kw88k@webmail.mit.edu>
    751 	<1346695742-sup-7650@javelin> <1346714661-sup-2638@javelin>
    752 	<1346774717-sup-3579@javelin>
    753 Message-ID: <1346781893-sup-7790@alvh.no-ip.org>
    754 
    755 Excerpts from Edward Z. Yang's message of mar sep 04 13:06:07 -0300 2012:
    756 > I've pushed a new revision of the branch: http://gitorious.org/~ezyang/sup/ezyang/commits/maildir-sync
    757 > which solves some bad interactions with sync-back.
    758 
    759 Okay, I'm running this now.  Having the emails show up instantly works
    760 great ... hooray for continuous, permanent distraction ;-)
    761 
    762 I haven't debugged the other issue yet.  I'll let you know.
    763 
    764 -- 
    765 ?lvaro Herrera <alvherre at alvh.no-ip.org>
    766 
    767 From matthieu.rakotojaona@gmail.com  Mon Sep 24 22:13:57 2012
    768 From: matthieu.rakotojaona@gmail.com (Matthieu Rakotojaona)
    769 Date: Tue, 25 Sep 2012 00:13:57 +0200
    770 Subject: [sup-devel] Update in heliotrope !
    771 Message-ID: <CAMiZLn3ipaH=j+-hZBmnb-XhFc1QwQFG7J-Mqw2oHmvuuKRKzw@mail.gmail.com>
    772 
    773 Hello everyone,
    774 
    775 I have updated my version of heliotrope on my imap branch
    776 (https://github.com/rakoo/heliotrope/tree/imap). You just have to
    777 clone it and checkout the branch.
    778 You may now synchronize your mails between heliotrope and a maildir
    779 with offlineimap. I tested most of  the situations that can happen
    780 when using it (Adding/removing a label on heliotrope side,
    781 adding/removing a mail on maildir side)
    782 
    783 A few notes :
    784 
    785 * It is not "multi-write-safe", in the sense that you should stop all
    786 other activity with heliotrope while you sync with offlinimap.
    787 Concurrent access will have unknown outcome.
    788 * To continue on this point, use offlineimap with the `-1` flag, which
    789 will turn off multithreading. It should work without it though,
    790 because AFAIK offlineimap threads do not share mailboxes. But I'm not
    791 sure.
    792 * There are a few hardcoded labels/dir that you should find in your
    793 maildir, such as INBOX/inbox and "All Mail", which has no equivalent
    794 in heliotrope. Fear not, this one is taken care of.
    795 * Syncing with another IMAP server should also work,because it is
    796 transparent to offlineimap (at least from the outside)
    797 
    798 Have fun !
    799 
    800 -- 
    801 Matthieu RAKOTOJAONA
    802 
    803 From sdothum@gmail.com  Tue Sep 25 19:15:01 2012
    804 From: sdothum@gmail.com (Steven Hum)
    805 Date: Tue, 25 Sep 2012 15:15:01 -0400
    806 Subject: [sup-devel] Update in heliotrope !
    807 In-Reply-To: <CAMiZLn3ipaH=j+-hZBmnb-XhFc1QwQFG7J-Mqw2oHmvuuKRKzw@mail.gmail.com>
    808 References: <CAMiZLn3ipaH=j+-hZBmnb-XhFc1QwQFG7J-Mqw2oHmvuuKRKzw@mail.gmail.com>
    809 Message-ID: <1348599178-sup-12@luna>
    810 
    811 Hello Matthieu,
    812 
    813 I am able to initialize the heliotrope mailstore directory easy enough 
    814 with
    815 
    816 ruby -Ilib bin/heliotrope-server -d <mailstore>
    817 
    818 (stuff gets created :-) but when I try to import with
    819 
    820 ruby -Ilib bin/heliotrope-import -a <maildir> -d <mailstore>
    821 
    822 <maildir> being the dir that contains the cur/new/tmp directories that 
    823 contain the actual emails downloaded by offlineimap,
    824 
    825 the import process hangs after the "Adding mail..." message and doesn't 
    826 load any emails (I just terminate the process with a ctrl-c). What am 
    827 I missing? I tried using heliotrope-add with the server running, but it 
    828 hung at the same point (not surpisingly).
    829 
    830 Secondly, your note "stop all other activity with heliotrope while you 
    831 sync with offlinimap". Does that mean heliotrope-server should not be 
    832 running if offlineimap is (updating the maildirs)?
    833 
    834 Thanks,
    835 Steven
    836 
    837 > 
    838 Excerpts from Matthieu Rakotojaona's message of 2012-09-24 18:13:57 -0400:
    839 > Hello everyone,
    840 > 
    841 > I have updated my version of heliotrope on my imap branch
    842 > (https://github.com/rakoo/heliotrope/tree/imap). You just have to
    843 > clone it and checkout the branch.
    844 > You may now synchronize your mails between heliotrope and a maildir
    845 > with offlineimap. I tested most of  the situations that can happen
    846 > when using it (Adding/removing a label on heliotrope side,
    847 > adding/removing a mail on maildir side)
    848 > 
    849 > A few notes :
    850 > 
    851 > * It is not "multi-write-safe", in the sense that you should stop all
    852 > other activity with heliotrope while you sync with offlinimap.
    853 > Concurrent access will have unknown outcome.
    854 > * To continue on this point, use offlineimap with the `-1` flag, which
    855 > will turn off multithreading. It should work without it though,
    856 > because AFAIK offlineimap threads do not share mailboxes. But I'm not
    857 > sure.
    858 > * There are a few hardcoded labels/dir that you should find in your
    859 > maildir, such as INBOX/inbox and "All Mail", which has no equivalent
    860 > in heliotrope. Fear not, this one is taken care of.
    861 > * Syncing with another IMAP server should also work,because it is
    862 > transparent to offlineimap (at least from the outside)
    863 > 
    864 > Have fun !
    865 > 
    866 -- 
    867 "Truth or die."
    868 
    869 Steven Hum  
    870 5 - 28 Gilmour St  
    871 Ottawa, ON K2P 0N3  
    872 email sdothum at gmail.com  
    873 tel 613.237.9058  
    874 
    875 From matthieu.rakotojaona@gmail.com  Tue Sep 25 19:44:49 2012
    876 From: matthieu.rakotojaona@gmail.com (Matthieu Rakotojaona)
    877 Date: Tue, 25 Sep 2012 21:44:49 +0200
    878 Subject: [sup-devel] Update in heliotrope !
    879 In-Reply-To: <1348599178-sup-12@luna>
    880 References: <CAMiZLn3ipaH=j+-hZBmnb-XhFc1QwQFG7J-Mqw2oHmvuuKRKzw@mail.gmail.com>
    881 	<1348599178-sup-12@luna>
    882 Message-ID: <CAMiZLn1LnxLAYmunVP+E5TP2OVATjrMH3=VWF7_EU_qnqCZq=w@mail.gmail.com>
    883 
    884 On Tue, Sep 25, 2012 at 9:15 PM, Steven Hum <sdothum at gmail.com> wrote:
    885 > Hello Matthieu,
    886 
    887 Hello,
    888 
    889 > I am able to initialize the heliotrope mailstore directory easy enough
    890 > with
    891 >
    892 > ruby -Ilib bin/heliotrope-server -d <mailstore>
    893 >
    894 > (stuff gets created :-) but when I try to import with
    895 >
    896 > ruby -Ilib bin/heliotrope-import -a <maildir> -d <mailstore>
    897 
    898 Did you even read the README ? =]
    899 
    900 It clearly says : "To bulk import mail, use heltrope-import. You must
    901 stop your server first."
    902 
    903 The thing is, both heliotrope-server and heliotrope-import will try to
    904 access the same resources, which are not designed for
    905 multi-concurrency. So, you have to shut heliotrope-server down before
    906 using heliotrope-import.
    907 
    908 > I tried using heliotrope-add with the server running, but it
    909 > hung at the same point (not surpisingly).
    910 
    911 This is more problematic, because it should work. Can you share some
    912 of your logs ?
    913 
    914 > Secondly, your note "stop all other activity with heliotrope while you
    915 > sync with offlinimap". Does that mean heliotrope-server should not be
    916 > running if offlineimap is (updating the maildirs)?
    917 
    918 No, on the contrary, it means that while heliotrope-server is running,
    919 you should either access it with a traditional http client, or with an
    920 IMAP client, but you should never do both the same time.
    921 But you will need heliotrope-server running for accessing IMAP.
    922 
    923 -- 
    924 Matthieu RAKOTOJAONA
    925 
    926 From sdothum@gmail.com  Tue Sep 25 21:01:45 2012
    927 From: sdothum@gmail.com (Steven Hum)
    928 Date: Tue, 25 Sep 2012 17:01:45 -0400
    929 Subject: [sup-devel] Update in heliotrope !
    930 In-Reply-To: <CAMiZLn1LnxLAYmunVP+E5TP2OVATjrMH3=VWF7_EU_qnqCZq=w@mail.gmail.com>
    931 References: <CAMiZLn3ipaH=j+-hZBmnb-XhFc1QwQFG7J-Mqw2oHmvuuKRKzw@mail.gmail.com>
    932 	<1348599178-sup-12@luna>
    933 	<CAMiZLn1LnxLAYmunVP+E5TP2OVATjrMH3=VWF7_EU_qnqCZq=w@mail.gmail.com>
    934 Message-ID: <1348605639-sup-455@luna>
    935 
    936 Yes! :-) (though I did try the import with the server running after the 
    937 persistent hanging without.. just in case the documentation wasn't 
    938 correct!).
    939 
    940 As nothing appeared to be working, I purged the heliotrope mailstore and 
    941 performed 2 actions: 1) the initial server startup to initialize the 
    942 mailstore and associated files, and 2) an import of a small maildir 
    943 (after stopping the server). I presume the logs you are referring to is 
    944 the LOG file in the <mailstore>/store dir.
    945 
    946 LOG from step 1) (or LOG.old after completing both steps)
    947 
    948 2012/09/25-16:42:13.770269 7ff2345bc700 Delete type=3 #1
    949 
    950 LOG from attempted import, step 2)
    951 
    952 2012/09/25-16:43:12.450537 7f0c45ff8700 Recovering log #3
    953 2012/09/25-16:43:12.450585 7f0c45ff8700 Level-0 table #5: started
    954 2012/09/25-16:43:12.458919 7f0c45ff8700 Level-0 table #5: 344 bytes OK
    955 2012/09/25-16:43:12.460026 7f0c45ff8700 Delete type=0 #3
    956 2012/09/25-16:43:12.460050 7f0c45ff8700 Delete type=3 #2
    957 
    958 Oh, the second thing you should know is that I run on debian (test 
    959 / wheezy) and have been using rvm 1.9.3. I know that the debian 
    960 environment doesn'd always make it easy for ruby installs at times. In 
    961 addition to the gems requirements listed in the README, I had to install 
    962 the mail and thin gems to get heliotrope to run.
    963 
    964 Regards,
    965 Steven
    966 
    967 
    968 Excerpts from Matthieu Rakotojaona's message of 2012-09-25 15:44:49 -0400:
    969 > On Tue, Sep 25, 2012 at 9:15 PM, Steven Hum <sdothum at gmail.com> wrote:
    970 > > Hello Matthieu,
    971 > 
    972 > Hello,
    973 > 
    974 > > I am able to initialize the heliotrope mailstore directory easy enough
    975 > > with
    976 > >
    977 > > ruby -Ilib bin/heliotrope-server -d <mailstore>
    978 > >
    979 > > (stuff gets created :-) but when I try to import with
    980 > >
    981 > > ruby -Ilib bin/heliotrope-import -a <maildir> -d <mailstore>
    982 > 
    983 > Did you even read the README ? =]
    984 > 
    985 > It clearly says : "To bulk import mail, use heltrope-import. You must
    986 > stop your server first."
    987 > 
    988 > The thing is, both heliotrope-server and heliotrope-import will try to
    989 > access the same resources, which are not designed for
    990 > multi-concurrency. So, you have to shut heliotrope-server down before
    991 > using heliotrope-import.
    992 > 
    993 > > I tried using heliotrope-add with the server running, but it
    994 > > hung at the same point (not surpisingly).
    995 > 
    996 > This is more problematic, because it should work. Can you share some
    997 > of your logs ?
    998 > 
    999 > > Secondly, your note "stop all other activity with heliotrope while you
   1000 > > sync with offlinimap". Does that mean heliotrope-server should not be
   1001 > > running if offlineimap is (updating the maildirs)?
   1002 > 
   1003 > No, on the contrary, it means that while heliotrope-server is running,
   1004 > you should either access it with a traditional http client, or with an
   1005 > IMAP client, but you should never do both the same time.
   1006 > But you will need heliotrope-server running for accessing IMAP.
   1007 > 
   1008 -- 
   1009 "Truth or die."
   1010 
   1011 Steven Hum  
   1012 5 - 28 Gilmour St  
   1013 Ottawa, ON K2P 0N3  
   1014 email sdothum at gmail.com  
   1015 tel 613.237.9058  
   1016 
   1017 From matthieu.rakotojaona@gmail.com  Thu Sep 27 15:50:40 2012
   1018 From: matthieu.rakotojaona@gmail.com (Matthieu Rakotojaona)
   1019 Date: Thu, 27 Sep 2012 17:50:40 +0200
   1020 Subject: [sup-devel] Update in heliotrope !
   1021 In-Reply-To: <1348605639-sup-455@luna>
   1022 References: <CAMiZLn3ipaH=j+-hZBmnb-XhFc1QwQFG7J-Mqw2oHmvuuKRKzw@mail.gmail.com>
   1023 	<1348599178-sup-12@luna>
   1024 	<CAMiZLn1LnxLAYmunVP+E5TP2OVATjrMH3=VWF7_EU_qnqCZq=w@mail.gmail.com>
   1025 	<1348605639-sup-455@luna>
   1026 Message-ID: <CAMiZLn2uzVmoSLvjjxR0M+uxSHPNXmgyMD7G+9g5aeCp7r+hmQ@mail.gmail.com>
   1027 
   1028 Sorry for the delay.
   1029 
   1030 On Tue, Sep 25, 2012 at 11:01 PM, Steven Hum <sdothum at gmail.com> wrote:
   1031 > I presume the logs you are referring to is
   1032 > the LOG file in the <mailstore>/store dir.
   1033 
   1034 Hmm no, I would like to see the heliotrope-server logs, or the
   1035 heliotrope-import logs. These could help solve the problem.
   1036 
   1037 > Oh, the second thing you should know is that I run on debian (test
   1038 > / wheezy) and have been using rvm 1.9.3. I know that the debian
   1039 > environment doesn'd always make it easy for ruby installs at times. In
   1040 > addition to the gems requirements listed in the README, I had to install
   1041 > the mail and thin gems to get heliotrope to run.
   1042 
   1043 Sorry, this is totally undocumented, but yes, you have to install the
   1044 mail gem. I decided to part from rmail, since it isn't maintained
   1045 anymore. Thin was just there to improve performance and is optional. I
   1046 just found the HTTP part stable enough fore my needs, so I changed to
   1047 it (And effectively saw some improvements)
   1048 
   1049 -- 
   1050 Matthieu RAKOTOJAONA
   1051 
   1052 From sdothum@gmail.com  Thu Sep 27 16:46:01 2012
   1053 From: sdothum@gmail.com (Steven Hum)
   1054 Date: Thu, 27 Sep 2012 12:46:01 -0400
   1055 Subject: [sup-devel] Update in heliotrope !
   1056 In-Reply-To: <CAMiZLn2uzVmoSLvjjxR0M+uxSHPNXmgyMD7G+9g5aeCp7r+hmQ@mail.gmail.com>
   1057 References: <CAMiZLn3ipaH=j+-hZBmnb-XhFc1QwQFG7J-Mqw2oHmvuuKRKzw@mail.gmail.com>
   1058 	<1348599178-sup-12@luna>
   1059 	<CAMiZLn1LnxLAYmunVP+E5TP2OVATjrMH3=VWF7_EU_qnqCZq=w@mail.gmail.com>
   1060 	<1348605639-sup-455@luna>
   1061 	<CAMiZLn2uzVmoSLvjjxR0M+uxSHPNXmgyMD7G+9g5aeCp7r+hmQ@mail.gmail.com>
   1062 Message-ID: <1348763216-sup-299@luna>
   1063 
   1064 I couldn't find any physical log files anywhere (at least any text files 
   1065 with an obvious name...). The only other file in the heliotrope 
   1066 mailstore directory I found was a 32 byte binary file named "000022.log"
   1067 
   1068 Where should these be located? Looks like there could be something 
   1069 fundamentally broken on my system if these log files aren't even being 
   1070 written!
   1071 
   1072 Regards,
   1073 Steven
   1074 
   1075 Excerpts from Matthieu Rakotojaona's message of 2012-09-27 11:50:40 -0400:
   1076 > Sorry for the delay.
   1077 > 
   1078 > On Tue, Sep 25, 2012 at 11:01 PM, Steven Hum <sdothum at gmail.com> wrote:
   1079 > > I presume the logs you are referring to is
   1080 > > the LOG file in the <mailstore>/store dir.
   1081 > 
   1082 > Hmm no, I would like to see the heliotrope-server logs, or the
   1083 > heliotrope-import logs. These could help solve the problem.
   1084 > 
   1085 > > Oh, the second thing you should know is that I run on debian (test
   1086 > > / wheezy) and have been using rvm 1.9.3. I know that the debian
   1087 > > environment doesn'd always make it easy for ruby installs at times. In
   1088 > > addition to the gems requirements listed in the README, I had to install
   1089 > > the mail and thin gems to get heliotrope to run.
   1090 > 
   1091 > Sorry, this is totally undocumented, but yes, you have to install the
   1092 > mail gem. I decided to part from rmail, since it isn't maintained
   1093 > anymore. Thin was just there to improve performance and is optional. I
   1094 > just found the HTTP part stable enough fore my needs, so I changed to
   1095 > it (And effectively saw some improvements)
   1096 > 
   1097 -- 
   1098 "Truth or die."
   1099 
   1100 Steven Hum  
   1101 5 - 28 Gilmour St  
   1102 Ottawa, ON K2P 0N3  
   1103 email sdothum at gmail.com  
   1104 tel 613.237.9058  
   1105