Archive of RubyForge sup-talk mailing list
 help / color / mirror / Atom feed

* [sup-talk] Crash while scrolling
@ 2009-09-11 16:58 Ben Gamari
  2009-09-12 16:35 ` William Morgan
  2009-10-04 19:59 ` Guillaume Quintard
  0 siblings, 2 replies; 14+ messages in thread
From: Ben Gamari @ 2009-09-11 16:58 UTC (permalink / raw)


Recently I've been seeing this crash[1] pretty consistently on the next
branch with the Xapian backend. All I need to do to reproduce the crash
is move to the last thread in the thread-index-mode and attempt to move
to the next thread.

I can also produce a very similar crash[2] by attempting to load all
threads. Thanks,

- Ben


[1] Crash while scrolling:

--- RuntimeError from thread: load threads for thread-index-mode
wrong id called on nil
/opt/exp/sup/lib/sup.rb:17:in `id'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'
/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `sort_by'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `each'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `sort_by'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `synchronize'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `update'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:637:in `__unprotected_load_n_threads'
(eval):12:in `load_n_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:609:in `load_n_threads_background'
/opt/exp/sup/lib/sup.rb:77:in `reporting_thread'
/opt/exp/sup/lib/sup.rb:75:in `initialize'
/opt/exp/sup/lib/sup.rb:75:in `new'
/opt/exp/sup/lib/sup.rb:75:in `reporting_thread'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:608:in `load_n_threads_background'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:679:in `__unprotected_load_threads'
(eval):12:in `load_threads'
/opt/exp/sup/lib/sup/mode.rb:51:in `send'
/opt/exp/sup/lib/sup/mode.rb:51:in `handle_input'
/opt/exp/sup/lib/sup/buffer.rb:267:in `handle_input'
/usr/bin/sup:238


[2] Crash after attempting to load all threads
--- RuntimeError from thread: load threads for thread-index-mode
wrong id called on nil
/opt/exp/sup/lib/sup.rb:17:in `id'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'
/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `sort_by'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `each'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `sort_by'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `synchronize'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `update'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:628:in `__unprotected_load_n_threads'
/opt/exp/sup/lib/sup/thread.rb:334:in `load_n_threads'
/opt/exp/sup/lib/sup/xapian_index.rb:151:in `each_id_by_date'
/opt/exp/sup/lib/sup/xapian_index.rb:144:in `each_id'
/opt/exp/sup/lib/sup/xapian_index.rb:144:in `each'
/opt/exp/sup/lib/sup/xapian_index.rb:144:in `each_id'
/opt/exp/sup/lib/sup/xapian_index.rb:151:in `each_id_by_date'
/opt/exp/sup/lib/sup/thread.rb:328:in `load_n_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:625:in `__unprotected_load_n_threads'
(eval):12:in `load_n_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:609:in `load_n_threads_background'
/opt/exp/sup/lib/sup.rb:77:in `reporting_thread'
/opt/exp/sup/lib/sup.rb:75:in `initialize'
/opt/exp/sup/lib/sup.rb:75:in `new'
/opt/exp/sup/lib/sup.rb:75:in `reporting_thread'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:608:in `load_n_threads_background'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:679:in `__unprotected_load_threads'
(eval):12:in `load_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:658:in `load_all_threads'
/opt/exp/sup/lib/sup/mode.rb:51:in `send'
/opt/exp/sup/lib/sup/mode.rb:51:in `handle_input'
/opt/exp/sup/lib/sup/buffer.rb:267:in `handle_input'
/usr/bin/sup:238


^ permalink raw reply	[flat|nested] 14+ messages in thread

• * [sup-talk] Crash while scrolling
    2009-09-11 16:58 [sup-talk] Crash while scrolling Ben Gamari
  @ 2009-09-12 16:35 ` William Morgan
    2009-09-13 15:02   ` Ben Gamari
    2009-09-16 17:23   ` Ben Gamari
    2009-10-04 19:59 ` Guillaume Quintard
    1 sibling, 2 replies; 14+ messages in thread
  From: William Morgan @ 2009-09-12 16:35 UTC (permalink / raw)
  
  
  Reformatted excerpts from Ben Gamari's message of 2009-09-11:
  > Recently I've been seeing this crash[1] pretty consistently on the next
  > branch with the Xapian backend.
  
  Is your Xapian index somewhat old? There have been index format changes
  that have caused this type of thing recently. You might try rebuilding
  it.
  -- 
  William <wmorgan-sup at masanjin.net>
  
  
  ^ permalink raw reply	[flat|nested] 14+ messages in thread

• • * [sup-talk] Crash while scrolling
      2009-09-12 16:35 ` William Morgan
    @ 2009-09-13 15:02   ` Ben Gamari
      2009-09-16 17:23   ` Ben Gamari
      1 sibling, 0 replies; 14+ messages in thread
    From: Ben Gamari @ 2009-09-13 15:02 UTC (permalink / raw)
    
    
    On Sat, Sep 12, 2009 at 09:35:21AM -0700, William Morgan wrote:
    > Reformatted excerpts from Ben Gamari's message of 2009-09-11:
    > > Recently I've been seeing this crash[1] pretty consistently on the next
    > > branch with the Xapian backend.
    > 
    > Is your Xapian index somewhat old? There have been index format changes
    > that have caused this type of thing recently. You might try rebuilding
    > it.
    
    I wish I could say it was unfortunately I just rebuilt it two days ago,
    suspecting that might be part of the issue. I'll try again, just to make
    sure but I'm fairly certain the index is up-to-date.
    
    - Ben
    
    
    
    ^ permalink raw reply	[flat|nested] 14+ messages in thread

  • * [sup-talk] Crash while scrolling
      2009-09-12 16:35 ` William Morgan
      2009-09-13 15:02   ` Ben Gamari
    @ 2009-09-16 17:23   ` Ben Gamari
      2009-09-26 14:31     ` William Morgan
      2009-09-30 21:21     ` [sup-talk] Crash while scrolling William Morgan
      1 sibling, 2 replies; 14+ messages in thread
    From: Ben Gamari @ 2009-09-16 17:23 UTC (permalink / raw)
    
    
    On Sat, Sep 12, 2009 at 09:35:21AM -0700, William Morgan wrote:
    > Reformatted excerpts from Ben Gamari's message of 2009-09-11:
    > > Recently I've been seeing this crash[1] pretty consistently on the next
    > > branch with the Xapian backend.
    > 
    > Is your Xapian index somewhat old? There have been index format changes
    > that have caused this type of thing recently. You might try rebuilding
    > it.
    
    Well, after several attempts at rebuilding my index, I finally got lucky
    and sup-sync ran to completion. Unfortunately, sup now fails to even
    start, failing out with the following exception while loading threads
    for inbox-mode,
    
    	$ sup
    	--- SystemExit from thread: main
    	Thread#join: deadlock 0x7f592eec4d78 - mutual join(0x7f592f01d6e8)
    	/opt/exp/sup/lib/sup/message.rb:240:in `select'
    	/opt/exp/sup/lib/sup/buffer.rb:35:in `nonblocking_getch'
    	/usr/bin/sup:213
    
    However, at this point during debugging I happened to pipe stderr to a
    file and accidentally found that most of the backtrace was actually
    being hidden by curses.  Examining the full output on stderr reveals,
    
    	/usr/bin/sup:213(eval):3:in `synchronize': Thread#join: deadlock 0x7f4d5da565c0 - mutual join(0x7f4d5daf8be0) (ThreadError)
    		from (eval):3:in `raw_header'
    		from /opt/exp/sup/lib/sup/imap.rb:98:in `load_header'
    		from /opt/exp/sup/lib/sup/util.rb:560:in `send'
    		from /opt/exp/sup/lib/sup/util.rb:560:in `__pass'
    		from /opt/exp/sup/lib/sup/util.rb:547:in `method_missing'
    		from /opt/exp/sup/lib/sup/message.rb:238:in `load_from_source!'
    		from /opt/exp/sup/lib/sup/message.rb:219:in `chunks'
    		from /opt/exp/sup/lib/sup/message.rb:164:in `snippet'
    		from /opt/exp/sup/lib/sup/thread.rb:91:in `snippet'
    		from /opt/exp/sup/lib/sup/imap.rb:259:in `select'
    		from /opt/exp/sup/lib/sup/thread.rb:68:in `each'
    		from /opt/exp/sup/lib/sup/thread.rb:168:in `each_with_stuff'
    		from /opt/exp/sup/lib/sup/thread.rb:67:in `each'
    		from /opt/exp/sup/lib/sup/thread.rb:91:in `select'
    		from /opt/exp/sup/lib/sup/thread.rb:91:in `snippet'
    		from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:840:in `text_for_thread_at'
    		from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:758:in `regen_text'
    		from /opt/exp/sup/lib/sup/util.rb:364:in `map_with_index'
    		from /opt/exp/sup/lib/sup/imap.rb:259:in `each_with_index'
    		from /opt/exp/sup/lib/sup/util.rb:364:in `each'
    		from /opt/exp/sup/lib/sup/util.rb:364:in `each_with_index'
    		from /opt/exp/sup/lib/sup/util.rb:364:in `map_with_index'
    		from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:758:in `regen_text'
    		from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:687:in `resize'
    		from /opt/exp/sup/lib/sup/buffer.rb:88:in `resize'
    		from /opt/exp/sup/lib/sup/buffer.rb:329:in `draw_screen'
    		from /opt/exp/sup/lib/sup/buffer.rb:745:in `clear'
    		from /opt/exp/sup/lib/sup/util.rb:520:in `send'
    		from /opt/exp/sup/lib/sup/util.rb:520:in `method_missing'
    		from /opt/exp/sup/lib/sup/imap.rb:283:in `shutup'
    		from /opt/exp/sup/lib/sup/imap.rb:270:in `unsafe_connect'
    		from /opt/exp/sup/lib/sup/imap.rb:244:in `initialize'
    		from /opt/exp/sup/lib/sup/imap.rb:244:in `new'
    		from /opt/exp/sup/lib/sup/imap.rb:244:in `unsafe_connect'
    		from /opt/exp/sup/lib/sup/imap.rb:331:in `safely'
    		from /opt/exp/sup/lib/sup/imap.rb:148:in `unsynchronized_connect'
    		from (eval):3:in `connect'
    		from (eval):3:in `synchronize'
    		from (eval):3:in `connect'
    		from /opt/exp/sup/lib/sup/util.rb:560:in `send'
    		from /opt/exp/sup/lib/sup/util.rb:560:in `__pass'
    		from /opt/exp/sup/lib/sup/util.rb:547:in `method_missing'
    		from /usr/bin/sup:189
    		from /opt/exp/sup/lib/sup.rb:77:in `reporting_thread'
    		from /opt/exp/sup/lib/sup.rb:75:in `initialize'
    		from /opt/exp/sup/lib/sup.rb:75:in `new'
    		from /opt/exp/sup/lib/sup.rb:75:in `reporting_thread'
    		from /usr/bin/sup:187
    		from /usr/bin/sup:185:in `each'
    		from /usr/bin/sup:185
    		[Wed Sep 16 13:01:11 -0400 2009] ERROR: oh crap, an exception
    		----------------------------------------------------------------
    		I'm very sorry. It seems that an error occurred in Sup. Please
    		accept my sincere apologies. If you don't mind, please send the
    		contents of ~/.sup/exception-log.txt and a brief report of the
    		circumstances to sup-talk at rubyforge dot orgs so that I might
    		address this problem. Thank you!
    
    		Sincerely,
    		William
    		----------------------------------------------------------------
    
    
    with the first stacktrace following this on stdout. With this
    information it looks like this bug should become much more debuggable.
    Being in the imap module, it seems likely that it is my gmail source
    that is causing the failure. Unfortunately I have no way to verify that
    the problem is limited to the gmail source as sup raises as exception
    when it encounters an unknown source, precluding the option of simply
    commenting out the source in sources.yaml.
    
    Anyways, this is the current status of things on my machine. William, do
    you have any ideas what might cause such a backtrace. At this point
    classes have started and I really have no more time to devote to
    getting sup working. If there isn't a fairly simple solution here I guess
    I'll just need to stick with mutt (ugh). Anyways, thanks for your work.
    
    Cheers,
    - Ben
    
    
    
    ^ permalink raw reply	[flat|nested] 14+ messages in thread

  • • * [sup-talk] Crash while scrolling
        2009-09-16 17:23   ` Ben Gamari
      @ 2009-09-26 14:31     ` William Morgan
        2009-09-28 18:10       ` Ben Gamari
        2009-09-30 21:21     ` [sup-talk] Crash while scrolling William Morgan
        1 sibling, 1 reply; 14+ messages in thread
      From: William Morgan @ 2009-09-26 14:31 UTC (permalink / raw)
      
      
      Sorry it's taken me so long to get back to this.
      
      Reformatted excerpts from Ben Gamari's message of 2009-09-16:
      >     Thread#join: deadlock 0x7f592eec4d78 - mutual join(0x7f592f01d6e8)
      >     /opt/exp/sup/lib/sup/message.rb:240:in `select'
      >     /opt/exp/sup/lib/sup/buffer.rb:35:in `nonblocking_getch'
      >     /usr/bin/sup:213
      > 
      > However, at this point during debugging I happened to pipe stderr to a
      > file and accidentally found that most of the backtrace was actually
      > being hidden by curses.  Examining the full output on stderr reveals,
      > 
      >     /usr/bin/sup:213(eval):3:in `synchronize': Thread#join: deadlock
      > 0x7f4d5da565c0 - mutual join(0x7f4d5daf8be0) (ThreadError)
      >         from (eval):3:in `raw_header'
      >         from /opt/exp/sup/lib/sup/imap.rb:98:in `load_header'
      
      I definitely am not sure why there's a deadlock happening, but I am
      guessing, based on the line numbers in that first exception, that it's
      being triggered by an underlying problem with the source. If you run
      sup with -n, does that change anything? (Or produce a better exception?)
      
      > Anyways, this is the current status of things on my machine. William, do
      > you have any ideas what might cause such a backtrace. At this point
      > classes have started and I really have no more time to devote to
      > getting sup working. If there isn't a fairly simple solution here I guess
      > I'll just need to stick with mutt (ugh). Anyways, thanks for your work.
      
      Well, there's a workaround, which is to switch over to an offlineimap
      setup where your Gmail is mirrored locally and Sup reads the mirror
      (which is what you want anyways if you're serious about using Sup with
      an IMAP client). I don't know what's causing the deadlock, but I will
      try and reproduce it on my end.
      -- 
      William <wmorgan-sup at masanjin.net>
      
      
      ^ permalink raw reply	[flat|nested] 14+ messages in thread

    • • * [sup-talk] Crash while scrolling
          2009-09-26 14:31     ` William Morgan
        @ 2009-09-28 18:10       ` Ben Gamari
          2009-10-01 13:43         ` William Morgan
          0 siblings, 1 reply; 14+ messages in thread
        From: Ben Gamari @ 2009-09-28 18:10 UTC (permalink / raw)
        
        
        Excerpts from William Morgan's message of Sat Sep 26 10:31:56 -0400 2009:
        > Sorry it's taken me so long to get back to this.
        
        Quite alright. I know we're all busy.
        
        > 
        > I definitely am not sure why there's a deadlock happening, but I am
        > guessing, based on the line numbers in that first exception, that it's
        > being triggered by an underlying problem with the source. If you run
        > sup with -n, does that change anything? (Or produce a better exception?)
        
        Well, things definitely failed a bit differently. I tried this a few
        times and after seeing no real improvement, I moved on to your
        suggestion below. 
        
        > 
        > > Anyways, this is the current status of things on my machine. William, do
        > > you have any ideas what might cause such a backtrace. At this point
        > > classes have started and I really have no more time to devote to
        > > getting sup working. If there isn't a fairly simple solution here I guess
        > > I'll just need to stick with mutt (ugh). Anyways, thanks for your work.
        > 
        > Well, there's a workaround, which is to switch over to an offlineimap
        > setup where your Gmail is mirrored locally and Sup reads the mirror
        > (which is what you want anyways if you're serious about using Sup with
        > an IMAP client). I don't know what's causing the deadlock, but I will
        > try and reproduce it on my end.
        
        This is definitely what I should have done from the beginning. Given
        that sup seems to work fine after removing my imap sources, it seems
        that that might be the source of the above deadlock. I think I'll just
        stick with offlineimap for now.
        
        This does raise one important question, however. It seems that
        offlineimap will delete messages if they are found to have been deleted
        in the remote respository. Is there any way that you know of to disable
        this behavior, such that it will only drop new messages into the local
        repository, thus making it somewhat compatible with sup's add-only
        source requirement? It is awfully nice to be able to use GMail's web
        interface when I'm away from my computer, but needing to re-sync sup
        afterwards becomes quite tiresome.
        
        This actually brings up a larger question. How difficult would it be to
        relax sup's assumption that sources are add-only? This seems like a
        horribly restriction to have in an email program. I understand that in
        the case of sources such as imap where there is no globally unique
        message identifier trying to track message changes could be quite
        difficult, but for local sources (especially maildirs) it seems like
        this should be quite possible.
        
        Anyways, thanks a ton for your work. Now since I finally have sup up and
        running reliably, it's been a joy to use. Leaps and bounds above my
        former mutt-based system.
        
        - Ben
        
        
        ^ permalink raw reply	[flat|nested] 14+ messages in thread

      • • * [sup-talk] Crash while scrolling
            2009-09-28 18:10       ` Ben Gamari
          @ 2009-10-01 13:43         ` William Morgan
            2009-10-01 17:44           ` Ben Gamari
            0 siblings, 1 reply; 14+ messages in thread
          From: William Morgan @ 2009-10-01 13:43 UTC (permalink / raw)
          
          
          Reformatted excerpts from Ben Gamari's message of 2009-09-28:
          > This does raise one important question, however. It seems that
          > offlineimap will delete messages if they are found to have been
          > deleted in the remote respository. Is there any way that you know of
          > to disable this behavior, such that it will only drop new messages
          > into the local repository, thus making it somewhat compatible with
          > sup's add-only source requirement?
          
          Maybe someone who uses offlineimap can comment on this. In the worst
          case, I'm sure the patch wouldn't be too hard.
          
          > This actually brings up a larger question. How difficult would it be
          > to relax sup's assumption that sources are add-only?
          
          It's not difficult per se, it just requires scanning over the entire
          source, which is slow. Removing this assumption would be tantamount to
          running sup-sync -c every time you start up sup.
          
          Here's the idea: scanning over a mailstore is slow. Much of this
          slowness is due to Ruby. So let's rewrite this code in C. Then we would
          have something as fast as, say, Mutt. But Mutt bogs down on my mbox file
          because it's too big. So my *only* reasonable choice with a large
          mailstore is Sup and the assumption that the source is add only.
          -- 
          William <wmorgan-sup at masanjin.net>
          
          
          ^ permalink raw reply	[flat|nested] 14+ messages in thread

        • • * [sup-talk] Crash while scrolling
              2009-10-01 13:43         ` William Morgan
            @ 2009-10-01 17:44           ` Ben Gamari
              2009-10-01 18:31             ` William Morgan
              2009-10-01 18:41             ` [sup-talk] Writing time-sensitive portions of sup in C Carl Worth
              0 siblings, 2 replies; 14+ messages in thread
            From: Ben Gamari @ 2009-10-01 17:44 UTC (permalink / raw)
            
            
            Excerpts from William Morgan's message of Thu Oct 01 09:43:48 -0400 2009:
            > Reformatted excerpts from Ben Gamari's message of 2009-09-28:
            > > This actually brings up a larger question. How difficult would it be
            > > to relax sup's assumption that sources are add-only?
            > 
            > It's not difficult per se, it just requires scanning over the entire
            > source, which is slow. Removing this assumption would be tantamount to
            > running sup-sync -c every time you start up sup.
            > 
            > Here's the idea: scanning over a mailstore is slow. Much of this
            > slowness is due to Ruby. So let's rewrite this code in C. Then we would
            > have something as fast as, say, Mutt. But Mutt bogs down on my mbox file
            > because it's too big. So my *only* reasonable choice with a large
            > mailstore is Sup and the assumption that the source is add only.
            
            It seems that C would definitely be a good start (or perhaps C++ would
            be a better idea as that is the language in which Xapian is written).
            However, I think one of the real issues is the exclusive nature of index
            access.  In fact, this is one of my primary gripes with the sup
            workflow. After processing a large number of messages, the write-out
            time can be quite substantial upon killing the buffer. This can be a
            noticeable interruption to workflow. It seems to me that index access
            should be asynchronous at least.
            
            If this were the case, then we could get support for mutable sources for
            free, as we could synchronize against sources without interrupting
            workflow (although keeping the view in sync with the backend would be a
            bit tricky).
            
            As an aside, it would be quite nice if one could run multiple
            simultaneous instances of sup. It seems that if one only held write access
            to the index during writes (is this the case presently?), there should
            be nothing preventing this from being possible.
            
            Correct me if I'm wrong in any part of my above assessment. Hope things
            are going well,
            
            - Ben
            
            
            ^ permalink raw reply	[flat|nested] 14+ messages in thread

          • • * [sup-talk] Crash while scrolling
                2009-10-01 17:44           ` Ben Gamari
              @ 2009-10-01 18:31             ` William Morgan
                2009-10-01 19:01               ` Ben Gamari
                2009-10-01 18:41             ` [sup-talk] Writing time-sensitive portions of sup in C Carl Worth
                1 sibling, 1 reply; 14+ messages in thread
              From: William Morgan @ 2009-10-01 18:31 UTC (permalink / raw)
              
              
              Reformatted excerpts from Ben Gamari's message of 2009-10-01:
              > It seems that C would definitely be a good start (or perhaps C++ would
              > be a better idea as that is the language in which Xapian is written).
              
              I was proposing that as a strawman argument. C does not solve my
              problem.
              
              > However, I think one of the real issues is the exclusive nature of
              > index access.  In fact, this is one of my primary gripes with the sup
              > workflow. After processing a large number of messages, the write-out
              > time can be quite substantial upon killing the buffer.
              
              It is possible to address this in a variety of ways, many of which have
              been discussed over the years, but yes, I agree that a delay is
              nonideal.
              
              > If this were the case, then we could get support for mutable sources
              > for free, as we could synchronize against sources without interrupting
              > workflow (although keeping the view in sync with the backend would be
              > a bit tricky).
              
              Making message state saving fast, or backgrounded, is a different beast
              from scanning over a mailstore.
              
              I have been working on a Sup server for quite some time that would
              address many of these problems, but progress is slow.
              
              > As an aside, it would be quite nice if one could run multiple
              > simultaneous instances of sup. It seems that if one only held write
              > access to the index during writes (is this the case presently?), there
              > should be nothing preventing this from being possible.
              
              It might be possible to do this with Xapian, especially if there were no
              expectation of updates being transmitted across processes. With Ferret,
              if it is possible, it's only with a tremendous amount of effort.
              
              -- 
              William <wmorgan-sup at masanjin.net>
              
              
              ^ permalink raw reply	[flat|nested] 14+ messages in thread

            • • * [sup-talk] Crash while scrolling
                  2009-10-01 18:31             ` William Morgan
                @ 2009-10-01 19:01               ` Ben Gamari
                  2009-10-11 20:31                 ` William Morgan
                  0 siblings, 1 reply; 14+ messages in thread
                From: Ben Gamari @ 2009-10-01 19:01 UTC (permalink / raw)
                
                
                Excerpts from William Morgan's message of Thu Oct 01 14:31:20 -0400 2009:
                > Reformatted excerpts from Ben Gamari's message of 2009-10-01:
                > > It seems that C would definitely be a good start (or perhaps C++ would
                > > be a better idea as that is the language in which Xapian is written).
                > 
                > I was proposing that as a strawman argument. C does not solve my
                > problem.
                
                Certainly, you will never be able to write a message indexer fast enough
                to index a source instantly. That is why we have an index to begin with.
                That being said, I think we can do substantially better than we are
                currently doing in a lower-level language. With mutt, I can come close
                to saturating my drive bandwidth while loading a folder. While
                synchronizing in sup, I am lucky to get a few hundred kilobytes/second.
                Certainly a large amount of that difference has to do with the amount of
                processing done by each, but even adjusting for this it seems to me that
                we should still be I/O bound (or at least close to it).
                
                > 
                > > However, I think one of the real issues is the exclusive nature of
                > > index access.  In fact, this is one of my primary gripes with the sup
                > > workflow. After processing a large number of messages, the write-out
                > > time can be quite substantial upon killing the buffer.
                > 
                > It is possible to address this in a variety of ways, many of which have
                > been discussed over the years, but yes, I agree that a delay is
                > nonideal.
                
                Glad we agree.
                
                > 
                > Making message state saving fast, or backgrounded, is a different beast
                > from scanning over a mailstore.
                
                If we are unable to update the index while the client is active,
                rescanning sources would destroy usability. I would argue that
                asynchronous writeout is very much a prerequisite for mutable sources.
                
                > 
                > I have been working on a Sup server for quite some time that would
                > address many of these problems, but progress is slow.
                
                This was largely what I had in mind. It seems like moving index
                manipulation out-of-process might be best, ultimately.
                
                > 
                > > As an aside, it would be quite nice if one could run multiple
                > > simultaneous instances of sup. It seems that if one only held write
                > > access to the index during writes (is this the case presently?), there
                > > should be nothing preventing this from being possible.
                > 
                > It might be possible to do this with Xapian, especially if there were no
                > expectation of updates being transmitted across processes.
                
                I think initially no updates between processes would be fine.
                > With Ferret,
                > if it is possible, it's only with a tremendous amount of effort.
                
                Is ferret even going to be supported after the Xapian backend
                stabilizes?
                
                Thanks for the comments and sup.
                
                - Ben
                
                
                ^ permalink raw reply	[flat|nested] 14+ messages in thread

              • • * Re: [sup-talk] Crash while scrolling
                    2009-10-01 19:01               ` Ben Gamari
                  @ 2009-10-11 20:31                 ` William Morgan
                    0 siblings, 0 replies; 14+ messages in thread
                  From: William Morgan @ 2009-10-11 20:31 UTC (permalink / raw)
                    To: sup-talk
                  
                  Reformatted excerpts from Ben Gamari's message of 2009-10-01:
                  > Is ferret even going to be supported after the Xapian backend
                  > stabilizes?
                  
                  No, I'm going to drop it at some point. I barely have enough time or
                  energy for Sup's current set of problems as is. :)
                  -- 
                  William <wmorgan-sup@masanjin.net>
                  _______________________________________________
                  sup-talk mailing list
                  sup-talk@rubyforge.org
                  http://rubyforge.org/mailman/listinfo/sup-talk
                  
                  
                  ^ permalink raw reply	[flat|nested] 14+ messages in thread

            • * [sup-talk] Writing time-sensitive portions of sup in C
                2009-10-01 17:44           ` Ben Gamari
                2009-10-01 18:31             ` William Morgan
              @ 2009-10-01 18:41             ` Carl Worth
                1 sibling, 0 replies; 14+ messages in thread
              From: Carl Worth @ 2009-10-01 18:41 UTC (permalink / raw)
              
              
              Excerpts from Ben Gamari's message of Thu Oct 01 10:44:19 -0700 2009:
              > Excerpts from William Morgan's message of Thu Oct 01 09:43:48 -0400 2009:
              > > Here's the idea: scanning over a mailstore is slow. Much of this
              > > slowness is due to Ruby. So let's rewrite this code in C. Then we would
              > > have something as fast as, say, Mutt. But Mutt bogs down on my mbox file
              > > because it's too big. So my *only* reasonable choice with a large
              > > mailstore is Sup and the assumption that the source is add only.
              > 
              > It seems that C would definitely be a good start (or perhaps C++ would
              > be a better idea as that is the language in which Xapian is written).
              
              I've started some experiments along these lines, (basically just
              writing C code (with C++ where necessary) using the Xapian tutorial
              and little peeks into sup/xapian-index.rb to get the right prefix
              values, etc.).
              
              > However, I think one of the real issues is the exclusive nature of index
              > access.  In fact, this is one of my primary gripes with the sup
              > workflow. After processing a large number of messages, the write-out
              > time can be quite substantial upon killing the buffer. This can be a
              > noticeable interruption to workflow. It seems to me that index access
              > should be asynchronous at least.
              
              What I'm hoping to end up with is a C library that provides
              asynchronous access to a sup-compatible index. I'll keep you posted if
              I make any actual progress on this front.
              
              -Carl
              -------------- next part --------------
              A non-text attachment was scrubbed...
              Name: signature.asc
              Type: application/pgp-signature
              Size: 190 bytes
              Desc: not available
              URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20091001/89e0a628/attachment.bin>
              
              
              ^ permalink raw reply	[flat|nested] 14+ messages in thread

    • * [sup-talk] Crash while scrolling
        2009-09-16 17:23   ` Ben Gamari
        2009-09-26 14:31     ` William Morgan
      @ 2009-09-30 21:21     ` William Morgan
        1 sibling, 0 replies; 14+ messages in thread
      From: William Morgan @ 2009-09-30 21:21 UTC (permalink / raw)
      
      
      Reformatted excerpts from Ben Gamari's message of 2009-09-16:
      > Well, after several attempts at rebuilding my index, I finally got
      > lucky and sup-sync ran to completion. Unfortunately, sup now fails to
      > even start, failing out with the following exception while loading
      > threads for inbox-mode,
      
      Hey, I just started seeing this too, Xapian only. I think I've fixed in
      master.
      -- 
      William <wmorgan-sup at masanjin.net>
      
      
      ^ permalink raw reply	[flat|nested] 14+ messages in thread

• * Re: [sup-talk] Crash while scrolling
    2009-09-11 16:58 [sup-talk] Crash while scrolling Ben Gamari
    2009-09-12 16:35 ` William Morgan
  @ 2009-10-04 19:59 ` Guillaume Quintard
    1 sibling, 0 replies; 14+ messages in thread
  From: Guillaume Quintard @ 2009-10-04 19:59 UTC (permalink / raw)
    To: sup-talk
  
  On Fri, Sep 11, 2009 at 6:58 PM, Ben Gamari <bgamari.foss@gmail.com> wrote:
  
  > I can also produce a very similar crash[2] by attempting to load all
  > threads. Thanks,
  
  I get the same thing when loading all the threads. Index has just been
  rebuilt, and I'm using an mbox file.
  
  -- 
  Guillaume
  _______________________________________________
  sup-talk mailing list
  sup-talk@rubyforge.org
  http://rubyforge.org/mailman/listinfo/sup-talk
  
  
  ^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2009-10-11 20:32 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-09-11 16:58 [sup-talk] Crash while scrolling Ben Gamari
2009-09-12 16:35 ` William Morgan
2009-09-13 15:02   ` Ben Gamari
2009-09-16 17:23   ` Ben Gamari
2009-09-26 14:31     ` William Morgan
2009-09-28 18:10       ` Ben Gamari
2009-10-01 13:43         ` William Morgan
2009-10-01 17:44           ` Ben Gamari
2009-10-01 18:31             ` William Morgan
2009-10-01 19:01               ` Ben Gamari
2009-10-11 20:31                 ` William Morgan
2009-10-01 18:41             ` [sup-talk] Writing time-sensitive portions of sup in C Carl Worth
2009-09-30 21:21     ` [sup-talk] Crash while scrolling William Morgan
2009-10-04 19:59 ` Guillaume Quintard

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox