Subject: Re: [sup-talk] Fwd: Re: Crash while in thread-view-mode

From: Ben Gamari <[email-masked]>
Date: Sun, 11 Oct 2009 17:04:53 -0400

Excerpts from William Morgan's message of Sun Oct 11 16:28:48 -0400 2009:
> Reformatted excerpts from Ben Gamari's message of 2009-10-07:
> > I understand that designing software around a contingency like this
> > might not be the best practice, but the frequency with which I've
> > needed to rebuild really does make me think that ruby isn't the best
> > language for the indexer.
> 
> The indexer isn't in Ruby, it's written in C++ in the case of Xapian and
> C in the case of Ferret.

Sorry, I was referring to the mail indexer (i.e. message,
mbox/maildir parser), not the backend indexing engine (e.g. Xapian).
Should have been more specific. 

> 
> > This is easily the fifth time I've needed to rebuild and each time it
> > has taken over 30 minutes for 1.5 GB of mail. That's substantially
> > less than 1MB/second for what should be an I/O bound operation. Ouch.
> 
> I think this isn't the indexer's fault so much as the mbox parsing,
> which is Ruby.

Exactly. This is where I think C++ is probably appropriate.
> 
> I'm sorry you've had to rebuild the index so many times. The Xapian side
> of things is very new, and I think you've had a run of bad luck. But I
> am personally not motivated to improve index time performance, because
> that's not a common event. At least, it shouldn't be.

Completely understandable. I really don't have a right to complain. It
does work a large majority of the time, after all. Just figured I'd let
you know of problems as they happen. Thanks for the awesome client.

- Ben
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Ben Gamari <[email-masked]>
Date: Tue, 13 Oct 2009 16:18:44 -0400

> that's not a common event. At least, it shouldn't be.

Unfortunately, it seems that the crashes have returned (In fact, they
returned only hours after I rebuilt). Interestingly, it is once again my
LKML label that is the culprit.

On the note of crashing, is it really necessary for the "wrong id called
on nil" exception to be fatal? Couldn't the client at least make some
attempt at recovering (skipping the problematic thread while catching
and logging the exception)? It seems like these issues could be rather
tricky to sort out (I've put an hour or so into tracking down the source
of the nils to no avail), and with a dynamically typed language like
Ruby, could occur at any time. Considering this, it seems like it might
be a good idea to handle these failures a bit more gracefully. Would
this be problematic?

Regardless, it might be a good idea to have a hierarchy of exception
classes instead of just throwing Strings. It seems that throwing
specialized classes would make the above substantially easier.

Thanks,

- Ben

--- RuntimeError from thread: load threads for thread-index-mode
wrong id called on nil
/opt/exp/sup/lib/sup.rb:17:in `id'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'
/usr/lib64/ruby/1.8/rubygems/custom_require.rb:31:in `sort_by'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `each'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `sort_by'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `synchronize'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `update'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:628:in `__unprotected_load_n_threads'
/opt/exp/sup/lib/sup/thread.rb:334:in `load_n_threads'
/opt/exp/sup/lib/sup/xapian_index.rb:117:in `each_id_by_date'
/opt/exp/sup/lib/sup/xapian_index.rb:110:in `each_id'
/opt/exp/sup/lib/sup/xapian_index.rb:110:in `each'
/opt/exp/sup/lib/sup/xapian_index.rb:110:in `each_id'
/opt/exp/sup/lib/sup/xapian_index.rb:117:in `each_id_by_date'
/opt/exp/sup/lib/sup/thread.rb:328:in `load_n_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:625:in `__unprotected_load_n_threads'
(eval):12:in `load_n_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:609:in `load_n_threads_background'
/opt/exp/sup/lib/sup.rb:77:in `reporting_thread'
/opt/exp/sup/lib/sup.rb:75:in `initialize'
/opt/exp/sup/lib/sup.rb:75:in `new'
/opt/exp/sup/lib/sup.rb:75:in `reporting_thread'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:608:in `load_n_threads_background'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:679:in `__unprotected_load_threads'
(eval):12:in `load_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:81:in `initialize'
/opt/exp/sup/lib/sup/modes/line-cursor-mode.rb:22:in `call'
/opt/exp/sup/lib/sup/modes/line-cursor-mode.rb:22:in `initialize'
/opt/exp/sup/lib/sup/modes/line-cursor-mode.rb:22:in `each'
/opt/exp/sup/lib/sup/modes/line-cursor-mode.rb:22:in `initialize'
/opt/exp/sup/lib/sup/modes/line-cursor-mode.rb:19:in `new'
/opt/exp/sup/lib/sup/modes/line-cursor-mode.rb:19:in `initialize'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:54:in `initialize'
/opt/exp/sup/lib/sup/modes/label-search-results-mode.rb:9:in `initialize'
/opt/exp/sup/lib/sup/modes/label-search-results-mode.rb:32:in `new'
/opt/exp/sup/lib/sup/modes/label-search-results-mode.rb:32:in `spawn_nicely'
/opt/exp/sup/lib/sup/buffer.rb:350:in `spawn_unless_exists'
/opt/exp/sup/lib/sup/util.rb:520:in `send'
/opt/exp/sup/lib/sup/util.rb:520:in `method_missing'
/opt/exp/sup/lib/sup/modes/label-search-results-mode.rb:32:in `spawn_nicely'
/opt/exp/sup/lib/sup/modes/label-list-mode.rb:134:in `select_label'
/opt/exp/sup/lib/sup/mode.rb:51:in `send'
/opt/exp/sup/lib/sup/mode.rb:51:in `handle_input'
/opt/exp/sup/lib/sup/buffer.rb:267:in `handle_input'
/usr/bin/sup:245
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: William Morgan <[email-masked]>
Date: Sun, 11 Oct 2009 13:28:48 -0700

-- 
William <[email-masked]>

---

From: Ben Gamari <[email-masked]>
Date: Wed, 07 Oct 2009 18:42:09 -0400

Excerpts from Guillaume Quintard's message of Wed Oct 07 04:48:56 -0400 2009:
> On Wed, Oct 7, 2009 at 8:44 AM, Rich Lane <[email-masked]> wrote:
> > You could try running sup with the -n flag to disable threading. The
> > major downside is that you have to hit P to poll manually.
> 
> That "solves" the problem for me.
> 

After going through the process of rebuilding my index (again), things
seem to be more stable (for now). I understand that designing software around a
contingency like this might not be the best practice, but the frequency
with which I've needed to rebuild really does make me think that ruby
isn't the best language for the indexer.

This is easily the fifth time I've needed to rebuild and each time it
has taken over 30 minutes for 1.5 GB of mail. That's substantially less than
1MB/second for what should be an I/O bound operation. Ouch. It'll be
really interesting to see how Carl's work comes out.

- Ben