Subject: [sup-talk] Crash while scrolling

From: [email-masked] (Ben Gamari)
Date: Fri, 11 Sep 2009 12:58:31 -0400

Recently I've been seeing this crash[1] pretty consistently on the next
branch with the Xapian backend. All I need to do to reproduce the crash
is move to the last thread in the thread-index-mode and attempt to move
to the next thread.

I can also produce a very similar crash[2] by attempting to load all
threads. Thanks,

- Ben

[1] Crash while scrolling:

--- RuntimeError from thread: load threads for thread-index-mode
wrong id called on nil
/opt/exp/sup/lib/sup.rb:17:in `id'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'
/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `sort_by'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `each'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `sort_by'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `synchronize'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `update'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:637:in `__unprotected_load_n_threads'
(eval):12:in `load_n_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:609:in `load_n_threads_background'
/opt/exp/sup/lib/sup.rb:77:in `reporting_thread'
/opt/exp/sup/lib/sup.rb:75:in `initialize'
/opt/exp/sup/lib/sup.rb:75:in `new'
/opt/exp/sup/lib/sup.rb:75:in `reporting_thread'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:608:in `load_n_threads_background'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:679:in `__unprotected_load_threads'
(eval):12:in `load_threads'
/opt/exp/sup/lib/sup/mode.rb:51:in `send'
/opt/exp/sup/lib/sup/mode.rb:51:in `handle_input'
/opt/exp/sup/lib/sup/buffer.rb:267:in `handle_input'
/usr/bin/sup:238

[2] Crash after attempting to load all threads
--- RuntimeError from thread: load threads for thread-index-mode
wrong id called on nil
/opt/exp/sup/lib/sup.rb:17:in `id'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'
/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `sort_by'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `each'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `sort_by'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `synchronize'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `update'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:628:in `__unprotected_load_n_threads'
/opt/exp/sup/lib/sup/thread.rb:334:in `load_n_threads'
/opt/exp/sup/lib/sup/xapian_index.rb:151:in `each_id_by_date'
/opt/exp/sup/lib/sup/xapian_index.rb:144:in `each_id'
/opt/exp/sup/lib/sup/xapian_index.rb:144:in `each'
/opt/exp/sup/lib/sup/xapian_index.rb:144:in `each_id'
/opt/exp/sup/lib/sup/xapian_index.rb:151:in `each_id_by_date'
/opt/exp/sup/lib/sup/thread.rb:328:in `load_n_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:625:in `__unprotected_load_n_threads'
(eval):12:in `load_n_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:609:in `load_n_threads_background'
/opt/exp/sup/lib/sup.rb:77:in `reporting_thread'
/opt/exp/sup/lib/sup.rb:75:in `initialize'
/opt/exp/sup/lib/sup.rb:75:in `new'
/opt/exp/sup/lib/sup.rb:75:in `reporting_thread'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:608:in `load_n_threads_background'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:679:in `__unprotected_load_threads'
(eval):12:in `load_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:658:in `load_all_threads'
/opt/exp/sup/lib/sup/mode.rb:51:in `send'
/opt/exp/sup/lib/sup/mode.rb:51:in `handle_input'
/opt/exp/sup/lib/sup/buffer.rb:267:in `handle_input'
/usr/bin/sup:238

---

From: [email-masked] (William Morgan)
Date: Sat, 12 Sep 2009 09:35:21 -0700

Reformatted excerpts from Ben Gamari's message of 2009-09-11:
> branch with the Xapian backend.

Is your Xapian index somewhat old? There have been index format changes
that have caused this type of thing recently. You might try rebuilding
it.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Ben Gamari)
Date: Sun, 13 Sep 2009 11:02:32 -0400

> that have caused this type of thing recently. You might try rebuilding

I wish I could say it was unfortunately I just rebuilt it two days ago,
suspecting that might be part of the issue. I'll try again, just to make
sure but I'm fairly certain the index is up-to-date.

- Ben

---

From: [email-masked] (Ben Gamari)
Date: Wed, 16 Sep 2009 13:23:40 -0400

> that have caused this type of thing recently. You might try rebuilding

Well, after several attempts at rebuilding my index, I finally got lucky
and sup-sync ran to completion. Unfortunately, sup now fails to even
start, failing out with the following exception while loading threads
for inbox-mode,

	$ sup
	--- SystemExit from thread: main
	Thread#join: deadlock 0x7f592eec4d78 - mutual join(0x7f592f01d6e8)
	/opt/exp/sup/lib/sup/message.rb:240:in `select'
	/opt/exp/sup/lib/sup/buffer.rb:35:in `nonblocking_getch'
	/usr/bin/sup:213

However, at this point during debugging I happened to pipe stderr to a
file and accidentally found that most of the backtrace was actually
being hidden by curses.  Examining the full output on stderr reveals,

	/usr/bin/sup:213(eval):3:in `synchronize': Thread#join: deadlock 0x7f4d5da565c0 - mutual join(0x7f4d5daf8be0) (ThreadError)
		from (eval):3:in `raw_header'
		from /opt/exp/sup/lib/sup/imap.rb:98:in `load_header'
		from /opt/exp/sup/lib/sup/util.rb:560:in `send'
		from /opt/exp/sup/lib/sup/util.rb:560:in `__pass'
		from /opt/exp/sup/lib/sup/util.rb:547:in `method_missing'
		from /opt/exp/sup/lib/sup/message.rb:238:in `load_from_source!'
		from /opt/exp/sup/lib/sup/message.rb:219:in `chunks'
		from /opt/exp/sup/lib/sup/message.rb:164:in `snippet'
		from /opt/exp/sup/lib/sup/thread.rb:91:in `snippet'
		from /opt/exp/sup/lib/sup/imap.rb:259:in `select'
		from /opt/exp/sup/lib/sup/thread.rb:68:in `each'
		from /opt/exp/sup/lib/sup/thread.rb:168:in `each_with_stuff'
		from /opt/exp/sup/lib/sup/thread.rb:67:in `each'
		from /opt/exp/sup/lib/sup/thread.rb:91:in `select'
		from /opt/exp/sup/lib/sup/thread.rb:91:in `snippet'
		from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:840:in `text_for_thread_at'
		from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:758:in `regen_text'
		from /opt/exp/sup/lib/sup/util.rb:364:in `map_with_index'
		from /opt/exp/sup/lib/sup/imap.rb:259:in `each_with_index'
		from /opt/exp/sup/lib/sup/util.rb:364:in `each'
		from /opt/exp/sup/lib/sup/util.rb:364:in `each_with_index'
		from /opt/exp/sup/lib/sup/util.rb:364:in `map_with_index'
		from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:758:in `regen_text'
		from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:687:in `resize'
		from /opt/exp/sup/lib/sup/buffer.rb:88:in `resize'
		from /opt/exp/sup/lib/sup/buffer.rb:329:in `draw_screen'
		from /opt/exp/sup/lib/sup/buffer.rb:745:in `clear'
		from /opt/exp/sup/lib/sup/util.rb:520:in `send'
		from /opt/exp/sup/lib/sup/util.rb:520:in `method_missing'
		from /opt/exp/sup/lib/sup/imap.rb:283:in `shutup'
		from /opt/exp/sup/lib/sup/imap.rb:270:in `unsafe_connect'
		from /opt/exp/sup/lib/sup/imap.rb:244:in `initialize'
		from /opt/exp/sup/lib/sup/imap.rb:244:in `new'
		from /opt/exp/sup/lib/sup/imap.rb:244:in `unsafe_connect'
		from /opt/exp/sup/lib/sup/imap.rb:331:in `safely'
		from /opt/exp/sup/lib/sup/imap.rb:148:in `unsynchronized_connect'
		from (eval):3:in `connect'
		from (eval):3:in `synchronize'
		from (eval):3:in `connect'
		from /opt/exp/sup/lib/sup/util.rb:560:in `send'
		from /opt/exp/sup/lib/sup/util.rb:560:in `__pass'
		from /opt/exp/sup/lib/sup/util.rb:547:in `method_missing'
		from /usr/bin/sup:189
		from /opt/exp/sup/lib/sup.rb:77:in `reporting_thread'
		from /opt/exp/sup/lib/sup.rb:75:in `initialize'
		from /opt/exp/sup/lib/sup.rb:75:in `new'
		from /opt/exp/sup/lib/sup.rb:75:in `reporting_thread'
		from /usr/bin/sup:187
		from /usr/bin/sup:185:in `each'
		from /usr/bin/sup:185
		[Wed Sep 16 13:01:11 -0400 2009] ERROR: oh crap, an exception
		----------------------------------------------------------------
		I'm very sorry. It seems that an error occurred in Sup. Please
		accept my sincere apologies. If you don't mind, please send the
		contents of ~/.sup/exception-log.txt and a brief report of the
		circumstances to sup-talk at rubyforge dot orgs so that I might
		address this problem. Thank you!

		Sincerely,
		William
		----------------------------------------------------------------

with the first stacktrace following this on stdout. With this
information it looks like this bug should become much more debuggable.
Being in the imap module, it seems likely that it is my gmail source
that is causing the failure. Unfortunately I have no way to verify that
the problem is limited to the gmail source as sup raises as exception
when it encounters an unknown source, precluding the option of simply
commenting out the source in sources.yaml.

Anyways, this is the current status of things on my machine. William, do
you have any ideas what might cause such a backtrace. At this point
classes have started and I really have no more time to devote to
getting sup working. If there isn't a fairly simple solution here I guess
I'll just need to stick with mutt (ugh). Anyways, thanks for your work.

Cheers,
- Ben

---

From: [email-masked] (William Morgan)
Date: Sat, 26 Sep 2009 07:31:56 -0700

Sorry it's taken me so long to get back to this.

Reformatted excerpts from Ben Gamari's message of 2009-09-16:
>     /usr/bin/sup:213(eval):3:in `synchronize': Thread#join: deadlock
> 0x7f4d5da565c0 - mutual join(0x7f4d5daf8be0) (ThreadError)

I definitely am not sure why there's a deadlock happening, but I am
guessing, based on the line numbers in that first exception, that it's
being triggered by an underlying problem with the source. If you run
sup with -n, does that change anything? (Or produce a better exception?)

> I'll just need to stick with mutt (ugh). Anyways, thanks for your work.

Well, there's a workaround, which is to switch over to an offlineimap
setup where your Gmail is mirrored locally and Sup reads the mirror
(which is what you want anyways if you're serious about using Sup with
an IMAP client). I don't know what's causing the deadlock, but I will
try and reproduce it on my end.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Ben Gamari)
Date: Mon, 28 Sep 2009 14:10:06 -0400

> Sorry it's taken me so long to get back to this.

Quite alright. I know we're all busy.

> sup with -n, does that change anything? (Or produce a better exception?)

Well, things definitely failed a bit differently. I tried this a few
times and after seeing no real improvement, I moved on to your
suggestion below. 

> try and reproduce it on my end.

This is definitely what I should have done from the beginning. Given
that sup seems to work fine after removing my imap sources, it seems
that that might be the source of the above deadlock. I think I'll just
stick with offlineimap for now.

This does raise one important question, however. It seems that
offlineimap will delete messages if they are found to have been deleted
in the remote respository. Is there any way that you know of to disable
this behavior, such that it will only drop new messages into the local
repository, thus making it somewhat compatible with sup's add-only
source requirement? It is awfully nice to be able to use GMail's web
interface when I'm away from my computer, but needing to re-sync sup
afterwards becomes quite tiresome.

This actually brings up a larger question. How difficult would it be to
relax sup's assumption that sources are add-only? This seems like a
horribly restriction to have in an email program. I understand that in
the case of sources such as imap where there is no globally unique
message identifier trying to track message changes could be quite
difficult, but for local sources (especially maildirs) it seems like
this should be quite possible.

Anyways, thanks a ton for your work. Now since I finally have sup up and
running reliably, it's been a joy to use. Leaps and bounds above my
former mutt-based system.

- Ben

---

From: [email-masked] (William Morgan)
Date: Wed, 30 Sep 2009 14:21:01 -0700

Reformatted excerpts from Ben Gamari's message of 2009-09-16:
> Well, after several attempts at rebuilding my index, I finally got
> lucky and sup-sync ran to completion. Unfortunately, sup now fails to
> even start, failing out with the following exception while loading
> threads for inbox-mode,

Hey, I just started seeing this too, Xapian only. I think I've fixed in
master.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (William Morgan)
Date: Thu, 01 Oct 2009 06:43:48 -0700

Reformatted excerpts from Ben Gamari's message of 2009-09-28:
> offlineimap will delete messages if they are found to have been
> deleted in the remote respository. Is there any way that you know of
> to disable this behavior, such that it will only drop new messages
> into the local repository, thus making it somewhat compatible with
> sup's add-only source requirement?

Maybe someone who uses offlineimap can comment on this. In the worst
case, I'm sure the patch wouldn't be too hard.

> This actually brings up a larger question. How difficult would it be
> to relax sup's assumption that sources are add-only?

It's not difficult per se, it just requires scanning over the entire
source, which is slow. Removing this assumption would be tantamount to
running sup-sync -c every time you start up sup.

Here's the idea: scanning over a mailstore is slow. Much of this
slowness is due to Ruby. So let's rewrite this code in C. Then we would
have something as fast as, say, Mutt. But Mutt bogs down on my mbox file
because it's too big. So my *only* reasonable choice with a large
mailstore is Sup and the assumption that the source is add only.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Ben Gamari)
Date: Thu, 01 Oct 2009 13:44:19 -0400

> mailstore is Sup and the assumption that the source is add only.

It seems that C would definitely be a good start (or perhaps C++ would
be a better idea as that is the language in which Xapian is written).
However, I think one of the real issues is the exclusive nature of index
access.  In fact, this is one of my primary gripes with the sup
workflow. After processing a large number of messages, the write-out
time can be quite substantial upon killing the buffer. This can be a
noticeable interruption to workflow. It seems to me that index access
should be asynchronous at least.

If this were the case, then we could get support for mutable sources for
free, as we could synchronize against sources without interrupting
workflow (although keeping the view in sync with the backend would be a
bit tricky).

As an aside, it would be quite nice if one could run multiple
simultaneous instances of sup. It seems that if one only held write access
to the index during writes (is this the case presently?), there should
be nothing preventing this from being possible.

Correct me if I'm wrong in any part of my above assessment. Hope things
are going well,

- Ben

---

From: [email-masked] (William Morgan)
Date: Thu, 01 Oct 2009 11:31:20 -0700

> be a better idea as that is the language in which Xapian is written).

I was proposing that as a strawman argument. C does not solve my
problem.

> However, I think one of the real issues is the exclusive nature of
> index access.  In fact, this is one of my primary gripes with the sup
> time can be quite substantial upon killing the buffer.

It is possible to address this in a variety of ways, many of which have
been discussed over the years, but yes, I agree that a delay is
nonideal.

> If this were the case, then we could get support for mutable sources
> for free, as we could synchronize against sources without interrupting
> workflow (although keeping the view in sync with the backend would be
> a bit tricky).

Making message state saving fast, or backgrounded, is a different beast
from scanning over a mailstore.

I have been working on a Sup server for quite some time that would
address many of these problems, but progress is slow.

> simultaneous instances of sup. It seems that if one only held write
> access to the index during writes (is this the case presently?), there
> should be nothing preventing this from being possible.

It might be possible to do this with Xapian, especially if there were no
expectation of updates being transmitted across processes. With Ferret,
if it is possible, it's only with a tremendous amount of effort.

---

From: [email-masked] (Carl Worth)
Date: Thu, 01 Oct 2009 11:41:59 -0700

> be a better idea as that is the language in which Xapian is written).

I've started some experiments along these lines, (basically just
writing C code (with C++ where necessary) using the Xapian tutorial
and little peeks into sup/xapian-index.rb to get the right prefix
values, etc.).

> should be asynchronous at least.

What I'm hoping to end up with is a C library that provides
asynchronous access to a sup-compatible index. I'll keep you posted if
I make any actual progress on this front.

-Carl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20091001/89e0a628/attachment.bin>

---

From: [email-masked] (Ben Gamari)
Date: Thu, 01 Oct 2009 15:01:35 -0400

> I was proposing that as a strawman argument. C does not solve my

Certainly, you will never be able to write a message indexer fast enough
to index a source instantly. That is why we have an index to begin with.
That being said, I think we can do substantially better than we are
currently doing in a lower-level language. With mutt, I can come close
to saturating my drive bandwidth while loading a folder. While
synchronizing in sup, I am lucky to get a few hundred kilobytes/second.
Certainly a large amount of that difference has to do with the amount of
processing done by each, but even adjusting for this it seems to me that
we should still be I/O bound (or at least close to it).

> been discussed over the years, but yes, I agree that a delay is

Glad we agree.

> from scanning over a mailstore.

If we are unable to update the index while the client is active,
rescanning sources would destroy usability. I would argue that
asynchronous writeout is very much a prerequisite for mutable sources.

> address many of these problems, but progress is slow.

This was largely what I had in mind. It seems like moving index
manipulation out-of-process might be best, ultimately.

> expectation of updates being transmitted across processes.

I think initially no updates between processes would be fine.
> With Ferret,

Is ferret even going to be supported after the Xapian backend
stabilizes?

Thanks for the comments and sup.

- Ben