* [sup-talk] Crash while scrolling
@ 2009-09-11 16:58 Ben Gamari
2009-09-12 16:35 ` William Morgan
2009-10-04 19:59 ` Guillaume Quintard
0 siblings, 2 replies; 14+ messages in thread
From: Ben Gamari @ 2009-09-11 16:58 UTC (permalink / raw)
Recently I've been seeing this crash[1] pretty consistently on the next
branch with the Xapian backend. All I need to do to reproduce the crash
is move to the last thread in the thread-index-mode and attempt to move
to the next thread.
I can also produce a very similar crash[2] by attempting to load all
threads. Thanks,
- Ben
[1] Crash while scrolling:
--- RuntimeError from thread: load threads for thread-index-mode
wrong id called on nil
/opt/exp/sup/lib/sup.rb:17:in `id'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'
/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `sort_by'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `each'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `sort_by'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `synchronize'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `update'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:637:in `__unprotected_load_n_threads'
(eval):12:in `load_n_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:609:in `load_n_threads_background'
/opt/exp/sup/lib/sup.rb:77:in `reporting_thread'
/opt/exp/sup/lib/sup.rb:75:in `initialize'
/opt/exp/sup/lib/sup.rb:75:in `new'
/opt/exp/sup/lib/sup.rb:75:in `reporting_thread'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:608:in `load_n_threads_background'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:679:in `__unprotected_load_threads'
(eval):12:in `load_threads'
/opt/exp/sup/lib/sup/mode.rb:51:in `send'
/opt/exp/sup/lib/sup/mode.rb:51:in `handle_input'
/opt/exp/sup/lib/sup/buffer.rb:267:in `handle_input'
/usr/bin/sup:238
[2] Crash after attempting to load all threads
--- RuntimeError from thread: load threads for thread-index-mode
wrong id called on nil
/opt/exp/sup/lib/sup.rb:17:in `id'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'
/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `sort_by'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `each'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `sort_by'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `synchronize'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `update'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:628:in `__unprotected_load_n_threads'
/opt/exp/sup/lib/sup/thread.rb:334:in `load_n_threads'
/opt/exp/sup/lib/sup/xapian_index.rb:151:in `each_id_by_date'
/opt/exp/sup/lib/sup/xapian_index.rb:144:in `each_id'
/opt/exp/sup/lib/sup/xapian_index.rb:144:in `each'
/opt/exp/sup/lib/sup/xapian_index.rb:144:in `each_id'
/opt/exp/sup/lib/sup/xapian_index.rb:151:in `each_id_by_date'
/opt/exp/sup/lib/sup/thread.rb:328:in `load_n_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:625:in `__unprotected_load_n_threads'
(eval):12:in `load_n_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:609:in `load_n_threads_background'
/opt/exp/sup/lib/sup.rb:77:in `reporting_thread'
/opt/exp/sup/lib/sup.rb:75:in `initialize'
/opt/exp/sup/lib/sup.rb:75:in `new'
/opt/exp/sup/lib/sup.rb:75:in `reporting_thread'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:608:in `load_n_threads_background'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:679:in `__unprotected_load_threads'
(eval):12:in `load_threads'
/opt/exp/sup/lib/sup/modes/thread-index-mode.rb:658:in `load_all_threads'
/opt/exp/sup/lib/sup/mode.rb:51:in `send'
/opt/exp/sup/lib/sup/mode.rb:51:in `handle_input'
/opt/exp/sup/lib/sup/buffer.rb:267:in `handle_input'
/usr/bin/sup:238
^ permalink raw reply [flat|nested] 14+ messages in thread
* [sup-talk] Crash while scrolling
2009-09-11 16:58 [sup-talk] Crash while scrolling Ben Gamari
@ 2009-09-12 16:35 ` William Morgan
2009-09-13 15:02 ` Ben Gamari
2009-09-16 17:23 ` Ben Gamari
2009-10-04 19:59 ` Guillaume Quintard
1 sibling, 2 replies; 14+ messages in thread
From: William Morgan @ 2009-09-12 16:35 UTC (permalink / raw)
Reformatted excerpts from Ben Gamari's message of 2009-09-11:
> Recently I've been seeing this crash[1] pretty consistently on the next
> branch with the Xapian backend.
Is your Xapian index somewhat old? There have been index format changes
that have caused this type of thing recently. You might try rebuilding
it.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [sup-talk] Crash while scrolling
2009-09-12 16:35 ` William Morgan
@ 2009-09-13 15:02 ` Ben Gamari
2009-09-16 17:23 ` Ben Gamari
1 sibling, 0 replies; 14+ messages in thread
From: Ben Gamari @ 2009-09-13 15:02 UTC (permalink / raw)
On Sat, Sep 12, 2009 at 09:35:21AM -0700, William Morgan wrote:
> Reformatted excerpts from Ben Gamari's message of 2009-09-11:
> > Recently I've been seeing this crash[1] pretty consistently on the next
> > branch with the Xapian backend.
>
> Is your Xapian index somewhat old? There have been index format changes
> that have caused this type of thing recently. You might try rebuilding
> it.
I wish I could say it was unfortunately I just rebuilt it two days ago,
suspecting that might be part of the issue. I'll try again, just to make
sure but I'm fairly certain the index is up-to-date.
- Ben
^ permalink raw reply [flat|nested] 14+ messages in thread
* [sup-talk] Crash while scrolling
2009-09-12 16:35 ` William Morgan
2009-09-13 15:02 ` Ben Gamari
@ 2009-09-16 17:23 ` Ben Gamari
2009-09-26 14:31 ` William Morgan
2009-09-30 21:21 ` [sup-talk] Crash while scrolling William Morgan
1 sibling, 2 replies; 14+ messages in thread
From: Ben Gamari @ 2009-09-16 17:23 UTC (permalink / raw)
On Sat, Sep 12, 2009 at 09:35:21AM -0700, William Morgan wrote:
> Reformatted excerpts from Ben Gamari's message of 2009-09-11:
> > Recently I've been seeing this crash[1] pretty consistently on the next
> > branch with the Xapian backend.
>
> Is your Xapian index somewhat old? There have been index format changes
> that have caused this type of thing recently. You might try rebuilding
> it.
Well, after several attempts at rebuilding my index, I finally got lucky
and sup-sync ran to completion. Unfortunately, sup now fails to even
start, failing out with the following exception while loading threads
for inbox-mode,
$ sup
--- SystemExit from thread: main
Thread#join: deadlock 0x7f592eec4d78 - mutual join(0x7f592f01d6e8)
/opt/exp/sup/lib/sup/message.rb:240:in `select'
/opt/exp/sup/lib/sup/buffer.rb:35:in `nonblocking_getch'
/usr/bin/sup:213
However, at this point during debugging I happened to pipe stderr to a
file and accidentally found that most of the backtrace was actually
being hidden by curses. Examining the full output on stderr reveals,
/usr/bin/sup:213(eval):3:in `synchronize': Thread#join: deadlock 0x7f4d5da565c0 - mutual join(0x7f4d5daf8be0) (ThreadError)
from (eval):3:in `raw_header'
from /opt/exp/sup/lib/sup/imap.rb:98:in `load_header'
from /opt/exp/sup/lib/sup/util.rb:560:in `send'
from /opt/exp/sup/lib/sup/util.rb:560:in `__pass'
from /opt/exp/sup/lib/sup/util.rb:547:in `method_missing'
from /opt/exp/sup/lib/sup/message.rb:238:in `load_from_source!'
from /opt/exp/sup/lib/sup/message.rb:219:in `chunks'
from /opt/exp/sup/lib/sup/message.rb:164:in `snippet'
from /opt/exp/sup/lib/sup/thread.rb:91:in `snippet'
from /opt/exp/sup/lib/sup/imap.rb:259:in `select'
from /opt/exp/sup/lib/sup/thread.rb:68:in `each'
from /opt/exp/sup/lib/sup/thread.rb:168:in `each_with_stuff'
from /opt/exp/sup/lib/sup/thread.rb:67:in `each'
from /opt/exp/sup/lib/sup/thread.rb:91:in `select'
from /opt/exp/sup/lib/sup/thread.rb:91:in `snippet'
from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:840:in `text_for_thread_at'
from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:758:in `regen_text'
from /opt/exp/sup/lib/sup/util.rb:364:in `map_with_index'
from /opt/exp/sup/lib/sup/imap.rb:259:in `each_with_index'
from /opt/exp/sup/lib/sup/util.rb:364:in `each'
from /opt/exp/sup/lib/sup/util.rb:364:in `each_with_index'
from /opt/exp/sup/lib/sup/util.rb:364:in `map_with_index'
from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:758:in `regen_text'
from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:687:in `resize'
from /opt/exp/sup/lib/sup/buffer.rb:88:in `resize'
from /opt/exp/sup/lib/sup/buffer.rb:329:in `draw_screen'
from /opt/exp/sup/lib/sup/buffer.rb:745:in `clear'
from /opt/exp/sup/lib/sup/util.rb:520:in `send'
from /opt/exp/sup/lib/sup/util.rb:520:in `method_missing'
from /opt/exp/sup/lib/sup/imap.rb:283:in `shutup'
from /opt/exp/sup/lib/sup/imap.rb:270:in `unsafe_connect'
from /opt/exp/sup/lib/sup/imap.rb:244:in `initialize'
from /opt/exp/sup/lib/sup/imap.rb:244:in `new'
from /opt/exp/sup/lib/sup/imap.rb:244:in `unsafe_connect'
from /opt/exp/sup/lib/sup/imap.rb:331:in `safely'
from /opt/exp/sup/lib/sup/imap.rb:148:in `unsynchronized_connect'
from (eval):3:in `connect'
from (eval):3:in `synchronize'
from (eval):3:in `connect'
from /opt/exp/sup/lib/sup/util.rb:560:in `send'
from /opt/exp/sup/lib/sup/util.rb:560:in `__pass'
from /opt/exp/sup/lib/sup/util.rb:547:in `method_missing'
from /usr/bin/sup:189
from /opt/exp/sup/lib/sup.rb:77:in `reporting_thread'
from /opt/exp/sup/lib/sup.rb:75:in `initialize'
from /opt/exp/sup/lib/sup.rb:75:in `new'
from /opt/exp/sup/lib/sup.rb:75:in `reporting_thread'
from /usr/bin/sup:187
from /usr/bin/sup:185:in `each'
from /usr/bin/sup:185
[Wed Sep 16 13:01:11 -0400 2009] ERROR: oh crap, an exception
----------------------------------------------------------------
I'm very sorry. It seems that an error occurred in Sup. Please
accept my sincere apologies. If you don't mind, please send the
contents of ~/.sup/exception-log.txt and a brief report of the
circumstances to sup-talk at rubyforge dot orgs so that I might
address this problem. Thank you!
Sincerely,
William
----------------------------------------------------------------
with the first stacktrace following this on stdout. With this
information it looks like this bug should become much more debuggable.
Being in the imap module, it seems likely that it is my gmail source
that is causing the failure. Unfortunately I have no way to verify that
the problem is limited to the gmail source as sup raises as exception
when it encounters an unknown source, precluding the option of simply
commenting out the source in sources.yaml.
Anyways, this is the current status of things on my machine. William, do
you have any ideas what might cause such a backtrace. At this point
classes have started and I really have no more time to devote to
getting sup working. If there isn't a fairly simple solution here I guess
I'll just need to stick with mutt (ugh). Anyways, thanks for your work.
Cheers,
- Ben
^ permalink raw reply [flat|nested] 14+ messages in thread
* [sup-talk] Crash while scrolling
2009-09-16 17:23 ` Ben Gamari
@ 2009-09-26 14:31 ` William Morgan
2009-09-28 18:10 ` Ben Gamari
2009-09-30 21:21 ` [sup-talk] Crash while scrolling William Morgan
1 sibling, 1 reply; 14+ messages in thread
From: William Morgan @ 2009-09-26 14:31 UTC (permalink / raw)
Sorry it's taken me so long to get back to this.
Reformatted excerpts from Ben Gamari's message of 2009-09-16:
> Thread#join: deadlock 0x7f592eec4d78 - mutual join(0x7f592f01d6e8)
> /opt/exp/sup/lib/sup/message.rb:240:in `select'
> /opt/exp/sup/lib/sup/buffer.rb:35:in `nonblocking_getch'
> /usr/bin/sup:213
>
> However, at this point during debugging I happened to pipe stderr to a
> file and accidentally found that most of the backtrace was actually
> being hidden by curses. Examining the full output on stderr reveals,
>
> /usr/bin/sup:213(eval):3:in `synchronize': Thread#join: deadlock
> 0x7f4d5da565c0 - mutual join(0x7f4d5daf8be0) (ThreadError)
> from (eval):3:in `raw_header'
> from /opt/exp/sup/lib/sup/imap.rb:98:in `load_header'
I definitely am not sure why there's a deadlock happening, but I am
guessing, based on the line numbers in that first exception, that it's
being triggered by an underlying problem with the source. If you run
sup with -n, does that change anything? (Or produce a better exception?)
> Anyways, this is the current status of things on my machine. William, do
> you have any ideas what might cause such a backtrace. At this point
> classes have started and I really have no more time to devote to
> getting sup working. If there isn't a fairly simple solution here I guess
> I'll just need to stick with mutt (ugh). Anyways, thanks for your work.
Well, there's a workaround, which is to switch over to an offlineimap
setup where your Gmail is mirrored locally and Sup reads the mirror
(which is what you want anyways if you're serious about using Sup with
an IMAP client). I don't know what's causing the deadlock, but I will
try and reproduce it on my end.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [sup-talk] Crash while scrolling
2009-09-26 14:31 ` William Morgan
@ 2009-09-28 18:10 ` Ben Gamari
2009-10-01 13:43 ` William Morgan
0 siblings, 1 reply; 14+ messages in thread
From: Ben Gamari @ 2009-09-28 18:10 UTC (permalink / raw)
Excerpts from William Morgan's message of Sat Sep 26 10:31:56 -0400 2009:
> Sorry it's taken me so long to get back to this.
Quite alright. I know we're all busy.
>
> I definitely am not sure why there's a deadlock happening, but I am
> guessing, based on the line numbers in that first exception, that it's
> being triggered by an underlying problem with the source. If you run
> sup with -n, does that change anything? (Or produce a better exception?)
Well, things definitely failed a bit differently. I tried this a few
times and after seeing no real improvement, I moved on to your
suggestion below.
>
> > Anyways, this is the current status of things on my machine. William, do
> > you have any ideas what might cause such a backtrace. At this point
> > classes have started and I really have no more time to devote to
> > getting sup working. If there isn't a fairly simple solution here I guess
> > I'll just need to stick with mutt (ugh). Anyways, thanks for your work.
>
> Well, there's a workaround, which is to switch over to an offlineimap
> setup where your Gmail is mirrored locally and Sup reads the mirror
> (which is what you want anyways if you're serious about using Sup with
> an IMAP client). I don't know what's causing the deadlock, but I will
> try and reproduce it on my end.
This is definitely what I should have done from the beginning. Given
that sup seems to work fine after removing my imap sources, it seems
that that might be the source of the above deadlock. I think I'll just
stick with offlineimap for now.
This does raise one important question, however. It seems that
offlineimap will delete messages if they are found to have been deleted
in the remote respository. Is there any way that you know of to disable
this behavior, such that it will only drop new messages into the local
repository, thus making it somewhat compatible with sup's add-only
source requirement? It is awfully nice to be able to use GMail's web
interface when I'm away from my computer, but needing to re-sync sup
afterwards becomes quite tiresome.
This actually brings up a larger question. How difficult would it be to
relax sup's assumption that sources are add-only? This seems like a
horribly restriction to have in an email program. I understand that in
the case of sources such as imap where there is no globally unique
message identifier trying to track message changes could be quite
difficult, but for local sources (especially maildirs) it seems like
this should be quite possible.
Anyways, thanks a ton for your work. Now since I finally have sup up and
running reliably, it's been a joy to use. Leaps and bounds above my
former mutt-based system.
- Ben
^ permalink raw reply [flat|nested] 14+ messages in thread
* [sup-talk] Crash while scrolling
2009-09-16 17:23 ` Ben Gamari
2009-09-26 14:31 ` William Morgan
@ 2009-09-30 21:21 ` William Morgan
1 sibling, 0 replies; 14+ messages in thread
From: William Morgan @ 2009-09-30 21:21 UTC (permalink / raw)
Reformatted excerpts from Ben Gamari's message of 2009-09-16:
> Well, after several attempts at rebuilding my index, I finally got
> lucky and sup-sync ran to completion. Unfortunately, sup now fails to
> even start, failing out with the following exception while loading
> threads for inbox-mode,
Hey, I just started seeing this too, Xapian only. I think I've fixed in
master.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [sup-talk] Crash while scrolling
2009-09-28 18:10 ` Ben Gamari
@ 2009-10-01 13:43 ` William Morgan
2009-10-01 17:44 ` Ben Gamari
0 siblings, 1 reply; 14+ messages in thread
From: William Morgan @ 2009-10-01 13:43 UTC (permalink / raw)
Reformatted excerpts from Ben Gamari's message of 2009-09-28:
> This does raise one important question, however. It seems that
> offlineimap will delete messages if they are found to have been
> deleted in the remote respository. Is there any way that you know of
> to disable this behavior, such that it will only drop new messages
> into the local repository, thus making it somewhat compatible with
> sup's add-only source requirement?
Maybe someone who uses offlineimap can comment on this. In the worst
case, I'm sure the patch wouldn't be too hard.
> This actually brings up a larger question. How difficult would it be
> to relax sup's assumption that sources are add-only?
It's not difficult per se, it just requires scanning over the entire
source, which is slow. Removing this assumption would be tantamount to
running sup-sync -c every time you start up sup.
Here's the idea: scanning over a mailstore is slow. Much of this
slowness is due to Ruby. So let's rewrite this code in C. Then we would
have something as fast as, say, Mutt. But Mutt bogs down on my mbox file
because it's too big. So my *only* reasonable choice with a large
mailstore is Sup and the assumption that the source is add only.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [sup-talk] Crash while scrolling
2009-10-01 13:43 ` William Morgan
@ 2009-10-01 17:44 ` Ben Gamari
2009-10-01 18:31 ` William Morgan
2009-10-01 18:41 ` [sup-talk] Writing time-sensitive portions of sup in C Carl Worth
0 siblings, 2 replies; 14+ messages in thread
From: Ben Gamari @ 2009-10-01 17:44 UTC (permalink / raw)
Excerpts from William Morgan's message of Thu Oct 01 09:43:48 -0400 2009:
> Reformatted excerpts from Ben Gamari's message of 2009-09-28:
> > This actually brings up a larger question. How difficult would it be
> > to relax sup's assumption that sources are add-only?
>
> It's not difficult per se, it just requires scanning over the entire
> source, which is slow. Removing this assumption would be tantamount to
> running sup-sync -c every time you start up sup.
>
> Here's the idea: scanning over a mailstore is slow. Much of this
> slowness is due to Ruby. So let's rewrite this code in C. Then we would
> have something as fast as, say, Mutt. But Mutt bogs down on my mbox file
> because it's too big. So my *only* reasonable choice with a large
> mailstore is Sup and the assumption that the source is add only.
It seems that C would definitely be a good start (or perhaps C++ would
be a better idea as that is the language in which Xapian is written).
However, I think one of the real issues is the exclusive nature of index
access. In fact, this is one of my primary gripes with the sup
workflow. After processing a large number of messages, the write-out
time can be quite substantial upon killing the buffer. This can be a
noticeable interruption to workflow. It seems to me that index access
should be asynchronous at least.
If this were the case, then we could get support for mutable sources for
free, as we could synchronize against sources without interrupting
workflow (although keeping the view in sync with the backend would be a
bit tricky).
As an aside, it would be quite nice if one could run multiple
simultaneous instances of sup. It seems that if one only held write access
to the index during writes (is this the case presently?), there should
be nothing preventing this from being possible.
Correct me if I'm wrong in any part of my above assessment. Hope things
are going well,
- Ben
^ permalink raw reply [flat|nested] 14+ messages in thread
* [sup-talk] Crash while scrolling
2009-10-01 17:44 ` Ben Gamari
@ 2009-10-01 18:31 ` William Morgan
2009-10-01 19:01 ` Ben Gamari
2009-10-01 18:41 ` [sup-talk] Writing time-sensitive portions of sup in C Carl Worth
1 sibling, 1 reply; 14+ messages in thread
From: William Morgan @ 2009-10-01 18:31 UTC (permalink / raw)
Reformatted excerpts from Ben Gamari's message of 2009-10-01:
> It seems that C would definitely be a good start (or perhaps C++ would
> be a better idea as that is the language in which Xapian is written).
I was proposing that as a strawman argument. C does not solve my
problem.
> However, I think one of the real issues is the exclusive nature of
> index access. In fact, this is one of my primary gripes with the sup
> workflow. After processing a large number of messages, the write-out
> time can be quite substantial upon killing the buffer.
It is possible to address this in a variety of ways, many of which have
been discussed over the years, but yes, I agree that a delay is
nonideal.
> If this were the case, then we could get support for mutable sources
> for free, as we could synchronize against sources without interrupting
> workflow (although keeping the view in sync with the backend would be
> a bit tricky).
Making message state saving fast, or backgrounded, is a different beast
from scanning over a mailstore.
I have been working on a Sup server for quite some time that would
address many of these problems, but progress is slow.
> As an aside, it would be quite nice if one could run multiple
> simultaneous instances of sup. It seems that if one only held write
> access to the index during writes (is this the case presently?), there
> should be nothing preventing this from being possible.
It might be possible to do this with Xapian, especially if there were no
expectation of updates being transmitted across processes. With Ferret,
if it is possible, it's only with a tremendous amount of effort.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [sup-talk] Writing time-sensitive portions of sup in C
2009-10-01 17:44 ` Ben Gamari
2009-10-01 18:31 ` William Morgan
@ 2009-10-01 18:41 ` Carl Worth
1 sibling, 0 replies; 14+ messages in thread
From: Carl Worth @ 2009-10-01 18:41 UTC (permalink / raw)
Excerpts from Ben Gamari's message of Thu Oct 01 10:44:19 -0700 2009:
> Excerpts from William Morgan's message of Thu Oct 01 09:43:48 -0400 2009:
> > Here's the idea: scanning over a mailstore is slow. Much of this
> > slowness is due to Ruby. So let's rewrite this code in C. Then we would
> > have something as fast as, say, Mutt. But Mutt bogs down on my mbox file
> > because it's too big. So my *only* reasonable choice with a large
> > mailstore is Sup and the assumption that the source is add only.
>
> It seems that C would definitely be a good start (or perhaps C++ would
> be a better idea as that is the language in which Xapian is written).
I've started some experiments along these lines, (basically just
writing C code (with C++ where necessary) using the Xapian tutorial
and little peeks into sup/xapian-index.rb to get the right prefix
values, etc.).
> However, I think one of the real issues is the exclusive nature of index
> access. In fact, this is one of my primary gripes with the sup
> workflow. After processing a large number of messages, the write-out
> time can be quite substantial upon killing the buffer. This can be a
> noticeable interruption to workflow. It seems to me that index access
> should be asynchronous at least.
What I'm hoping to end up with is a C library that provides
asynchronous access to a sup-compatible index. I'll keep you posted if
I make any actual progress on this front.
-Carl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20091001/89e0a628/attachment.bin>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [sup-talk] Crash while scrolling
2009-10-01 18:31 ` William Morgan
@ 2009-10-01 19:01 ` Ben Gamari
2009-10-11 20:31 ` William Morgan
0 siblings, 1 reply; 14+ messages in thread
From: Ben Gamari @ 2009-10-01 19:01 UTC (permalink / raw)
Excerpts from William Morgan's message of Thu Oct 01 14:31:20 -0400 2009:
> Reformatted excerpts from Ben Gamari's message of 2009-10-01:
> > It seems that C would definitely be a good start (or perhaps C++ would
> > be a better idea as that is the language in which Xapian is written).
>
> I was proposing that as a strawman argument. C does not solve my
> problem.
Certainly, you will never be able to write a message indexer fast enough
to index a source instantly. That is why we have an index to begin with.
That being said, I think we can do substantially better than we are
currently doing in a lower-level language. With mutt, I can come close
to saturating my drive bandwidth while loading a folder. While
synchronizing in sup, I am lucky to get a few hundred kilobytes/second.
Certainly a large amount of that difference has to do with the amount of
processing done by each, but even adjusting for this it seems to me that
we should still be I/O bound (or at least close to it).
>
> > However, I think one of the real issues is the exclusive nature of
> > index access. In fact, this is one of my primary gripes with the sup
> > workflow. After processing a large number of messages, the write-out
> > time can be quite substantial upon killing the buffer.
>
> It is possible to address this in a variety of ways, many of which have
> been discussed over the years, but yes, I agree that a delay is
> nonideal.
Glad we agree.
>
> Making message state saving fast, or backgrounded, is a different beast
> from scanning over a mailstore.
If we are unable to update the index while the client is active,
rescanning sources would destroy usability. I would argue that
asynchronous writeout is very much a prerequisite for mutable sources.
>
> I have been working on a Sup server for quite some time that would
> address many of these problems, but progress is slow.
This was largely what I had in mind. It seems like moving index
manipulation out-of-process might be best, ultimately.
>
> > As an aside, it would be quite nice if one could run multiple
> > simultaneous instances of sup. It seems that if one only held write
> > access to the index during writes (is this the case presently?), there
> > should be nothing preventing this from being possible.
>
> It might be possible to do this with Xapian, especially if there were no
> expectation of updates being transmitted across processes.
I think initially no updates between processes would be fine.
> With Ferret,
> if it is possible, it's only with a tremendous amount of effort.
Is ferret even going to be supported after the Xapian backend
stabilizes?
Thanks for the comments and sup.
- Ben
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [sup-talk] Crash while scrolling
2009-09-11 16:58 [sup-talk] Crash while scrolling Ben Gamari
2009-09-12 16:35 ` William Morgan
@ 2009-10-04 19:59 ` Guillaume Quintard
1 sibling, 0 replies; 14+ messages in thread
From: Guillaume Quintard @ 2009-10-04 19:59 UTC (permalink / raw)
To: sup-talk
On Fri, Sep 11, 2009 at 6:58 PM, Ben Gamari <bgamari.foss@gmail.com> wrote:
> I can also produce a very similar crash[2] by attempting to load all
> threads. Thanks,
I get the same thing when loading all the threads. Index has just been
rebuilt, and I'm using an mbox file.
--
Guillaume
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [sup-talk] Crash while scrolling
2009-10-01 19:01 ` Ben Gamari
@ 2009-10-11 20:31 ` William Morgan
0 siblings, 0 replies; 14+ messages in thread
From: William Morgan @ 2009-10-11 20:31 UTC (permalink / raw)
To: sup-talk
Reformatted excerpts from Ben Gamari's message of 2009-10-01:
> Is ferret even going to be supported after the Xapian backend
> stabilizes?
No, I'm going to drop it at some point. I barely have enough time or
energy for Sup's current set of problems as is. :)
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2009-10-11 20:32 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-09-11 16:58 [sup-talk] Crash while scrolling Ben Gamari
2009-09-12 16:35 ` William Morgan
2009-09-13 15:02 ` Ben Gamari
2009-09-16 17:23 ` Ben Gamari
2009-09-26 14:31 ` William Morgan
2009-09-28 18:10 ` Ben Gamari
2009-10-01 13:43 ` William Morgan
2009-10-01 17:44 ` Ben Gamari
2009-10-01 18:31 ` William Morgan
2009-10-01 19:01 ` Ben Gamari
2009-10-11 20:31 ` William Morgan
2009-10-01 18:41 ` [sup-talk] Writing time-sensitive portions of sup in C Carl Worth
2009-09-30 21:21 ` [sup-talk] Crash while scrolling William Morgan
2009-10-04 19:59 ` Guillaume Quintard
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox