Subject: Re: [sup-talk] DatabaseCorruptError

From: William Morgan <[email-masked]>
Date: Wed, 11 May 2011 00:36:54 +0000

Reformatted excerpts from Andrew Pimlott's message of 2011-05-06:
> Sup had been running for a while, and I just came back to my terminal to
> find it dead:
> 
> --- IOError from thread: periodic poll
> DatabaseError: Error reading block 4279173482: got end of file
> I'm running an old git pull from next, commit 1bafcf6 dated 2010-10-12.

Yipes. Are you out of disk space, or anything like that?

> Anything I should do to try to diagnose and recover this?  Or just
> load from sup-dump?

Not knowing very much about Xapian, I suggest sup-dump. Not a great
solution, I know. Wait for heliotrope to solve your problems (or at
least be something I understand and can fix).
-- 
William <[email-masked]>
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Andrew Pimlott <[email-masked]>
Date: Fri, 6 May 2011 15:06:50 -0700

--- IOError from thread: periodic poll
DatabaseError: Error reading block 4279173482: got end of file
./lib/sup/index.rb:271:in `_dangerous_allterms_begin'
./lib/sup/index.rb:271:in `each_prefixed_term'
./lib/sup/index.rb:284:in `each_source_info'
./lib/sup/mbox.rb:147:in `each'
./lib/sup/mbox.rb:147:in `map'
./lib/sup/mbox.rb:147:in `last_indexed_message'
./lib/sup/util.rb:204:in `call'
./lib/sup/util.rb:204:in `benchmark'
/usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
./lib/sup/util.rb:204:in `benchmark'
./lib/sup/mbox.rb:147:in `last_indexed_message'
./lib/sup/mbox.rb:152:in `first_new_message'
./lib/sup/mbox.rb:124:in `poll'
./lib/sup/poll.rb:154:in `poll_from'
./lib/sup/poll.rb:113:in `do_poll'
./lib/sup/poll.rb:103:in `each'
./lib/sup/poll.rb:103:in `do_poll'
./lib/sup/poll.rb:102:in `synchronize'
./lib/sup/poll.rb:102:in `do_poll'
./lib/sup/util.rb:609:in `send'
./lib/sup/util.rb:609:in `method_missing'
./lib/sup/modes/poll-mode.rb:15:in `poll'
./lib/sup/poll.rb:49:in `poll_with_sources'
./lib/sup/poll.rb:68:in `poll'
./lib/sup/poll.rb:86:in `start'
./lib/sup.rb:78:in `reporting_thread'
./lib/sup.rb:76:in `initialize'
./lib/sup.rb:76:in `new'
./lib/sup.rb:76:in `reporting_thread'
./lib/sup/poll.rb:83:in `start'
./lib/sup/util.rb:609:in `send'
./lib/sup/util.rb:609:in `method_missing'
bin/sup:222

I tried to start it again, and got

--- IOError from thread: poll after loading inbox
DatabaseCorruptError: Db block overwritten - are there multiple writers?
/usr/lib/ruby/1.8/xapian.rb:259:in `_dangerous_postlist_begin'
/usr/lib/ruby/1.8/xapian.rb:259:in `postlist'
./lib/sup/index.rb:530:in `term_docids'
./lib/sup/index.rb:717:in `index_message_threading'
./lib/sup/index.rb:633:in `sync_message'
./lib/sup/util.rb:609:in `send'
./lib/sup/util.rb:609:in `method_missing'
./lib/sup/poll.rb:168:in `poll_from'
./lib/sup/mbox.rb:127:in `poll'
./lib/sup/poll.rb:154:in `poll_from'
./lib/sup/poll.rb:113:in `do_poll'
./lib/sup/poll.rb:103:in `each'
./lib/sup/poll.rb:103:in `do_poll'
./lib/sup/poll.rb:102:in `synchronize'
./lib/sup/poll.rb:102:in `do_poll'
./lib/sup/util.rb:609:in `send'
./lib/sup/util.rb:609:in `method_missing'
./lib/sup/modes/poll-mode.rb:15:in `poll'
./lib/sup/poll.rb:49:in `poll_with_sources'
./lib/sup/poll.rb:68:in `poll'
./lib/sup/util.rb:609:in `send'
./lib/sup/util.rb:609:in `method_missing'
bin/sup:212
./lib/sup.rb:78:in `reporting_thread'
./lib/sup.rb:76:in `initialize'
./lib/sup.rb:76:in `new'
./lib/sup.rb:76:in `reporting_thread'
bin/sup:212
./lib/sup/modes/thread-index-mode.rb:684:in `call'
./lib/sup/modes/thread-index-mode.rb:684:in `__unprotected_load_threads'
./lib/sup/modes/thread-index-mode.rb:625:in `call'
./lib/sup/modes/thread-index-mode.rb:625:in `load_n_threads_background'
./lib/sup.rb:78:in `reporting_thread'
./lib/sup.rb:76:in `initialize'
./lib/sup.rb:76:in `new'
./lib/sup.rb:76:in `reporting_thread'
./lib/sup/modes/thread-index-mode.rb:623:in `load_n_threads_background'
./lib/sup/modes/thread-index-mode.rb:694:in `__unprotected_load_threads'
(eval):12:in `load_threads'
bin/sup:212

My xapian dir looks like:

total 334396
-rw------- 1 andrew andrew         0 May  6 14:43 flintlock
-rw-rw-r-- 1 andrew andrew        12 May 31  2010 iamflint
-rw-rw-r-- 1 andrew andrew 103014400 May  6 13:09 position.DB
-rw-rw-r-- 1 andrew andrew      1594 May  6 11:22 position.baseA
-rw-rw-r-- 1 andrew andrew 161333248 May  6 13:29 postlist.DB
-rw-rw-r-- 1 andrew andrew      2475 May  6 11:22 postlist.baseA
-rw-rw-r-- 1 andrew andrew  20807680 May  6 13:04 record.DB
-rw-rw-r-- 1 andrew andrew       339 May  6 11:22 record.baseA
-rw-rw-r-- 1 andrew andrew  50692096 May  6 13:29 termlist.DB
-rw-rw-r-- 1 andrew andrew       795 May  6 11:22 termlist.baseA
-rw-rw-r-- 1 andrew andrew   6184960 May  6 13:04 value.DB
-rw-rw-r-- 1 andrew andrew       115 May  6 11:22 value.baseA

I'm running an old git pull from next, commit 1bafcf6 dated 2010-10-12.
Anything I should do to try to diagnose and recover this?  Or just load
from sup-dump?

Andrew

---

From: Andrew Pimlott <[email-masked]>
Date: Tue, 10 May 2011 23:14:01 -0700

> Yipes. Are you out of disk space, or anything like that?

I wasn't out of space--unless possibly Xapian caused some transient
spike in disk use that was freed when it died.  The system does run
without a ton of free space though, maybe 100 MB for a Xapian database
that is ~300 MB total.

I didn't get any disk errors either.  This is on a VPS that could
conceivably be hiding some errors.

> solution, I know.

It turned out that sup-dump still worked.  Probably some index was
corrupt but the data could still be walked.  So I was able to restore
without losing any data.  (Well, it did use most of the 160 MB of memory
on the VPS.)

> Wait for heliotrope to solve your problems (or at

Dum de dum dum...

Andrew