Subject: Re: [sup-talk] Crash, bad index, and ensuing misery

From: Guillaume Quintard <[email-masked]>
Date: Sat, 3 Oct 2009 23:54:28 +0200

On Sat, Oct 3, 2009 at 11:39 PM, Rich Lane <[email-masked]> wrote:
> Please post the output of "git log --pretty=oneline 'origin/next^'.."
>
here you go
80789d873ac555bb1979f7734e95f104d848007e Merge branch 'next' of
git://gitorious.org/sup/mainline into next
0eee0973223b625b66e30c196ccd45d2f7bf358b Merge branch 'master' into next
93b5552730c10e2a352bd33f5ee98800bcd8679e more release-script updates
d1eabf9cb21940b933464ab3efc25df3c1a5f7e1 Merge branch 'next' of
git://gitorious.org/sup/mainline into next
df96980d0573cf91742a8f9ae5e8a49366d338b8 Merge branch 'next' of
git://gitorious.org/sup/mainline into next
a6dcb02dda7dd8bb1742890d460b1b0abfc28454 Merge branch 'next' of
git://gitorious.org/sup/mainline into next
823148627f554fbf5a7fb13379567276eeee14c7 Merge branch 'next' of
git://gitorious.org/sup/mainline into next
40ecc407a8aacf16d7f9f104501311cfd1fb2eb7 Revert "Merge branch
'after-add-message-hook' into next"

-- 
Guillaume
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Guillaume Quintard <[email-masked]>
Date: Tue, 6 Oct 2009 15:10:44 +0200

On Tue, Oct 6, 2009 at 2:40 PM, Olly Betts <[email-masked]> wrote:
> NEVER EVER remove the flintlock file.
Ooops, didn't know, won't do it again.

> My guess is that removing the flintlock file is the cause of the corruption
> you're seeing.  Can you reproduce it on a database where you haven't removed
> this file?
Unfortunately, no

> which you have is whether there's a file called "iamflint" or "iamchert" in
Flint then

---

From: Rich Lane <[email-masked]>
Date: Sat, 03 Oct 2009 16:17:54 -0400

Excerpts from Guillaume Quintard's message of Sat Oct 03 16:04:14 -0400 2009:
> Hi, I crash my computer with sup running, and it seems the index
> didn't really like. sup-sync -a didn't help, what can I do?

I'd hoped this kind of corruption wouldn't be possible with newer
xapian-index versions. What sup commit are you on? What version of
Xapian are you using? Which Xapian backend, Flint or Chert?
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Rich Lane <[email-masked]>
Date: Sat, 03 Oct 2009 18:25:19 -0400

Excerpts from Guillaume Quintard's message of Sat Oct 03 16:04:14 -0400 2009:
> --- RuntimeError from thread: load threads for thread-index-mode
> DocNotFoundError: Document 2461178145 not found.
> ./lib/sup/xapian_index.rb:132:in `document'

The relevant line:
docs = term_docids(mkterm(:thread, thread_id)).map { |x| @xapian.document x }

Basically expands to:
@xapian.postlist(term).map { |x| @xapian.document x.docid }

So, Xapian is giving us docids it can't turn into documents. At this
point I think you should just do a sup-dump (which might still work),
move the old xapian directory out of the way, and re-sync. You can try
running xapian-check on the corrupted version to see if it finds
anything interesting.
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Guillaume Quintard <[email-masked]>
Date: Sat, 3 Oct 2009 22:04:14 +0200

--- RuntimeError from thread: load threads for thread-index-mode
DocNotFoundError: Document 2461178145 not found.
./lib/sup/xapian_index.rb:132:in `document'
./lib/sup/xapian_index.rb:132:in `each_message_in_thread_for'
./lib/sup/xapian_index.rb:132:in `map'
./lib/sup/xapian_index.rb:132:in `each_message_in_thread_for'
./lib/sup/thread.rb:341:in `load_thread_for_message'
./lib/sup/thread.rb:333:in `load_n_threads'
./lib/sup/xapian_index.rb:117:in `each_id_by_date'
./lib/sup/xapian_index.rb:110:in `each_id'
./lib/sup/xapian_index.rb:110:in `each'
./lib/sup/xapian_index.rb:110:in `each_id'
./lib/sup/xapian_index.rb:117:in `each_id_by_date'
./lib/sup/thread.rb:328:in `load_n_threads'
./lib/sup/modes/thread-index-mode.rb:625:in `__unprotected_load_n_threads'
(eval):12:in `load_n_threads'
./lib/sup/modes/thread-index-mode.rb:609:in `load_n_threads_background'
./lib/sup.rb:77:in `reporting_thread'
./lib/sup.rb:75:in `initialize'
./lib/sup.rb:75:in `new'
./lib/sup.rb:75:in `reporting_thread'
./lib/sup/modes/thread-index-mode.rb:608:in `load_n_threads_background'
./lib/sup/modes/thread-index-mode.rb:679:in `__unprotected_load_threads'
(eval):12:in `load_threads'
bin/sup:197

---

From: Olly Betts <[email-masked]>
Date: Tue, 6 Oct 2009 12:40:25 +0000 (UTC)

Guillaume Quintard writes:
> On Sat, Oct 3, 2009 at 10:17 PM, Rich Lane <rlane <at> club.cc.cmu.edu> wrote:
> ubuntu 9.10
> libxapian-ruby1.8    1.0.14-1
> I'd say flink since I often have to remove flintlock after sup
> commiting suicide.

NEVER EVER remove the flintlock file.  It's not the presence of the file which
determines the lock, but rather whether there's an fcntl() lock on it, so
removing it smashes any lock which is currently held, but leaves the process
which held it thinking it still has exclusive write access to the database,
which will likely quickly lead to a corrupt database, especially if you're
doing so because Xapian says the database is already locked.  If Xapian says
that, then there really is a process which still has the database open for
writing.

Also, both chert and flint use the same locking approach with a file called
flintlock, so that doesn't discriminate between them.  The easy way to tell
which you have is whether there's a file called "iamflint" or "iamchert" in
the database directory.

Cheers,
    Olly

---

From: Guillaume Quintard <[email-masked]>
Date: Sat, 3 Oct 2009 23:06:36 +0200

ubuntu 9.10

and git status says
# On branch next
# Your branch is ahead of 'origin/next' by 6 commits.

(ahead?)

---

From: Rich Lane <[email-masked]>
Date: Sat, 03 Oct 2009 17:39:42 -0400