Subject: [sup-talk] Crash while syncing

From: [email-masked] (Nicolas Pouillard)
Date: Sat, 29 Aug 2009 11:18:52 +0200

Hello,

While doing a from scratch sup-sync to benefit for the new version of
the xapian index, sup-sync
crashed like this:

/usr/lib/ruby/site_ruby/1.8/xapian.rb:240:in `doclength':
DocNotFoundError: Document 3318104838 not found (RuntimeError)
        from /usr/lib/ruby/site_ruby/1.8/xapian.rb:240:in `postlist'
        from /usr/lib/ruby/site_ruby/1.8/xapian.rb:60:in `_safelyIterate'
        from /usr/lib/ruby/site_ruby/1.8/xapian.rb:238:in `postlist'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/xapian_index.rb:365:in
`term_docids'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/xapian_index.rb:369:in
`find_docid'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/xapian_index.rb:373:in
`find_doc'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/xapian_index.rb:469:in
`index_message'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/xapian_index.rb:469:in
`map'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/xapian_index.rb:469:in
`index_message'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/xapian_index.rb:118:in
`sync_message'
        from /usr/lib/ruby/1.8/monitor.rb:242:in `synchronize'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/xapian_index.rb:392:in
`synchronize'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/xapian_index.rb:117:in
`sync_message'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/xapian_index.rb:87:in
`add_message'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/bin/sup-sync:202
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/poll.rb:152:in
`each_message_from'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/source.rb:104:in `each'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/util.rb:560:in `send'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/util.rb:560:in `__pass'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/util.rb:547:in
`method_missing'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/poll.rb:140:in
`each_message_from'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/util.rb:520:in `send'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/util.rb:520:in
`method_missing'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/bin/sup-sync:146
        from /usr/lib/ruby/gems/1.8/gems/sup-999/bin/sup-sync:141:in `each'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/bin/sup-sync:141
        from /usr/bin/sup-sync:19:in `load'
        from /usr/bin/sup-sync:19

The last status message was:

## read 9996m (about 30%) @ 11.1m/s. 0:15:03 elapsed, about 0:35:00 remaining

And it was processing a 500MB mbox.

I would like to know what are my options from here?

Best regards,

-- 
Nicolas Pouillard aka Ertai

---

From: [email-masked] (Nicolas Pouillard)
Date: Sat, 29 Aug 2009 14:03:04 +0200

On Sat, Aug 29, 2009 at 11:18 AM, Nicolas
> the xapian index, sup-sync

I've lauched sup-sync on another bunch of mboxes and got this crash at the end:

Deleting missing messages from the index...
/usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup.rb:17:in `id': wrong id
called on nil (RuntimeError)
        from /usr/lib/ruby/gems/1.8/gems/sup-999/bin/sup-sync:237
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/index.rb:150:in
`each_message'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/xapian_index.rb:141:in
`each_id'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/xapian_index.rb:141:in
`each'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/xapian_index.rb:141:in
`each_id'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/lib/sup/index.rb:149:in
`each_message'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/bin/sup-sync:235
        from /usr/lib/ruby/gems/1.8/gems/sup-999/bin/sup-sync:233:in `each'
        from /usr/lib/ruby/gems/1.8/gems/sup-999/bin/sup-sync:233
        from /usr/bin/sup-sync:19:in `load'
        from /usr/bin/sup-sync:19

---

From: [email-masked] (Rich Lane)
Date: Sat, 29 Aug 2009 12:25:33 -0400

>         from /usr/bin/sup-sync:19

Are you using the Flint or Chert Xapian backend?

---

From: [email-masked] (Nicolas Pouillard)
Date: Sat, 29 Aug 2009 18:45:29 +0200

On Sat, Aug 29, 2009 at 6:25 PM, Rich Lane<rlane at club.cc.cmu.edu> wrote:
>> On Sat, Aug 29, 2009 at 11:18 AM, Nicolas Pouillard<nicolas.pouillard at gmail.com> wrote:
>> > Hello,
>> >

[...]

> Are you using the Flint or Chert Xapian backend?

The Chert one.

---

From: [email-masked] (Rich Lane)
Date: Sat, 29 Aug 2009 12:59:59 -0400

> > Are you using the Flint or Chert Xapian backend?

Could you try it with Flint? I've seen this (nondeterministic) bug when
using Chert.

---

From: [email-masked] (Nicolas Pouillard)
Date: Sat, 29 Aug 2009 19:05:33 +0200

> Could you try it with Flint? I've seen this (nondeterministic) bug when

I was currently running it again with no problems so far.

Are the nondeterministic bugs only during sync?

Chert is supposed to faster?

Thanks for your help,

-- 
Nicolas Pouillard

---

From: [email-masked] (Rich Lane)
Date: Sat, 29 Aug 2009 20:22:15 -0400

> Are the nondeterministic bugs only during sync?

I've only seen it during sup-sync, but I'd guess if you ran sup long
enough you'd see it there too (after changing labels / polling).

> Chert is supposed to faster?

Yes, and more space efficient.