Subject: [sup-talk] Too many clauses

From: [email-masked] (Dennis Griffith)
Date: Mon, 3 Dec 2007 13:52:38 -0600

I started getting this crash upon start after updating to 0.3 from 0.1.
-- 
Dennis Griffith
-------------- next part --------------
--- Ferret::StateError from thread: load threads for thread-index-mode
State Error occured at <except.c>:93 in xraise
Error occured in q_boolean.c:1595 - bq_add_query_nr
	Two many clauses. The max clause limit is set to <1024> but your query has <1024> clauses. You can try increasing :max_clause_count for the BooleanQuery or using a different type of query.

/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/index.rb:270:in `add_query'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/index.rb:270:in `each_message_in_thread_for'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/index.rb:268:in `each'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/index.rb:268:in `each_message_in_thread_for'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/thread.rb:321:in `load_thread_for_message'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/thread.rb:313:in `load_n_threads'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/index.rb:217:in `each_id_by_date'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/index.rb:217:in `each'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/index.rb:217:in `each_id_by_date'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/thread.rb:308:in `load_n_threads'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/modes/thread-index-mode.rb:415:in `__unprotected_load_n_threads'
(eval):12:in `load_n_threads'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/modes/thread-index-mode.rb:404:in `load_n_threads_background'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup.rb:69:in `reporting_thread'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup.rb:67:in `initialize'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup.rb:67:in `new'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup.rb:67:in `reporting_thread'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/modes/thread-index-mode.rb:403:in `load_n_threads_background'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/modes/thread-index-mode.rb:454:in `__unprotected_load_threads'
(eval):12:in `load_threads'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/bin/sup:185
/usr/bin/sup:16:in `load'
/usr/bin/sup:16
--- SystemExit from thread: main
State Error occured at <except.c>:93 in xraise
Error occured in q_boolean.c:1595 - bq_add_query_nr
	Two many clauses. The max clause limit is set to <1024> but your query has <1024> clauses. You can try increasing :max_clause_count for the BooleanQuery or using a different type of query.

/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup.rb:73:in `select'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/buffer.rb:31:in `nonblocking_getch'
/usr/lib/ruby/gems/1.8/gems/sup-0.3/bin/sup:198
/usr/bin/sup:16:in `load'
/usr/bin/sup:16

---

From: [email-masked] (William Morgan)
Date: Sun, 09 Dec 2007 11:44:43 -0800

> /usr/lib/ruby/gems/1.8/gems/sup-0.3/lib/sup/index.rb:270:in `add_query'

At first, I thought, "Great! An error message that contains its own
solution!" But the max_clause_count thing seems to have disappeared in
recent versions of Ferret.

Can you try this patch, please?

  --- lib/sup/index.rb    (revision 748)
  +++ lib/sup/index.rb    (working copy)
  @@ -236,6 +236,7 @@
     ## true, stops loading any thread if a message with a :killed flag
     ## is found.
     SAME_SUBJECT_DATE_LIMIT = 7
  +  MAX_CLAUSES = 1000
     def each_message_in_thread_for m, opts={}
       #Redwood::log "Building thread for #{m.id}: #{m.subj}"
       messages = {}
  @@ -264,13 +265,16 @@
   
       until pending.empty? || (opts[:limit] && messages.size >= opts[:limit])
         q = Ferret::Search::BooleanQuery.new true
  +      # this disappeared in newer ferrets... wtf.
  +      # q.max_clause_count = 2048
   
  -      pending.each do |id|
  +      lim = [MAX_CLAUSES / 2, pending.length].min
  +      pending[0 ... lim].each do |id|
           searched[id] = true
           q.add_query Ferret::Search::TermQuery.new(:message_id, id), :should
           q.add_query Ferret::Search::TermQuery.new(:refs, id), :should
         end
  -      pending = []
  +      pending = pending[lim .. -1]
   
         q = build_query :qobj => q

Thanks,

-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Dennis Griffith)
Date: Sun, 09 Dec 2007 19:31:23 -0600

> recent versions of Ferret.

I thought that too and tried to fix it. However not knowing hardly any ruby proved a bit of a hinderence.

> Can you try this patch, please?

I could, although I wouldn't be able to tell you if it works anymore. After installing 0.3 again, in preparation for trying the patch, I found I no
longer hit the problem. I haven't updated ferret during this time, so perhaps my message count has fallen low enough not to trigger the problem, if
I hit this again I will try the patch and let you know.

-- 
Dennis Griffith

---

From: [email-masked] (William Morgan)
Date: Sun, 09 Dec 2007 18:54:43 -0800

Excerpts from Dennis Griffith's message of Sun Dec 09 17:31:23 -0800 2007:
> I could, although I wouldn't be able to tell you if it works anymore.
> After installing 0.3 again, in preparation for trying the patch, I
> found I no longer hit the problem. I haven't updated ferret during
> this time, so perhaps my message count has fallen low enough not to
> trigger the problem, if I hit this again I will try the patch and let
> you know.

Weird. I think this problem surfaces with very large threads, so it
shouldn't have anything to do with raw message count.

But all's well that ends well. I've committed the patch to SVN anyways
because it's probably the right thing to do. (And is a better solution
than just increasing the Ferret thingy.)