From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by 10.142.241.17 with SMTP id o17cs123990wfh; Mon, 16 May 2011 11:15:43 -0700 (PDT) Received: by 10.224.188.193 with SMTP id db1mr3572003qab.85.1305569743095; Mon, 16 May 2011 11:15:43 -0700 (PDT) Return-Path: Received: from rubyforge.org (rubyforge.org [205.234.109.19]) by mx.google.com with ESMTP id s3si10207176qco.124.2011.05.16.11.15.42; Mon, 16 May 2011 11:15:43 -0700 (PDT) Received-SPF: pass (google.com: domain of sup-talk-bounces@rubyforge.org designates 205.234.109.19 as permitted sender) client-ip=205.234.109.19; Authentication-Results: mx.google.com; spf=pass (google.com: domain of sup-talk-bounces@rubyforge.org designates 205.234.109.19 as permitted sender) smtp.mail=sup-talk-bounces@rubyforge.org Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id 8B48F3C8082; Mon, 16 May 2011 14:15:42 -0400 (EDT) X-Greylist: delayed 2329 seconds by postgrey-1.31 at rubyforge.org; Mon, 16 May 2011 14:09:45 EDT Received: from mailgw.es.net (mail3.es.net [198.129.252.242]) by rubyforge.org (Postfix) with ESMTP id 6FD631858361 for ; Mon, 16 May 2011 14:09:45 -0400 (EDT) Received: from localhost (dugan.mcs.anl.gov [140.221.13.64]) (authenticated bits=0) by mailgw.es.net (8.14.3/8.14.3) with ESMTP id p4GHUpl7024294 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Mon, 16 May 2011 10:30:51 -0700 From: Jon Dugan To: sup talk Date: Mon, 16 May 2011 12:30:50 -0500 Message-Id: <1305566071-sup-6625@arrakis.es.net> User-Agent: Sup/git X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.2.15, 1.0.148, 0.0.0000 definitions=2011-05-16_03:2011-05-13, 2011-05-16, 1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=5.0.0-1012030000 definitions=main-1105160125 Subject: [sup-talk] Multiple documents for a single docid in Xapian? X-BeenThere: sup-talk@rubyforge.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: User & developer discussion of Sup List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: sup-talk-bounces@rubyforge.org Errors-To: sup-talk-bounces@rubyforge.org Recently I've gotten into a situation where sup is crashing. The traceback is at the end. The first time I did a little digging, a git pull and finallly gave up and rebuilt my index. It happened again this morning and I dug in a bit more. (What follows is based on a pull from master as of a few minutes ago.) I added a bit of debugging at the point of the crash: (around line 555 of index.rb) def find_docid id docids = term_docids(mkterm(:msgid,id)) #fail unless docids.size <= 1 warn "got #{docids.size} (expected 1 or less) for #{id}" unless docids.size <= 1 docids.first end Basically instead of failing I log a warning and move on. For some reason I am seeing 2 entries in the index for that particular docid. Any idea how this could happen? Is there a recommended way to fix this? My little warning hack makes me nervous. I assume I can just delete one of them from the index but I haven't spent much time understanding how the index is set up and I'm concerned about unintended consequences... Interestingly I get the warning 4 times: [Mon May 16 12:14:26 -0500 2011] WARNING: got 2 (expected 1 or less) for 1305558622-sup-33@arrakis.es.net [Mon May 16 12:14:26 -0500 2011] WARNING: got 2 (expected 1 or less) for 1305558622-sup-33@arrakis.es.net [Mon May 16 12:14:29 -0500 2011] WARNING: got 2 (expected 1 or less) for 1305558622-sup-33@arrakis.es.net [Mon May 16 12:14:29 -0500 2011] WARNING: got 2 (expected 1 or less) for 1305558622-sup-33@arrakis.es.net I originally got this after an offline IMAP run when it was scanning for new messages to add to the index. The message in question is actually in my ~/.sup/sent.mbox file and so it wasn't pulled from IMAP -- I'm guessing it must have just checked the sent mbox file after it finished checking the offline IMAP directories. I'm not really sure. Any ideas? Thanks, Jon --- RuntimeError from thread: poll after loading inbox /Users/jdugan/projects/sup/mainline/lib/sup/index.rb:556:in `find_docid' /Users/jdugan/projects/sup/mainline/lib/sup/index.rb:562:in `find_doc' /Users/jdugan/projects/sup/mainline/lib/sup/index.rb:572:in `get_entry' /Users/jdugan/projects/sup/mainline/lib/sup/index.rb:200:in `build_message' /opt/local/lib/ruby/1.8/monitor.rb:242:in `synchronize' /Users/jdugan/projects/sup/mainline/lib/sup/index.rb:581:in `synchronize' /Users/jdugan/projects/sup/mainline/lib/sup/index.rb:200:in `build_message' /Users/jdugan/projects/sup/mainline/lib/sup/util.rb:618:in `send' /Users/jdugan/projects/sup/mainline/lib/sup/util.rb:618:in `method_missing' /Users/jdugan/projects/sup/mainline/lib/sup/poll.rb:159:in `poll_from' /Users/jdugan/projects/sup/mainline/lib/sup/maildir.rb:106:in `poll' /Users/jdugan/projects/sup/mainline/lib/sup/util.rb:615:in `each_with_index' /Users/jdugan/projects/sup/mainline/lib/sup/maildir.rb:105:in `each' /Users/jdugan/projects/sup/mainline/lib/sup/maildir.rb:105:in `each_with_index' /Users/jdugan/projects/sup/mainline/lib/sup/maildir.rb:105:in `poll' /Users/jdugan/projects/sup/mainline/lib/sup/maildir.rb:90:in `each' /Users/jdugan/projects/sup/mainline/lib/sup/maildir.rb:90:in `poll' /Users/jdugan/projects/sup/mainline/lib/sup/poll.rb:155:in `poll_from' /Users/jdugan/projects/sup/mainline/lib/sup/poll.rb:113:in `do_poll' /Users/jdugan/projects/sup/mainline/lib/sup/poll.rb:103:in `each' /Users/jdugan/projects/sup/mainline/lib/sup/poll.rb:103:in `do_poll' /Users/jdugan/projects/sup/mainline/lib/sup/poll.rb:102:in `synchronize' /Users/jdugan/projects/sup/mainline/lib/sup/poll.rb:102:in `do_poll' /Users/jdugan/projects/sup/mainline/lib/sup/util.rb:618:in `send' /Users/jdugan/projects/sup/mainline/lib/sup/util.rb:618:in `method_missing' /Users/jdugan/projects/sup/mainline/lib/sup/modes/poll-mode.rb:15:in `poll' /Users/jdugan/projects/sup/mainline/lib/sup/poll.rb:49:in `poll_with_sources' /Users/jdugan/projects/sup/mainline/lib/sup/poll.rb:68:in `poll' /Users/jdugan/projects/sup/mainline/lib/sup/util.rb:618:in `send' /Users/jdugan/projects/sup/mainline/lib/sup/util.rb:618:in `method_missing' /Users/jdugan/projects/sup/mainline/bin/sup:223 /Users/jdugan/projects/sup/mainline/lib/sup.rb:78:in `reporting_thread' /Users/jdugan/projects/sup/mainline/lib/sup.rb:76:in `initialize' /Users/jdugan/projects/sup/mainline/lib/sup.rb:76:in `new' /Users/jdugan/projects/sup/mainline/lib/sup.rb:76:in `reporting_thread' /Users/jdugan/projects/sup/mainline/bin/sup:223 /Users/jdugan/projects/sup/mainline/lib/sup/modes/thread-index-mode.rb:684:in `call' /Users/jdugan/projects/sup/mainline/lib/sup/modes/thread-index-mode.rb:684:in `__unprotected_load_threads' /Users/jdugan/projects/sup/mainline/lib/sup/modes/thread-index-mode.rb:625:in `call' /Users/jdugan/projects/sup/mainline/lib/sup/modes/thread-index-mode.rb:625:in `load_n_threads_background' /Users/jdugan/projects/sup/mainline/lib/sup.rb:78:in `reporting_thread' /Users/jdugan/projects/sup/mainline/lib/sup.rb:76:in `initialize' /Users/jdugan/projects/sup/mainline/lib/sup.rb:76:in `new' /Users/jdugan/projects/sup/mainline/lib/sup.rb:76:in `reporting_thread' /Users/jdugan/projects/sup/mainline/lib/sup/modes/thread-index-mode.rb:623:in `load_n_threads_background' /Users/jdugan/projects/sup/mainline/lib/sup/modes/thread-index-mode.rb:694:in `__unprotected_load_threads' (eval):12:in `load_threads' /Users/jdugan/projects/sup/mainline/bin/sup:223 _______________________________________________ sup-talk mailing list sup-talk@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-talk