Subject: [sup-talk] U (unread) search is off

From: [email-masked] (Ben Walton)
Date: Wed, 09 Sep 2009 09:54:55 -0400

Since my update to Xapian, my U searches to display on unread mail
aren't working correctly any more.  I get lots of mail that is
definitely read (doesn't show as new in the search buffer).  Is anyone
else seeing this?  I'm hoping to dig into it later today, but wanted
to fire of the query first.

Thanks
-Ben
-- 
Ben Walton
Systems Programmer - CHASS
University of Toronto
C:416.407.5610 | W:416.978.4302

GPG Key Id: 8E89F6D2; Key Server: pgp.mit.edu
Contact me to arrange for a CAcert assurance meeting.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20090909/f80057be/attachment.bin>

---

From: [email-masked] (Ingmar Vanhassel)
Date: Thu, 10 Sep 2009 13:22:30 +0200

> to fire of the query first.

Known issue, see http://mid.gmane.org/1251792282-sup-2057 at cannonball and
Rich's answer to my mail.

-- 
Exherbo KDE, X.org maintainer

---

From: [email-masked] (William Morgan)
Date: Thu, 10 Sep 2009 06:59:13 -0700

> Rich's answer to my mail.

Actually, I haven't been following this too closely, but what is the status?
I noticed we have this in xapian_index.rb:
    qp.default_op = Xapian::Query::OP_AND

Is that not sufficient (combined with a newish Xapian, I guess?) to fix
the problem?
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Ben Walton)
Date: Thu, 10 Sep 2009 10:04:02 -0400

> Is that not sufficient (combined with a newish Xapian, I guess?) to fix

I'm running 1.0.14.  Maybe I need to update to 1.0.15 then?

Thanks
-Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20090910/d6b22755/attachment.bin>

---

From: [email-masked] (Ingmar Vanhassel)
Date: Thu, 10 Sep 2009 16:08:15 +0200

> Is that not sufficient (combined with a newish Xapian, I guess?) to fix

Refining a query still doesn't work: Searching for label:foo, then
piping through label:bar gives messages that even have neither label.

This is on xapian 1.0.15

---

From: [email-masked] (Rich Lane)
Date: Thu, 10 Sep 2009 12:16:35 -0400

There are actually 2 bugs in this thread. Ben is still using Xapian
1.0.14 which doesn't have my fix for the phantom labels problem.
Ingmar's message was about the surprise Xapian::QueryParser feature.

> Is that not sufficient (combined with a newish Xapian, I guess?) to fix

I thought it was, but it turns out that unless you explicitly add AND
operators Xapian will OR terms over the same field such that
"label:foo label:bar" gives you the union instead of the intersection.

We could fix this by patching Xapian to make this behavior configurable,
or we could come up with an index-agnostic simple query language that
doesn't support boolean operators. The native query language would still
be available, of course, but the simple one would suffice for most usage
and potentially have Sup-specific features.

---

From: [email-masked] (Rich Lane)
Date: Thu, 10 Sep 2009 12:17:11 -0400

> I'm running 1.0.14.  Maybe I need to update to 1.0.15 then?

Yes, upgrading to 1.0.15 will fix this problem.

---

From: [email-masked] (Ben Walton)
Date: Thu, 10 Sep 2009 13:13:32 -0400

> Yes, upgrading to 1.0.15 will fix this problem.

Ok, I rolled rpms for 1.0.16 (available to anyone else if they
want/need them) and updated.  This has resolved the issue for me.

Thanks Rich! :)
-Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20090910/ec4dc234/attachment.bin>

---

From: [email-masked] (William Morgan)
Date: Sat, 12 Sep 2009 09:47:14 -0700

> "label:foo label:bar" gives you the union instead of the intersection.

Ok. I only ask because I'm considering how to present the Xapian index
option in 0.9---the options are to not say anything about it, to force
everyone to switch to it, or anywhere in between. 

> We could fix this by patching Xapian to make this behavior
> configurable, or we could come up with an index-agnostic simple query
> language that doesn't support boolean operators.

The former sounds easier to me. Especially since it seems like this is
something Xapian should have...
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Rich Lane)
Date: Sun, 13 Sep 2009 20:31:23 -0400

> everyone to switch to it, or anywhere in between. 

Yeah, we do need the query behavior to be reasonable before advertising
the Xapian index. What's your timeline on 0.9?

> something Xapian should have...

Agreed that Xapian should have this, and I'll probably implement it. We
still need to support people using older Xapian versions though. Whether
this means a log message telling them to use explicit ANDs in their
query, or automatically transforming simple-enough queries into what we
think they actually want, I don't know. I'll also need to add a
workaround for the earlier Xapian bug before this index gets wider
usage.

As far as which index to make the default, Ferret has the advantage of
being installable as a gem dependency. I think this ease of installation
is important enough to block Xapian for now, unless someone can create a
Xapian gem. There's been interest in this before:
http://article.gmane.org/gmane.comp.search.xapian.devel/1408/

---

From: [email-masked] (Olly Betts)
Date: Tue, 15 Sep 2009 07:18:51 +0000 (UTC)

> workaround for the earlier Xapian bug before this index gets wider

I've now opened a Xapian ticket for this:

http://trac.xapian.org/ticket/402

Patches are certainly most welcome, especially as I'm currently trying to
focus on getting Xapian 1.2.0 out of the door.

I don't see a satisfactory workaround for it, sadly.

> http://article.gmane.org/gmane.comp.search.xapian.devel/1408/

I recently noticed this gem version of (modified) xapian-bindings:

http://groups.google.com/group/acts_as_xapian/msg/1bbb1e0d3753a0b7

I've not had a chance to look at it yet though, so I don't know if the
changes mentioned are compatible or not.  Assuming they're useful, they
probably should be folded in upstream - more idiomatic wrappers are a
desirable improvement (though if they're incompatible we'd need some
sort of migration plan).  It doesn't seem productive to have forked
versions like this either.

Cheers,
    Olly