Subject: [sup-talk] Amazon.com messages can't be added to index

From: [email-masked] (jenny w)
Date: Sat, 13 Oct 2007 20:32:23 -0700

Hi,

I just installed sup and am quite excited to check it out in action. I
haven't gotten much further than my first sync (27k messages over
IMAPS), but I've noticed something that I didn't in the archives
(relying on Google for searching).

Messages I've received from Amazon.com seem to consistently bring up
an exception on line 157 of 188 of index.rb in the 0.1 release. I saw
in the troubleshooting tips to try the version from svn but I get a
similar exception. So I just changed "raise" to "puts", under the
assumption this means that I just won't be able to search old
Amazon.com messages. Right now I don't know if this is all Amazon.com
messages or just some of them. After the sync is done I'll see if I
can reproduce with just one message and try to narrow down what about
the message is causing the problem.

Anyway, thought I'd mention it in case someone knows about this
problem, or in case it helps track down what the problem is (the
comment above line 188 is 'this hasn't been triggered in a long time.
TODO: decide whether it's still a problem.')

Sample error message:

    just added message .AAA-notification-24668,
9630.1162603428 at na-rte-app-5104.iad5.amazon.com but couldn't find it
in a search

---

From: [email-masked] (Kevin Mark)
Date: Sun, 14 Oct 2007 01:20:03 -0400

> 9630.1162603428 at na-rte-app-5104.iad5.amazon.com but couldn't find it

It would be helpful to the list to show a portion of the actual error
message(or all of it if its small) instead of a restatement of it. A
restatement of an error message is rarely adequet to solve an error.

-- 
|  .''`.  == Debian GNU/Linux == |       my web site:           |
| : :' :      The  Universal     |mysite.verizon.net/kevin.mark/|
| `. `'      Operating System    | go to counter.li.org and     |
|   `-    http://www.debian.org/ |    be counted! #238656       |
|  my keyserver: subkeys.pgp.net |     my NPO: cfsg.org         |
|join the new debian-community.org to help Debian!              |
|_______  Unless I ask to be CCd, assume I am subscribed _______|

---

From: [email-masked] (jenny w)
Date: Sun, 14 Oct 2007 02:04:04 -0700

> restatement of an error message is rarely adequet to solve an error.

Okay, more info. I ended up running this under ruby-debug to see what
was going on.  It seems the problem I'm having is that Amazon.com
sends me messages with message_ids like:

    AAA-notification-70925, 4617.1234123412 at na-rte-app-5105.iad5.amazon.com

The space after the comma is what's throwing ferret off.

I don't know if it helps, but I noticed that if I try doing
@index.search("message_id:#{m.id}") ferret will find the doc. This
seems to be because Ferret is turning the string into a Boolean (as
opposed to Term) query.

---

From: [email-masked] (Christopher Warrington)
Date: Sun, 14 Oct 2007 16:23:57 -0500

> The space after the comma is what's throwing ferret off.

I know that this doesn't solve the problem, but that's not a valid
RFC822 MID. The space and comma are not allowed in the part to the left
of the at-sign (unless the part to the left is a quoted string, which it
is not in this case). Section 3.4.1 talks explicitly about this.

-- 
Christopher Warrington <chrisw at rice.edu>
Jones College

---

From: [email-masked] (jenny w)
Date: Sun, 14 Oct 2007 15:32:08 -0700

> is not in this case). Section 3.4.1 talks explicitly about this.

Yes, that's a problem, and I think they've fixed it in their current
e-mails, but I suspect violations of rfc822 will continue happen.

I think the problem is that the field gets tokenized. I thought using
a PhraseQuery would solve the problem, but that didn't help. Sorry I
can't provide more useful information!

---

From: [email-masked] (William Morgan)
Date: Sat, 27 Oct 2007 20:06:26 -0700

Excerpts from veganjenny's message of Sun Oct 14 15:32:08 -0700 2007:
> I think the problem is that the field gets tokenized.

Precisely so. I think I have finally fixed this problem in SVN (by
stripping all spaces from message ids). Update to r637 and see if it
still happens.

-- 
William <wmorgan-sup at masanjin.net>