Subject: [sup-talk] Amazon.com messages can't be added to index

From: [email-masked] (Luis Villa)
Date: Fri, 22 Feb 2008 10:46:55 -0500

>From a long ago thread:
http://rubyforge.org/pipermail/sup-talk/2007-October/000326.html

It looks like I've found a similar problem. In importing my mail (with
0.4, haven't tried next yet) I get:

/usr/lib/ruby/gems/1.8/gems/sup-0.4/lib/sup/index.rb:200:in
`sync_message': just added message
"!~!UENERkVCMDkAAQACAPYAAAAAAAAAOKG7EAXlEBqhuwgAKypWwgAAbXNwc3QuZGxsAAAAAABOSVRB+b+4AQCqADfZbgAAAABDADoAXABEAG8AYwB1AG0AZQBuAHQAcwAgAGEAbgBkACAAUwBlAHQAdABpAG4AZwBzAFwAawBiAGUAbgB0AG8AbgBcAEwAbwBjAGEAbAAgAFMAZQB0AHQAaQBuAGcAcwBcAEEAcABwAGwAaQBjAGEAdABpAG8AbgAgAEQAYQB0AGEAXABNAGkAYwByAG8AcwBvAGYAdABcAE8AdQB0AGwAbwBvAGsAXABPAHUAdABsAG8AbwBrAC4AcABzAHQAAAAYAAAAAAAAALLH/vR9UMVCgMck3LV+0wHCgAAAGAAAAAAAAACyx/70fVDFQoDHJNy1ftMBhLcgAAAAAAAQAAAAqTDfdQ6dIEawbQUxhNxqVz4AAABSRTogQnVnemlsbGE6IEhhcyBhbnlvbmUgc3VjY2Vzc2Z1bGx5IGNyZWF0ZWQgU3ViLUNvbXBvbmVudHM/AA==@amd.com"
but couldn't find it in a search (RuntimeError)

Is this still a tokenization problem, or...? Any additional debugging
information I can grab to help debug?

Thanks-
Luis

---

From: [email-masked] (Luis Villa)
Date: Fri, 22 Feb 2008 11:25:01 -0500

Thanks-
Luis

---

From: [email-masked] (William Morgan)
Date: Sun, 24 Feb 2008 21:10:57 -0800

Reformatted excerpts from Luis Villa's message of 2008-02-22:
> /usr/lib/ruby/gems/1.8/gems/sup-0.4/lib/sup/index.rb:200:in `sync_message': just added message "!~!UENERkVCMDkAAQACAPYAAAAAAAAAOKG7EAXlEBqhuwgAKypWwgAAbXNwc3QuZGxsAAAAAABOSVRB+b+4AQCqADfZbgAAAABDADoAXABEAG8AYwB1AG0AZQBuAHQAcwAgAGEAbgBkACAAUwBlAHQAdABpAG4AZwBzAFwAawBiAGUAbgB0AG8AbgBcAEwAbwBjAGEAbAAgAFMAZQB0AHQAaQBuAGcAcwBcAEEAcABwAGwAaQBjAGEAdABpAG8AbgAgAEQAYQB0AGEAXABNAGkAYwByAG8AcwBvAGYAdABcAE8AdQB0AGwAbwBvAGsAXABPAHUAdABsAG8AbwBrAC4AcABzAHQAAAAYAAAAAAAAALLH/vR9UMVCgMck3LV+0wHCgAAAGAAAAAAAAACyx/70fVDFQoDHJNy1ftMBhLcgAAAAAAAQAAAAqTDfdQ6dIEawbQUxhNxqVz4AAABSRTogQnVnemlsbGE6IEhhcyBhbnlvbmUgc3VjY2Vzc2Z1bGx5IGNyZWF0ZWQgU3ViLUNvbXBvbmVudHM/AA==@amd.com" but couldn't find it in a search (RuntimeError)

Sigh. Why would anyone generate a message id like that?

There were two problems causing your error. I've fixed them both in git
next. You can probably apply the attached patches to your 0.4 release if
you don't want to use git just yet.

The first problem was that marking the message_id field as non-tokenized
in Ferret just solves all sorts of tokenization problems. So that's in.

The second problem is a Ferret bug, where apparently TermQuery values of
more than 255 characters never match anything. The current workaround
just lops off anything after the 255th character. And that may very well
screw things up if falsely uniquefies things.

The right long-term answer is probably to take the hex SHA1 of every
message id and just use that instead of the original value. Then all of
these issues will be solved. That will require an index rebuild for
everyone, so I'm going to hold off on that for now.

-- 
William <wmorgan-sup at masanjin.net>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-don-t-tokenize-message_id-field-in-index.patch
Type: application/octet-stream
Size: 945 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/sup-talk/attachments/20080224/94955be9/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-only-use-the-first-255-characters-of-a-message-id-f.patch
Type: application/octet-stream
Size: 1046 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/sup-talk/attachments/20080224/94955be9/attachment-0001.obj

---

From: [email-masked] (Christopher Warrington)
Date: Mon, 25 Feb 2008 11:08:22 -0600

William Morgan @ 2008-2-24 11:10:57 PM
"[sup-talk] Amazon.com messages can't be added to index" <mid:1203915874-sup-5504 at south>

> Sigh. Why would anyone generate a message id like that?

I've seen some Netscape.com (webmail, I assume) messages with ids
like that. In fact, I just got one today:

Message-ID: <!~!UENERkVCMDkAAQACAAAAAAAAAAAAAAAAABgAAAAAAAAAXMhTt2UCrU6Chc6GzOTGusKAAAAQAAAASrY7ZritsEOjEOY8QXLoDQEAAAAA at netscape.com>

-- 
Christopher Warrington <chrisw at rice.edu>

"Be careful of reading health books: you might die of a misprint."
-Mark Twain
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 183 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/sup-talk/attachments/20080225/fa4b01ba/attachment.bin

---

From: [email-masked] (William Morgan)
Date: Mon, 25 Feb 2008 09:13:35 -0800

Reformatted excerpts from Christopher Warrington's message of 2008-02-25:
> Message-ID:
> <!~!UENERkVCMDkAAQACAAAAAAAAAAAAAAAAABgAAAAAAAAAXMhTt2UCrU6Chc6GzOTGusKAAAAQAAAASrY7ZritsEOjEOY8QXLoDQEAAAAA at netscape.com>

Hey, at least that one's under 255 characters. :)

Does seem to be the same software, though. !~! my ass.