* [sup-talk] [PATCH] xapian: drop excessively long terms
@ 2009-08-01 22:56 Rich Lane
2009-08-05 13:55 ` William Morgan
0 siblings, 1 reply; 9+ messages in thread
From: Rich Lane @ 2009-08-01 22:56 UTC (permalink / raw)
---
lib/sup/xapian_index.rb | 4 +++-
1 files changed, 3 insertions(+), 1 deletions(-)
diff --git a/lib/sup/xapian_index.rb b/lib/sup/xapian_index.rb
index 6358a20..5a5dfc1 100644
--- a/lib/sup/xapian_index.rb
+++ b/lib/sup/xapian_index.rb
@@ -304,6 +304,8 @@ class XapianIndex < BaseIndex
DATE_VALUENO = 0
+ MAX_TERM_LENGTH = 245
+
# Xapian can very efficiently sort in ascending docid order. Sup always wants
# to sort by descending date, so this method maps between them. In order to
# handle multiple messages per second, we use a logistic curve centered
@@ -428,7 +430,7 @@ class XapianIndex < BaseIndex
@term_generator.document = doc
text.each { |text,prefix| @term_generator.index_text text, 1, prefix }
- terms.each { |term| doc.add_term term }
+ terms.each { |term| doc.add_term term if term.length <= MAX_TERM_LENGTH }
doc.add_value DATE_VALUENO, date_value
doc.data = m.id
--
1.6.0.4
^ permalink raw reply [flat|nested] 9+ messages in thread
* [sup-talk] [PATCH] xapian: drop excessively long terms
2009-08-01 22:56 [sup-talk] [PATCH] xapian: drop excessively long terms Rich Lane
@ 2009-08-05 13:55 ` William Morgan
2009-08-05 14:54 ` Edward Z. Yang
2009-08-05 15:36 ` Rich Lane
0 siblings, 2 replies; 9+ messages in thread
From: William Morgan @ 2009-08-05 13:55 UTC (permalink / raw)
Applied, thanks!
BTW I've merged the xapian stuff into master.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 9+ messages in thread
* [sup-talk] [PATCH] xapian: drop excessively long terms
2009-08-05 13:55 ` William Morgan
@ 2009-08-05 14:54 ` Edward Z. Yang
2009-08-05 15:36 ` Rich Lane
1 sibling, 0 replies; 9+ messages in thread
From: Edward Z. Yang @ 2009-08-05 14:54 UTC (permalink / raw)
Excerpts from William Morgan's message of Wed Aug 05 09:55:47 -0400 2009:
> BTW I've merged the xapian stuff into master.
Exciting days we live in. :-) Maybe I'll wait another week and update my local
checkout.
Cheers,
Edward
^ permalink raw reply [flat|nested] 9+ messages in thread
* [sup-talk] [PATCH] xapian: drop excessively long terms
2009-08-05 13:55 ` William Morgan
2009-08-05 14:54 ` Edward Z. Yang
@ 2009-08-05 15:36 ` Rich Lane
2009-08-05 15:48 ` William Morgan
2009-08-05 17:29 ` Nicolas Pouillard
1 sibling, 2 replies; 9+ messages in thread
From: Rich Lane @ 2009-08-05 15:36 UTC (permalink / raw)
Excerpts from William Morgan's message of Wed Aug 05 09:55:47 -0400 2009:
> BTW I've merged the xapian stuff into master.
Let's not advertise this as stable just yet - I've got a patchset that
removes the GDBM databases which will break compatibility. Using just
Xapian means better consistency if your kernel panics and potentially
better performance. However, I have some Xapian bugs to squash before
it's usable.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [sup-talk] [PATCH] xapian: drop excessively long terms
2009-08-05 15:36 ` Rich Lane
@ 2009-08-05 15:48 ` William Morgan
2009-08-05 17:29 ` Nicolas Pouillard
1 sibling, 0 replies; 9+ messages in thread
From: William Morgan @ 2009-08-05 15:48 UTC (permalink / raw)
Reformatted excerpts from Rich Lane's message of 2009-08-05:
> Let's not advertise this as stable just yet - I've got a patchset that
> removes the GDBM databases which will break compatibility.
Since the xapian stuff is only enabled via an obscure environment
variable, I'm not too concerned about it being uber-stable. I've merged
this more for the internal API changes, which seem stable and which I'm
now building off of.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 9+ messages in thread
* [sup-talk] [PATCH] xapian: drop excessively long terms
2009-08-05 15:36 ` Rich Lane
2009-08-05 15:48 ` William Morgan
@ 2009-08-05 17:29 ` Nicolas Pouillard
2009-08-05 17:48 ` Rich Lane
1 sibling, 1 reply; 9+ messages in thread
From: Nicolas Pouillard @ 2009-08-05 17:29 UTC (permalink / raw)
Excerpts from Rich Lane's message of Wed Aug 05 17:36:19 +0200 2009:
> Excerpts from William Morgan's message of Wed Aug 05 09:55:47 -0400 2009:
> > BTW I've merged the xapian stuff into master.
>
> Let's not advertise this as stable just yet -
> I've got a patchset that
> removes the GDBM databases which will break compatibility.
I was planning to try a migration today, should I better wait a little?
--
Nicolas Pouillard
http://nicolaspouillard.fr
^ permalink raw reply [flat|nested] 9+ messages in thread
* [sup-talk] [PATCH] xapian: drop excessively long terms
2009-08-05 17:29 ` Nicolas Pouillard
@ 2009-08-05 17:48 ` Rich Lane
2009-08-05 17:58 ` Nicolas Pouillard
0 siblings, 1 reply; 9+ messages in thread
From: Rich Lane @ 2009-08-05 17:48 UTC (permalink / raw)
Excerpts from Nicolas Pouillard's message of Wed Aug 05 13:29:48 -0400 2009:
> I was planning to try a migration today, should I better wait a little?
I'd say go for it, just be aware you'll need to reindex soon. For the
full experience compile svn xapian and export XAPIAN_PREFER_CHERT=1.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [sup-talk] [PATCH] xapian: drop excessively long terms
2009-08-05 17:48 ` Rich Lane
@ 2009-08-05 17:58 ` Nicolas Pouillard
2009-08-05 18:06 ` Rich Lane
0 siblings, 1 reply; 9+ messages in thread
From: Nicolas Pouillard @ 2009-08-05 17:58 UTC (permalink / raw)
Excerpts from Rich Lane's message of Wed Aug 05 19:48:11 +0200 2009:
> Excerpts from Nicolas Pouillard's message of Wed Aug 05 13:29:48 -0400 2009:
> > I was planning to try a migration today, should I better wait a little?
>
> I'd say go for it, just be aware you'll need to reindex soon. For the
> full experience compile svn xapian and export XAPIAN_PREFER_CHERT=1.
When you say the svn version you mean the trunk or the 1.0-branch ?
--
Nicolas Pouillard
http://nicolaspouillard.fr
^ permalink raw reply [flat|nested] 9+ messages in thread
* [sup-talk] [PATCH] xapian: drop excessively long terms
2009-08-05 17:58 ` Nicolas Pouillard
@ 2009-08-05 18:06 ` Rich Lane
0 siblings, 0 replies; 9+ messages in thread
From: Rich Lane @ 2009-08-05 18:06 UTC (permalink / raw)
Excerpts from Nicolas Pouillard's message of Wed Aug 05 13:58:43 -0400 2009:
> Excerpts from Rich Lane's message of Wed Aug 05 19:48:11 +0200 2009:
> > Excerpts from Nicolas Pouillard's message of Wed Aug 05 13:29:48 -0400 2009:
> > > I was planning to try a migration today, should I better wait a little?
> >
> > I'd say go for it, just be aware you'll need to reindex soon. For the
> > full experience compile svn xapian and export XAPIAN_PREFER_CHERT=1.
>
> When you say the svn version you mean the trunk or the 1.0-branch ?
Trunk.
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2009-08-05 18:06 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-08-01 22:56 [sup-talk] [PATCH] xapian: drop excessively long terms Rich Lane
2009-08-05 13:55 ` William Morgan
2009-08-05 14:54 ` Edward Z. Yang
2009-08-05 15:36 ` Rich Lane
2009-08-05 15:48 ` William Morgan
2009-08-05 17:29 ` Nicolas Pouillard
2009-08-05 17:48 ` Rich Lane
2009-08-05 17:58 ` Nicolas Pouillard
2009-08-05 18:06 ` Rich Lane
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox