* [sup-talk] Trouble loading dump into Xapian index
@ 2009-09-03 14:24 Marc Weber
2009-09-03 15:25 ` William Morgan
0 siblings, 1 reply; 10+ messages in thread
From: Marc Weber @ 2009-09-03 14:24 UTC (permalink / raw)
I've used this sh script to get a new SUP_BASE using xapian only:
However somehow i couldn't import the tags? What have I done wrong?
dump2: The dump file created from the xapian index.
set -x
OLD=~/.sup-old/
NEW=~/.sup-new-test
DUMP=/tmp/dump-file-new
DUMP2=/tmp/dump-file-new2
SYNC_LOG=/tmp/sup-sync-log
GIT_REPO=~/managed_repos/sup_mainline
rm -fr $NEW
export SUP_BASE=$OLD
sup-dump > $DUMP
# from now on operate on the new base only
export SUP_BASE=$NEW
cp -r $OLD $NEW
rm -fr $NEW/ferret
mv $NEW/{hooks,hooks-disabled}
# echo loading stuff..
export SUP_INDEX=xapian
cd $GIT_REPO
echo "syncing.. this will take long. use tail -f $SYNC_LOG to watch progress"
ruby -Ilib bin/sup-sync --all --all-sources --restore $DUMP &> $SYNC_LOG
echo "writing dump"
ruby -Ilib bin/sup-dump > $DUMP2
dump: (first line after sorting)
000001c9fe57$346bf530$9d43df90$@com.au (unread openeeg)
dump2: (first line after sorting)
000001c9fe57$346bf530$9d43df90$@com.au (inbox unread)
Marc Weber
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] Trouble loading dump into Xapian index
2009-09-03 14:24 [sup-talk] Trouble loading dump into Xapian index Marc Weber
@ 2009-09-03 15:25 ` William Morgan
2009-09-03 16:21 ` Marc Weber
0 siblings, 1 reply; 10+ messages in thread
From: William Morgan @ 2009-09-03 15:25 UTC (permalink / raw)
Reformatted excerpts from Marc Weber's message of 2009-09-03:
> I've used this sh script to get a new SUP_BASE using xapian only:
> However somehow i couldn't import the tags? What have I done wrong?
This *might* be related to a patch that Rich sent that I thought I had
applied, but apparently did not. Can you try once more with the latest
master? If it still doesn't work we'll take it from there. (Also, check
that the number of messages sup-sync reports it has restored state on is
the number of messages you'd expect, e.g. roughly the number of lines in
the dumpfile.)
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] Trouble loading dump into Xapian index
2009-09-03 15:25 ` William Morgan
@ 2009-09-03 16:21 ` Marc Weber
2009-09-03 16:42 ` Rich Lane
0 siblings, 1 reply; 10+ messages in thread
From: Marc Weber @ 2009-09-03 16:21 UTC (permalink / raw)
Hi William.
I guess it was.
The second dump looks very similar to the first one now.
Only the order of flags is different now which doesn't matter.
However threads which have been killed do show up in inbox:
http://mawercer.de/~marc/sup.png
All seven mails of the first thread are labeled by
killed inbox deleted
So they shouldn't appear, should they?
Same happens to thread where all mails are labeled by
killed, inbox
Did the view behaviour change here?
Sincerly
Marc
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] Trouble loading dump into Xapian index
2009-09-03 16:21 ` Marc Weber
@ 2009-09-03 16:42 ` Rich Lane
2009-09-03 17:16 ` William Morgan
2009-09-03 17:26 ` Marc Weber
0 siblings, 2 replies; 10+ messages in thread
From: Rich Lane @ 2009-09-03 16:42 UTC (permalink / raw)
Excerpts from Marc Weber's message of Thu Sep 03 12:21:31 -0400 2009:
> However threads which have been killed do show up in inbox:
> http://mawercer.de/~marc/sup.png
> All seven mails of the first thread are labeled by
> killed inbox deleted
>
> So they shouldn't appear, should they?
>
> Same happens to thread where all mails are labeled by
> killed, inbox
>
> Did the view behaviour change here?
I added support for thread killing in 4d82ef88, which hasn't been merged
to master yet. If you'd like to use the Xapian index I suggest using
next for now.
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] Trouble loading dump into Xapian index
2009-09-03 16:42 ` Rich Lane
@ 2009-09-03 17:16 ` William Morgan
2009-09-04 17:00 ` Rich Lane
2009-09-03 17:26 ` Marc Weber
1 sibling, 1 reply; 10+ messages in thread
From: William Morgan @ 2009-09-03 17:16 UTC (permalink / raw)
Reformatted excerpts from Rich Lane's message of 2009-09-03:
> I added support for thread killing in 4d82ef88, which hasn't been
> merged to master yet. If you'd like to use the Xapian index I suggest
> using next for now.
If you feel it's reasonably stable, I can merge it into master.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] Trouble loading dump into Xapian index
2009-09-03 16:42 ` Rich Lane
2009-09-03 17:16 ` William Morgan
@ 2009-09-03 17:26 ` Marc Weber
2009-09-03 18:35 ` Rich Lane
1 sibling, 1 reply; 10+ messages in thread
From: Marc Weber @ 2009-09-03 17:26 UTC (permalink / raw)
> I added support for thread killing in 4d82ef88, which hasn't been merged
> to master yet. If you'd like to use the Xapian index I suggest using
> next for now.
Hi Rich, using next I get the following error:
+ ruby -Ilib bin/sup-sync --all --all-sources --restore /tmp/dump-file-new
./lib/sup/crypto.rb:17: warning: Insecure world writable dir /pr/webkos
in PATH, mode 040777
Loading state dump from /tmp/dump-file-new...
Read 39048 entries from dump file.
./lib/sup/xapian_index.rb:32:in `load_index': This Sup version expects a
v1 index, but you have an existing v0 index. Please downgrade to your
previous version and dump your labels before upgrading to this version
(then run sup-sync --restore). (RuntimeError)
from ./lib/sup/index.rb:67:in `load'
from bin/sup-sync:117
I looked at strace to and noticed that sup only accessed the new
~/.sup-new-test direcotry which was empty. So there is no v0 at all)
Marc Weber
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] Trouble loading dump into Xapian index
2009-09-03 17:26 ` Marc Weber
@ 2009-09-03 18:35 ` Rich Lane
2009-09-04 16:25 ` Marc Weber
0 siblings, 1 reply; 10+ messages in thread
From: Rich Lane @ 2009-09-03 18:35 UTC (permalink / raw)
Excerpts from Marc Weber's message of Thu Sep 03 13:26:59 -0400 2009:
>
> > I added support for thread killing in 4d82ef88, which hasn't been merged
> > to master yet. If you'd like to use the Xapian index I suggest using
> > next for now.
>
> Hi Rich, using next I get the following error:
>
> + ruby -Ilib bin/sup-sync --all --all-sources --restore /tmp/dump-file-new
> ./lib/sup/crypto.rb:17: warning: Insecure world writable dir /pr/webkos
> in PATH, mode 040777
> Loading state dump from /tmp/dump-file-new...
> Read 39048 entries from dump file.
> ./lib/sup/xapian_index.rb:32:in `load_index': This Sup version expects a
> v1 index, but you have an existing v0 index. Please downgrade to your
> previous version and dump your labels before upgrading to this version
> (then run sup-sync --restore). (RuntimeError)
> from ./lib/sup/index.rb:67:in `load'
> from bin/sup-sync:117
>
> I looked at strace to and noticed that sup only accessed the new
> ~/.sup-new-test direcotry which was empty. So there is no v0 at all)
That's strange, because this codepath (xapian_index.rb:32) is only hit
when the xapian directory already exists. Can you add a log message to
output the "path" variable and see if it looks correct?
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] Trouble loading dump into Xapian index
2009-09-03 18:35 ` Rich Lane
@ 2009-09-04 16:25 ` Marc Weber
0 siblings, 0 replies; 10+ messages in thread
From: Marc Weber @ 2009-09-04 16:25 UTC (permalink / raw)
Sorry.
It was my fault. I had a ~/.sup/xapian I forgot about. That made the
trouble causing the correct error message.
So I think it's safe to switch to Xapian. You may use this script.
It will copy the existing ~/.sup and create a new directory $NEW with
the xapian index. You cane set SUP_BASE to run the new sup to try it
out. Next feels so much better in various ways (speed, search in threads etc.)
Thanks for your all this work!
set -x
set -e
OLD=~/.sup
NEW=/pr/sup-new-test
DUMP=/tmp/dump-file-new
DUMP2=/tmp/dump-file-new2
SYNC_LOG=/tmp/sup-sync-log
GIT_REPO=~/managed_repos/sup_mainline
rm -fr $NEW
export SUP_BASE=$OLD
sup-dump > $DUMP
# from now on operate on the new base only
export SUP_BASE=$NEW
cp -r $OLD $NEW
rm -fr $NEW/ferret
rm -fr $NEW/xapian # remove old xapian cruft!
mv $NEW/{hooks,hooks-disabled}
# echo loading stuff..
export SUP_INDEX=xapian
cd $GIT_REPO
echo "syncing.. this will take long. use tail -f $SYNC_LOG to watch progress"
ruby -Ilib bin/sup-sync --all --all-sources --restore $DUMP &> $SYNC_LOG
echo "writing dump"
ruby -Ilib bin/sup-dump # > $DUMP2
Marc Weber
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] Trouble loading dump into Xapian index
2009-09-03 17:16 ` William Morgan
@ 2009-09-04 17:00 ` Rich Lane
2009-09-04 17:29 ` William Morgan
0 siblings, 1 reply; 10+ messages in thread
From: Rich Lane @ 2009-09-04 17:00 UTC (permalink / raw)
Excerpts from William Morgan's message of Thu Sep 03 13:16:41 -0400 2009:
> Reformatted excerpts from Rich Lane's message of 2009-09-03:
> > I added support for thread killing in 4d82ef88, which hasn't been
> > merged to master yet. If you'd like to use the Xapian index I suggest
> > using next for now.
>
> If you feel it's reasonably stable, I can merge it into master.
I think it's ok to merge.
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] Trouble loading dump into Xapian index
2009-09-04 17:00 ` Rich Lane
@ 2009-09-04 17:29 ` William Morgan
0 siblings, 0 replies; 10+ messages in thread
From: William Morgan @ 2009-09-04 17:29 UTC (permalink / raw)
Reformatted excerpts from Rich Lane's message of 2009-09-04:
> I think it's ok to merge.
Merged!
If you're running a Xapian from master (weird!) you will have to rebuild
your index after removing ~/.sup/{xapian,*.db}.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2009-09-04 17:29 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-09-03 14:24 [sup-talk] Trouble loading dump into Xapian index Marc Weber
2009-09-03 15:25 ` William Morgan
2009-09-03 16:21 ` Marc Weber
2009-09-03 16:42 ` Rich Lane
2009-09-03 17:16 ` William Morgan
2009-09-04 17:00 ` Rich Lane
2009-09-04 17:29 ` William Morgan
2009-09-03 17:26 ` Marc Weber
2009-09-03 18:35 ` Rich Lane
2009-09-04 16:25 ` Marc Weber
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox