Subject: [sup-talk] Trouble loading dump into Xapian index

From: [email-masked] (Marc Weber)
Date: Thu, 03 Sep 2009 16:24:32 +0200

I've used this sh script to get a new SUP_BASE using xapian only:
However somehow i couldn't import the tags? What have I done wrong?
dump2: The dump file created from the xapian index.

  set -x
  OLD=~/.sup-old/
  NEW=~/.sup-new-test
  DUMP=/tmp/dump-file-new
  DUMP2=/tmp/dump-file-new2
  SYNC_LOG=/tmp/sup-sync-log
  GIT_REPO=~/managed_repos/sup_mainline

  rm -fr $NEW

  export SUP_BASE=$OLD
  sup-dump > $DUMP

  # from now on operate on the new base only
  export SUP_BASE=$NEW
  cp -r $OLD $NEW
  rm -fr $NEW/ferret
  mv $NEW/{hooks,hooks-disabled}

  # echo loading stuff..
  export SUP_INDEX=xapian
  cd $GIT_REPO 
  echo "syncing.. this will take long. use tail -f $SYNC_LOG to watch progress"
  ruby -Ilib bin/sup-sync --all --all-sources --restore $DUMP &> $SYNC_LOG

  echo "writing dump"
  ruby -Ilib bin/sup-dump > $DUMP2

dump: (first line after sorting)
[email-masked] (unread openeeg)

dump2: (first line after sorting)
[email-masked] (inbox unread)

Marc Weber

---

From: [email-masked] (William Morgan)
Date: Thu, 03 Sep 2009 08:25:50 -0700

> However somehow i couldn't import the tags? What have I done wrong?

This *might* be related to a patch that Rich sent that I thought I had
applied, but apparently did not. Can you try once more with the latest
master? If it still doesn't work we'll take it from there. (Also, check
that the number of messages sup-sync reports it has restored state on is
the number of messages you'd expect, e.g. roughly the number of lines in
the dumpfile.)
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Marc Weber)
Date: Thu, 03 Sep 2009 18:21:31 +0200

Hi William.

I guess it was.

The second dump looks very similar to the first one now.
Only the order of flags is different now which doesn't matter.

However threads which have been killed do show up in inbox:
http://mawercer.de/~marc/sup.png
All seven mails of the first thread are labeled by
  killed inbox deleted

So they shouldn't appear, should they?

Same happens to thread where all mails are labeled by 
  killed, inbox

Did the view behaviour change here?

Sincerly
Marc

---

From: [email-masked] (Rich Lane)
Date: Thu, 03 Sep 2009 12:42:55 -0400

> Did the view behaviour change here?

I added support for thread killing in 4d82ef88, which hasn't been merged
to master yet. If you'd like to use the Xapian index I suggest using
next for now.

---

From: [email-masked] (William Morgan)
Date: Thu, 03 Sep 2009 10:16:41 -0700

Reformatted excerpts from Rich Lane's message of 2009-09-03:
> I added support for thread killing in 4d82ef88, which hasn't been
> merged to master yet. If you'd like to use the Xapian index I suggest
> using next for now.

If you feel it's reasonably stable, I can merge it into master.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Marc Weber)
Date: Thu, 03 Sep 2009 19:26:59 +0200

> to master yet. If you'd like to use the Xapian index I suggest using

Hi Rich, using next I get the following error:

+ ruby -Ilib bin/sup-sync --all --all-sources --restore /tmp/dump-file-new
./lib/sup/crypto.rb:17: warning: Insecure world writable dir /pr/webkos
in PATH, mode 040777
Loading state dump from /tmp/dump-file-new...
Read 39048 entries from dump file.
./lib/sup/xapian_index.rb:32:in `load_index': This Sup version expects a
v1 index, but you have an existing v0 index. Please downgrade to your
previous version and dump your labels before upgrading to this version
(then run sup-sync --restore). (RuntimeError)
        from ./lib/sup/index.rb:67:in `load'
        from bin/sup-sync:117

I looked at strace to and noticed that sup only accessed the new
~/.sup-new-test direcotry which was empty. So there is no v0 at all)

Marc Weber

---

From: [email-masked] (Rich Lane)
Date: Thu, 03 Sep 2009 14:35:05 -0400

> ~/.sup-new-test direcotry which was empty. So there is no v0 at all)

That's strange, because this codepath (xapian_index.rb:32) is only hit
when the xapian directory already exists. Can you add a log message to
output the "path" variable and see if it looks correct?

---

From: [email-masked] (Marc Weber)
Date: Fri, 04 Sep 2009 18:25:39 +0200

Sorry.

It was my fault. I had a ~/.sup/xapian I forgot about. That made the
trouble causing the correct error message.

So I think it's safe to switch to Xapian. You may use this script.
It will copy the existing  ~/.sup and create a new directory $NEW with
the xapian index. You cane set SUP_BASE to run the new sup to try it
out. Next feels so much better in various ways (speed, search in threads etc.)

Thanks for your all this work!

  set -x
  set -e
  OLD=~/.sup
  NEW=/pr/sup-new-test
  DUMP=/tmp/dump-file-new

  rm -fr $NEW

  export SUP_BASE=$OLD
  sup-dump > $DUMP

  # from now on operate on the new base only
  export SUP_BASE=$NEW
  cp -r $OLD $NEW
  rm -fr $NEW/ferret
  rm -fr $NEW/xapian # remove old xapian cruft!
  mv $NEW/{hooks,hooks-disabled}

  # echo loading stuff..
  export SUP_INDEX=xapian
  cd $GIT_REPO 

  echo "writing dump"
  ruby -Ilib bin/sup-dump # > $DUMP2

Marc Weber

---

From: [email-masked] (Rich Lane)
Date: Fri, 04 Sep 2009 13:00:57 -0400

> If you feel it's reasonably stable, I can merge it into master.

I think it's ok to merge.

---

From: [email-masked] (William Morgan)
Date: Fri, 04 Sep 2009 10:29:23 -0700

> I think it's ok to merge.

Merged!

If you're running a Xapian from master (weird!) you will have to rebuild
your index after removing ~/.sup/{xapian,*.db}.
-- 
William <wmorgan-sup at masanjin.net>