Subject: [sup-devel] after second heliotrope import, new messages not appearing

From: Hamish D <[email-masked]>
Date: Thu, 27 Dec 2012 16:19:40 +0000

Hello

I've just been trying to carry on from where I left off with migrating
my work email into heliotrope (which I last worked on in May).

So I've rsynced the mbox files across to where I will run heliotrope
and tried running heliotrope-import to put all the new messages in the
index. It appeared to work properly, reporting that messages were
being added to the index and the index files themselves have had their
timestamps updated. But when I start turnsole, I can't find any email
after May.

Any ideas on where to start looking to work out what is going on?

Hamish
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Sun, 30 Dec 2012 17:22:51 +0100

Hello,

Your problem may have multiple origins, and I have no idea where you could
start :

- Did you use the correct directories for messages and index for _both_ the
import and the server ? (sorry if this sounds stupid)
- Did the index size in heliotrope change ? (as given by a GET to
/size.json)
- Did the messages file size change ?
- Are the messages absent in inbox only or from the whole turnsole ?
- As a last resort, could you send the mbox file so that I can try this at
home, if your mails aren't too personal ?

-- 
Matthieu RAKOTOJAONA

---

From: Hamish D <[email-masked]>
Date: Tue, 1 Jan 2013 18:33:09 +0000

> import and the server ? (sorry if this sounds stupid)

Always good to check the basics, but yes, I do use the same directory.
I have scripts for the import and the server and both use the -d
argument to ensure they use the same directory.

> - Did the index size in heliotrope change ? (as given by a GET to

No - it remained the same. (And is about half of the count that sup reports).

> - Did the messages file size change ?

It did on one import, but not the most recent.

> - Are the messages absent in inbox only or from the whole turnsole ?

All searches that I do stop in May.

> home, if your mails aren't too personal ?

It's my work email, with a total of over 100,000 messages. So I'd
prefer not to send it over.

Any logs that might be worth looking at? While doing the import, I do
get various lines along the line of:

    ; scanned 71, indexed 71, skipped 0 bad and 0 seen messages in
22.1s = 3.2 m/s

I do use the state argument (-t) to save reimporting old mail. I have
also tried doing a reindex (as I import from multiple mboxes) but to
no avail.

I could just try another import of the whole lot into a clean directory I guess.

I'm also happy to type stuff into heliotrope-console if that will help
diagnose stuff.

Thanks for your suggestions so far
Hamish

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Wed, 2 Jan 2013 21:47:06 +0100

So, it appears heliotrope tells you your mails are indexed, yet they don't
appear. All I see is `MetaIndex#add_message` returning in the test that
checks if the mail is already in the store. Problem is, the exact same
check is already done before that moment, and if the mail really was
already present, it should be marked as "seen" and not "indexed".

Another possibility could be `MetaIndex#gen_new_docid` which, for an
unknown reason, returns a wrong value, so it would always overwrite an old
message.

I'm not really good at debugging efficiently, but I would re-launch the
import after having placed a few "puts" in this `add_message` method, see
if the existence test returns true or not. If not, print the docid, and see
what the message with that docid is. If the message with that docid is the
one you just imported, then there is a problem with how we generate them
(because it was already given to another message).