* [sup-devel] after second heliotrope import, new messages not appearing
@ 2012-12-27 16:19 Hamish D
2012-12-30 16:22 ` Matthieu Rakotojaona
0 siblings, 1 reply; 4+ messages in thread
From: Hamish D @ 2012-12-27 16:19 UTC (permalink / raw)
To: Sup developer discussion
Hello
I've just been trying to carry on from where I left off with migrating
my work email into heliotrope (which I last worked on in May).
So I've rsynced the mbox files across to where I will run heliotrope
and tried running heliotrope-import to put all the new messages in the
index. It appeared to work properly, reporting that messages were
being added to the index and the index files themselves have had their
timestamps updated. But when I start turnsole, I can't find any email
after May.
Any ideas on where to start looking to work out what is going on?
Hamish
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [sup-devel] after second heliotrope import, new messages not appearing
2012-12-27 16:19 [sup-devel] after second heliotrope import, new messages not appearing Hamish D
@ 2012-12-30 16:22 ` Matthieu Rakotojaona
2013-01-01 18:33 ` Hamish D
0 siblings, 1 reply; 4+ messages in thread
From: Matthieu Rakotojaona @ 2012-12-30 16:22 UTC (permalink / raw)
To: Sup developer discussion
[-- Attachment #1.1: Type: text/plain, Size: 1378 bytes --]
Hello,
Your problem may have multiple origins, and I have no idea where you could
start :
- Did you use the correct directories for messages and index for _both_ the
import and the server ? (sorry if this sounds stupid)
- Did the index size in heliotrope change ? (as given by a GET to
/size.json)
- Did the messages file size change ?
- Are the messages absent in inbox only or from the whole turnsole ?
- As a last resort, could you send the mbox file so that I can try this at
home, if your mails aren't too personal ?
On Thu, Dec 27, 2012 at 5:19 PM, Hamish D <dmishd@gmail.com> wrote:
> Hello
>
> I've just been trying to carry on from where I left off with migrating
> my work email into heliotrope (which I last worked on in May).
>
> So I've rsynced the mbox files across to where I will run heliotrope
> and tried running heliotrope-import to put all the new messages in the
> index. It appeared to work properly, reporting that messages were
> being added to the index and the index files themselves have had their
> timestamps updated. But when I start turnsole, I can't find any email
> after May.
>
> Any ideas on where to start looking to work out what is going on?
>
> Hamish
> _______________________________________________
> Sup-devel mailing list
> Sup-devel@rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-devel
>
--
Matthieu RAKOTOJAONA
[-- Attachment #1.2: Type: text/html, Size: 2034 bytes --]
[-- Attachment #2: Type: text/plain, Size: 143 bytes --]
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [sup-devel] after second heliotrope import, new messages not appearing
2012-12-30 16:22 ` Matthieu Rakotojaona
@ 2013-01-01 18:33 ` Hamish D
2013-01-02 20:47 ` Matthieu Rakotojaona
0 siblings, 1 reply; 4+ messages in thread
From: Hamish D @ 2013-01-01 18:33 UTC (permalink / raw)
To: Sup developer discussion
> Your problem may have multiple origins, and I have no idea where you could
> start :
>
> - Did you use the correct directories for messages and index for _both_ the
> import and the server ? (sorry if this sounds stupid)
Always good to check the basics, but yes, I do use the same directory.
I have scripts for the import and the server and both use the -d
argument to ensure they use the same directory.
> - Did the index size in heliotrope change ? (as given by a GET to
> /size.json)
No - it remained the same. (And is about half of the count that sup reports).
> - Did the messages file size change ?
It did on one import, but not the most recent.
> - Are the messages absent in inbox only or from the whole turnsole ?
All searches that I do stop in May.
> - As a last resort, could you send the mbox file so that I can try this at
> home, if your mails aren't too personal ?
It's my work email, with a total of over 100,000 messages. So I'd
prefer not to send it over.
Any logs that might be worth looking at? While doing the import, I do
get various lines along the line of:
; scanned 71, indexed 71, skipped 0 bad and 0 seen messages in
22.1s = 3.2 m/s
I do use the state argument (-t) to save reimporting old mail. I have
also tried doing a reindex (as I import from multiple mboxes) but to
no avail.
I could just try another import of the whole lot into a clean directory I guess.
I'm also happy to type stuff into heliotrope-console if that will help
diagnose stuff.
Thanks for your suggestions so far
Hamish
> On Thu, Dec 27, 2012 at 5:19 PM, Hamish D <dmishd@gmail.com> wrote:
>>
>> Hello
>>
>> I've just been trying to carry on from where I left off with migrating
>> my work email into heliotrope (which I last worked on in May).
>>
>> So I've rsynced the mbox files across to where I will run heliotrope
>> and tried running heliotrope-import to put all the new messages in the
>> index. It appeared to work properly, reporting that messages were
>> being added to the index and the index files themselves have had their
>> timestamps updated. But when I start turnsole, I can't find any email
>> after May.
>>
>> Any ideas on where to start looking to work out what is going on?
>>
>> Hamish
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [sup-devel] after second heliotrope import, new messages not appearing
2013-01-01 18:33 ` Hamish D
@ 2013-01-02 20:47 ` Matthieu Rakotojaona
0 siblings, 0 replies; 4+ messages in thread
From: Matthieu Rakotojaona @ 2013-01-02 20:47 UTC (permalink / raw)
To: Sup developer discussion
[-- Attachment #1.1: Type: text/plain, Size: 3534 bytes --]
So, it appears heliotrope tells you your mails are indexed, yet they don't
appear. All I see is `MetaIndex#add_message` returning in the test that
checks if the mail is already in the store. Problem is, the exact same
check is already done before that moment, and if the mail really was
already present, it should be marked as "seen" and not "indexed".
Another possibility could be `MetaIndex#gen_new_docid` which, for an
unknown reason, returns a wrong value, so it would always overwrite an old
message.
I'm not really good at debugging efficiently, but I would re-launch the
import after having placed a few "puts" in this `add_message` method, see
if the existence test returns true or not. If not, print the docid, and see
what the message with that docid is. If the message with that docid is the
one you just imported, then there is a problem with how we generate them
(because it was already given to another message).
On Tue, Jan 1, 2013 at 7:33 PM, Hamish D <dmishd@gmail.com> wrote:
> > Your problem may have multiple origins, and I have no idea where you
> could
> > start :
> >
> > - Did you use the correct directories for messages and index for _both_
> the
> > import and the server ? (sorry if this sounds stupid)
>
> Always good to check the basics, but yes, I do use the same directory.
> I have scripts for the import and the server and both use the -d
> argument to ensure they use the same directory.
>
> > - Did the index size in heliotrope change ? (as given by a GET to
> > /size.json)
>
> No - it remained the same. (And is about half of the count that sup
> reports).
>
> > - Did the messages file size change ?
>
> It did on one import, but not the most recent.
>
> > - Are the messages absent in inbox only or from the whole turnsole ?
>
> All searches that I do stop in May.
>
> > - As a last resort, could you send the mbox file so that I can try this
> at
> > home, if your mails aren't too personal ?
>
> It's my work email, with a total of over 100,000 messages. So I'd
> prefer not to send it over.
>
> Any logs that might be worth looking at? While doing the import, I do
> get various lines along the line of:
>
> ; scanned 71, indexed 71, skipped 0 bad and 0 seen messages in
> 22.1s = 3.2 m/s
>
> I do use the state argument (-t) to save reimporting old mail. I have
> also tried doing a reindex (as I import from multiple mboxes) but to
> no avail.
>
> I could just try another import of the whole lot into a clean directory I
> guess.
>
> I'm also happy to type stuff into heliotrope-console if that will help
> diagnose stuff.
>
> Thanks for your suggestions so far
> Hamish
>
>
> > On Thu, Dec 27, 2012 at 5:19 PM, Hamish D <dmishd@gmail.com> wrote:
> >>
> >> Hello
> >>
> >> I've just been trying to carry on from where I left off with migrating
> >> my work email into heliotrope (which I last worked on in May).
> >>
> >> So I've rsynced the mbox files across to where I will run heliotrope
> >> and tried running heliotrope-import to put all the new messages in the
> >> index. It appeared to work properly, reporting that messages were
> >> being added to the index and the index files themselves have had their
> >> timestamps updated. But when I start turnsole, I can't find any email
> >> after May.
> >>
> >> Any ideas on where to start looking to work out what is going on?
> >>
> >> Hamish
> _______________________________________________
> Sup-devel mailing list
> Sup-devel@rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-devel
>
--
Matthieu RAKOTOJAONA
[-- Attachment #1.2: Type: text/html, Size: 4724 bytes --]
[-- Attachment #2: Type: text/plain, Size: 143 bytes --]
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2013-01-02 20:52 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-12-27 16:19 [sup-devel] after second heliotrope import, new messages not appearing Hamish D
2012-12-30 16:22 ` Matthieu Rakotojaona
2013-01-01 18:33 ` Hamish D
2013-01-02 20:47 ` Matthieu Rakotojaona
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox