* [sup-devel] after second heliotrope import, new messages not appearing @ 2012-12-27 16:19 Hamish D 2012-12-30 16:22 ` Matthieu Rakotojaona 0 siblings, 1 reply; 4+ messages in thread From: Hamish D @ 2012-12-27 16:19 UTC (permalink / raw) To: Sup developer discussion Hello I've just been trying to carry on from where I left off with migrating my work email into heliotrope (which I last worked on in May). So I've rsynced the mbox files across to where I will run heliotrope and tried running heliotrope-import to put all the new messages in the index. It appeared to work properly, reporting that messages were being added to the index and the index files themselves have had their timestamps updated. But when I start turnsole, I can't find any email after May. Any ideas on where to start looking to work out what is going on? Hamish _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [sup-devel] after second heliotrope import, new messages not appearing 2012-12-27 16:19 [sup-devel] after second heliotrope import, new messages not appearing Hamish D @ 2012-12-30 16:22 ` Matthieu Rakotojaona 2013-01-01 18:33 ` Hamish D 0 siblings, 1 reply; 4+ messages in thread From: Matthieu Rakotojaona @ 2012-12-30 16:22 UTC (permalink / raw) To: Sup developer discussion [-- Attachment #1.1: Type: text/plain, Size: 1378 bytes --] Hello, Your problem may have multiple origins, and I have no idea where you could start : - Did you use the correct directories for messages and index for _both_ the import and the server ? (sorry if this sounds stupid) - Did the index size in heliotrope change ? (as given by a GET to /size.json) - Did the messages file size change ? - Are the messages absent in inbox only or from the whole turnsole ? - As a last resort, could you send the mbox file so that I can try this at home, if your mails aren't too personal ? On Thu, Dec 27, 2012 at 5:19 PM, Hamish D <dmishd@gmail.com> wrote: > Hello > > I've just been trying to carry on from where I left off with migrating > my work email into heliotrope (which I last worked on in May). > > So I've rsynced the mbox files across to where I will run heliotrope > and tried running heliotrope-import to put all the new messages in the > index. It appeared to work properly, reporting that messages were > being added to the index and the index files themselves have had their > timestamps updated. But when I start turnsole, I can't find any email > after May. > > Any ideas on where to start looking to work out what is going on? > > Hamish > _______________________________________________ > Sup-devel mailing list > Sup-devel@rubyforge.org > http://rubyforge.org/mailman/listinfo/sup-devel > -- Matthieu RAKOTOJAONA [-- Attachment #1.2: Type: text/html, Size: 2034 bytes --] [-- Attachment #2: Type: text/plain, Size: 143 bytes --] _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [sup-devel] after second heliotrope import, new messages not appearing 2012-12-30 16:22 ` Matthieu Rakotojaona @ 2013-01-01 18:33 ` Hamish D 2013-01-02 20:47 ` Matthieu Rakotojaona 0 siblings, 1 reply; 4+ messages in thread From: Hamish D @ 2013-01-01 18:33 UTC (permalink / raw) To: Sup developer discussion > Your problem may have multiple origins, and I have no idea where you could > start : > > - Did you use the correct directories for messages and index for _both_ the > import and the server ? (sorry if this sounds stupid) Always good to check the basics, but yes, I do use the same directory. I have scripts for the import and the server and both use the -d argument to ensure they use the same directory. > - Did the index size in heliotrope change ? (as given by a GET to > /size.json) No - it remained the same. (And is about half of the count that sup reports). > - Did the messages file size change ? It did on one import, but not the most recent. > - Are the messages absent in inbox only or from the whole turnsole ? All searches that I do stop in May. > - As a last resort, could you send the mbox file so that I can try this at > home, if your mails aren't too personal ? It's my work email, with a total of over 100,000 messages. So I'd prefer not to send it over. Any logs that might be worth looking at? While doing the import, I do get various lines along the line of: ; scanned 71, indexed 71, skipped 0 bad and 0 seen messages in 22.1s = 3.2 m/s I do use the state argument (-t) to save reimporting old mail. I have also tried doing a reindex (as I import from multiple mboxes) but to no avail. I could just try another import of the whole lot into a clean directory I guess. I'm also happy to type stuff into heliotrope-console if that will help diagnose stuff. Thanks for your suggestions so far Hamish > On Thu, Dec 27, 2012 at 5:19 PM, Hamish D <dmishd@gmail.com> wrote: >> >> Hello >> >> I've just been trying to carry on from where I left off with migrating >> my work email into heliotrope (which I last worked on in May). >> >> So I've rsynced the mbox files across to where I will run heliotrope >> and tried running heliotrope-import to put all the new messages in the >> index. It appeared to work properly, reporting that messages were >> being added to the index and the index files themselves have had their >> timestamps updated. But when I start turnsole, I can't find any email >> after May. >> >> Any ideas on where to start looking to work out what is going on? >> >> Hamish _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [sup-devel] after second heliotrope import, new messages not appearing 2013-01-01 18:33 ` Hamish D @ 2013-01-02 20:47 ` Matthieu Rakotojaona 0 siblings, 0 replies; 4+ messages in thread From: Matthieu Rakotojaona @ 2013-01-02 20:47 UTC (permalink / raw) To: Sup developer discussion [-- Attachment #1.1: Type: text/plain, Size: 3534 bytes --] So, it appears heliotrope tells you your mails are indexed, yet they don't appear. All I see is `MetaIndex#add_message` returning in the test that checks if the mail is already in the store. Problem is, the exact same check is already done before that moment, and if the mail really was already present, it should be marked as "seen" and not "indexed". Another possibility could be `MetaIndex#gen_new_docid` which, for an unknown reason, returns a wrong value, so it would always overwrite an old message. I'm not really good at debugging efficiently, but I would re-launch the import after having placed a few "puts" in this `add_message` method, see if the existence test returns true or not. If not, print the docid, and see what the message with that docid is. If the message with that docid is the one you just imported, then there is a problem with how we generate them (because it was already given to another message). On Tue, Jan 1, 2013 at 7:33 PM, Hamish D <dmishd@gmail.com> wrote: > > Your problem may have multiple origins, and I have no idea where you > could > > start : > > > > - Did you use the correct directories for messages and index for _both_ > the > > import and the server ? (sorry if this sounds stupid) > > Always good to check the basics, but yes, I do use the same directory. > I have scripts for the import and the server and both use the -d > argument to ensure they use the same directory. > > > - Did the index size in heliotrope change ? (as given by a GET to > > /size.json) > > No - it remained the same. (And is about half of the count that sup > reports). > > > - Did the messages file size change ? > > It did on one import, but not the most recent. > > > - Are the messages absent in inbox only or from the whole turnsole ? > > All searches that I do stop in May. > > > - As a last resort, could you send the mbox file so that I can try this > at > > home, if your mails aren't too personal ? > > It's my work email, with a total of over 100,000 messages. So I'd > prefer not to send it over. > > Any logs that might be worth looking at? While doing the import, I do > get various lines along the line of: > > ; scanned 71, indexed 71, skipped 0 bad and 0 seen messages in > 22.1s = 3.2 m/s > > I do use the state argument (-t) to save reimporting old mail. I have > also tried doing a reindex (as I import from multiple mboxes) but to > no avail. > > I could just try another import of the whole lot into a clean directory I > guess. > > I'm also happy to type stuff into heliotrope-console if that will help > diagnose stuff. > > Thanks for your suggestions so far > Hamish > > > > On Thu, Dec 27, 2012 at 5:19 PM, Hamish D <dmishd@gmail.com> wrote: > >> > >> Hello > >> > >> I've just been trying to carry on from where I left off with migrating > >> my work email into heliotrope (which I last worked on in May). > >> > >> So I've rsynced the mbox files across to where I will run heliotrope > >> and tried running heliotrope-import to put all the new messages in the > >> index. It appeared to work properly, reporting that messages were > >> being added to the index and the index files themselves have had their > >> timestamps updated. But when I start turnsole, I can't find any email > >> after May. > >> > >> Any ideas on where to start looking to work out what is going on? > >> > >> Hamish > _______________________________________________ > Sup-devel mailing list > Sup-devel@rubyforge.org > http://rubyforge.org/mailman/listinfo/sup-devel > -- Matthieu RAKOTOJAONA [-- Attachment #1.2: Type: text/html, Size: 4724 bytes --] [-- Attachment #2: Type: text/plain, Size: 143 bytes --] _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2013-01-02 20:52 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-12-27 16:19 [sup-devel] after second heliotrope import, new messages not appearing Hamish D 2012-12-30 16:22 ` Matthieu Rakotojaona 2013-01-01 18:33 ` Hamish D 2013-01-02 20:47 ` Matthieu Rakotojaona
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox