* [sup-devel] Invalid meta data error in OklahomaMixer
@ 2011-06-11 15:30 Horacio Sanson
2011-06-14 2:14 ` William Morgan
0 siblings, 1 reply; 6+ messages in thread
From: Horacio Sanson @ 2011-06-11 15:30 UTC (permalink / raw)
To: Sup developer discussion
After finishing sync of my gmail account to heliotrope I tried to
start the server but was received with this error message:
ruby1.9.1 -Ilib bin/heliotrope-server -d ~/.heliotrope
/var/lib/gems/1.9.1/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/hash_database.rb:337:in
`try': invalid meta data (error code 5)
(OklahomaMixer::Error::CabinetError)
from /var/lib/gems/1.9.1/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/hash_database.rb:44:in
`initialize'
from /var/lib/gems/1.9.1/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer.rb:17:in
`new'
from /var/lib/gems/1.9.1/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer.rb:17:in
`open'
from /media/DATA/Apps/heliotrope/lib/heliotrope/index.rb:44:in
`initialize'
from bin/heliotrope-server:576:in `new'
from bin/heliotrope-server:576:in `<main>'
seems something got corrupted and I have no idea how to recover. Any
tips that would fix this without resync my whole account would be
greatly appreciated.
Horacio
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [sup-devel] Invalid meta data error in OklahomaMixer
2011-06-11 15:30 [sup-devel] Invalid meta data error in OklahomaMixer Horacio Sanson
@ 2011-06-14 2:14 ` William Morgan
2011-06-14 2:14 ` William Morgan
0 siblings, 1 reply; 6+ messages in thread
From: William Morgan @ 2011-06-14 2:14 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-06-11:
> ruby1.9.1 -Ilib bin/heliotrope-server -d ~/.heliotrope
> /var/lib/gems/1.9.1/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/hash_database.rb:337:in
> `try': invalid meta data (error code 5)
> (OklahomaMixer::Error::CabinetError)
Very disturbing. I found this:
http://www.supermind.org/blog/557/tokyocabinet-hdb-slowdown
Did you hit 2GB on your store.tch file, by any chance?
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [sup-devel] Invalid meta data error in OklahomaMixer
2011-06-14 2:14 ` William Morgan
@ 2011-06-14 2:14 ` William Morgan
2011-06-14 2:37 ` Horacio Sanson
0 siblings, 1 reply; 6+ messages in thread
From: William Morgan @ 2011-06-14 2:14 UTC (permalink / raw)
To: sup-devel
Whoops, I mean this:
http://www.supermind.org/blog/554/2gb-limit-with-tokyo-cabinet-aka-invalid-meta-data
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [sup-devel] Invalid meta data error in OklahomaMixer
2011-06-14 2:14 ` William Morgan
@ 2011-06-14 2:37 ` Horacio Sanson
2011-06-14 3:30 ` William Morgan
0 siblings, 1 reply; 6+ messages in thread
From: Horacio Sanson @ 2011-06-14 2:37 UTC (permalink / raw)
To: Sup developer discussion
Not even close... my store.tch is 76MB only. This metadata corruption
in Tokyo Cabinet seems to be a common occurrence and what is scary
about this is that there seems to be no way to recover from it (as far
as google can tell). The only option I see is delete my store/index
and restart sync again.
regards,
Horacio
On Tue, Jun 14, 2011 at 11:14 AM, William Morgan
<wmorgan-sup@masanjin.net> wrote:
> Whoops, I mean this:
>
> http://www.supermind.org/blog/554/2gb-limit-with-tokyo-cabinet-aka-invalid-meta-data
> --
> William <wmorgan-sup@masanjin.net>
> _______________________________________________
> Sup-devel mailing list
> Sup-devel@rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-devel
>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [sup-devel] Invalid meta data error in OklahomaMixer
2011-06-14 2:37 ` Horacio Sanson
@ 2011-06-14 3:30 ` William Morgan
2011-06-14 4:21 ` Horacio Sanson
0 siblings, 1 reply; 6+ messages in thread
From: William Morgan @ 2011-06-14 3:30 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-06-14:
> Not even close... my store.tch is 76MB only. This metadata corruption
> in Tokyo Cabinet seems to be a common occurrence and what is scary
> about this is that there seems to be no way to recover from it (as far
> as google can tell). The only option I see is delete my store/index
> and restart sync again.
Can you point me to some links about this type of corruption?
I'm far from wedded to Oklahoma Mixer / Tokyo Cabinet. It was just the
easiest thing to get started with. Any kind of mutable persistent
key-value store will do. Other alternatives to consider: another
TokyoCabinet gem besides OH, KyotoCabinet, BDB... I kinda don't want to
move to separate processes like Redis though. If only there were a
libredis (well there is, but it's not what you want.)
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [sup-devel] Invalid meta data error in OklahomaMixer
2011-06-14 3:30 ` William Morgan
@ 2011-06-14 4:21 ` Horacio Sanson
0 siblings, 0 replies; 6+ messages in thread
From: Horacio Sanson @ 2011-06-14 4:21 UTC (permalink / raw)
To: Sup developer discussion
I managed to get the store.tch working again by installing
tokyocabinet-bin and running this command on the file:
tchmgr optimize -nl store.tch # without -nl nothing works...
at the end of this command I got a write error but the file got back
to a usable state. I am afraid this command simply truncated the file
causing some missing emails but not really sure. Is there a way to
check if the index and the messages are in sync? That is all indexed
messages have a corresponding email message in the store.tch?
Here is a good blog post with tips for using Tokyo Cabinet. It
recommends enabling logging and running optimize from time to time.
Logging would allow us to see what caused the metadata corruption:
http://www.bigdbahead.com/?p=700
Some posts of people with the same invalid metadata problem:
http://twitter.com/#!/ono_matope/statuses/17650765308
http://groups.google.com/group/tokyocabinet-users/browse_thread/thread/f114f078848e20ed
Tokyo Cabinet is pretty good and is actually used for the largest
social network in Japan
(http://alpha.mixi.co.jp/blog/?s=tchdb&paged=2) so I would not change
it. Even big databases (e.g. MySQL, Postgres and Oracle) are not free
of table corruption. The difference is that they provide tools to
detect and recover from such corruption state.
regards,
Horacio
On Tue, Jun 14, 2011 at 12:30 PM, William Morgan
<wmorgan-sup@masanjin.net> wrote:
> Reformatted excerpts from Horacio Sanson's message of 2011-06-14:
>> Not even close... my store.tch is 76MB only. This metadata corruption
>> in Tokyo Cabinet seems to be a common occurrence and what is scary
>> about this is that there seems to be no way to recover from it (as far
>> as google can tell). The only option I see is delete my store/index
>> and restart sync again.
>
> Can you point me to some links about this type of corruption?
>
> I'm far from wedded to Oklahoma Mixer / Tokyo Cabinet. It was just the
> easiest thing to get started with. Any kind of mutable persistent
> key-value store will do. Other alternatives to consider: another
> TokyoCabinet gem besides OH, KyotoCabinet, BDB... I kinda don't want to
> move to separate processes like Redis though. If only there were a
> libredis (well there is, but it's not what you want.)
> --
> William <wmorgan-sup@masanjin.net>
> _______________________________________________
> Sup-devel mailing list
> Sup-devel@rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-devel
>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-06-14 4:48 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-06-11 15:30 [sup-devel] Invalid meta data error in OklahomaMixer Horacio Sanson
2011-06-14 2:14 ` William Morgan
2011-06-14 2:14 ` William Morgan
2011-06-14 2:37 ` Horacio Sanson
2011-06-14 3:30 ` William Morgan
2011-06-14 4:21 ` Horacio Sanson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox