From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by 10.142.125.10 with SMTP id x10cs229038wfc; Mon, 13 Jun 2011 21:48:16 -0700 (PDT) Received: by 10.224.197.132 with SMTP id ek4mr4528980qab.382.1308026895826; Mon, 13 Jun 2011 21:48:15 -0700 (PDT) Return-Path: Received: from rubyforge.org (rubyforge.org [205.234.109.19]) by mx.google.com with ESMTP id k2si10020526qct.140.2011.06.13.21.48.15; Mon, 13 Jun 2011 21:48:15 -0700 (PDT) Received-SPF: pass (google.com: domain of sup-devel-bounces@rubyforge.org designates 205.234.109.19 as permitted sender) client-ip=205.234.109.19; Authentication-Results: mx.google.com; spf=pass (google.com: domain of sup-devel-bounces@rubyforge.org designates 205.234.109.19 as permitted sender) smtp.mail=sup-devel-bounces@rubyforge.org; dkim=neutral (body hash did not verify) header.i=@gmail.com Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id 6055E18583A5 for ; Tue, 14 Jun 2011 00:48:15 -0400 (EDT) Received: from mail-vw0-f50.google.com (mail-vw0-f50.google.com [209.85.212.50]) by rubyforge.org (Postfix) with ESMTP id 785E41858372 for ; Tue, 14 Jun 2011 00:21:57 -0400 (EDT) Received: by vws14 with SMTP id 14so5578831vws.23 for ; Mon, 13 Jun 2011 21:21:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=eDZKtTicLAwst2yf8yXG56ett3U2FGceZZIAlj9Yx1g=; b=MOiYgg5AoWiZdrAgdJ/gSUPgjGibffaVvcNry1sYsF5yMxo84W92WZyeHfNqGoPYVJ LyQBE5clEt6Oj3ST2yZ8a2mrLwTH7gUMcg8QoZa59sxyMc244uaR9mJdNnDZB3BfGL0k CL4oWCpRIhmL2YRXFULNl6t8ZCq9HP9ot68hA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=KlXM52l2HjeqDC9aCy7hHIkOODtDgHP5WDq0JnocpJWc9Eh0HLGh5L6Z6dYKQRnzWT EHDTWcSf2QzCPecxCT5CxG92IromvvaryhriAWKcmB1UOmRpu/D0IQX73tnan7ZTDvBx v/6TV8ctOhFugEuOz8xf47WQ1gI7uNEwyN+o0= MIME-Version: 1.0 Received: by 10.52.96.5 with SMTP id do5mr1079250vdb.67.1308025316513; Mon, 13 Jun 2011 21:21:56 -0700 (PDT) Received: by 10.52.184.225 with HTTP; Mon, 13 Jun 2011 21:21:56 -0700 (PDT) In-Reply-To: <1308022018-sup-5391@masanjin.net> References: <1308017442-sup-8849@masanjin.net> <1308017654-sup-350@masanjin.net> <1308022018-sup-5391@masanjin.net> Date: Tue, 14 Jun 2011 13:21:56 +0900 Message-ID: From: Horacio Sanson To: Sup developer discussion Subject: Re: [sup-devel] Invalid meta data error in OklahomaMixer X-BeenThere: sup-devel@rubyforge.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Sup developer discussion List-Id: Sup developer discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: sup-devel-bounces@rubyforge.org Errors-To: sup-devel-bounces@rubyforge.org I managed to get the store.tch working again by installing tokyocabinet-bin and running this command on the file: tchmgr optimize -nl store.tch # without -nl nothing works... at the end of this command I got a write error but the file got back to a usable state. I am afraid this command simply truncated the file causing some missing emails but not really sure. Is there a way to check if the index and the messages are in sync? That is all indexed messages have a corresponding email message in the store.tch? Here is a good blog post with tips for using Tokyo Cabinet. It recommends enabling logging and running optimize from time to time. Logging would allow us to see what caused the metadata corruption: http://www.bigdbahead.com/?p=700 Some posts of people with the same invalid metadata problem: http://twitter.com/#!/ono_matope/statuses/17650765308 http://groups.google.com/group/tokyocabinet-users/browse_thread/thread/f114f078848e20ed Tokyo Cabinet is pretty good and is actually used for the largest social network in Japan (http://alpha.mixi.co.jp/blog/?s=tchdb&paged=2) so I would not change it. Even big databases (e.g. MySQL, Postgres and Oracle) are not free of table corruption. The difference is that they provide tools to detect and recover from such corruption state. regards, Horacio On Tue, Jun 14, 2011 at 12:30 PM, William Morgan wrote: > Reformatted excerpts from Horacio Sanson's message of 2011-06-14: >> Not even close... my store.tch is 76MB only. This metadata corruption >> in Tokyo Cabinet seems to be a common occurrence and what is scary >> about this is that there seems to be no way to recover from it (as far >> as google can tell). The only option I see is delete my store/index >> and restart sync again. > > Can you point me to some links about this type of corruption? > > I'm far from wedded to Oklahoma Mixer / Tokyo Cabinet. It was just the > easiest thing to get started with. Any kind of mutable persistent > key-value store will do. Other alternatives to consider: another > TokyoCabinet gem besides OH, KyotoCabinet, BDB... I kinda don't want to > move to separate processes like Redis though. If only there were a > libredis (well there is, but it's not what you want.) > -- > William > _______________________________________________ > Sup-devel mailing list > Sup-devel@rubyforge.org > http://rubyforge.org/mailman/listinfo/sup-devel > _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel