* [sup-devel] Tokyo Cabinet fails with invalid record header
@ 2011-06-14 8:36 Horacio Sanson
2011-06-14 17:30 ` William Morgan
0 siblings, 1 reply; 24+ messages in thread
From: Horacio Sanson @ 2011-06-14 8:36 UTC (permalink / raw)
To: Sup developer discussion
Well I will retract what I said earlier... maybe is a good idea to
replace TokyoCabinet with something else.
Now the store.tch got corrupted again for some reason and the only
error I get is "invalid record header". Some googling does not help
and my previous fix (tchmgr optimiza -nl) does not fix this problem.
What is interesting is that the store.tch file is at 76MB now that is
exactly the same size it had when the previous corruption occured.
=========== error ============
; found 496204 new messages...
; found 496204 messages to scan
; requesting messages 79228..79327 from imap server
/var/lib/gems/1.9.1/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/hash_database.rb:337:in
`try': invalid record header (error code 6)
(OklahomaMixer::Error::CabinetError)
from /var/lib/gems/1.9.1/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/hash_database.rb:142:in
`fetch'
from /var/lib/gems/1.9.1/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/hash_database.rb:205:in
`include?'
from /media/DATA/Apps/heliotrope/lib/heliotrope/index.rb:633:in
`contains_key?'
from /media/DATA/Apps/heliotrope/lib/heliotrope/index.rb:165:in
`contains_msgid?'
from bin/heliotrope-add:139:in `<main>'
regards,
Horacio
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-14 8:36 [sup-devel] Tokyo Cabinet fails with invalid record header Horacio Sanson
@ 2011-06-14 17:30 ` William Morgan
2011-06-14 22:36 ` Horacio Sanson
0 siblings, 1 reply; 24+ messages in thread
From: William Morgan @ 2011-06-14 17:30 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-06-14:
> Now the store.tch got corrupted again for some reason and the only
> error I get is "invalid record header". Some googling does not help
> and my previous fix (tchmgr optimiza -nl) does not fix this problem.
> What is interesting is that the store.tch file is at 76MB now that is
> exactly the same size it had when the previous corruption occured.
Hm. And you're not running out of disk space or anything wierd like
that, are you?
Maybe you're encountering a limitation on the number of keys? What does
this say for you?
$ ruby -Ilib bin/heliotrope-console -d store
Heliotrope console started.
[...]
> s = index.instance_eval { @store }
[...]
> s.size
=> 234201
> s.keys.size
=> 234201
(My store.tch is only 30mb.)
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-14 17:30 ` William Morgan
@ 2011-06-14 22:36 ` Horacio Sanson
2011-06-15 15:11 ` William Morgan
2011-06-15 15:16 ` William Morgan
0 siblings, 2 replies; 24+ messages in thread
From: Horacio Sanson @ 2011-06-14 22:36 UTC (permalink / raw)
To: Sup developer discussion
Interesting.... yesterday I started again to sync my gmail account
but this time using ruby1.8.7 instead of ruby1.9.1 and without any
hooks configured and it finished without problems. I think the problem
is the hooks so I will rerun again with ruby 1.9.1 but without the
hooks.
On Wed, Jun 15, 2011 at 2:30 AM, William Morgan
<wmorgan-sup@masanjin.net> wrote:
> Reformatted excerpts from Horacio Sanson's message of 2011-06-14:
>> Now the store.tch got corrupted again for some reason and the only
>> error I get is "invalid record header". Some googling does not help
>> and my previous fix (tchmgr optimiza -nl) does not fix this problem.
>> What is interesting is that the store.tch file is at 76MB now that is
>> exactly the same size it had when the previous corruption occured.
>
> Hm. And you're not running out of disk space or anything wierd like
> that, are you?
>
> Maybe you're encountering a limitation on the number of keys? What does
> this say for you?
>
> $ ruby -Ilib bin/heliotrope-console -d store
> Heliotrope console started.
> [...]
> > s = index.instance_eval { @store }
> [...]
> > s.size
> => 234201
> > s.keys.size
> => 234201
>
> (My store.tch is only 30mb.)
irb> s = index.instance_eval { @store }
irb> s.size
=> 3469212
irb> s.keys.size
=> 3469212
$ ls -lh ~/.heliotrope
合計 3.8G
drwxrwxr-x 2 ryujin ryujin 4.0K 2011-06-14 17:36 hooks
-rw-r----- 1 ryujin ryujin 4.1M 2011-06-15 07:26 index0.lb
-rw-r----- 1 ryujin ryujin 513M 2011-06-15 07:27 index0.pr
-rw-r----- 1 ryujin ryujin 13M 2011-06-15 07:28 index0.sh_
-rw-r----- 1 ryujin ryujin 65M 2011-06-15 07:27 index0.sp
-rw-r----- 1 ryujin ryujin 37M 2011-06-15 07:28 index0.th
-rw-rw-r-- 1 ryujin ryujin 3.0G 2011-06-15 07:27 messages
-rw-r--r-- 1 ryujin ryujin 378M 2011-06-15 07:28 store.tch
> --
> William <wmorgan-sup@masanjin.net>
> _______________________________________________
> Sup-devel mailing list
> Sup-devel@rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-devel
>
regards
Horacio
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-14 22:36 ` Horacio Sanson
@ 2011-06-15 15:11 ` William Morgan
2011-06-18 5:02 ` William Morgan
2011-06-15 15:16 ` William Morgan
1 sibling, 1 reply; 24+ messages in thread
From: William Morgan @ 2011-06-15 15:11 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-06-14:
> Interesting.... yesterday I started again to sync my gmail account
> but this time using ruby1.8.7 instead of ruby1.9.1 and without any
> hooks configured and it finished without problems. I think the problem
> is the hooks so I will rerun again with ruby 1.9.1 but without the
> hooks.
There is a difference in how strings are marshalled between Ruby 1.8 and
1.9, and you will obviously get different strings if you tokenize
differently, but if either of these two changes is the difference, it's
a bug in TC.
One option is to write some Ruby bindings to LevelDB and try that, but
it would be a bunch of work, would require C++0x which means every OS X
user would be complaining all the time, etc etc. Not sure if I want to
go down that route.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-15 15:11 ` William Morgan
@ 2011-06-18 5:02 ` William Morgan
2011-06-19 13:25 ` Horacio Sanson
0 siblings, 1 reply; 24+ messages in thread
From: William Morgan @ 2011-06-18 5:02 UTC (permalink / raw)
To: sup-devel
I have pushed a branch called 'leveldb' that uses leveldb instead of
TokyoCabinet. Can you try this please? You will need to install the
'leveldb-ruby' gem.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-18 5:02 ` William Morgan
@ 2011-06-19 13:25 ` Horacio Sanson
2011-06-19 18:30 ` William Morgan
0 siblings, 1 reply; 24+ messages in thread
From: Horacio Sanson @ 2011-06-19 13:25 UTC (permalink / raw)
To: Sup developer discussion
On Saturday 18 June 2011 14:02:59 William Morgan wrote:
> I have pushed a branch called 'leveldb' that uses leveldb instead of
> TokyoCabinet. Can you try this please? You will need to install the
> 'leveldb-ruby' gem.
Sorry but I am unable to install the leveldb-gem. It complains about a missing
header and I have already installed all the packages that contain that file and
still no luck:
###########################################
sudo gem1.9.1 install leveldb-ruby
Building native extensions. This could take a while...
ERROR: Error installing leveldb-ruby:
ERROR: Failed to build gem native extension.
/usr/bin/ruby1.9.1 extconf.rb
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 db/builder.cc -o db/builder.o
In file included from ./port/port.h:14:0,
from ./db/filename.h:14,
from db/builder.cc:7:
./port/port_posix.h:14:22: fatal error: cstdatomic: そのようなファイルやディレクトリはありません
compilation terminated.
make: *** [db/builder.o] エラー 1
##############################################
The error message says that the cstdatomic file does not exists. I installed
the libstdc++6-4.4-dev package that has the file in
(/usr/include/c++/4.4/cstdatomic) but the build system fails to find it.
--
regards,
Horacio Sanson
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-19 13:25 ` Horacio Sanson
@ 2011-06-19 18:30 ` William Morgan
2011-06-19 21:59 ` William Morgan
0 siblings, 1 reply; 24+ messages in thread
From: William Morgan @ 2011-06-19 18:30 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-06-19:
> ./port/port_posix.h:14:22: fatal error: cstdatomic: そのようなファイルやディレクトリはありません
What is the output of g++ --version on your system, please?
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-19 18:30 ` William Morgan
@ 2011-06-19 21:59 ` William Morgan
2011-06-19 22:48 ` Horacio Sanson
0 siblings, 1 reply; 24+ messages in thread
From: William Morgan @ 2011-06-19 21:59 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from William Morgan's message of 2011-06-19:
> Reformatted excerpts from Horacio Sanson's message of 2011-06-19:
> > ./port/port_posix.h:14:22: fatal error: cstdatomic: そのようなファイルやディレクトリはありません
>
> What is the output of g++ --version on your system, please?
Try version 0.4. I believe it should work for oyu.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-19 21:59 ` William Morgan
@ 2011-06-19 22:48 ` Horacio Sanson
2011-06-20 3:54 ` William Morgan
0 siblings, 1 reply; 24+ messages in thread
From: Horacio Sanson @ 2011-06-19 22:48 UTC (permalink / raw)
To: Sup developer discussion
This morning I tried with g++-4.4 and got a different error. When I return from
work I will look into it. In the meantime this is the error:
rm -f libleveldb.a
ar -rs libleveldb.a ./db/builder.o ./db/db_impl.o ./db/db_iter.o
./db/filename.o ./db/dbformat.o ./db/log_reader.o ./db/log_writer.o
./db/memtable.o ./db/repair.o ./db/table_cache.o ./db/version_edit.o
./db/version_set.o ./db/write_batch.o ./port/port_posix.o ./table/block.o
./table/block_builder.o ./table/format.o ./table/iterator.o ./table/merger.o
./table/table.o ./table/table_builder.o ./table/two_level_iterator.o
./util/arena.o ./util/cache.o ./util/coding.o ./util/comparator.o
./util/crc32c.o ./util/env.o ./util/env_posix.o ./util/hash.o
./util/histogram.o ./util/logging.o ./util/options.o ./util/status.o
ar: creating libleveldb.a
creating Makefile
make
g++ -I. -I/usr/lib/ruby/1.8/x86_64-linux -I/usr/lib/ruby/1.8/x86_64-linux -I.
-fPIC -fno-strict-aliasing -g -g -O2 -fPIC -I../../leveldb/include -c
leveldb.cc
gcc -shared -o leveldb.so leveldb.o -L. -L/usr/lib -L. -Wl,-Bsymbolic-
functions -rdynamic -Wl,-export-dynamic -lruby1.8 -lpthread -lrt -ldl -
lcrypt -lm -lc -L../../leveldb -lleveldb
/usr/bin/ld: ../../leveldb/libleveldb.a(db_impl.o): relocation R_X86_64_32S
against `vtable for leveldb::DBImpl::RecoverLogFile(unsigned long,
leveldb::VersionEdit*, unsigned long*)::LogReporter' can not be used when
making a shared object; recompile with -fPIC
../../leveldb/libleveldb.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [leveldb.so] エラー 1
This is my g++:
g++ --version
g++ (Ubuntu/Linaro 4.4.5-15ubuntu1) 4.4.5
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
On Monday 20 June 2011 06:59:02 William Morgan wrote:
> Reformatted excerpts from William Morgan's message of 2011-06-19:
> > Reformatted excerpts from Horacio Sanson's message of 2011-06-19:
> > > ./port/port_posix.h:14:22: fatal error: cstdatomic:
> > > そのようなファイルやディレクトリはありません
> >
> > What is the output of g++ --version on your system, please?
>
> Try version 0.4. I believe it should work for oyu.
--
regards,
Horacio Sanson
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-19 22:48 ` Horacio Sanson
@ 2011-06-20 3:54 ` William Morgan
2011-06-20 5:23 ` Horacio Sanson
0 siblings, 1 reply; 24+ messages in thread
From: William Morgan @ 2011-06-20 3:54 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-06-19:
> This morning I tried with g++-4.4 and got a different error. When I
> return from work I will look into it. In the meantime this is the
> error:
Hm. Try with 0.4. If that doesn't work then I can try modifying the
original leveldb Makefile to use -fPIC. I didn't think it would be
necessary, but you apparently have some funny computer that is breaking
all my assumptions.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-20 3:54 ` William Morgan
@ 2011-06-20 5:23 ` Horacio Sanson
2011-06-20 16:42 ` William Morgan
0 siblings, 1 reply; 24+ messages in thread
From: Horacio Sanson @ 2011-06-20 5:23 UTC (permalink / raw)
To: Sup developer discussion
On Monday 20 June 2011 12:54:22 William Morgan wrote:
> Reformatted excerpts from Horacio Sanson's message of 2011-06-19:
> > This morning I tried with g++-4.4 and got a different error. When I
> > return from work I will look into it. In the meantime this is the
>
> > error:
> Hm. Try with 0.4. If that doesn't work then I can try modifying the
> original leveldb Makefile to use -fPIC. I didn't think it would be
> necessary, but you apparently have some funny computer that is breaking
> all my assumptions.
When you say try 0.4 you mean leveldb-ruby version 0.4 right?, because I
downloaded the source from github and the latest version I get is 0.3. Also
rubygems only has version 0.3.
https://github.com/wmorgan/leveldb-ruby
http://rubygems.org/gems/leveldb-ruby
--
regards,
Horacio Sanson
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-20 5:23 ` Horacio Sanson
@ 2011-06-20 16:42 ` William Morgan
2011-06-20 22:26 ` Horacio Sanson
0 siblings, 1 reply; 24+ messages in thread
From: William Morgan @ 2011-06-20 16:42 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-06-20:
> When you say try 0.4 you mean leveldb-ruby version 0.4 right?, because
> I downloaded the source from github and the latest version I get is
> 0.3. Also rubygems only has version 0.3.
Whoops, forgot to upload it. Try now. (0.4 should have the fix for
cstdatomic issue, but nothing with -fPIC.)
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-20 16:42 ` William Morgan
@ 2011-06-20 22:26 ` Horacio Sanson
2011-06-20 22:52 ` William Morgan
0 siblings, 1 reply; 24+ messages in thread
From: Horacio Sanson @ 2011-06-20 22:26 UTC (permalink / raw)
To: Sup developer discussion
On Tuesday 21 June 2011 01:42:24 William Morgan wrote:
> Reformatted excerpts from Horacio Sanson's message of 2011-06-20:
> > When you say try 0.4 you mean leveldb-ruby version 0.4 right?, because
> > I downloaded the source from github and the latest version I get is
> > 0.3. Also rubygems only has version 0.3.
>
> Whoops, forgot to upload it. Try now. (0.4 should have the fix for
> cstdatomic issue, but nothing with -fPIC.)
Indeed it fixed the cstdatomic issue but not the linking one:
####################################
sudo gem1.9.1 install leveldb-ruby
Building native extensions. This could take a while...
ERROR: Error installing leveldb-ruby:
ERROR: Failed to build gem native extension.
/usr/bin/ruby1.9.1 extconf.rb
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 db/builder.cc -o db/builder.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 db/db_impl.cc -o db/db_impl.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 db/db_iter.cc -o db/db_iter.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 db/filename.cc -o db/filename.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 db/dbformat.cc -o db/dbformat.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 db/log_reader.cc -o db/log_reader.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 db/log_writer.cc -o db/log_writer.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 db/memtable.cc -o db/memtable.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 db/repair.cc -o db/repair.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 db/table_cache.cc -o db/table_cache.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 db/version_edit.cc -o db/version_edit.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 db/version_set.cc -o db/version_set.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 db/write_batch.cc -o db/write_batch.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 port/port_posix.cc -o port/port_posix.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 table/block.cc -o table/block.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 table/block_builder.cc -o table/block_builder.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 table/format.cc -o table/format.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 table/iterator.cc -o table/iterator.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 table/merger.cc -o table/merger.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 table/table.cc -o table/table.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 table/table_builder.cc -o table/table_builder.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 table/two_level_iterator.cc -o table/two_level_iterator.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 util/arena.cc -o util/arena.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 util/cache.cc -o util/cache.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 util/coding.cc -o util/coding.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 util/comparator.cc -o util/comparator.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 util/crc32c.cc -o util/crc32c.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 util/env.cc -o util/env.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 util/env_posix.cc -o util/env_posix.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 util/hash.cc -o util/hash.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 util/histogram.cc -o util/histogram.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 util/logging.cc -o util/logging.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 util/options.cc -o util/options.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG
-DUSE_SNAPPY=1 util/status.cc -o util/status.o
rm -f libleveldb.a
ar -rs libleveldb.a ./db/builder.o ./db/db_impl.o ./db/db_iter.o
./db/filename.o ./db/dbformat.o ./db/log_reader.o ./db/log_writer.o
./db/memtable.o ./db/repair.o ./db/table_cache.o ./db/version_edit.o
./db/version_set.o ./db/write_batch.o ./port/port_posix.o ./table/block.o
./table/block_builder.o ./table/format.o ./table/iterator.o ./table/merger.o
./table/table.o ./table/table_builder.o ./table/two_level_iterator.o
./util/arena.o ./util/cache.o ./util/coding.o ./util/comparator.o
./util/crc32c.o ./util/env.o ./util/env_posix.o ./util/hash.o
./util/histogram.o ./util/logging.o ./util/options.o ./util/status.o
ar: creating libleveldb.a
creating Makefile
make
g++ -I. -I/usr/include/ruby-1.9.1/x86_64-linux -
I/usr/include/ruby-1.9.1/ruby/backward -I/usr/include/ruby-1.9.1 -I. -fPIC -
fno-strict-aliasing -g -g -O2 -fPIC -I../../leveldb/include -fno-strict-
aliasing -g -o leveldb.o -c leveldb.cc
g++ -shared -o leveldb.so leveldb.o -L. -L/usr/lib -L. -rdynamic -Wl,-export-
dynamic -lruby-1.9.1 -lpthread -lrt -ldl -lcrypt -lm -lc -L../../leveldb
-lleveldb
/usr/bin/ld: ../../leveldb/libleveldb.a(db_impl.o): relocation R_X86_64_32S
against `vtable for leveldb::DB' can not be used when making a shared object;
recompile with -fPIC
../../leveldb/libleveldb.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [leveldb.so] エラー 1
Gem files will remain installed in /var/lib/gems/1.9.1/gems/leveldb-ruby-0.4
for inspection.
Results logged to /var/lib/gems/1.9.1/gems/leveldb-
ruby-0.4/ext/leveldb/gem_make.out
################################
I tried qwith both g++-4.4 and g++-4.5 that come with standard Ubuntu 11.04
distribution.
--
regards,
Horacio Sanson
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-20 22:26 ` Horacio Sanson
@ 2011-06-20 22:52 ` William Morgan
2011-06-22 3:25 ` William Morgan
0 siblings, 1 reply; 24+ messages in thread
From: William Morgan @ 2011-06-20 22:52 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-06-20:
> /usr/bin/ld: ../../leveldb/libleveldb.a(db_impl.o): relocation
> R_X86_64_32S against `vtable for leveldb::DB' can not be used when
> making a shared object;
Ok, I have a potential fix for this that I'll try and get out tonight.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-20 22:52 ` William Morgan
@ 2011-06-22 3:25 ` William Morgan
2011-06-22 12:38 ` Horacio Sanson
2011-06-22 22:41 ` Horacio Sanson
0 siblings, 2 replies; 24+ messages in thread
From: William Morgan @ 2011-06-22 3:25 UTC (permalink / raw)
To: sup-devel
Hi Horacio,
Reformatted excerpts from William Morgan's message of 2011-06-20:
> Ok, I have a potential fix for this that I'll try and get out tonight.
Can you please:
1. git clone git://github.com/wmorgan/leveldb-ruby.git
2. git checkout fpic
3. rake gem
4. gem install pkg/leveldb-ruby-0.4.gem
And tell me if that works?
If so, I will release that as 0.5.
Thanks!
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-22 3:25 ` William Morgan
@ 2011-06-22 12:38 ` Horacio Sanson
2011-06-22 18:06 ` William Morgan
2011-06-22 22:41 ` Horacio Sanson
1 sibling, 1 reply; 24+ messages in thread
From: Horacio Sanson @ 2011-06-22 12:38 UTC (permalink / raw)
To: Sup developer discussion
On Wednesday 22 June 2011 12:25:16 William Morgan wrote:
> Hi Horacio,
>
> Reformatted excerpts from William Morgan's message of 2011-06-20:
> > Ok, I have a potential fix for this that I'll try and get out tonight.
>
> Can you please:
>
> 1. git clone git://github.com/wmorgan/leveldb-ruby.git
> 2. git checkout fpic
> 3. rake gem
> 4. gem install pkg/leveldb-ruby-0.4.gem
>
> And tell me if that works?
>
It works!!
> If so, I will release that as 0.5.
>
I am downloading my emails again using the leveldb branch.
> Thanks!
Thanks to you....
--
regards,
Horacio Sanson
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-22 12:38 ` Horacio Sanson
@ 2011-06-22 18:06 ` William Morgan
2011-06-28 17:42 ` Alvaro Herrera
0 siblings, 1 reply; 24+ messages in thread
From: William Morgan @ 2011-06-22 18:06 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-06-22:
> It works!!
Great. I will release that as leveldb-ruby 0.5. Hopefully the leveldb
branch of heliotrope will work for you too.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-22 18:06 ` William Morgan
@ 2011-06-28 17:42 ` Alvaro Herrera
2011-06-29 5:18 ` William Morgan
0 siblings, 1 reply; 24+ messages in thread
From: Alvaro Herrera @ 2011-06-28 17:42 UTC (permalink / raw)
To: sup-devel
Excerpts from William Morgan's message of mié jun 22 14:06:44 -0400 2011:
> Reformatted excerpts from Horacio Sanson's message of 2011-06-22:
> > It works!!
>
> Great. I will release that as leveldb-ruby 0.5. Hopefully the leveldb
> branch of heliotrope will work for you too.
Hi, I was just trying to get the latest Heliotrope, but I'm getting the
same -fPIC problem with leveldb-ruby. Gem says it's trying to install
0.4 though; maybe you forgot to actually release 0.5?
Last line of failing gem install:
/usr/bin/ld: ../../leveldb/libleveldb.a(db_impl.o): relocation R_X86_64_32S against `vtable for leveldb::DB' can not be used when making a shared object; recompile with -fPIC
../../leveldb/libleveldb.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [leveldb.so] Error 1
Gem files will remain installed in /var/lib/gems/1.9.1/gems/leveldb-ruby-0.4 for inspection.
Results logged to /var/lib/gems/1.9.1/gems/leveldb-ruby-0.4/ext/leveldb/gem_make.out
--
Álvaro Herrera <alvherre@alvh.no-ip.org>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-22 3:25 ` William Morgan
2011-06-22 12:38 ` Horacio Sanson
@ 2011-06-22 22:41 ` Horacio Sanson
2011-06-22 23:36 ` William Morgan
1 sibling, 1 reply; 24+ messages in thread
From: Horacio Sanson @ 2011-06-22 22:41 UTC (permalink / raw)
To: Sup developer discussion
I left heliotrope-add running but it did not finished. I got some strange nil
errors (see below):
I think this error happens when find_msgids is unable to find the message's id.
Not sure what is the best action in this case but skipping it sounds a good
option.
###################################################
; requesting messages 141445..141564 from imap server
scanned 130401, indexed 130278, skipped 25 bad and 98 seen messages in 9566.0s
= 13.6 m/s
; requesting messages 141565..141688 from imap server
scanned 130501, indexed 130378, skipped 25 bad and 98 seen messages in 9572.0s
= 13.6 m/s
/media/DATA/Apps/heliotrope/lib/heliotrope/message.rb:155:in `digest': can't
convert nil into String (TypeError)
from /media/DATA/Apps/heliotrope/lib/heliotrope/message.rb:155:in
`hexdigest'
from /media/DATA/Apps/heliotrope/lib/heliotrope/message.rb:155:in
`munge_msgid'
from /media/DATA/Apps/heliotrope/lib/heliotrope/message.rb:20:in
`parse!'
from bin/heliotrope-add:138:in `<main>'
####################################################
This error happens when @imap.uid_fetch doesn't raise any exception and
returns nil instead of empty array. Was thinking to check for nil and set it
to an empty array if true but it would likely result in an infinite loop.
####################################################
; loading mail...
; connecting...
; login as horacio.sanson@gmail.com ...
; found 293004 new messages...
; found 293004 messages to scan
; requesting messages 283664..283763 from imap server
scanned 1, indexed 1, skipped 0 bad and 0 seen messages in 7.9s = 0.1 m/s
; requesting messages 283764..283863 from imap server
/media/DATA/Apps/heliotrope/lib/heliotrope/gmail-dumper.rb:171:in
`next_message': undefined method `empty?' for nil:NilClass (NoMethodError)
from bin/heliotrope-add:128:in `<main>'
#####################################################
Finally the leveldb branch is not working that well. After reaching 130378
indexed messages I started the heliotrope server and the web interface only
showed two messages.
I will try again without the hooks for Japanese and see how it goes.
On Wednesday 22 June 2011 12:25:16 William Morgan wrote:
> Hi Horacio,
>
> Reformatted excerpts from William Morgan's message of 2011-06-20:
> > Ok, I have a potential fix for this that I'll try and get out tonight.
>
> Can you please:
>
> 1. git clone git://github.com/wmorgan/leveldb-ruby.git
> 2. git checkout fpic
> 3. rake gem
> 4. gem install pkg/leveldb-ruby-0.4.gem
>
> And tell me if that works?
>
> If so, I will release that as 0.5.
>
> Thanks!
--
regards,
Horacio Sanson
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-22 22:41 ` Horacio Sanson
@ 2011-06-22 23:36 ` William Morgan
2011-06-23 13:29 ` Horacio Sanson
0 siblings, 1 reply; 24+ messages in thread
From: William Morgan @ 2011-06-22 23:36 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-06-22:
> Finally the leveldb branch is not working that well. After reaching
> 130378 indexed messages I started the heliotrope server and the web
> interface only showed two messages.
What if you search for "*"?
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-22 23:36 ` William Morgan
@ 2011-06-23 13:29 ` Horacio Sanson
2011-06-23 15:44 ` William Morgan
0 siblings, 1 reply; 24+ messages in thread
From: Horacio Sanson @ 2011-06-23 13:29 UTC (permalink / raw)
To: Sup developer discussion
On Thursday 23 June 2011 08:36:07 William Morgan wrote:
> Reformatted excerpts from Horacio Sanson's message of 2011-06-22:
> > Finally the leveldb branch is not working that well. After reaching
> > 130378 indexed messages I started the heliotrope server and the web
> > interface only showed two messages.
>
> What if you search for "*"?
Search for "*" return zero items...
wget http://localhost:8042/search?q=*
search(body:"*", 0, 20) took 0.0ms
127.0.0.1 - - [23/Jun/2011 22:26:30] "GET /search?q=* HTTP/1.1" 200 813 0.0010
localhost - - [23/Jun/2011:22:26:30 JST] "GET /search?q=* HTTP/1.1" 200 813
http://localhost:8042/search?q=%7Einbox -> /search?q=*
127.0.0.1 - - [23/Jun/2011 22:26:30] "GET /favicon.ico HTTP/1.1" 404 447
0.0006
localhost - - [23/Jun/2011:22:26:30 JST] "GET /favicon.ico HTTP/1.1" 404 447
- -> /favicon.ico
I ran the heliotrope-add with and without hooks and both resulted in no
messages even though it reached 130378 messages indexed before crashing with a
nil error.
--
regards,
Horacio Sanson
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [sup-devel] Tokyo Cabinet fails with invalid record header
2011-06-14 22:36 ` Horacio Sanson
2011-06-15 15:11 ` William Morgan
@ 2011-06-15 15:16 ` William Morgan
1 sibling, 0 replies; 24+ messages in thread
From: William Morgan @ 2011-06-15 15:16 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-06-14:
> irb> s.size
> => 3469212
Ok, squarely between 2^21 and 2^22. It doesn't seem like you're bumping
up against some kind of key size limit.
> -rw-r--r-- 1 ryujin ryujin 378M 2011-06-15 07:28 store.tch
Looks like 378mb here. You were sayings 78mb earlier. Was that a
different set of mail?
If you have a corrupted store.tch file, keep it around. Maybe we can get
one of the TC people to help us.
But maybe the easiest next thing to try is another TC gem besides
OklahomaMixer.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2011-06-29 5:08 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-06-14 8:36 [sup-devel] Tokyo Cabinet fails with invalid record header Horacio Sanson
2011-06-14 17:30 ` William Morgan
2011-06-14 22:36 ` Horacio Sanson
2011-06-15 15:11 ` William Morgan
2011-06-18 5:02 ` William Morgan
2011-06-19 13:25 ` Horacio Sanson
2011-06-19 18:30 ` William Morgan
2011-06-19 21:59 ` William Morgan
2011-06-19 22:48 ` Horacio Sanson
2011-06-20 3:54 ` William Morgan
2011-06-20 5:23 ` Horacio Sanson
2011-06-20 16:42 ` William Morgan
2011-06-20 22:26 ` Horacio Sanson
2011-06-20 22:52 ` William Morgan
2011-06-22 3:25 ` William Morgan
2011-06-22 12:38 ` Horacio Sanson
2011-06-22 18:06 ` William Morgan
2011-06-28 17:42 ` Alvaro Herrera
2011-06-29 5:18 ` William Morgan
2011-06-22 22:41 ` Horacio Sanson
2011-06-22 23:36 ` William Morgan
2011-06-23 13:29 ` Horacio Sanson
2011-06-23 15:44 ` William Morgan
2011-06-15 15:16 ` William Morgan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox