Subject: Re: [sup-devel] Tokyo Cabinet fails with invalid record header

From: William Morgan <[email-masked]>
Date: Mon, 20 Jun 2011 16:42:24 +0000

Reformatted excerpts from Horacio Sanson's message of 2011-06-20:
> When you say try 0.4 you mean leveldb-ruby version 0.4 right?, because
> I downloaded the source from github and the latest version I get is
> 0.3. Also rubygems only has version 0.3.

Whoops, forgot to upload it. Try now. (0.4 should have the fix for
cstdatomic issue, but nothing with -fPIC.)
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Horacio Sanson <[email-masked]>
Date: Wed, 15 Jun 2011 07:36:51 +0900

Interesting....  yesterday I started again to sync my gmail account
but this time using ruby1.8.7 instead of ruby1.9.1 and without any
hooks configured and it finished without problems. I think the problem
is the hooks so I will rerun again with ruby 1.9.1 but without the
hooks.

On Wed, Jun 15, 2011 at 2:30 AM, William Morgan
<[email-masked]> wrote:
> Reformatted excerpts from Horacio Sanson's message of 2011-06-14:
>> Now the store.tch got corrupted again for some reason and the only
>> error I get is "invalid record header". Some googling does not help
>> and my previous fix (tchmgr optimiza -nl) does not fix this problem.
>> What is interesting is that the store.tch file is at 76MB now that is
>> exactly the same size it had when the previous corruption occured.
>
> Hm. And you're not running out of disk space or anything wierd like
> that, are you?
>
> Maybe you're encountering a limitation on the number of keys? What does
> this say for you?
>
>  $ ruby -Ilib bin/heliotrope-console -d store
>  Heliotrope console started.
>  [...]
>  > s = index.instance_eval { @store }
>  [...]
>  > s.size
>  => 234201
>  > s.keys.size
>  => 234201
>
> (My store.tch is only 30mb.)

irb> s = index.instance_eval { @store }
irb> s.size
=> 3469212
irb> s.keys.size
=> 3469212

$ ls -lh ~/.heliotrope
合計 3.8G
drwxrwxr-x 2 ryujin ryujin 4.0K 2011-06-14 17:36 hooks
-rw-r----- 1 ryujin ryujin 4.1M 2011-06-15 07:26 index0.lb
-rw-r----- 1 ryujin ryujin 513M 2011-06-15 07:27 index0.pr
-rw-r----- 1 ryujin ryujin  13M 2011-06-15 07:28 index0.sh_
-rw-r----- 1 ryujin ryujin  65M 2011-06-15 07:27 index0.sp
-rw-r----- 1 ryujin ryujin  37M 2011-06-15 07:28 index0.th
-rw-rw-r-- 1 ryujin ryujin 3.0G 2011-06-15 07:27 messages
-rw-r--r-- 1 ryujin ryujin 378M 2011-06-15 07:28 store.tch

regards
Horacio

---

From: William Morgan <[email-masked]>
Date: Tue, 14 Jun 2011 17:30:32 +0000

> s = index.instance_eval { @store }
  [...]
  > s.size
  => 234201
  > s.keys.size
  => 234201

---

From: Horacio Sanson <[email-masked]>
Date: Wed, 22 Jun 2011 21:38:50 +0900

On Wednesday 22 June 2011 12:25:16 William Morgan wrote:
> Hi Horacio,
> 
> Reformatted excerpts from William Morgan's message of 2011-06-20:
> > Ok, I have a potential fix for this that I'll try and get out tonight.
> 
> Can you please:
> 
> 1. git clone git://github.com/wmorgan/leveldb-ruby.git
> 2. git checkout fpic
> 3. rake gem
> 4. gem install pkg/leveldb-ruby-0.4.gem
> 
> And tell me if that works?
> 
It works!!

> If so, I will release that as 0.5.
>

I am downloading my emails again using the leveldb branch.
 
> Thanks!

Thanks to you....

-- 
regards,                                                                                                                                                                                                       
Horacio Sanson

---

From: William Morgan <[email-masked]>
Date: Thu, 23 Jun 2011 15:44:20 +0000

Reformatted excerpts from Horacio Sanson's message of 2011-06-23:
> Search for "*" return zero items...

I forgot to add: make sure you have whistlepig 0.7 installed.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Horacio Sanson <[email-masked]>
Date: Tue, 14 Jun 2011 17:36:02 +0900

Well I will retract what I said earlier... maybe is a good idea to
replace TokyoCabinet with something else.

===========  error ============
; found 496204 new messages...
; found 496204 messages to scan
; requesting messages 79228..79327 from imap server
/var/lib/gems/1.9.1/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/hash_database.rb:337:in
`try': invalid record header (error code 6)
(OklahomaMixer::Error::CabinetError)
        from /var/lib/gems/1.9.1/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/hash_database.rb:142:in
`fetch'
        from /var/lib/gems/1.9.1/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/hash_database.rb:205:in
`include?'
        from /media/DATA/Apps/heliotrope/lib/heliotrope/index.rb:633:in
`contains_key?'
        from /media/DATA/Apps/heliotrope/lib/heliotrope/index.rb:165:in
`contains_msgid?'
        from bin/heliotrope-add:139:in `<main>'

regards,
Horacio

---

From: William Morgan <[email-masked]>
Date: Wed, 15 Jun 2011 15:11:54 +0000

> is the hooks so I will rerun again with ruby 1.9.1 but without the

There is a difference in how strings are marshalled between Ruby 1.8 and
1.9, and you will obviously get different strings if you tokenize
differently, but if either of these two changes is the difference, it's
a bug in TC.

One option is to write some Ruby bindings to LevelDB and try that, but
it would be a bunch of work, would require C++0x which means every OS X
user would be complaining all the time, etc etc. Not sure if I want to
go down that route.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Horacio Sanson <[email-masked]>
Date: Mon, 20 Jun 2011 07:48:38 +0900

This morning I tried with g++-4.4 and got a different error. When I return from 
work I will look into it. In the meantime this is the error:

rm -f libleveldb.a
ar -rs libleveldb.a ./db/builder.o ./db/db_impl.o ./db/db_iter.o 
./db/filename.o ./db/dbformat.o ./db/log_reader.o ./db/log_writer.o 
./db/memtable.o ./db/repair.o ./db/table_cache.o ./db/version_edit.o 
./db/version_set.o ./db/write_batch.o ./port/port_posix.o ./table/block.o 
./table/block_builder.o ./table/format.o ./table/iterator.o ./table/merger.o 
./table/table.o ./table/table_builder.o ./table/two_level_iterator.o 
./util/arena.o ./util/cache.o ./util/coding.o ./util/comparator.o 
./util/crc32c.o ./util/env.o ./util/env_posix.o ./util/hash.o 
./util/histogram.o ./util/logging.o ./util/options.o ./util/status.o
ar: creating libleveldb.a
creating Makefile

make
g++ -I. -I/usr/lib/ruby/1.8/x86_64-linux -I/usr/lib/ruby/1.8/x86_64-linux -I.   
-fPIC -fno-strict-aliasing -g -g -O2  -fPIC  -I../../leveldb/include   -c 
leveldb.cc
gcc -shared -o leveldb.so leveldb.o -L. -L/usr/lib -L. -Wl,-Bsymbolic-
functions -rdynamic -Wl,-export-dynamic    -lruby1.8  -lpthread -lrt -ldl -
lcrypt -lm   -lc -L../../leveldb -lleveldb
/usr/bin/ld: ../../leveldb/libleveldb.a(db_impl.o): relocation R_X86_64_32S 
against `vtable for leveldb::DBImpl::RecoverLogFile(unsigned long, 
leveldb::VersionEdit*, unsigned long*)::LogReporter' can not be used when 
making a shared object; recompile with -fPIC
../../leveldb/libleveldb.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [leveldb.so] エラー 1

This is my g++:

g++ --version
g++ (Ubuntu/Linaro 4.4.5-15ubuntu1) 4.4.5
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

On Monday 20 June 2011 06:59:02 William Morgan wrote:
> Reformatted excerpts from William Morgan's message of 2011-06-19:
> > Reformatted excerpts from Horacio Sanson's message of 2011-06-19:
> > > ./port/port_posix.h:14:22: fatal error: cstdatomic:
> > > そのようなファイルやディレクトリはありません
> > 
> > What is the output of g++ --version on your system, please?
> 
> Try version 0.4. I believe it should work for oyu.

---

From: William Morgan <[email-masked]>
Date: Wed, 29 Jun 2011 05:18:55 +0000

Reformatted excerpts from Alvaro Herrera's message of 2011-06-28:
> Hi, I was just trying to get the latest Heliotrope, but I'm getting the
> same -fPIC problem with leveldb-ruby.  Gem says it's trying to install
> 0.4 though; maybe you forgot to actually release 0.5?

Yeah, if you try again you should get it. Sorry about that.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Wed, 15 Jun 2011 15:16:20 +0000

> irb> s.size

Ok, squarely between 2^21 and 2^22. It doesn't seem like you're bumping
up against some kind of key size limit.

> -rw-r--r-- 1 ryujin ryujin 378M 2011-06-15 07:28 store.tch

Looks like 378mb here. You were sayings 78mb earlier. Was that a
different set of mail?

If you have a corrupted store.tch file, keep it around. Maybe we can get
one of the TC people to help us.

But maybe the easiest next thing to try is another TC gem besides
OklahomaMixer.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Horacio Sanson <[email-masked]>
Date: Mon, 20 Jun 2011 14:23:26 +0900

On Monday 20 June 2011 12:54:22 William Morgan wrote:
> > This morning I tried with g++-4.4 and got a different error. When I
> > return from work I will look into it. In the meantime this is the
> 
> > error:
> Hm. Try with 0.4. If that doesn't work then I can try modifying the
> original leveldb Makefile to use -fPIC. I didn't think it would be
> necessary, but you apparently have some funny computer that is breaking
> all my assumptions.

When you say try 0.4 you mean leveldb-ruby version 0.4 right?, because I 
downloaded the source from github and the latest version I get is 0.3. Also 
rubygems only has version 0.3.

https://github.com/wmorgan/leveldb-ruby
http://rubygems.org/gems/leveldb-ruby

---

From: William Morgan <[email-masked]>
Date: Wed, 22 Jun 2011 18:06:44 +0000

> It works!!

Great. I will release that as leveldb-ruby 0.5. Hopefully the leveldb
branch of heliotrope will work for you too.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Horacio Sanson <[email-masked]>
Date: Sun, 19 Jun 2011 22:25:34 +0900

On Saturday 18 June 2011 14:02:59 William Morgan wrote:
> I have pushed a branch called 'leveldb' that uses leveldb instead of
> TokyoCabinet. Can you try this please? You will need to install the
> 'leveldb-ruby' gem.

Sorry but I am unable to install the leveldb-gem. It complains about a missing 
header and I have already installed all the packages that contain that file and 
still no luck:

###########################################
sudo gem1.9.1 install leveldb-ruby
Building native extensions.  This could take a while...
ERROR:  Error installing leveldb-ruby:
        ERROR: Failed to build gem native extension.

/usr/bin/ruby1.9.1 extconf.rb
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 db/builder.cc -o db/builder.o
In file included from ./port/port.h:14:0,
                 from ./db/filename.h:14,
                 from db/builder.cc:7:
./port/port_posix.h:14:22: fatal error: cstdatomic: そのようなファイルやディレクトリはありません
compilation terminated.
make: *** [db/builder.o] エラー 1
##############################################

The error message says that the cstdatomic file does not exists. I installed 
the  libstdc++6-4.4-dev package that has the file in 
(/usr/include/c++/4.4/cstdatomic) but the build system fails to find it.

---

From: William Morgan <[email-masked]>
Date: Sun, 19 Jun 2011 21:59:02 +0000

---

From: Alvaro Herrera <[email-masked]>
Date: Tue, 28 Jun 2011 13:42:59 -0400

Last line of failing gem install:

/usr/bin/ld: ../../leveldb/libleveldb.a(db_impl.o): relocation R_X86_64_32S against `vtable for leveldb::DB' can not be used when making a shared object; recompile with -fPIC
../../leveldb/libleveldb.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [leveldb.so] Error 1

Gem files will remain installed in /var/lib/gems/1.9.1/gems/leveldb-ruby-0.4 for inspection.
Results logged to /var/lib/gems/1.9.1/gems/leveldb-ruby-0.4/ext/leveldb/gem_make.out

-- 
Álvaro Herrera <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Sun, 19 Jun 2011 18:30:22 +0000

---

From: William Morgan <[email-masked]>
Date: Mon, 20 Jun 2011 03:54:22 +0000

---

From: William Morgan <[email-masked]>
Date: Wed, 22 Jun 2011 03:25:16 +0000

Hi Horacio,

Can you please:

Thanks!

---

From: William Morgan <[email-masked]>
Date: Mon, 20 Jun 2011 22:52:05 +0000

Reformatted excerpts from Horacio Sanson's message of 2011-06-20:
> /usr/bin/ld: ../../leveldb/libleveldb.a(db_impl.o): relocation
> R_X86_64_32S against `vtable for leveldb::DB' can not be used when
> making a shared object;

---

From: Horacio Sanson <[email-masked]>
Date: Thu, 23 Jun 2011 22:29:25 +0900

On Thursday 23 June 2011 08:36:07 William Morgan wrote:
> > Finally the leveldb branch is not working that well. After reaching
> > 130378 indexed messages I started the heliotrope server and the web
> > interface only showed two messages.
> 
> What if you search for "*"?

wget http://localhost:8042/search?q=*

search(body:"*", 0, 20) took 0.0ms
127.0.0.1 - - [23/Jun/2011 22:26:30] "GET /search?q=* HTTP/1.1" 200 813 0.0010
localhost - - [23/Jun/2011:22:26:30 JST] "GET /search?q=* HTTP/1.1" 200 813
http://localhost:8042/search?q=%7Einbox -> /search?q=*
127.0.0.1 - - [23/Jun/2011 22:26:30] "GET /favicon.ico HTTP/1.1" 404 447 
0.0006
localhost - - [23/Jun/2011:22:26:30 JST] "GET /favicon.ico HTTP/1.1" 404 447
- -> /favicon.ico

I ran the heliotrope-add with and without hooks and both resulted in no 
messages even though it reached 130378 messages indexed before crashing with a 
nil error.

---

From: Horacio Sanson <[email-masked]>
Date: Tue, 21 Jun 2011 07:26:18 +0900

> cstdatomic issue, but nothing with -fPIC.)

Indeed it fixed the cstdatomic issue but not the linking one:

####################################

/usr/bin/ruby1.9.1 extconf.rb
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 db/builder.cc -o db/builder.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 db/db_impl.cc -o db/db_impl.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 db/db_iter.cc -o db/db_iter.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 db/filename.cc -o db/filename.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 db/dbformat.cc -o db/dbformat.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 db/log_reader.cc -o db/log_reader.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 db/log_writer.cc -o db/log_writer.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 db/memtable.cc -o db/memtable.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 db/repair.cc -o db/repair.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 db/table_cache.cc -o db/table_cache.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 db/version_edit.cc -o db/version_edit.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 db/version_set.cc -o db/version_set.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 db/write_batch.cc -o db/write_batch.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 port/port_posix.cc -o port/port_posix.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 table/block.cc -o table/block.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 table/block_builder.cc -o table/block_builder.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 table/format.cc -o table/format.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 table/iterator.cc -o table/iterator.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 table/merger.cc -o table/merger.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 table/table.cc -o table/table.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 table/table_builder.cc -o table/table_builder.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 table/two_level_iterator.cc -o table/two_level_iterator.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 util/arena.cc -o util/arena.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 util/cache.cc -o util/cache.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 util/coding.cc -o util/coding.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 util/comparator.cc -o util/comparator.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 util/crc32c.cc -o util/crc32c.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 util/env.cc -o util/env.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 util/env_posix.cc -o util/env_posix.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 util/hash.cc -o util/hash.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 util/histogram.cc -o util/histogram.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 util/logging.cc -o util/logging.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 util/options.cc -o util/options.o
g++ -c -I. -I./include -DLEVELDB_PLATFORM_POSIX -std=c++0x -O2 -DNDEBUG        
-DUSE_SNAPPY=1 util/status.cc -o util/status.o
rm -f libleveldb.a
ar -rs libleveldb.a ./db/builder.o ./db/db_impl.o ./db/db_iter.o 
./db/filename.o ./db/dbformat.o ./db/log_reader.o ./db/log_writer.o 
./db/memtable.o ./db/repair.o ./db/table_cache.o ./db/version_edit.o 
./db/version_set.o ./db/write_batch.o ./port/port_posix.o ./table/block.o 
./table/block_builder.o ./table/format.o ./table/iterator.o ./table/merger.o 
./table/table.o ./table/table_builder.o ./table/two_level_iterator.o 
./util/arena.o ./util/cache.o ./util/coding.o ./util/comparator.o 
./util/crc32c.o ./util/env.o ./util/env_posix.o ./util/hash.o 
./util/histogram.o ./util/logging.o ./util/options.o ./util/status.o
ar: creating libleveldb.a
creating Makefile

make
g++ -I. -I/usr/include/ruby-1.9.1/x86_64-linux -
I/usr/include/ruby-1.9.1/ruby/backward -I/usr/include/ruby-1.9.1 -I.   -fPIC -
fno-strict-aliasing -g -g -O2 -fPIC -I../../leveldb/include  -fno-strict-
aliasing -g -o leveldb.o -c leveldb.cc
g++ -shared -o leveldb.so leveldb.o -L. -L/usr/lib -L.  -rdynamic -Wl,-export-
dynamic   -lruby-1.9.1  -lpthread -lrt -ldl -lcrypt -lm   -lc -L../../leveldb 
-lleveldb
/usr/bin/ld: ../../leveldb/libleveldb.a(db_impl.o): relocation R_X86_64_32S 
against `vtable for leveldb::DB' can not be used when making a shared object; 
recompile with -fPIC
../../leveldb/libleveldb.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [leveldb.so] エラー 1

Gem files will remain installed in /var/lib/gems/1.9.1/gems/leveldb-ruby-0.4 
for inspection.
Results logged to /var/lib/gems/1.9.1/gems/leveldb-
ruby-0.4/ext/leveldb/gem_make.out
################################

I tried qwith both g++-4.4 and g++-4.5 that come with standard Ubuntu 11.04 
distribution.

---

From: Horacio Sanson <[email-masked]>
Date: Thu, 23 Jun 2011 07:41:19 +0900

I left heliotrope-add running but it did not finished. I got some strange nil 
errors (see below):

I think this error happens when find_msgids is unable to find the message's id. 
Not sure what is the best action in this case but skipping it sounds a good 
option.
###################################################
; requesting messages 141445..141564 from imap server
scanned 130401, indexed 130278, skipped 25 bad and 98 seen messages in 9566.0s 
= 13.6 m/s
; requesting messages 141565..141688 from imap server
scanned 130501, indexed 130378, skipped 25 bad and 98 seen messages in 9572.0s 
= 13.6 m/s
/media/DATA/Apps/heliotrope/lib/heliotrope/message.rb:155:in `digest': can't 
convert nil into String (TypeError)
        from /media/DATA/Apps/heliotrope/lib/heliotrope/message.rb:155:in 
`hexdigest'
        from /media/DATA/Apps/heliotrope/lib/heliotrope/message.rb:155:in 
`munge_msgid'
        from /media/DATA/Apps/heliotrope/lib/heliotrope/message.rb:20:in 
`parse!'
        from bin/heliotrope-add:138:in `<main>'
####################################################

This error happens when @imap.uid_fetch doesn't raise any exception and 
returns nil instead of empty array. Was thinking to check for nil and set it
to an empty array if true but it would likely result in an infinite loop.
####################################################
; loading mail...
; connecting...
; login as [email-masked] ...
; found 293004 new messages...
; found 293004 messages to scan
; requesting messages 283664..283763 from imap server
scanned 1, indexed 1, skipped 0 bad and 0 seen messages in 7.9s = 0.1 m/s
; requesting messages 283764..283863 from imap server
/media/DATA/Apps/heliotrope/lib/heliotrope/gmail-dumper.rb:171:in 
`next_message': undefined method `empty?' for nil:NilClass (NoMethodError)
        from bin/heliotrope-add:128:in `<main>'
#####################################################

Finally the leveldb branch is not working that well. After reaching 130378 
indexed messages I started the heliotrope server and the web interface only 
showed two messages. 

I will try again without the hooks for Japanese and see how it goes.

---

From: William Morgan <[email-masked]>
Date: Sat, 18 Jun 2011 05:02:59 +0000

---

From: William Morgan <[email-masked]>
Date: Wed, 22 Jun 2011 23:36:07 +0000