Subject: [sup-talk] Fwd: Re: [Ferret-trac] #279: IO Error - Error reading the segment infos

From: [email-masked] (William Morgan)
Date: Tue, 14 Aug 2007 22:17:46 -0700

Fore those following the index.c:901 error, here's a potential reason
why Ferret might be doing this: too many open filehandles. If you can
reproduce this, try increasing the max open filehandle limit and see if
that helps things.

--- Begin forwarded message from Ferret ---
From: Ferret <trac at davebalmain.com>
Cc: dbalmain <dbalmain at gmail.com>
Date: Wed, 15 Aug 2007 05:52:27 -0000
Subject: Re: [Ferret-trac] #279: IO Error - Error reading the segment infos

#279: IO Error - Error reading the segment infos
--------------------------------------+-------------------------------------
  Reporter:  dennis at dennisschaaf.com  |       Owner:  somebody
      Type:  defect                   |      Status:  reopened
  Priority:  major                    |   Milestone:          
 Component:  component1               |     Version:          
Resolution:                           |    Keywords:          
--------------------------------------+-------------------------------------
Comment (by dennis at dennisschaaf.com):

 This guy mentions that there seems to be a limit with too many open files.
 Could it be that ferret is not closing files properly? I don't know where
 in the source i'd look but i'll try.

 http://www.ruby-forum.com/topic/120609

--- End forwarded message ---

-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (jeff covey)
Date: Sat, 18 Aug 2007 08:18:15 -0400

on Tue, Aug 14, 2007 at 10:17:46PM -0700%, William Morgan said:

> Fore those following the index.c:901 error, here's a potential reason why
> Ferret might be doing this: too many open filehandles. If you can

i tried this, setting it to 32768 as he recommended, then adding my archive
sources from the beginning of the year (i had been working with just
messages from the past month).  it caused a lot of exceptions in sup and
some segmentation faults in ruby.  i tried resetting the limit to 1024, and
got the exception below.

i think i'll have to assume sup/ferret can't handle many sources and change
my archiving system to accommodate.

[Sat Aug 18 08:07:54 -0400 2007] stopped cursing
[Sat Aug 18 08:07:54 -0400 2007] oh crap, an exception
[Sat Aug 18 08:07:54 -0400 2007] unlocking /home/jeff/.sup/lock...
----------------------------------------------------------------
I'm very sorry, but it seems that an error occurred in Sup.
Please accept my sincere apologies. If you don't mind, please
send the backtrace below and a brief report of the circumstances
to sup-talk at rubyforge dot orgs so that I might address this
problem. Thank you!

Sincerely,
William
----------------------------------------------------------------

The problem was: 'Too many open files - /home/jeff/mail/turbotax at mailca.custhelp.com-2007-02' (error type Errno::EMFILE)
A backtrace follows:
./lib/sup/mbox/loader.rb:20:in `initialize': Too many open files - /home/jeff/mail/turbotax at mailca.custhelp.com-2007-02 (Errno::EMFILE)
        from ./lib/sup/mbox/loader.rb:20:in `open'
        from ./lib/sup/mbox/loader.rb:20:in `initialize'
        from ./lib/sup.rb:28:in `new'
        from ./lib/sup.rb:28:in `yaml_properties'
        from /usr/lib/ruby/1.8/yaml.rb:133:in `call'
        from /usr/lib/ruby/1.8/yaml.rb:133:in `transfer'
        from /usr/lib/ruby/1.8/yaml.rb:133:in `node_import'
        from /usr/lib/ruby/1.8/yaml.rb:133:in `load'
         ... 6 levels...
        from ./lib/sup/index.rb:101:in `load'
        from ./lib/sup/util.rb:401:in `send'
        from ./lib/sup/util.rb:401:in `method_missing'
        from bin/sup:89
checking for messages to delete...
./lib/sup/util.rb:7: warning: method redefined; discarding old gen_lock_id
./lib/sup/util.rb:18: warning: method redefined; discarding old dump_lock_id
./lib/sup/mbox/loader.rb:20:in `initialize': Too many open files - /home/jeff/mail/turbotax at mailca.custhelp.com-2007-02 (Errno::EMFILE)
        from ./lib/sup/mbox/loader.rb:20:in `open'
        from ./lib/sup/mbox/loader.rb:20:in `initialize'
        from ./lib/sup.rb:28:in `new'
        from ./lib/sup.rb:28:in `yaml_properties'
        from /usr/lib/ruby/1.8/yaml.rb:133:in `call'
        from /usr/lib/ruby/1.8/yaml.rb:133:in `transfer'
        from /usr/lib/ruby/1.8/yaml.rb:133:in `node_import'
        from /usr/lib/ruby/1.8/yaml.rb:133:in `load'
        from /usr/lib/ruby/1.8/yaml.rb:133:in `load'
        from /usr/lib/ruby/1.8/yaml.rb:144:in `load_file'
        from /usr/lib/ruby/1.8/yaml.rb:143:in `open'
        from /usr/lib/ruby/1.8/yaml.rb:143:in `load_file'
        from ./lib/sup.rb:99:in `load_yaml_obj'
        from ./lib/sup/index.rb:348:in `load_sources'
        from ./lib/sup/index.rb:101:in `load'
        from bin/sup-dump:25

thanks,

-- 
jeff covey
http://jeffcovey.net/

---

From: [email-masked] (William Morgan)
Date: Mon, 27 Aug 2007 13:32:56 -0700

Excerpts from jeff.covey's message of Sat Aug 18 05:18:15 -0700 2007:
> i tried this, setting it to 32768 as he recommended, then adding my
> archive sources from the beginning of the year (i had been working
> with just messages from the past month).  it caused a lot of
> exceptions in sup and some segmentation faults in ruby.

Sup does keep a filehandle open to every mbox source. Since sources
aren't read except when you open a thread-view-mode, it probably
wouldn't be too bad to just open them on demand. That might help.

---

From: [email-masked] (jeff covey)
Date: Tue, 28 Aug 2007 17:48:01 -0400

Excerpts from William Morgan's message of Mon Aug 27 16:32:56 -0400 2007:

> Sup does keep a filehandle open to every mbox source.

yikes, no wonder it complains!  :)

-- 
jeff covey
http://jeffcovey.net/