Subject: [sup-talk] sup-sync dies; memory limit?

From: [email-masked] (Andrew Pimlott)
Date: Fri, 8 May 2009 09:42:26 -0700

I have been running sup-sync on a new mailbox, and it tends to get
partway through and then die, but not in the same place each time.
Twice, the error has been like

/var/lib/gems/1.8/gems/sup-0.7/lib/sup/crypto.rb:162:in ``': Cannot allocate memory - /usr/bin/gpg --quiet --batch --no-verbose --logger-fd 1 --use-agent --verify /tmp/28751-0-redwood.signature /tmp/28751-0-redwood.payload 2> /dev/null (Errno::ENOMEM)

followed by a backtrace.  Another time, it was just

zsh: killed     /var/lib/gems/1.8/bin/sup-sync --all-sources

I monitored the sup-sync process the last time.  It's memory use grew
slowly, and the process died at around 64M of virtual memory.  I don't
have any memory limits set.  Is it possible that sup-sync or ruby uses
some itself?  Any other ideas?

Andrew

---

From: [email-masked] (Andrew Pimlott)
Date: Mon, 11 May 2009 11:26:15 -0700

I forgot to mention, this was the latest release (0.7) installed with
gem.  I just tried the current git mainline (clearing all my .sup state
first), and it got through my whole mailbox without a problem.

Andrew

On Fri, May 08, 2009 at 09:42:26AM -0700, Andrew Pimlott wrote:
> _______________________________________________
> sup-talk mailing list
> sup-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-talk

---

From: [email-masked] (William Morgan)
Date: Tue, 12 May 2009 10:19:48 -0700

Reformatted excerpts from Andrew Pimlott's message of 2009-05-11:
> > I don't have any memory limits set.  Is it possible that sup-sync or
> > ruby uses some itself?  Any other ideas?

Very mysterious. Sup-sync doesn't do anything fancy with memory; it's a
fairly straight-forward Ruby script. Maybe one of the C extensions (like
Ferret) has a memory leak.

> gem.  I just tried the current git mainline (clearing all my .sup
> state first), and it got through my whole mailbox without a problem.

Evey more mysterious. The differences between these two versions are
trivial:

diff --git a/origin/release-0.7:bin/sup-sync b/origin/master:bin/sup-sync
index ac5caf6..91710d4 100644
--- a/origin/release-0.7:bin/sup-sync
+++ b/origin/master:bin/sup-sync
@@ -143,12 +143,7 @@ begin
       next if target == :changed && entry && entry[:source_id].to_i == source.id && entry[:source_info].to_i == offset
 
       ## get the state currently in the index
-      index_state =
-        if entry
-          entry[:label].split(/\s+/).map { |x| x.intern }
-        else
-          nil
-        end
+      index_state = entry[:label].split(/\s+/).map { |x| x.intern } if entry
 
       ## skip if we're operating on restored messages, and this one
       ## ain't.
@@ -163,7 +158,7 @@ begin
       ## assign message labels based on the operation we're performing
       case op
       when :asis
-        m.labels = index_state if index_state
+        m.labels = ((m.labels - [:unread, :inbox]) + index_state).uniq if index_state
       when :restore
         ## if the entry exists on disk
         if restored_state[m.id]

I'm at a loss.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Andrew Pimlott)
Date: Tue, 12 May 2009 11:31:58 -0700

On Tue, May 12, 2009 at 10:19:48AM -0700, William Morgan wrote:
> Very mysterious.

I cannot reproduce the problem anymore, but as far as I know, nothing
changed on my system (other than the inbox I am testing with evolving
routinely).  sup-sync from the 0.7 gem now runs to completion and never
uses more than ~36M virtual memory (before it got up around 60M before
crashing).  It doesn't seem possible that the git copy of sup affected
the installed gem.  So I'm at a complete loss too.  

Andrew

---

From: [email-masked] (William Morgan)
Date: Tue, 12 May 2009 12:54:35 -0700

Reformatted excerpts from Andrew Pimlott's message of 2009-05-12:
> It doesn't seem possible that the git copy of sup affected the
> installed gem.  So I'm at a complete loss too.  

Let's just pretend this never happened.
-- 
William <wmorgan-sup at masanjin.net>