* [sup-talk] Increase poll frequency
2008-12-11 20:20 ` William Morgan
2008-12-11 20:30 ` Florian Bender
@ 2008-12-11 20:41 ` Marc Hartstein
2008-12-12 1:37 ` William Morgan
1 sibling, 1 reply; 9+ messages in thread
From: Marc Hartstein @ 2008-12-11 20:41 UTC (permalink / raw)
Excerpts from William Morgan's message of Thu Dec 11 15:20:08 -0500 2008:
> Reformatted excerpts from Florian Bender's message of 2008-12-08:
> > after a quick glance at the documentation, I didn't notice any option
> > to increase the frequency sup polls a source for new E-Mails. For my
> > GMail Account I'll like it, if it'd be polled every minute or so.
>
> There's no way to do this now, but it would be pretty easy to add.
Christopher Warrington sent a patch to this list in the spring which
allows configuration of poll interval on a per-source basis. This
should do what's being looked for.
I run it locally, mostly so sup doesn't waste time polling static
sources (like the mailing list archive from before I joined this list),
and it seems to work nicely.
Here's the email as I have it saved in my sup-patches/:
From chrisw at rice.edu Sat Mar 29 06:32:05 2008
From: chrisw@rice.edu (Christopher Warrington)
Date: Sat, 29 Mar 2008 05:32:05 -0500
Subject: [sup-talk] [PATCH] polling is now done per source
In-Reply-To: <1206784680-sup-3052 at chris-tablet>
References: <1206784680-sup-3052 at chris-tablet>
Message-ID: <1206786725-5456-1-git-send-email-chrisw at rice.edu>
Each source has a poll_interval property. This property is used to
determine whether the source should be polled. The user can still for a
poll of all sources.
---
bin/sup | 4 +-
bin/sup-add | 12 ++-
lib/sup.rb | 4 +-
lib/sup/imap.rb | 6 +-
lib/sup/maildir.rb | 6 +-
lib/sup/mbox/loader.rb | 4 +-
lib/sup/modes/poll-mode.rb | 4 +-
lib/sup/poll.rb | 211 +++++++++++++++++++++++++++++++++-----------
lib/sup/source.rb | 8 +-
9 files changed, 189 insertions(+), 70 deletions(-)
mode change 100755 => 100644 lib/sup/buffer.rb
diff --git a/bin/sup b/bin/sup
index 84fd77c..86a2d9f 100644
--- a/bin/sup
+++ b/bin/sup
@@ -197,7 +197,7 @@ begin
end
end unless $opts[:no_initial_poll]
- imode.load_threads :num => ibuf.content_height, :when_done => lambda { reporting_thread("poll after loading inbox") { sleep 1; PollManager.poll } unless $opts[:no_threads] || $opts[:no_initial_poll] }
+ imode.load_threads :num => ibuf.content_height, :when_done => lambda { reporting_thread("poll after loading inbox") { sleep 1; PollManager.auto_poll } unless $opts[:no_threads] || $opts[:no_initial_poll] }
if $opts[:compose]
ComposeMode.spawn_nicely :to_default => $opts[:compose]
@@ -263,7 +263,7 @@ begin
when :compose
ComposeMode.spawn_nicely
when :poll
- reporting_thread("user-invoked poll") { PollManager.poll }
+ reporting_thread("user-invoked poll") { PollManager.forced_poll }
when :recall_draft
case Index.num_results_for :label => :draft
when 0
diff --git a/bin/sup-add b/bin/sup-add
index 50bbb29..88705e1 100644
--- a/bin/sup-add
+++ b/bin/sup-add
@@ -39,6 +39,7 @@ EOS
opt :unusual, "Do not automatically poll these sources for new messages."
opt :labels, "A comma-separated set of labels to apply to all messages from this source", :type => String
opt :force_new, "Create a new account for this source, even if one already exists."
+ opt :poll_interval, "The interval (in seconds) between new message polls. The default is #{Redwood::DEFAULT_POLL_INTERVAL}.", :type => :int
end
Trollop::die "require one or more sources" if ARGV.empty?
@@ -84,6 +85,9 @@ index.lock_or_die
begin
index.load_sources
+ Trollop::die "The poll interval must be a positive integer." if $opts[:poll_interval] <= 0
+ poll_interval = $opts[:poll_interval] || Redwood::DEFAULT_POLL_INTERVAL
+
ARGV.each do |uri|
labels = $opts[:labels] ? $opts[:labels].split(/\s*,\s*/).uniq : []
@@ -100,14 +104,14 @@ begin
say "For SSH connections, if you will use public key authentication, you may leave the username and password blank."
say ""
username, password = get_login_info uri, index.sources
- Redwood::MBox::SSHLoader.new uri, username, password, nil, !$opts[:unusual], $opts[:archive], nil, labels
+ Redwood::MBox::SSHLoader.new uri, username, password, nil, !$opts[:unusual], $opts[:archive], nil, poll_interval, labels
when "imap", "imaps"
username, password = get_login_info uri, index.sources
- Redwood::IMAP.new uri, username, password, nil, !$opts[:unusual], $opts[:archive], nil, labels
+ Redwood::IMAP.new uri, username, password, nil, !$opts[:unusual], $opts[:archive], nil, poll_interval, labels
when "maildir"
- Redwood::Maildir.new uri, nil, !$opts[:unusual], $opts[:archive], nil, labels
+ Redwood::Maildir.new uri, nil, !$opts[:unusual], $opts[:archive], nil, poll_interval, labels
when "mbox"
- Redwood::MBox::Loader.new uri, nil, !$opts[:unusual], $opts[:archive], nil, labels
+ Redwood::MBox::Loader.new uri, nil, !$opts[:unusual], $opts[:archive], nil, poll_interval, labels
when nil
Trollop::die "Sources must be specified with an URI"
else
diff --git a/lib/sup.rb b/lib/sup.rb
index 1946f3c..c27a4bc 100644
--- a/lib/sup.rb
+++ b/lib/sup.rb
@@ -50,6 +50,8 @@ module Redwood
YAML_DOMAIN = "masanjin.net"
YAML_DATE = "2006-10-01"
+ DEFAULT_POLL_INTERVAL = 300
+
## record exceptions thrown in threads nicely
def reporting_thread name
if $opts[:no_threads]
@@ -72,7 +74,7 @@ module Redwood
def save_yaml_obj object, fn, safe=false
if safe
safe_fn = "#{File.dirname fn}/safe_#{File.basename fn}"
- mode = File.stat(fn) if File.exists? fn
+ mode = File.stat(fn).mode if File.exists? fn
File.open(safe_fn, "w", mode) { |f| f.puts object.to_yaml }
FileUtils.mv safe_fn, fn
else
diff --git a/lib/sup/buffer.rb b/lib/sup/buffer.rb
old mode 100755
new mode 100644
diff --git a/lib/sup/imap.rb b/lib/sup/imap.rb
index 1d36976..8b58cba 100644
--- a/lib/sup/imap.rb
+++ b/lib/sup/imap.rb
@@ -51,13 +51,13 @@ class IMAP < Source
attr_accessor :username, :password
yaml_properties :uri, :username, :password, :cur_offset, :usual,
- :archived, :id, :labels
+ :archived, :id, :poll_interval, :labels
- def initialize uri, username, password, last_idate=nil, usual=true, archived=false, id=nil, labels=[]
+ def initialize uri, username, password, last_idate=nil, usual=true, archived=false, id=nil, poll_interval=nil, labels=[]
raise ArgumentError, "username and password must be specified" unless username && password
raise ArgumentError, "not an imap uri" unless uri =~ %r!imaps?://!
- super uri, last_idate, usual, archived, id
+ super uri, last_idate, usual, archived, id, [poll_interval, SCAN_INTERVAL].max
@parsed_uri = URI(uri)
@username = username
diff --git a/lib/sup/maildir.rb b/lib/sup/maildir.rb
index 584e657..ef71e0b 100644
--- a/lib/sup/maildir.rb
+++ b/lib/sup/maildir.rb
@@ -12,9 +12,9 @@ class Maildir < Source
SCAN_INTERVAL = 30 # seconds
## remind me never to use inheritance again.
- yaml_properties :uri, :cur_offset, :usual, :archived, :id, :labels
- def initialize uri, last_date=nil, usual=true, archived=false, id=nil, labels=[]
- super uri, last_date, usual, archived, id
+ yaml_properties :uri, :cur_offset, :usual, :archived, :id, :poll_interval, :labels
+ def initialize uri, last_date=nil, usual=true, archived=false, id=nil, poll_interval=nil, labels=[]
+ super uri, last_date, usual, archived, id, [poll_interval, SCAN_INTERVAL].max
uri = URI(Source.expand_filesystem_uri(uri))
raise ArgumentError, "not a maildir URI" unless uri.scheme == "maildir"
diff --git a/lib/sup/mbox/loader.rb b/lib/sup/mbox/loader.rb
index 7fe9129..44317d5 100644
--- a/lib/sup/mbox/loader.rb
+++ b/lib/sup/mbox/loader.rb
@@ -9,7 +9,7 @@ class Loader < Source
attr_accessor :labels
## uri_or_fp is horrific. need to refactor.
- def initialize uri_or_fp, start_offset=nil, usual=true, archived=false, id=nil, labels=[]
+ def initialize uri_or_fp, start_offset=nil, usual=true, archived=false, id=nil, poll_interval=nil, labels=[]
@mutex = Mutex.new
@labels = ((labels || []) - LabelManager::RESERVED_LABELS).uniq.freeze
@@ -26,7 +26,7 @@ class Loader < Source
@path = uri_or_fp.path
end
- super uri_or_fp, start_offset, usual, archived, id
+ super uri_or_fp, start_offset, usual, archived, id, poll_interval
end
def file_path; @path end
diff --git a/lib/sup/modes/poll-mode.rb b/lib/sup/modes/poll-mode.rb
index 5849f3e..5521cdc 100644
--- a/lib/sup/modes/poll-mode.rb
+++ b/lib/sup/modes/poll-mode.rb
@@ -10,11 +10,11 @@ class PollMode < LogMode
self << s + "\n"
end
- def poll
+ def poll sources
puts unless @new
@new = false
puts "Poll started at #{Time.now}"
- PollManager.do_poll { |s| puts s }
+ PollManager.do_poll_sources(sources) { |s| puts s }
end
end
diff --git a/lib/sup/poll.rb b/lib/sup/poll.rb
index d32c893..1ff8014 100644
--- a/lib/sup/poll.rb
+++ b/lib/sup/poll.rb
@@ -12,13 +12,22 @@ Variables:
EOS
HookManager.register "before-poll", <<EOS
+Executes immediately before any poll for new messages commences.
+Variables:
+ accts: an array of source URIs that will be polled
+EOS
+
+ HookManager.register "before-acct-poll", <<EOS
Executes immediately before a poll for new messages commences.
-No variables.
+Variables:
+ acct: the URI of the account being polled
EOS
- HookManager.register "after-poll", <<EOS
-Executes immediately after a poll for new messages completes.
+ HookManager.register "after-acct-poll", <<EOS
+Executes immediately after a poll for new messages completes. The statistics are
+only for the account specified.
Variables:
+ acct: the URI of the account being polled
num: the total number of new messages added in this poll
num_inbox: the number of new messages added in this poll which
appear in the inbox (i.e. were not auto-archived).
@@ -28,14 +37,25 @@ num_inbox_total_unread: the total number of unread messages in the inbox
only those messages appearing in the inbox
EOS
- DELAY = 300
+ HookManager.register "after-poll", <<EOS
+Executes immediately after any poll for new messages completes. The statistics
+are for all sources polled.
+Variables:
+ accts: an array of source URIs that were polled
+ num: the total number of new messages added in this poll
+ num_inbox: the number of new messages added in this poll which
+ appear in the inbox (i.e. were not auto-archived).
+num_inbox_total_unread: the total number of unread messages in the inbox
+ from_and_subj: an array of (from email address, subject) pairs
+ from_and_subj_inbox: an array of (from email address, subject) pairs for
+ only those messages appearing in the inbox
+EOS
def initialize
@mutex = Mutex.new
@thread = nil
- @last_poll = nil
@polling = false
-
+
self.class.i_am_the_instance self
end
@@ -44,47 +64,113 @@ EOS
b
end
- def poll
+ def start
+ ## Periodically calls auto_poll, effectively automatically polling
+ ## in the background.
+ ##
+ ## auto_poll is called more frequently than the smallest poll
+ ## interval. auto_poll is not guaranteed to be called in multiples
+ ## of poll intervals. I.e., don't rely on auto_poll to be called
+ ## every x seconds: when auto_poll is called assume an
+ ## indeterminate amount of time has passed.
+ @min_poll_interval = Index.usual_sources.collect{|s| s.poll_interval}.select{|pi| !pi.nil?}.min || Redwood::DEFAULT_POLL_INTERVAL
+
+ Redwood::log "Background poll interval is #{@min_poll_interval} seconds."
+
+ @thread = Redwood::reporting_thread("periodic poll") do
+ while true
+ Redwood::log "Sleeping for #{@min_poll_interval / 2} seconds."
+
+ sleep @min_poll_interval / 2
+ auto_poll
+ end
+ end
+ end
+
+ def stop
+ @thread.kill if @thread
+ @thread = nil
+ end
+
+ def auto_poll
+ ## This method is called by the thread spawned in start. It
+ ## collects the sources that should be polled
+ ## (source.poll_interval has expired or the source has never been
+ ## polled) and polls them.
+ ##
+ ## Returns an array [# of new messages, # of new messages to
+ ## inbox, new message subject/name array pairs, subject/name array
+ ## pairs loaded to inbox]
+
+ sources_to_poll = []
+ @mutex.synchronize do # William, do we need to synchronize here?
+ begin
+ sources_to_poll = Index.usual_sources.select do |source|
+ begin
+ source.last_poll.nil? || (Time.now - source.last_poll) >= source.poll_interval
+ end
+ end
+ end
+ end
+
+ poll_sources sources_to_poll
+ end
+
+ def forced_poll
+ ## This method is called when the user explicitly requests a poll.
+ ##
+ ## Returns an array [# of new messages, # of new messages to
+ ## inbox, new message subject/name array pairs, subject/name array
+ ## pairs loaded to inbox]
+ poll_sources Index.usual_sources
+ end
+
+ def poll_sources sources
+ ## Polls the given sources. Clients of PollManager should call
+ ## this method, not do_poll_sources or poll_source. (Well, only
+ ## PollMode should call do_poll_sources)
+ ##
+ ## Returns an array [# of new messages, # of new messages to
+ ## inbox, new message subject/name array pairs, subject/name array
+ ## pairs loaded to inbox]
return if @polling
@polling = true
- HookManager.run "before-poll"
+
+ source_uris = sources.map{|s| s.uri}
+
+ HookManager.run "before-acct-poll", :accts => source_uris
BufferManager.flash "Polling for new messages..."
- num, numi, from_and_subj, from_and_subj_inbox = buffer.mode.poll
+ num, numi, from_subj, from_subj_inbox = buffer.mode.poll sources
if num > 0
BufferManager.flash "Loaded #{num.pluralize 'new message'}, #{numi} to inbox."
else
BufferManager.flash "No new messages."
end
- HookManager.run "after-poll", :num => num, :num_inbox => numi, :from_and_subj => from_and_subj, :from_and_subj_inbox => from_and_subj_inbox, :num_inbox_total_unread => lambda { Index.num_results_for :labels => [:inbox, :unread] }
+ HookManager.run "after-poll", :accts => source_uris, :num => num, :num_inbox => numi, :from_and_subj => from_subj, :from_and_subj_inbox => from_subj_inbox, :num_inbox_total_unread => lambda { Index.num_results_for :labels => [:inbox, :unread] }
@polling = false
- [num, numi]
- end
-
- def start
- @thread = Redwood::reporting_thread("periodic poll") do
- while true
- sleep DELAY / 2
- poll if @last_poll.nil? || (Time.now - @last_poll) >= DELAY
- end
- end
+ [num, numi, from_subj, from_subj_inbox]
end
- def stop
- @thread.kill if @thread
- @thread = nil
- end
+ def do_poll_sources sources, &block
+ ## Polls each source, keeping track of vital statistics (number
+ ## loaded, name/e-mail pairs, &c.) about the poll results.
+ ##
+ ## We need explicit access to the block so that we can pass it to
+ ## poll_source.
+ ##
+ ## Returns an array [total # of new messages, total # of new
+ ## messages to inbox, all new message subject/name array pairs,
+ ## all subject/name array pairs loaded to inbox]
- def do_poll
total_num = total_numi = 0
- from_and_subj = []
- from_and_subj_inbox = []
+ total_from_and_subj = []
+ total_from_and_subj_inbox = []
@mutex.synchronize do
- Index.usual_sources.each do |source|
-# yield "source #{source} is done? #{source.done?} (cur_offset #{source.cur_offset} >= #{source.end_offset})"
+ sources.each do |source|
begin
yield "Loading from #{source}... " unless source.done? || source.has_errors?
rescue SourceError => e
@@ -93,32 +179,57 @@ EOS
next
end
- num = 0
- numi = 0
- add_messages_from source do |m, offset, entry|
- ## always preserve the labels on disk.
- m.labels = entry[:label].split(/\s+/).map { |x| x.intern } if entry
- yield "Found message at #{offset} with labels {#{m.labels * ', '}}"
- unless entry
- num += 1
- from_and_subj << [m.from.longname, m.subj]
- if m.has_label?(:inbox) && ([:spam, :deleted, :killed] & m.labels).empty?
- from_and_subj_inbox << [m.from.longname, m.subj]
- numi += 1
- end
- end
- m
- end
- yield "Found #{num} messages, #{numi} to inbox." unless num == 0
+ HookManager.run "before-acct-poll", :acct => source.uri
+
+ BufferManager.flash "Polling #{source.uri} for new messages..."
+ num, numi, from_and_subj, from_and_subj_inbox = poll_source source, &block
+
total_num += num
total_numi += numi
+ total_from_and_subj += from_and_subj
+ total_from_and_subj_inbox += from_and_subj_inbox
+
+ HookManager.run "after-acct-poll", :acct => source.uri, :num => num, :num_inbox => numi, :from_and_subj => from_and_subj, :from_and_subj_inbox => from_and_subj_inbox, :num_inbox_total_unread => lambda { Index.num_results_for :labels => [:inbox, :unread] }
end
+ end
+
+ [total_num, total_numi, total_from_and_subj, total_from_and_subj_inbox]
+ end
- yield "Done polling; loaded #{total_num} new messages total"
- @last_poll = Time.now
- @polling = false
+ def poll_source source
+ ## Polls the given source for new messages.
+ ##
+ ## @mutex must be held before calling! See do_poll_sources.
+ ##
+ ## Returns an array [# of new messages, # of new messages to
+ ## inbox, new message subject/name array pairs, subject/name array
+ ## pairs loaded to inbox]
+
+ num = 0
+ numi = 0
+ from_and_subj = []
+ from_and_subj_inbox = []
+
+ add_messages_from source do |m, offset, entry|
+ ## always preserve the labels on disk.
+ m.labels = entry[:label].split(/\s+/).map { |x| x.intern } if entry
+ yield "Found message at #{offset} with labels {#{m.labels * ', '}}"
+ unless entry
+ num += 1
+ from_and_subj << [m.from.longname, m.subj]
+ if m.has_label?(:inbox) && ([:spam, :deleted, :killed] & m.labels).empty?
+ from_and_subj_inbox << [m.from.longname, m.subj]
+ numi += 1
+ end
+ end
+ m
end
- [total_num, total_numi, from_and_subj, from_and_subj_inbox]
+
+ yield "For source #{source.uri}, found #{num} messages, #{numi} to inbox." unless num == 0
+
+ source.last_poll = Time.now
+
+ [num, numi, from_and_subj, from_and_subj_inbox]
end
## this is the main mechanism for adding new messages to the
diff --git a/lib/sup/source.rb b/lib/sup/source.rb
index 6510aae..fd8d381 100644
--- a/lib/sup/source.rb
+++ b/lib/sup/source.rb
@@ -62,10 +62,10 @@ class Source
## dirty? means cur_offset has changed, so the source info needs to
## be re-saved to sources.yaml.
bool_reader :usual, :archived, :dirty
- attr_reader :uri, :cur_offset
- attr_accessor :id
+ attr_reader :uri, :cur_offset, :poll_interval
+ attr_accessor :id, :last_poll
- def initialize uri, initial_offset=nil, usual=true, archived=false, id=nil
+ def initialize uri, initial_offset=nil, usual=true, archived=false, id=nil, poll_interval=nil
raise ArgumentError, "id must be an integer: #{id.inspect}" unless id.is_a? Fixnum if id
@uri = uri
@@ -73,6 +73,8 @@ class Source
@usual = usual
@archived = archived
@id = id
+ @poll_interval = poll_interval || Redwood::DEFAULT_POLL_INTERVAL #seconds
+ @last_poll = nil
@dirty = false
end
--
1.5.4
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20081211/d2fdb6d5/attachment.bin>
^ permalink raw reply [flat|nested] 9+ messages in thread