Subject: Re: [sup-devel] sup-server revisited

From: Sascha Silbe <[email-masked]>
Date: Thu, 03 Mar 2011 20:38:14 +0100

Excerpts from William Morgan's message of Wed Feb 23 19:43:34 +0100 2011:

> - Individual messages have "state", including unread, starred, and all the
>   other things you've mentioned.

Are you going to support user-defined "states"? I have quite a few
labels that would make much more sense on the message than on the thread
(e.g. todo-{low,normal,high}).

I sometimes "reply" to messages without setting a recipient in order to
record private notes. It would be nice if this capability was either
preserved (i.e. don't choke if there's no recipient) or maybe even
explicit support for non-mail objects (notes, attached files, reminder
dates, etc.) added.

> - The user can set the labels for a thread, but can't create labels that have
>   the same name as a message state.

Hmm, that has a chance of conflicting with user-defined message states...

Sascha

-- 
http://sascha.silbe.org/
http://www.infra-silbe.de/
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Sat, 26 Feb 2011 22:04:13 +0000

Reformatted excerpts from Hamish's message of 2011-02-26:
> /var/lib/gems/1.8/gems/ffi-1.0.6/lib/ffi/library.rb:93:in `ffi_libraries': no library specified (LoadError)
>         from /var/lib/gems/1.8/gems/ffi-1.0.6/lib/ffi/library.rb:129:in `attach_function'

You need to install your system's TokyoTyrant / TokyoCabinet libraries. The
OklahomaMixer gem is a ffi wrapper on those. Then it should work.

> I also wasn't sure how to build and run whistlepig from source.

`make rubygem` and then `gem install ruby/pkg/*.gem` will do it, but installing the public gem should be sufficient.

I'm about to push some changes to Heliotrope that should make it easy to get
started with. I will announce here.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Tero Tilus <[email-masked]>
Date: Tue, 22 Feb 2011 22:17:04 +0200

William Morgan, 2011-02-21 23:02:
> I have been making a lot of
> progress in a short amount of time.

Cool!  I'm thrilled.  \o/

> - Precompute threads, so that search requires only moderate effort,
>   instead of the large effort it does now. This will make search
>   much, much faster, at the expense of a little more effort at index
>   time.

I guess this offers the possibility to add threading to "external
state" of messages, together with labels.  I'd do a whole a lot of
more prune'n'crafting with my threads if there only was more
comfortable interface to doing it (admit it, been too lazy to
implement it in sup).

Do you have considerations wrt to programmatical thread browsing and
editing API?  Something I've thought of, on the top of my head.

first_msg
|-- second_msg
|-- third_msg
|   `-- fifth_msg
`-- fourth_msg

sixth_msg
`-- seventh_msg

thread = Thread.find :second
subthread = thread.find { |msg| msg == :third_msg }.prune!
Message.find(:sixth_msg).craft! subthread

first
|-- second
`-- fourth

sixth
|-- third
|   `-- fifth
`-- seventh

> - Allow concurrent access from multiple clients.

Since the first post on sup-server I've been nurturing the idea of
"fat" sup-client for maemo/meego.

> - Borrow as much code as possible from the current Sup, because I
>   sure as shit don't want to have to rewrite it all.

Would it be time to go for https://github.com/mikel/mail now?  It
would be supported and actively developed and I really like the API.
As a downside, depending on activesupport pulls in a whole a lot of
fluff and using treetop may suggest that parsing might hog a little
more cpu and memory than is absolutely necessary.

Who would give us bindings to GMime and wrap it inside Mail API...

-- 
Tero Tilus ## 050 3635 235 ## http://tero.tilus.net/
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Sun, 17 Apr 2011 17:55:06 +0000

Reformatted excerpts from William Morgan's message of 2011-04-17:
> Reformatted excerpts from Gregor Hoffleit's message of 2011-04-15:
> > Bugfix: uint16_t is too small for posarray->size and posarray->next
> 
> Applied. Thanks!

I have released this as 0.5. Also, I have pushed a forced update to
whistlepig master in order to correct some erroneous email addresses in
the commit logs, so you'll have to reset your local master branch next
time you fetch.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Gregor Hoffleit <[email-masked]>
Date: Wed, 02 Mar 2011 13:43:51 +0100

I keep getting stalls from the heliotrope-server process.

When adding a certain mbox file, the server reproducably hangs after
1167 messages, leading to an RestClient::RequestTimeout exception in
heliotrope-add.

At this point I have to kill -KILL the heliotrope-server process.

I can't give away the mbox file. Any hints how I could debug the
heliotrope-server script?

Nice work otherwise! The web client is a nice demonstration of what
might come out of this project ;-)

    Gregor

* Hamish <[email-masked]> [Di Mär 01 20:38:38 +0100 2011]
> Excerpts from William Morgan's message of Sat Feb 26 23:15:50 +0000 2011:
> > Reformatted excerpts from William Morgan's message of 2011-02-26:
> > Check out the README and see if the instructions work for you.
> 
> They do indeed, thank you. It's fun to play about with. I look forward
> to watching it develop and hopefully contributing :)

---

From: Nicolas Pouillard <[email-masked]>
Date: Wed, 23 Feb 2011 01:48:12 -0800 (PST)

On Tue, 22 Feb 2011 21:29:07 +0000, William Morgan <[email-masked]> wrote:
> Reformatted excerpts from Tero Tilus's message of 2011-02-22:
> > I guess this offers the possibility to add threading to "external state"
> > of messages, together with labels.  I'd do a whole a lot of more
> > prune'n'crafting with my threads if there only was more comfortable
> > interface to doing it (admit it, been too lazy to implement it in sup).
> 
> Not quite sure I follow, but the intention is certainly to allow clients to
> create and break threads and to present labels as per-thread, rather than
> per-message, data.

I would not go for "labels on threads" only, here is a list of labels that are
more message based than thread based: unread, starred, draft, sent,
attachment. And while it would make sense to have special cases for those this
also prevents from user defined per message labels.

-- 
Nicolas Pouillard
http://nicolaspouillard.fr
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Wed, 23 Feb 2011 18:43:34 +0000

Reformatted excerpts from Nicolas Pouillard's message of 2011-02-23:
> I would not go for "labels on threads" only, here is a list of labels that
> are more message based than thread based: unread, starred, draft, sent,
> attachment. And while it would make sense to have special cases for those
> this also prevents from user defined per message labels.

One of the major goals of Heliotrope is to formalize this type of thing and to
implement it correctly. Here's the model:

- Individual messages have "state", including unread, starred, and all the
  other things you've mentioned.
- Threads have "labels", which are user-defined.
- Message state is mapped onto thread labels by taking the union of the states
  of all the messages in a thread. E.g. if one message is unread, the entire
  thread has the unread label.
- Sup maintains that mapping for you.
- The user can set (some of) the state of a particular message.
- The user can set the labels for a thread, but can't create labels that have
  the same name as a message state.
- The labels of a thread match all messages in a thread at search time.

Sup conflates these things, maintains labels on messages only, and does some UI
stuff to make it seem like they apply to threads. It's easy to find corner
cases where you get weird behavior.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Tue, 22 Feb 2011 21:29:07 +0000

> editing API?

I suspect won't get into that level of detail. The API will be more along
the lines of "give me thread 6" and you'll get a tree of message ids back,
and on the client side you can represent that however you like. Unless
there's a compelling case for handling portions of threads at the API level.

> Since the first post on sup-server I've been nurturing the idea of "fat"
> sup-client for maemo/meego.

That would be great. I would like an Android client while you're at it. :)

> Would it be time to go for https://github.com/mikel/mail now?  It would be
> supported and actively developed and I really like the API.  As a
> downside, depending on activesupport pulls in a whole a lot of fluff and
> using treetop may suggest that parsing might hog a little more cpu and
> memory than is absolutely necessary.

I looked at it, but the dependency on activesupport is a dealbreaker for me.
There's no way I would pull that pile of Rails shit into Sup.

RMail is not great and I don't like using it, but it really is the best
thing I've found so far. I think I have whipped it into shape pretty well in
heliotrope.

> Who would give us bindings to GMime and wrap it inside Mail API...

I would use GMime bindings in a heartbeat.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Sat, 26 Feb 2011 23:15:50 +0000

---

From: William Morgan <[email-masked]>
Date: Wed, 23 Feb 2011 21:08:07 +0000

Reformatted excerpts from Alvaro Herrera's message of 2011-02-23:
> I like this, except the part that thread_labels = union(states).  It
> doesn't work correctly for "attachment" for example: it makes it look
> like the whole thread "has attachments" which doesn't make sense.

The thread has the label "attachment", but when you look at the
individual messages inside of it, you see which ones actually have the
attachment state. I think this makes sense.

> Also, you said upthread that Heliotrope was going to provide ways to
> break threads and form new ones from pieces.  I badly want this
> feature :-)  (I haven't checked Heliotrope out yet).

Yes, this will definitely be there.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Gregor Hoffleit <[email-masked]>
Date: Fri, 15 Apr 2011 22:53:21 +0200

* William Morgan <[email-masked]> [Fr Apr 15 19:16:43 +0200 2011]
> > The source of this problem is in whistlepig's tokenizer. I'm hurt by an
> > overflow in posarray->next and posarray->size, which are defined as
> > uint16_t. I was able to fix my problem by defining these to uint32_t:
> 
> Thanks! This is totally good. I will admit I got distracted by some
> other tasks and forgot to come back to this. Maybe that's a sign I
> should start using Github's vaunted issue tracker.
> 
> What I don't understand is that the posarray stuff is all for token
> offsets, isn't it? So why would the number of lines actually matter?
> Large messages can overflow this but it should be in terms of tokens,
> not lines.

Yep. In fact the number of lines is a pure coincidence.

The mail that got stuck was a logcheck report, which had 35.000 lines
all starting with the current date. I guess whistlepig tried to append
some 35.000 instances of "Feb" to the posarray, which lead to an
infinite loop.

Regards,
    Gregor

---

From: William Morgan <[email-masked]>
Date: Sun, 17 Apr 2011 16:58:06 +0000

Applied. Thanks!

---

From: Hamish <[email-masked]>
Date: Sat, 26 Feb 2011 19:15:40 +0000

Excerpts from William Morgan's message of Mon Feb 21 21:02:28 +0000 2011:
> As I alluded to in an earlier email to sup-devel, I have been working recently
> on the ancient goal of splitting Sup functionality out into separate client and
> server programs. This will have many advantages; in particular it will remove
> email lock-in by making it possible to have non-Sup clients interact with Sup.

Excellent :) I look forward to an android client. I will definitely
contribute to it, and I *might* even get it started. Or might not ...

> You can find the work in progress here: https://github.com/wmorgan/heliotrope
> Currently it is not actually a server, but accomplishes many of the goals
> above. Run bin/email-indexer and then bin/email-searcher to play around with
> a simple preview of things to come. Stay tuned for more on this.

I've had a quick look at heliotrope (and whistlepig) and not got very
far:

$ ruby -I lib bin/email-indexer ~/path/to/mbox
/var/lib/gems/1.8/gems/ffi-1.0.6/lib/ffi/library.rb:93:in `ffi_libraries': no library specified (LoadError)
        from /var/lib/gems/1.8/gems/ffi-1.0.6/lib/ffi/library.rb:129:in `attach_function'
        from /var/lib/gems/1.8/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/utilities.rb:39:in `func'
        from /var/lib/gems/1.8/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/utilities.rb:214
        from /usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
        from /usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from /var/lib/gems/1.8/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/extensible_string/c.rb:1
        from /usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
        from /usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from /var/lib/gems/1.8/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/extensible_string.rb:1
        from /usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
        from /usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from /var/lib/gems/1.8/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/hash_database.rb:2
        from /var/lib/gems/1.8/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer.rb:11:in `open'
        from ./lib/heliotrope/store.rb:40:in `initialize'
        from bin/email-indexer:20:in `new'
        from bin/email-indexer:20

$ ruby --version
ruby 1.8.7 (2010-06-23 patchlevel 299) [x86_64-linux]

Am I missing something? It appears that the oklahoma_mixer gem is
broken. I installed it though rubygems. Any ideas?

I also wasn't sure how to build and run whistlepig from source. I'm not
familiar with the extconf stuff. I can type make happily enough, but it
appears more steps are required. If someone shows me the way I'll
happily contribute an improved README for anyone else who wants to use
it. (I installed whistlepig from the gem in order to try out heliotrope,
in case you're wondering).

Hamish

---

From: William Morgan <[email-masked]>
Date: Fri, 15 Apr 2011 17:16:43 +0000

---

From: Nicolas Pouillard <[email-masked]>
Date: Tue, 22 Feb 2011 06:09:43 -0800 (PST)

On Mon, 21 Feb 2011 21:02:28 +0000, William Morgan <[email-masked]> wrote:
> Hello Sup fans,

[...]

> I'm focusing on the server component first. My specific goals are:
> 
> - Encapsulate all storage and indexing logic.
> - Formalize the notion of labels and state, and the relationship between them,
>   messages, and threads. Currently these relationships are fuzzy in Sup, and
>   things suffer for it.
> - Precompute threads, so that search requires only moderate effort, instead of
>   the large effort it does now. This will make search much, much faster, at the
>   expense of a little more effort at index time.
> - Provide something that is transparent and easy to modify without having to go
>   through an ncurses client.

On the server side, and more precisely on the interaction between the two,
I would love to see a simple Unix/command/CLI defined for the server. To make
myself clear, a bit like the notmuch CLI. This would allow for a greater
modularity and reusablity between components. For instance I would like to
index my mails with differents backends (disk is cheap), to get a way to
compare different tools on real data, when searching emails, and thus allow
to debug the different tools.

---

From: Alvaro Herrera <[email-masked]>
Date: Wed, 23 Feb 2011 18:30:26 -0300

> attachment state. I think this makes sense.

Yes, this sounds good to me.

> Yes, this will definitely be there.

Great.

-- 
Álvaro Herrera <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Mark Alexander <[email-masked]>
Date: Tue, 22 Feb 2011 08:46:56 -0500

> You can find the work in progress here: https://github.com/wmorgan/heliotrope

I took a very quick look, and it appears to support mbox only at the moment.
Presumably maildir will be supported eventually?
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Gregor Hoffleit <[email-masked]>
Date: Fri, 15 Apr 2011 12:46:40 +0200

* William Morgan <[email-masked]> [Do Mär 03 20:17:36 +0100 2011]
> Reformatted excerpts from Gregor Hoffleit's message of 2011-03-03:
> > William, I was able to isolate my problem: For me, heliotrope-add hangs
> > for messages with more than 32768 content lines.
> 
> Crazy! I will take a look. Thanks for the good debugging.

William, I noticed that heliotrope still has a problem with mails with
my mail with more than 32768 lines of content.

commit 42dbc087260074513af22078b77e65b91d3318d9
Author: Gregor Hoffleit <[email-masked]>
Date:   Fri Apr 15 12:19:10 2011 +0200

    Change the type of posarray->size and posarray->next from uint16_t
    to uint32_t. 16 bits are not enough to hold really large messages.

diff --git a/entry.h b/entry.h
index 9799e15..56673a0 100644
--- a/entry.h
+++ b/entry.h
@@ -15,8 +15,8 @@
 #include "khash.h"
 
 typedef struct posarray {
-  uint16_t size;
-  uint16_t next;
+  uint32_t size;
+  uint32_t next;
   pos_t* data;
 } posarray;

Regards,
    Gregor Hoffleit

---

From: Hamish <[email-masked]>
Date: Tue, 01 Mar 2011 19:38:38 +0000

Hamish Downer

---

From: Gregor Hoffleit <[email-masked]>
Date: Thu, 03 Mar 2011 18:32:34 +0100

I have attached two mbox files, each containing a single message.
The message in good.mbox has 32768 content lines (excluding header) and
has no problem in heliotrope-add. The message in bad.mbox has 32769
content lines, and heliotrope-add hangs when trying to add this message.

I haven't yet tried to locate the locate the problem in the source.

Regards,
    Gregor

* William Morgan <[email-masked]> [Mi Mär 02 19:51:58 +0100 2011]
> Reformatted excerpts from Gregor Hoffleit's message of 2011-03-02:
> Can you isolate the message? It might be that it has a huge attachment
> that RMail is taking forever to parse, and that results in a client
> timeout.
> 
> I need to move the message processing out of the request loop,
> obviously.
> 
> As a workaround (which also is much faster), kill the server, wipe
> everything, and use `heliotrope-add -d <dir>` to import an mbox or
> maildir without having to go through HTTP.
> 
> Thanks! I'm currently working on stripping everything out of Sup that
> will be handled by the server. The good news is that this is a LOT, and
> that Sup will be much leaner and meaner than before.

---

From: William Morgan <[email-masked]>
Date: Tue, 22 Feb 2011 19:00:01 +0000

Reformatted excerpts from Tero Tilus's message of 2011-02-22:
> You mean something like configuring your MDA to do a pipe-deliver to
> "sup-indexer"?

Yes.

> Does this mean you intend to implement IMAP capability to sup-server?

Yeah. I'm not really looking forward to actually doing it, but I would like to
have an IMAP compatibility layer on the server. That goes a long way to
eliminating what I think is the biggest blocker for Sup adoption, which is
lockin.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Tue, 22 Feb 2011 17:55:23 +0000

> Presumably maildir will be supported eventually?

It's more like: you will give new emails to the server using some tool, and it
will store them some way that you don't care about, and you can then use a sup
client to view them, or (eventually) an IMAP client, or you can export them
from the server and leave Sup entirely.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Thu, 03 Mar 2011 19:17:36 +0000

---

From: William Morgan <[email-masked]>
Date: Wed, 02 Mar 2011 18:51:58 +0000

---

From: Alvaro Herrera <[email-masked]>
Date: Wed, 23 Feb 2011 15:53:23 -0300

I like this, except the part that thread_labels = union(states).  It
doesn't work correctly for "attachment" for example: it makes it look
like the whole thread "has attachments" which doesn't make sense.  The
user really needs to have a way of telling messages that really have
attachments from ones that don't.

Also, you said upthread that Heliotrope was going to provide ways to
break threads and form new ones from pieces.  I badly want this feature
:-)  (I haven't checked Heliotrope out yet).

> cases where you get weird behavior.

Agreed.  Someone mentioned some weeks ago that "undo" badly messed up
labels that you're now separating as "state" (again, attachment).  Seems
like the current proposal fixes that problem.

---

From: William Morgan <[email-masked]>
Date: Mon, 21 Feb 2011 21:02:28 +0000

Hello Sup fans,

I've tried various approaches to this goal at various times, without much
success. But with the release of Whistlepig, I have been making a lot of
progress in a short amount of time.

I wanted to lay out my general plan in this email, so that everyone can follow
along and try things out if they feel inclined.

Once the server is in a reasonable state, my goals for the curses client reboot
are:

- Move to an event-based model rather than the polling + many threads +
  observable model we have now. Dump all events into one big queue,
  and block on it.
- Have one, and only one, non-main thread, which it to communicate with the
  server. No more mutex insanity.
- Work from the ground up with Ruby 1.9, which means finding the many sources
  of encoding errors and punishing them.
- Require the console and ncurses gems so that i18n works out of the box.
- Borrow as much code as possible from the current Sup, because I sure as shit
  don't want to have to rewrite it all.

I haven't started on this at all.

So stay tuned, and check out heliotrope if you are so inclined.

---

From: Tero Tilus <[email-masked]>
Date: Tue, 22 Feb 2011 20:44:28 +0200

William Morgan, 2011-02-22 19:55:
> It's more like: you will give new emails to the server using some
> tool

> you can then use a sup client to view them, or (eventually) an IMAP
> client