Subject: Re: [sup-talk] [sup-devel] sup-server revisited

From: William Morgan <[email-masked]>
Date: Wed, 23 Feb 2011 18:23:33 +0000

Reformatted excerpts from Nicolas Pouillard's message of 2011-02-23:
> I agree with you and don't to expose such details. What I want is a
> way to separate the frontend from the backend completely.

Yes, absolutely.

> I also want to be able to script batch operations on the mail store
> easily.

Agreed. A definite goal. The prevalence of [bug reports about]
sup-tweak-labels usage is a sign that other people desire this too.
-- 
William <[email-masked]>
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Nicolas Pouillard <[email-masked]>
Date: Tue, 22 Feb 2011 06:09:43 -0800 (PST)

On Mon, 21 Feb 2011 21:02:28 +0000, William Morgan <[email-masked]> wrote:
> Hello Sup fans,

[...]

> I'm focusing on the server component first. My specific goals are:
> 
> - Encapsulate all storage and indexing logic.
> - Formalize the notion of labels and state, and the relationship between them,
>   messages, and threads. Currently these relationships are fuzzy in Sup, and
>   things suffer for it.
> - Precompute threads, so that search requires only moderate effort, instead of
>   the large effort it does now. This will make search much, much faster, at the
>   expense of a little more effort at index time.
> - Allow concurrent access from multiple clients.
> - Provide something that is transparent and easy to modify without having to go
>   through an ncurses client.

On the server side, and more precisely on the interaction between the two,
I would love to see a simple Unix/command/CLI defined for the server. To make
myself clear, a bit like the notmuch CLI. This would allow for a greater
modularity and reusablity between components. For instance I would like to
index my mails with differents backends (disk is cheap), to get a way to
compare different tools on real data, when searching emails, and thus allow
to debug the different tools.

-- 
Nicolas Pouillard
http://nicolaspouillard.fr
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Tue, 22 Feb 2011 18:00:06 +0000

[resend to list]

Reformatted excerpts from Nicolas Pouillard's message of 2011-02-22:
> I would love to see a simple Unix/command/CLI defined for the server. To
> make myself clear, a bit like the notmuch CLI. This would allow for a
> greater modularity and reusablity between components.

I am definitely interested in a set of composable CLI tools that talk to the
server, for basic operations like search, show, add/remove labels, etc.

> For instance I would like to index my mails with differents backends (disk
> is cheap), to get a way to compare different tools on real data, when
> searching emails, and thus allow to debug the different tools.

The server implementation is pretty heavily tied to a particular index and
store mechanism. It's not clear to me how (or whether) the CLI tools will
allow you to swap out different implementations.
-- 
William <[email-masked]>
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Nicolas Pouillard <[email-masked]>
Date: Wed, 23 Feb 2011 01:59:39 -0800 (PST)

> server, for basic operations like search, show, add/remove labels, etc.

Ok, then I think it would make not much sense to make it different than
notmuch. Of course with good arguments changes can go the other direction as
well.

> allow you to swap out different implementations.

I agree with you and don't to expose such details. What I want is a way to
separate the frontend from the backend completely. This would allow me to
use the frontend with any of:
  - heliotrope
  - sup (maybe)
  - notmuch (not written in ruby)
  - my own toy project (not in ruby either)
  - any combination of both, where "writes" are dispatched to all of them,
    and "reads" are dispatched to one of them (that I can change easily).

I also want to be able to script batch operations on the mail store easily.

Best regards,

---

From: William Morgan <[email-masked]>
Date: Mon, 21 Feb 2011 21:02:28 +0000

Hello Sup fans,

As I alluded to in an earlier email to sup-devel, I have been working recently
on the ancient goal of splitting Sup functionality out into separate client and
server programs. This will have many advantages; in particular it will remove
email lock-in by making it possible to have non-Sup clients interact with Sup.

I've tried various approaches to this goal at various times, without much
success. But with the release of Whistlepig, I have been making a lot of
progress in a short amount of time.

I wanted to lay out my general plan in this email, so that everyone can follow
along and try things out if they feel inclined.

You can find the work in progress here: https://github.com/wmorgan/heliotrope
Currently it is not actually a server, but accomplishes many of the goals
above. Run bin/email-indexer and then bin/email-searcher to play around with
a simple preview of things to come. Stay tuned for more on this.

Once the server is in a reasonable state, my goals for the curses client reboot
are:

- Move to an event-based model rather than the polling + many threads +
  observable model we have now. Dump all events into one big queue,
  and block on it.
- Have one, and only one, non-main thread, which it to communicate with the
  server. No more mutex insanity.
- Work from the ground up with Ruby 1.9, which means finding the many sources
  of encoding errors and punishing them.
- Require the console and ncurses gems so that i18n works out of the box.
- Borrow as much code as possible from the current Sup, because I sure as shit
  don't want to have to rewrite it all.

I haven't started on this at all.

So stay tuned, and check out heliotrope if you are so inclined.