Subject: [sup-talk] rethinking sup part ii

From: [email-masked] (William Morgan)
Date: Sun, 20 Jul 2008 14:33:24 -0700

In which the new version of Sup is described!

http://all-thing.net/2008/07/rethinking-sup-part-ii.html
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Guillaume Quintard)
Date: Sun, 20 Jul 2008 16:28:44 -0700

> http://all-thing.net/2008/07/rethinking-sup-part-ii.html

looks cool to me (mostly the server/client idea). I just have one
question: I am currently using gmail+mpop, and giving the mbox file to
sup. The new sup won't change a lot for me, will it? 
(maybe a stupid question, but I'd like to be sure I understood the
implications).

-- 
Guillaume

---

From: [email-masked] (William Morgan)
Date: Sun, 20 Jul 2008 17:04:35 -0700

Reformatted excerpts from guillaume.quintard's message of 2008-07-20:
> I just have one question: I am currently using gmail+mpop, and giving
> the mbox file to sup. The new sup won't change a lot for me, will it?

It should be basically the same, except that you can throw away the mbox
file when you're done (because STS will handle the storage for you).
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Guillaume Quintard)
Date: Sun, 20 Jul 2008 22:43:40 -0700

> file when you're done (because STS will handle the storage for you).

Ok, that was what I understood. I'd keep the mbox anyway, having a
standard backup format is always nice.

-- 
Guillaume

---

From: [email-masked] (Peter Krenn)
Date: Mon, 21 Jul 2008 14:14:05 +0200

Sounds great.
What kind of interface will this service provide? Or will it be more
like a library which one could use to access the sup storage from any
application? Something like a specialized database?

---

From: [email-masked] (William Morgan)
Date: Mon, 21 Jul 2008 08:22:35 -0700

> application? Something like a specialized database?

It will be a network service with Thrift defining the interface.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Stephen Patterson)
Date: Mon, 21 Jul 2008 16:26:59 +0100

> It will be a network service with Thrift defining the interface.

So we'd then need a collection of custom client applications to search
and/or browse the index? 

This sounds quite a bit like the beagle project -
http://beagle-project.org/Main_Page

-- 
Stephen Patterson :: steve at patter.mine.nu :: http://patter.mine.nu/
GPG: B416F0DE :: Jabber: patter at jabber.earth.li 
"Don't be silly, Minnie. Who'd be walking round these cliffs with a gas oven?"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20080721/aade96ff/attachment.bin>

---

From: [email-masked] (John Bent)
Date: Mon, 21 Jul 2008 10:04:33 -0600

> http://beagle-project.org/Main_Page
And google desktop:
http://desktop.google.com
And spotlight:
http://en.wikipedia.org/wiki/Spotlight_(software)
And MS Fast Find:
http://support.microsoft.com/support/kb/articles/Q166/3/02.ASP

There is room for improvement in all of these tools but the margin seems
considerably smaller than it did for the sup email client which is vastly
superior to anything else.

John

---

From: [email-masked] (Lionel Ott)
Date: Mon, 21 Jul 2008 20:43:49 +0200

> http://all-thing.net/2008/07/rethinking-sup-part-ii.html

Sounds nice overall but something I couldn't infer from the post was if
you're thinking about doing just the reading and searching part of sup or
if some of the mail functionality is going to be kept.
Basically will sts be able to send mail or would one have to send it
using some other client and sts then would present all that nicely to
one, given you feed it with your mails.

---

From: [email-masked] (William Morgan)
Date: Mon, 21 Jul 2008 19:12:09 -0700

> http://beagle-project.org/Main_Page

Thanks for the pointer. It does sound quite similar. Having played with
Beagle a bit, I think they key difference is that I'm envisioning STS as
being the primary interface to your precious infos, whereas Beagle seems
more of a supplemental index of data that's primarily handled by other
apps.

This has a couple implications, including who's responsible for storing
the files (STS: STS; Beagle: you and your apps), and what the user
experience looks like (Beagle: use Thunderbird, then call up Beagle for
help; STS: browse everything through your client, starting with a
search for label "inbox", and call up other applications for handling
foreign mime types).

If anything, Beagle is closer to Sup classic, and STS brings us closer
to Gmail.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Richard Heycock)
Date: Wed, 23 Jul 2008 18:45:06 +1000

> http://all-thing.net/2008/07/rethinking-sup-part-ii.html

I really like the idea and I have to say that I'm *very* glad that you
are keeping the ncurses interface.

My one concern is that you are thinking of using Sphinx and as a
consequence you are relient on a relational database. Is that really
what you want for a mail client? I find this disturbing on two levels:
having a full blown database system for email seems like overkill and
the idea of an index bring based on a relational model quite appaling.

There are a number of other indexs such as Xapian or Hyper Estraier
both have ruby bindings and both are quite mature and widely used in
other projects.

Just my two cents.

rgh

-- 
+61 (0) 410 646 369
[e]:  rgh at neoss.com.au
[im]: rgh at jabber.org

You're worried criminals will continue to penetrate into cyberspace, and
I'm worried complexity, poor design and mismanagement will be there to meet
them - Marcus Ranum

---

From: [email-masked] (Nicolas Pouillard)
Date: Wed, 23 Jul 2008 10:58:10 +0200

> both have ruby bindings and both are quite mature and widely used in

I'm also concerned by using relying on relational databases for a mail
service.

-- 
Nicolas Pouillard aka Ertai
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20080723/0a4a45de/attachment.bin>

---

From: [email-masked] (teroz)
Date: Wed, 23 Jul 2008 11:27:46 +0200

Well surely if we r eschewing typical mail storage formats like mbox and
IMAP then a database is the way to go
unless your suggesting we write some form of storage library as well.

On Wed, Jul 23, 2008 at 10:58 AM, Nicolas Pouillard <
nicolas.pouillard at gmail.com> wrote:

> _______________________________________________
> sup-talk mailing list
> sup-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-talk
>
>

-- 
--Teroz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20080723/0dfa1626/attachment.html>

---

From: [email-masked] (William Morgan)
Date: Wed, 23 Jul 2008 10:09:35 -0700

> the idea of an index bring based on a relational model quite appaling.

I feel exactly the same way. I'm using Sphinx's XML interface to add
things to the index, so there's no RDBMS involved. I hate XML but not as
much as I hate RDBMS.

The current plan is to actually store email/documents in some kind of
persistent key-value store. I have a preliminary very simple
implementation that just uses files on disk, but it will be possible to
swap that out to something like TokyoCabinet or AWS.

> both have ruby bindings and both are quite mature and widely used in

Interesting. I will check those out. I only picked Sphinx because it
seemed popular and robust, but it is currently making my life more difficult
than it really needs to be.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Nicolas Pouillard)
Date: Wed, 23 Jul 2008 19:40:35 +0200

> If anything, Beagle is closer to Sup classic, and STS brings us closer

Another design choice, that could (and perhaps *should*) be made; is that
stored data don't change in the case of mails, IM..., only their labels and
children (threading) do.

Then, this makes even more differences, general file system indexers have an
harder task since, files change all the time this imply more complex data
structures that often provides worse complexity than one could wish.

Cheers,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20080723/4e09fbde/attachment.bin>

---

From: [email-masked] (Richard Heycock)
Date: Thu, 24 Jul 2008 07:41:52 +1000

> unless your suggesting we write some form of storage library as well.

Two things: I do not have a problem with a database per se, I do however
have a problem with a standalone relational database. There are many
embedded databases that work a treat and the user never need know about
them. That's pretty much what mbox, Maildir, etc are.

The other point is that in using Sphinx the STS would not be using
the database directly but would require it simply because the index
required it.

As I mentioned in my previous email there are a number of other indexes
which will serve the purpose without the unnecessary complexity of a
standalone database system.

rgh

-- 
+61 (0) 410 646 369
[e]:  rgh at neoss.com.au
[im]: rgh at jabber.org

---

From: [email-masked] (William Morgan)
Date: Wed, 23 Jul 2008 15:26:18 -0700

Reformatted excerpts from nicolas.pouillard's message of 2008-07-23:
> Another design choice, that could (and perhaps *should*) be made; is
> that stored data don't change in the case of mails, IM..., only their
> labels and children (threading) do.

I think that's a true fact. Draft messages are sort of an exception, but
editing can always be simulated as deleting the old one and adding the
new one.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Guarded Identity)
Date: Wed, 23 Jul 2008 23:55:38 -0500

> http://all-thing.net/2008/07/rethinking-sup-part-ii.html

Your first blog was intriguing, but this second one is putting Sup in a
direction I'm really happy with.  I'm happy about:

    Central Server
    --------------
    I like having a central server, so I can run clients from multiple
    locations.

    Modifiable Messages
    -------------------
    There was a post I read in this thread indicating that people didn't want
    to modify documents.   But if we handle this as a "delete/re-add"
    operation, I'm hoping it will be fine.

    In all honesty, the modifications I'd like to do are

        - remove attachments.  Sometimes at work, people send out some
          absolutely useless, gigantic attachments.

        - delete/expire documents.

    Documents Beyond E-mail
    -----------------------
    I'm /really/ interested in storing different types of "documents" like RSS.
    . . maybe newsgroups too?  Then I could stop using snownews and slrn.  One
    client. . . actually, for this reason, I'd tried out Emacs Gnus, but I
    wasn't really happy with the experience/complexity.

However, I was hoping to see if I could get in a request while you're in the
beginning stages of STS development.

    True Thread-level Labels
    ------------------------
    I was using rss2email, so the number of messages I had was enormous
    (although I disabled that due to performance problems, Ferret corruption,
    and the problem I'm discussion here).  I hate wasting disk space
    needlessly, or cluttering the index for that matter.  There's some mailing
    lists that are mostly fluff, but I wanted to save the occasional thread and
    expire the rest.  But because messages attach to messages really, and not
    to the thread, negation in search logic breaks.  If I search for threads
    that have "!label:save" I can get threads that have "save" messages if they
    also have a message not labeled "save".

    Attaching labels to message always bothered me.  I'm hoping that with STS,
    you can consider designing an architecture that lets you attach labels to a
    thread grouping.  I'm hoping it can be done in a way that's not too
    terribly performing.

While I'm at it, I also have a feature request for STC (Sup The Client)

    Improved Multiple-Label Support
    -------------------------------
    If I'm only using one label for each thread, then really there's no benefit
    to having labels over folders.  So to really get the benefit of labels, it
    would be nice to be able to attach more than one of them in a more
    convenient way.

    There's a couple of ideas I had on how to do this:

        Label Hierarchies
        -----------------
        if you use one label, it automatically pulls in another.  But I wasn't
        sure how to manage this with the Sup UI.

        Label Recommendations
        ---------------------
        When using tab-complete for labeling, I see all my labels initially.
        By now, it's an overwhelming list, and I have to hunt for the label I
        really wanted.  What would be nice is if Sup keeps an index of all
        label combinations (and perhaps their frequencies).  Then when one hits
        tab the first time only labels that are relevant are presented.  If
        those aren't adequate, then one might hit tab to cycle to a listing
        including the rest of the labels.  Or perhaps without cycling, all
        labels can be listed by frequency; that way, the hunt for the most
        relevant label won't be so bad.

    So Sup probably didn't have this kind of feature because of the stronger
    advocacy for:

        - automated label attachment with hooks

        - not relying on labels as much as one relies on the search engine for
          keyword queries (the GMail way)

    But at work, people send me too many messages with too many different types
    of contexts.  It would take quite a bit of AI to auto-categorize these
    messages.  Keyword searches are maybe okay for some of these messages, but
    it's not really perfect.  Sometimes I have to run through synonyms for
    certain concepts.  Better label management would really help me at work.

Okay, so those are the requests I have thus far.  Hopefully they aren't too
far off base from what you are interested in doing with Sup/STS.  I'm happy to
talk through concepts on this list.  And although I'm still working up my Ruby
skills, I'm happy to try to contribute code.

By the way, are you set up to get these kinds of feature requests with ditz?  I
didn't see a ditz "bugs" database (folder) in my git clone of Sup.  Or are you
just using ditz internally for your own personal tracking of Sup development?

Seems like there's a number of places to issue feature requests:

    - this mailing list

    - ditz, if you can help me figure out how best to use it

    - a FeatureRequest wiki page, but it would be moot if you didn't make a
      habit of looking at it

Just let us know what you prefer.

-Sukant

---

From: [email-masked] (Nicolas Pouillard)
Date: Thu, 24 Jul 2008 11:16:45 +0200

> editing can always be simulated as deleting the old one and adding the

Perhaps  that  drafts  should  be marked as "short-lived", and left in another
store.  This  reminds me how generational Garbage Collectors works, there is a
kind of relation...

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20080724/2cc5f51a/attachment.bin>

---

From: [email-masked] (Nicolas Pouillard)
Date: Thu, 24 Jul 2008 11:21:28 +0200

> direction I'm really happy with.  I'm happy about:

Here is another big design decision, note that some labels are truly message
based:

  * unread: of course
  * spam: to be sure that a spam message a response to a ham message don't
          throw away the whole thread.
  * starred: sometime it's there to highlight a message and sometimes to
             select the whole thread.

And some others are truly thread based:

  * inbox
  * most of users labels
  * killed

Since  there  is  mandatory  requirements  for message based labels and thread
based  labels,  one can provide both. Perhaps a syntactic distinction would be
sufficient:
  * a case distinction: INBOX vs unread
  * a special mark: inbox* vs unread (here the star means the repetition on
    all messages of the thread).
  * more ideas to find...

>     Improved Multiple-Label Support

Hum,  that's  a  very  interesting feature. This reminds me delicio.us. In the
same  line,  be  able  to  share popular/public labels for discussion could be
very nice.

Let's  imagine  that  public  tagging  is  done using public:foo labels, users
could  share  some  labels  using  public  ones.  Then  in  answer  message an
X-Public-Labels  header  is  added  and  then suggested to users when applying
labels.

> just using ditz internally for your own personal tracking of Sup development?

It's in the bugs branch.

> Just let us know what you prefer.

I'm not William but I would say that the mailing list is the preferred way.
Moreover attaching a git-patch for the ditz issue tracker could help.

Cheers,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20080724/598bc93c/attachment.bin>

---

From: [email-masked] (William Morgan)
Date: Fri, 25 Jul 2008 00:18:14 -0700

>   - delete/expire documents.

Sup 2.0 will definitely handle both of these cases.

> I'm /really/ interested in storing different types of "documents" like
> RSS.  . . maybe newsgroups too?

Sure. Anything you can munge into Sup's expected format will be
supported. I'm expecting, in addition to a variety of clients (well
if other people get excited!), a variety of readers, each of which
understands some how to get and import into Sup some kind of
information.

> There's some mailing lists that are mostly fluff, but I wanted to save
> the occasional thread and expire the rest.  But because messages
> attach to messages really, and not to the thread, negation in search
> logic breaks.  If I search for threads that have "!label:save" I can
> get threads that have "save" messages if they also have a message not
> labeled "save".

The problem here is not really that there are labels on messages, it's
that the current search operation is: for each message that matches the
query, return the thread that contains it. That works for positive
searches, but clearly doesn't work for negative ones.

Making negative search efficient is hard. But I don't think it matters
whether the labels are per-message or per-thread... 

> Attaching labels to message always bothered me.

How come? Every other property of a thread is calculated from its
component messages---subject, recipients, date---so why not the labels
as well? It's a nice symmetry.

Anyways, I'm not sure what the right answer is yet. It certainly could
be per-thread labels.

> Label Hierarchies

It would be doable... probably through a hook.

> label combinations (and perhaps their frequencies).  Then when one
> hits tab the first time only labels that are relevant are presented.
> If those aren't adequate, then one might hit tab to cycle to a listing

The last of those options (list all but by frequency) is definitely
doable. The other options are probably a fair amount of work. You can
play around with LabelSearchMode and LabelManager if you want to try and
mock something up. I don't expect those to change too much, except that
LabelManager will probably become a thin shell talking to the server.

> By the way, are you set up to get these kinds of feature requests with
> ditz?  I didn't see a ditz "bugs" database (folder) in my git clone of
> Sup. Or are you just using ditz internally for your own personal
> tracking of Sup development?

As Nicolas pointed out, it's currently on the bugs branch, but I'm
planning to move it back into master and starting to add issues there. I
also have tentative plans to move Sup's git repo over to gitorious so
that people can see pretty pictures when they clone.

> Just let us know what you prefer.

The mailing list, ditz patches, or ditz merge requests are all perfectly
acceptable.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Marcus Williams)
Date: Fri, 25 Jul 2008 13:11:06 +0100

> than it really needs to be.

You might enjoy datamapper  - http://datamapper.org

I've been knocking up web apps in ruby/sinatra/datamapper in stupidly
few lines with absolutely no contact with the db underlying the system
other than through datamapper.

Marcus

---

From: [email-masked] (William Morgan)
Date: Fri, 25 Jul 2008 09:00:40 -0700

Reformatted excerpts from nicolas.pouillard's message of 2008-07-24:
> Here is another big design decision, note that some labels are truly
> message based:
> 
>   * unread: of course
>   * spam: to be sure that a spam message a response to a ham message
>     don't throw away the whole thread.
>   * starred: sometime it's there to highlight a message and sometimes
>     to select the whole thread.

What's interesting about the labels you've listed is that they all have
special semantics, i.e. the server or the client does something special
based on whether they're present.

Do people have "user" labels that they use to distinguish individual
messages, as opposed to individual threads?

If not, we could have labels only on threads by having a bunch of
boolean flags on messages: read/unread, spam/ham, starred/regular,
draft/not-draft, etc. The search syntax would probably change for those
features.

> Since  there  is  mandatory  requirements  for message based labels
> and thread based  labels,  one can provide both. Perhaps a syntactic
> distinction would be sufficient:
>   * a special mark: inbox* vs unread (here the star means the
>     repetition on all messages of the thread).

Having both would be possible too, but it seems overly complicated to
me. I think that's my least favorite option right now.

This is a good discussion though. I haven't really thought this all
through.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Guarded Identity)
Date: Sat, 26 Jul 2008 02:32:17 -0500

> messages, as opposed to individual threads?

I got away with applying the "spam" label at the thread level because I never
get any spam that's actually part of a thread; it's normally just a stand-alone
piece of mail.

I'm not sure what other people use starring for, but I just use it to highlight
things I need to reply to, and for that purpose, it works fine at a
thread-level because most threads only have one response I'm interested in
issuing.  But I guess I've seen some of those gigantic threads, so maybe
starring at the message-level /might/ be useful, but I'd not sweat it if it
wasn't possible.

Even for "unread", I get away just managing it at the thread-level, because I'm
generally marking an entire thread as read.  However, I'd imagine someone might
want to revert the unread status of part of a thread.  However, that would be a
nightmare for a large thread.  In that case, I'd probably just use the "@"
key-binding to revert labels.

So all in all, I never use message-level labels, even though I get understand
the instance in which they're needed.  The major reason for this is that the
presentation of the search is at the thread level, so it makes little sense to
think of labels at the message level.

> draft/not-draft, etc. The search syntax would probably change for those

This is /exactly/ what I'm hoping for.  I could probably come up with other
hypothetical abstractions for indexing, but I'm happy as long as at the end of
the day I get negation in reasonably fast searches.  I know it sounds like a
feature that not everyone might use, but the more messages I put in the index,
the more I find I myself trying to make use of negation to prune searches.
Also, sometimes I like to manage Sup with scripts and having negation allows me
to do some more cool things without having to bother William with feature
requests for the odd message/label-managing tasks I have.

Hopefully you can continue brain-storming what design you want to do to support
this.  If you run into a problem, maybe we can jump into the design then.

> me. I think that's my least favorite option right now.

Yeah, I was trying to think of a way to manage both, and it felt like a kludge.
Personally, I like the idea of managing as much as we can at the thread-level.

-Sukant