Mailing-List: contact esme-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: esme-dev@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of dakoller@googlemail.com
 designates 209.85.218.165 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=v0kcnyBk2DBqjoQ/sNqdRA60zmSgETWfoyi+oGVpr+SDtC76jHyAQDrUrgpxfkGPHC
         4TLGmMnymWWHW0vf1OFsM+z/yGKle6563dCvyDkZsz7lrWCubtN6PEXc102VYqPKt5q0
         PH8js/o4YOHcRKFSaOKzWcVmaz8u4zYqmKELo=
MIME-Version: 1.0
In-Reply-To: <p06240801c5c38c6c1c5b@192.168.0.65>
References: <p06240801c5c38c6c1c5b@192.168.0.65>
Date: Fri, 20 Feb 2009 14:27:25 +0100
Message-ID: <2bca8c350902200527rbd88227n668cc3c98b48d788@mail.gmail.com>
Subject: Re: [UI] data model and functionality
From: Daniel Koller <dakoller@googlemail.com>
To: esme-dev@incubator.apache.org
Content-Type: multipart/alternative; boundary=00163646be021809790463599d51

--00163646be021809790463599d51
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Bill,

thx for compiling such an overview: my comments follow inline:

On Thu, Feb 19, 2009 at 11:25 PM, Bill Fernandez <bill@billfernandez.com>wr=
ote:


>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> THOUGHTS AND QUESTIONS RE. THE END-USER DATA-MODEL AND RELATED
> FUNCTIONALITY FOR ESME
>
> by Bill Fernandez, 19feb09
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> ____________________________________________________________
> SERVERS, DIRECTORIES, USERS, PROFILES
>
> SCOPE
> o Let's assume that we're limiting our discussion to a single server, and
> that we'll deal with federations of servers in a future discussion.
>
> USER DATABASE ASSUMPTIONS
> o I'm assuming that for each ESME server there will be one and only one
> user directory server (such as an LDAP server).

In my opinion we should allow more sources as a user database: jdbc, ldap,
active directory also in combination.

>
>
> o I'm assuming that an ESME server will only recognize users listed in th=
e
> directory server.
>
> o I'm assuming that the directory server is managed by organization-level
> administrators, and that only these administrators can add, edit and dele=
te
> users.
>
> IMPLICATIONS OF THE USER DATABASE ASSUMPTIONS
> o WHO can use a given ESME server is thus out of control of the ESME
> server, and in control of the user directory administrator(s).
>
> o Since the ESME server will have to keep track of a number of
> ESME-specific data for each user (e.g. "who do I follow?"), the ESME serv=
er
> will have to maintain a database of "preferences" that are keyed to each
> user in the directory server.

The ESME server should use an user identifier (from a directory), but shoul=
d
keep the preferences in an own data store independent from the directoy.
(--> I am thinking a environments, where ESME can read/use the directory,
but not modify it)

>
>
> o Within the ESME end-user UI, this results in there being some info that
> comes from the directory server and thus that the end user CAN'T change
> (e.g. name, work phone, department, mailstop, etc.) and info that only
> applies within the ESME context and that the end user CAN change (e.g.
> nickname, maybe his/her picture or icon, list of people he/she follows,
> etc.).
>
> QUESTIONS, ISSUES
> o Are the above assumptions and implications correct?
>
> o Do we want to allow the administrator of an ESME server to be able to
> create and maintain it's own private database of authorized users in
> addition to those provided by the organization's directory server?
>
>      PROs: Makes it easy to set up a test/evaluation ESME server. Makes i=
t
> possible for small organizations (those without corporate LDAP, etc.
> directories) to use ESME.
>
>      CONs: Introduces the possibility of duplicate user entries.  If used
> extensively makes synchronization with the organizational directory
> impossible.  Provides an end run around policies that the organization is
> counting on the organizational directory to enforce.
>
> o What ESME-specific, end-user-defined, user-profile data do we want to
> support?
>
>      ESME Nickname? -- But what if organizational policy dictates the use
> of only a person's "official" user name?

We should support an ESME specific nicknames, because username policy might
also mean that I am "Z000EC1G", which is ok for windows domains, but not fo=
r
a microblogging tool.

>
>
>      Picture/Icon? -- But what if the organizational directory server
> supplies one?  Do we use the "official" image by default, and let any
> user-defined image override it?

If you let the image_url property be either editable or not, this topic
should be solved.

>
>
> o Would our strategy be that the ESME administrator could enable/disable
> any of these user-profile-customization fields as required by organizatio=
nal
> policy, that we use the organizational-server-supplied data by default, b=
ut
> override it with the user-supplied values (if any)?

Good point.

>
>
>
>
> ____________________________________________________________
> GROUPS, POOLS, USERS, PRIVILEGES, ACCESS CONTROL
>
> David says he hates the idea of groups (though he likes "pools") and of
> sending things to groups (though he likes "posting to pools"). Neverthele=
ss
> I'm going to use the term "groups" because it helps me describe certain
> functionality, and I hope we can decide on terminology after the
> functionality's agreed to.
>
> With that, I'd like to suggest some concepts and see how the team feels
> about them.
>
> GROUPS
> o I'd like to propose that each server can support an abitrary number of
> groups.
>
> o A group is a logical construct consisting of a set of data:
>    -- A set of users.
>    -- A pool of messages.
>    -- A set of metadata.
>    -- Perhaps a set of business rules.
>
> GROUP MEMBERSHIP
> o All users are members of a group called something like "Public", which =
by
> definition contains the set of all users of the ESME server.
>
> o A user may be a member of as many additional groups as desired.
>
> GROUP CREATION, DEFINITION, DELETION
> o Any user can create a group (or would we like the ESME administrator to
> be able to grant a "create group" privilege to only certain users?).
>
> o The creator of the group is it's first (and initially only)
> administrator.
>
> o A group's administrator(s) can add and remove users at will, and sets t=
he
> privileges for each user.
>
> o A group's administrator(s) can edit the group's metadata (e.g. it's
> name), and edit any business rules we support for groups.


I would love a feature to populate groups automatically based on business
rules such as "create a group with all people with jobtitle=3Dcontroller in
department=3D"XYZ" (based on regular expressions this could be quite powerf=
ul.

>
>
> USER PRIVILEGES
> o Within a group, each user has one of the following privilege levels:
>      -- Read.  (Let's you read messages in the group's pool)
>      -- Read and Write. (adds the ability to post messages)
>      -- Read, Write and Administer (adds all admin privileges)
>
> o The above model establishes three roles:  those of viewer, participant
> and administrator.
>
> o Any user having at least READ privileges against a group has PERMISSION
> to see ALL messages in the group's pool.
>
> ADMINISTRATOR PRIVILEGES
>
> o Anyone with admin privileges is an administrator.  Thus a group can hav=
e
> as many admins as desired.
>
> o There's no concept of "ownership", only of "control".  All administrato=
rs
> have equal control, and the creator of the group can drop off the set of
> administrators to pass control on to others.
>
> o Over time, organizations will inevitably want finer-grained admin
> privileges.  Do we want to try to build them in from the beginning? For
> example:
>      -- Moderate group (lets user delete messages?)
>      -- Delete group (lets user destroy the group)
>      -- Add/remove read and/or write users
>      -- Add/remove, promote/demote, admin users
>
> MESSAGE POOLS
> o Before posting a message you must choose a group.  This defines the set
> of users who will have permission to read the message.
>
> o By default, all users are members of the Public group, the Public group
> is the only group that is required for an ESME server, and thus in the
> simplest case all messages are posted to the public pool and accessible t=
o
> everyone.
>
> o Every message posted to a group goes into that group's message pool;
> which is the set of all messages posted to/within that group.
>
> o Which messages a given user sees depends upon what the user has chosen =
to
> see (who he/she follows, what tags he/she tracks, etc.), and/or what he/s=
he
> searches for.
>
> PRIVACY
> o So each message exists in one and only one pool, and only users of that
> pool's group have permission to access that message.
>
> o Therefore by choosing a group before you post a message you are choosin=
g
> the scope of privacy for that message; because (a) ALL members of that gr=
oup
> will have a way (by searching, tracking, following, etc.) to SEE that
> message, (b) NO users outside that group will ever be able to see that
> message.
>
> JUMPING THE WALLS OF PRIVACY
> o Note that since a given message can only exist in one pool over its
> lifetime, if you want multiple groups to recieve the same message you mus=
t
> post it to each of those groups separately.  From an intuitive, end-user
> standpoint this appears to be a case of posting the same message to multi=
ple
> groups, but to the system each posting is a separate logical entity (each
> has it's own unique ID, even if the text of each is the same).
>
> o So if you see a message in one pool that you want users of another pool
> to see, you have to "copy" it from the one and "paste" it into the other.
>
> o This gives us a simple, safe privacy model, yet makes it easy and
> convenient for users to circumvent the rules when necessary.  It takes th=
e
> burden off the system to manage access control of cross-pool messaging, a=
nd
> puts the burden of responibility on the user.
>
> ACCESS CONTROL
> o Thus the fundamental access control model is:  (a) create groups of use=
rs
> that need to communicate with each other, (b) post messages to the
> appropriate group, (c) the system will keep your messages private to the
> group.
>
> PRIVATE, INTERPERSONAL MESSAGING
> o I think we also need to make it possible to have private conversations
> between any two users without having to formally set up a group.  It shou=
ld
> be easy to make it so that you choose a user in the UI as the "group", po=
st
> a message, and see all the messages in this ad-hoc, private pool.
>
> o But then, of course, the two users are going to want to bring others in
> on some conversations, so maybe it would be better to make private, ad-ho=
c
> group messaging a property of conversations....  I don't yet have a
> suggestion for how to handle this anticipated end-user desire.
>
> ____________________________________________________________
> STREAMS, TRACKS, SEARCHES, VIEWS
>
> o OK, so I've suggested that: messages be clustered into pools, each pool
> has a wall of privacy around it that ony allows users in it's group to
> "fish" for messages from that pool, and that a given user may be a member=
 of
> multiple groups and thus have access to multiple pools of messages.
>
> o So how to we convert the user's permissions against this multi-pooled
> dataspace into useful and usable views?
>
> HOW ARE MESSAGES STORED?
> o I'm assuming that messages are basically records in a relational
> database, and therefore that creating end-user views is (mechanically) a
> matter of running a query and formatting the output.
>
> STREAMS
> o Most of the time the view a user wants is an ordered list of messages
> meeting certain criteria.  I call these lists "streams".
>
> o In various contexts I have heard the use of terms like "timeline",
> "mailbox", "tracking", "conversations", etc.  In all cases the common
> element is that they all consist of a query against the set of messages, =
a
> results set that is limted by a user's access controls, a sort order
> (usually reverse-chronological order), and the results displayed as a lis=
t
> of messages.  I would call all of these streams.
>
> o Here are some common streams:
>      -- The Public timeline.
>      -- A conversation.
>      -- A track (a timeline defined by your tracking criteria).
>      -- The messages directed (via @yourname) to you.
>      -- Replies to your messages.
>      -- Each group would have its own timeline.
>      -- The set of messages sent by users you follow.
>
> SYSTEM-DEFINED STREAMS
> o The system should create certain predefined streams for you on demand.
>  You should simply be able to ask to view, and get a stream for:
>
>      o The messages in a selected group's pool.
>      o The messages sent by a selected user.
>      o The messages with a seleted tag.
>      o The messages posted by users you follow.
>      o All messages in the Public pool.
>      o The messages you've sent.
>      o etc.
>
All messages containing a defined expression?

>
> ACCESS CONTROL IN STREAMS
> o In all cases, each user only gets to see those messages that their acce=
ss
> privileges allow them to see.
>
> o For example, if you follow someone who posts to groups of which you are
> not a member, you will never under any circumstances be able to see THOSE
> messages.  On the other hand, any messages posted to the Public group (of
> which you ARE a member), and posted to groups of which you are a member;
> you'll be able to see in the streams you request.
>
> DIRECTED MESSAGES
> o So what happens if a poster directs a message (via @username) to a give=
n
> user, but does so when posting to a group of which the user is not a memb=
er.
>  In that case, by the rule of group privacy, the targeted user would neve=
r
> see the message.

The sneder should get a feedback on that.

>
>
> FILTERING STREAMS
> o We should make it possible (and easy) for users to filter a stream.
>
> o For example, let's say that a given server has only one group defined s=
o
> far, the Public group, making all messages in the server available (in
> theory) to all users.
>
> o By default, on viewing the public stream, we would display only the mos=
t
> recent (say) 20 messages, and let the user page back in history to see
> previous messages.
>
> o We should also let the user apply ad-hoc filtering criteria to the stre=
am
> without having to formally do a search.  So the user could ask to see onl=
y
> messages with a certain tag, and/or by a certain user, and the stream sho=
uld
> re-display with most recent (say) 20 messages from the now-filtered list.
>  Underr the covers I expect this to be implemented as an ad-hoc query,
> taking some search criteria from the basic stream definition and adding s=
ome
> terms from the user-selected criteria.
>
>
> TRACKING
> o Tracking seems to me to be nothing more than stored queries that produc=
e
> streams.  So for example when you say "show me my track of the ESME tag",
> the server runs a query against all the groups to which you have access f=
or
> messages tagged with ESME, sorts them in reverse chronological order, and
> shows you the first (say) 20 results.
>
> SEARCHING
> o So far I've described timelines and tracking as ways to generate stream=
s
> on-demand by running stored queries.  The only real difference is that at
> the UI level we are calling them different things for the user's benefit.
>
> o I think we should also make it possible for users to explicitly "do a
> search".
>
> o users should be able to create a query by specifying:
>        o which group pool(s) to search.
>        o what tags to look for.
>        o what date range to search.
>        o whether files, URLs, etc. are attached.
>        o what user sent them.
>        o etc.
> And to be able to show the results in various ways:
>        o list of messages.
>        o list of files or URLs sent as attachments.
>        o a graph of message volume over time.
>        o a chart showing the most/least active users.
>        o a cloud of tags, or users, or keywords, etc.
>
> CLOUDS
> o I think of a cloud as another kind of view onto the results of a query;
> although a cloud is special in that for performance reasons you need to
> pre-compute much of the results set (e.g. by indexing.).
>
> o So, for example, the cloud of tags used by JohnDoe would be the
> precompiled list of tags, filtered by their association with JohnDow.
>
> o Since cloud data has to be pre-compiled, we have to define what kinds o=
f
> clouds we're interested in.  We currently have tag clouds, and that seems
> like it should stay.  We currently have word-frequency clouds, and I'm no=
t
> sure we should keep this.  If we want to have user-participation clouds,
> etc. we'll need to say so up front.


Additionally we should provide the content of clouds also via an api. (this
decouples ESME from means to present the cloud content.

>
>
> CHARTS
> o System administrators will need to see charts of different kinds to
> monitor system loading, performance, error rates, etc.
>
> o Maybe there are charts that end-users would like to see too.  For
> example, would group administrators want to see charts of activity levels
> over time?
>
> ____________________________________________________________
> CONVERSATIONS
>
> o My idea for conversations is that if you reply to a message you now hav=
e
> a two-message conversation: which illustrates the rule that a conversatio=
n
> is a set of messages linked in a tree-shaped structure, linked from child=
 to
> parent by virtue of the child being a reply to the parent.
>
> o Within any conversation tree there will thus be parents, children and
> siblings, and users might want to see any/all of those in "viewing" a
> conversation.
>
> o I could see one view of a conversation that is simple a
> forward-chronological listing (a stream) of the messages in the
> conversation.
>
> o But I could also image users wanting to navigate the tree to follow the
> threads of discourse; which means a different UI than just a list of
> messages.  Perhaps, then, we'd have to open conversations into their own
> windows to view them...
>
>
> ____________________________________________________________
> MESSAGES, PAYLOADS, METADATA
>
> o So what is a message (and consequently, what can you do with it)?
>
> o To a first approximation, a message is a short text string, posted by a
> single user, to a designated message pool.
>
> SYSTEM META-DATA
> o But there is also a lot of metadata that's automatically associated wit=
h
> the message:
>      -- Sender
>      -- Posting timepoint
>      -- Group (or pool)
>      -- is this a reply?
>      -- if so, then to what message?
>      -- etc.
>
>  USER-SUPPLIED METADATA
>  o There is also metadata that the user can add at his/her discretion:
>       -- tags
>       -- directed recipients (via @username)
>       -- expiration timepoint?
>
>  PAYLOAD
>  o I suggest that we also allow users to attach "payloads" to messages.
>
>  o One form of payload is a URL.  A URL is a long text string, that if
> contained in the message body would make the message too long.  So we sho=
uld
> make it possible to select a run of text in the message to use as the "na=
me"
> of the URL, then to define the text of the URL elsewhere.  There could be
> two UIs for this:  For the novice, let the user type the URL in a separat=
e
> text box.  For the expert, let the user use a special syntax to type the =
URL
> inline with the message body.
>
> o Another type of payload would me an email address...
>
> o Another type of payload could be a file...
>
> o In all cases we might require that each payload have a run of text in t=
he
> message body to act as the name, anchor and access point for the payload.
>  payload names might be styled to look like HTML links; and clicking on t=
hem
> might invoke the payload (opening a link in a new window or tab, opening =
an
> email address in the user's mail client, downloading a file to the user's
> computer, etc.).
>
> o It should thus be possible for a message to have an arbitrary number of
> payloads.
>
> o If we require that every payload have some "name" in the body of the
> message, then it should be easy to copy and paste these names (and thus t=
he
> payloads) between messages.  This would be useful, for example, when you
> want to share a file or a link from a message you received with another
> user; or to include it in a new message you are sending out.
>
> FILE PAYLOADS
> o My first idea about payloads that are files is that the ESME server
> should act like a file store and forward server.  As long as there is any
> message in any pool that references a given file we should store it, and
> allow it to be retrieved whenever a user asks for it.

Good point: if we see this extension, we should provide an interface for
virus scanners.
Another one: if there is a library to shorten urls by default, we could
incorporate it here.

>
>
> o Privacy and security is, to a first approximation, enforced by the grou=
p
> pools:  if a file is attached to a message, it can only be accessed by us=
ers
> in that pool.  If a user chooses to copy the name of (and hence a link to=
) a
> file and to paste it into a message posted to another pool, then it becom=
es
> available to that pool too; at the responsibility of the user.
>
> o One might ask how to remove a file from circulation once it's been
> attached to an intial message and uploaded to the ESME server.  Once migh=
t
> ask if the file has an owner, who can get a list of his/her files and for=
ce
> them to be deleted.  Reasonable questions.  But I'm wondering if it makes
> sense to treat files the same way we do URLs. Each URL points to a resour=
ce
> (typically a file of SOME sort) on the internet, and once published there=
's
> no way to retract the distribution of a URL string.  So maybe we could tr=
eat
> files the same way....

How would it be if we supply a personlized list of documents posted and an
option for the user (and the admin) to explicitly delete a certain file.

>
>
> o Or we might say that when the original message either expires (because
> the sender put an expiration timepoint on it) or is purged from the syste=
m
> (because there's a server policy in place that purges all messages older
> than a certain age), that only then does the file get deleted from the fi=
le
> store.

We should provide a means to archive conversations from ESME.

>
>
> TEXT MESSAGES, ETC.
> o It's easy for a payload reference (in essence, perhaps just a URL) to b=
e
> embedded in the message string itself if the message string is HTML.  If =
so,
> then we embed URLs as part of anchor tags and the web browser automagical=
ly
> displays just the name of the anchor while carrying with it the full URL
> string.
>
> o On the other hand, what happens when we forward a message to a
> non-HTML-based system; such as cellphone text messaging (SMS)?  In these
> cases we'd have to strip out the payload, and perhaps just insert a paylo=
ad
> stub -- a special symbol that means "there used to be a payload attached =
to
> this word", etc.
>
>
> ____________________________________________________________
> NAMESPACES, COMMAND LINE MESSAGES
>
> o I'd also like us to nail down the at least the basic set of special
> characters and commands that can be embedded into messages.
>
> o We use @username to make sure a user sees a message (subject to access
> rules and depending on what timeline he/she's viewing).
>
> o We also use #tagname to mark embedded tags.
>
> o It also occurs to me that we could use $servicename to direct a message
> to a service (a machine that will respond to our message)..

Depends on whether we want to handle bots different from users, otherwise w=
e
could call bots also via @

> .
>
> o Twitter has a number of letter commands, such as "d" at the beginning o=
f
> a message to make it private. I remember seeing a long list of special
> commands recognized by Twitter, but they seemed: (a) fragile, and (b) har=
d
> to extend the command set.
>
> o I wonder about a command language with a syntax like "p: @fred @joe
> here's my message" to cause a private message to be sent to only fred and
> joe?  Does anyone have a handle on the set of commands we envision for ou=
r
> command-line language?
>
>
>
>
> Kind regards,


Daniel

--=20
---
Daniel Koller
Jahnstrasse 20
80469 M=FCnchen * dakoller@googlemail.com

--00163646be021809790463599d51--