esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Fernandez <>
Subject [UI] data model and functionality
Date Thu, 19 Feb 2009 22:25:50 GMT

Greetings ESME team:

In trying to design a proposed end-user web UI for ESME last week, I 
realized there were too many open issues related to the ESME 
dataspace (as seen from the end user's perspective), and the 
functionality associated therewith.  So I spent most of my time last 
week thinking about those issues.  I have a lot of ideas to suggest, 
and questions to ask, and thought for the team to review and discuss. 
It all seems to long and complicated for email, but the esme-dev 
mailing list seems like the only thing we've got -- so here goes (I 
do wish we had a discussion board...).

I hope this all makes sense that that you don't mind me raising so 
many issues at once.



by Bill Fernandez, 19feb09


o Let's assume that we're limiting our discussion to a single server, 
and that we'll deal with federations of servers in a future 

o I'm assuming that for each ESME server there will be one and only 
one user directory server (such as an LDAP server).

o I'm assuming that an ESME server will only recognize users listed 
in the directory server.

o I'm assuming that the directory server is managed by 
organization-level administrators, and that only these administrators 
can add, edit and delete users.

o WHO can use a given ESME server is thus out of control of the ESME 
server, and in control of the user directory administrator(s).

o Since the ESME server will have to keep track of a number of 
ESME-specific data for each user (e.g. "who do I follow?"), the ESME 
server will have to maintain a database of "preferences" that are 
keyed to each user in the directory server.

o Within the ESME end-user UI, this results in there being some info 
that comes from the directory server and thus that the end user CAN'T 
change (e.g. name, work phone, department, mailstop, etc.) and info 
that only applies within the ESME context and that the end user CAN 
change (e.g. nickname, maybe his/her picture or icon, list of people 
he/she follows, etc.).

o Are the above assumptions and implications correct?

o Do we want to allow the administrator of an ESME server to be able 
to create and maintain it's own private database of authorized users 
in addition to those provided by the organization's directory server?

       PROs: Makes it easy to set up a test/evaluation ESME server. 
Makes it possible for small organizations (those without corporate 
LDAP, etc. directories) to use ESME.

       CONs: Introduces the possibility of duplicate user entries.  If 
used extensively makes synchronization with the organizational 
directory impossible.  Provides an end run around policies that the 
organization is counting on the organizational directory to enforce.

o What ESME-specific, end-user-defined, user-profile data do we want 
to support?

       ESME Nickname? -- But what if organizational policy dictates 
the use of only a person's "official" user name?

       Picture/Icon? -- But what if the organizational directory 
server supplies one?  Do we use the "official" image by default, and 
let any user-defined image override it?

o Would our strategy be that the ESME administrator could 
enable/disable any of these user-profile-customization fields as 
required by organizational policy, that we use the 
organizational-server-supplied data by default, but override it with 
the user-supplied values (if any)?


David says he hates the idea of groups (though he likes "pools") and 
of sending things to groups (though he likes "posting to pools"). 
Nevertheless I'm going to use the term "groups" because it helps me 
describe certain functionality, and I hope we can decide on 
terminology after the functionality's agreed to.

With that, I'd like to suggest some concepts and see how the team 
feels about them.

o I'd like to propose that each server can support an abitrary number 
of groups.

o A group is a logical construct consisting of a set of data:
     -- A set of users.
     -- A pool of messages.
     -- A set of metadata.
     -- Perhaps a set of business rules.

o All users are members of a group called something like "Public", 
which by definition contains the set of all users of the ESME server.

o A user may be a member of as many additional groups as desired.

o Any user can create a group (or would we like the ESME 
administrator to be able to grant a "create group" privilege to only 
certain users?).

o The creator of the group is it's first (and initially only) administrator.

o A group's administrator(s) can add and remove users at will, and 
sets the privileges for each user.

o A group's administrator(s) can edit the group's metadata (e.g. it's 
name), and edit any business rules we support for groups.

o Within a group, each user has one of the following privilege levels:
       -- Read.  (Let's you read messages in the group's pool)
       -- Read and Write. (adds the ability to post messages)
       -- Read, Write and Administer (adds all admin privileges)

o The above model establishes three roles:  those of viewer, 
participant and administrator.

o Any user having at least READ privileges against a group has 
PERMISSION to see ALL messages in the group's pool.


o Anyone with admin privileges is an administrator.  Thus a group can 
have as many admins as desired.

o There's no concept of "ownership", only of "control".  All 
administrators have equal control, and the creator of the group can 
drop off the set of administrators to pass control on to others.

o Over time, organizations will inevitably want finer-grained admin 
privileges.  Do we want to try to build them in from the beginning? 
For example:
       -- Moderate group (lets user delete messages?)
       -- Delete group (lets user destroy the group)
       -- Add/remove read and/or write users
       -- Add/remove, promote/demote, admin users

o Before posting a message you must choose a group.  This defines the 
set of users who will have permission to read the message.

o By default, all users are members of the Public group, the Public 
group is the only group that is required for an ESME server, and thus 
in the simplest case all messages are posted to the public pool and 
accessible to everyone.

o Every message posted to a group goes into that group's message 
pool; which is the set of all messages posted to/within that group.

o Which messages a given user sees depends upon what the user has 
chosen to see (who he/she follows, what tags he/she tracks, etc.), 
and/or what he/she searches for.

o So each message exists in one and only one pool, and only users of 
that pool's group have permission to access that message.

o Therefore by choosing a group before you post a message you are 
choosing the scope of privacy for that message; because (a) ALL 
members of that group will have a way (by searching, tracking, 
following, etc.) to SEE that message, (b) NO users outside that group 
will ever be able to see that message.

o Note that since a given message can only exist in one pool over its 
lifetime, if you want multiple groups to recieve the same message you 
must post it to each of those groups separately.  From an intuitive, 
end-user standpoint this appears to be a case of posting the same 
message to multiple groups, but to the system each posting is a 
separate logical entity (each has it's own unique ID, even if the 
text of each is the same).

o So if you see a message in one pool that you want users of another 
pool to see, you have to "copy" it from the one and "paste" it into 
the other.

o This gives us a simple, safe privacy model, yet makes it easy and 
convenient for users to circumvent the rules when necessary.  It 
takes the burden off the system to manage access control of 
cross-pool messaging, and puts the burden of responibility on the 

o Thus the fundamental access control model is:  (a) create groups of 
users that need to communicate with each other, (b) post messages to 
the appropriate group, (c) the system will keep your messages private 
to the group.

o I think we also need to make it possible to have private 
conversations between any two users without having to formally set up 
a group.  It should be easy to make it so that you choose a user in 
the UI as the "group", post a message, and see all the messages in 
this ad-hoc, private pool.

o But then, of course, the two users are going to want to bring 
others in on some conversations, so maybe it would be better to make 
private, ad-hoc group messaging a property of conversations....  I 
don't yet have a suggestion for how to handle this anticipated 
end-user desire.


o OK, so I've suggested that: messages be clustered into pools, each 
pool has a wall of privacy around it that ony allows users in it's 
group to "fish" for messages from that pool, and that a given user 
may be a member of multiple groups and thus have access to multiple 
pools of messages.

o So how to we convert the user's permissions against this 
multi-pooled dataspace into useful and usable views?

o I'm assuming that messages are basically records in a relational 
database, and therefore that creating end-user views is 
(mechanically) a matter of running a query and formatting the output.

o Most of the time the view a user wants is an ordered list of 
messages meeting certain criteria.  I call these lists "streams".

o In various contexts I have heard the use of terms like "timeline", 
"mailbox", "tracking", "conversations", etc.  In all cases the common 
element is that they all consist of a query against the set of 
messages, a results set that is limted by a user's access controls, a 
sort order (usually reverse-chronological order), and the results 
displayed as a list of messages.  I would call all of these streams.

o Here are some common streams:
       -- The Public timeline.
       -- A conversation.
       -- A track (a timeline defined by your tracking criteria).
       -- The messages directed (via @yourname) to you.
       -- Replies to your messages.
       -- Each group would have its own timeline.
       -- The set of messages sent by users you follow.

o The system should create certain predefined streams for you on 
demand.  You should simply be able to ask to view, and get a stream 

       o The messages in a selected group's pool.
       o The messages sent by a selected user.
       o The messages with a seleted tag.
       o The messages posted by users you follow.
       o All messages in the Public pool.
       o The messages you've sent.
       o etc.

o In all cases, each user only gets to see those messages that their 
access privileges allow them to see.

o For example, if you follow someone who posts to groups of which you 
are not a member, you will never under any circumstances be able to 
see THOSE messages.  On the other hand, any messages posted to the 
Public group (of which you ARE a member), and posted to groups of 
which you are a member; you'll be able to see in the streams you 

o So what happens if a poster directs a message (via @username) to a 
given user, but does so when posting to a group of which the user is 
not a member.  In that case, by the rule of group privacy, the 
targeted user would never see the message.

o We should make it possible (and easy) for users to filter a stream.

o For example, let's say that a given server has only one group 
defined so far, the Public group, making all messages in the server 
available (in theory) to all users.

o By default, on viewing the public stream, we would display only the 
most recent (say) 20 messages, and let the user page back in history 
to see previous messages.

o We should also let the user apply ad-hoc filtering criteria to the 
stream without having to formally do a search.  So the user could ask 
to see only messages with a certain tag, and/or by a certain user, 
and the stream should re-display with most recent (say) 20 messages 
from the now-filtered list.  Underr the covers I expect this to be 
implemented as an ad-hoc query, taking some search criteria from the 
basic stream definition and adding some terms from the user-selected 

o Tracking seems to me to be nothing more than stored queries that 
produce streams.  So for example when you say "show me my track of 
the ESME tag", the server runs a query against all the groups to 
which you have access for messages tagged with ESME, sorts them in 
reverse chronological order, and shows you the first (say) 20 results.

o So far I've described timelines and tracking as ways to generate 
streams on-demand by running stored queries.  The only real 
difference is that at the UI level we are calling them different 
things for the user's benefit.

o I think we should also make it possible for users to explicitly "do 
a search".

o users should be able to create a query by specifying:
	o which group pool(s) to search.
	o what tags to look for.
	o what date range to search.
	o whether files, URLs, etc. are attached.
	o what user sent them.
	o etc.
And to be able to show the results in various ways:
	o list of messages.
	o list of files or URLs sent as attachments.
	o a graph of message volume over time.
	o a chart showing the most/least active users.
	o a cloud of tags, or users, or keywords, etc.

o I think of a cloud as another kind of view onto the results of a 
query; although a cloud is special in that for performance reasons 
you need to pre-compute much of the results set (e.g. by indexing.).

o So, for example, the cloud of tags used by JohnDoe would be the 
precompiled list of tags, filtered by their association with JohnDow.

o Since cloud data has to be pre-compiled, we have to define what 
kinds of clouds we're interested in.  We currently have tag clouds, 
and that seems like it should stay.  We currently have word-frequency 
clouds, and I'm not sure we should keep this.  If we want to have 
user-participation clouds, etc. we'll need to say so up front.

o System administrators will need to see charts of different kinds to 
monitor system loading, performance, error rates, etc.

o Maybe there are charts that end-users would like to see too.  For 
example, would group administrators want to see charts of activity 
levels over time?


o My idea for conversations is that if you reply to a message you now 
have a two-message conversation: which illustrates the rule that a 
conversation is a set of messages linked in a tree-shaped structure, 
linked from child to parent by virtue of the child being a reply to 
the parent.

o Within any conversation tree there will thus be parents, children 
and siblings, and users might want to see any/all of those in 
"viewing" a conversation.

o I could see one view of a conversation that is simple a 
forward-chronological listing (a stream) of the messages in the 

o But I could also image users wanting to navigate the tree to follow 
the threads of discourse; which means a different UI than just a list 
of messages.  Perhaps, then, we'd have to open conversations into 
their own windows to view them...


o So what is a message (and consequently, what can you do with it)?

o To a first approximation, a message is a short text string, posted 
by a single user, to a designated message pool.

o But there is also a lot of metadata that's automatically associated 
with the message:
       -- Sender
       -- Posting timepoint
       -- Group (or pool)
       -- is this a reply?
       -- if so, then to what message?
       -- etc.

  o There is also metadata that the user can add at his/her discretion:
        -- tags
        -- directed recipients (via @username)
        -- expiration timepoint?

  o I suggest that we also allow users to attach "payloads" to messages.

  o One form of payload is a URL.  A URL is a long text string, that 
if contained in the message body would make the message too long.  So 
we should make it possible to select a run of text in the message to 
use as the "name" of the URL, then to define the text of the URL 
elsewhere.  There could be two UIs for this:  For the novice, let the 
user type the URL in a separate text box.  For the expert, let the 
user use a special syntax to type the URL inline with the message 

o Another type of payload would me an email address...

o Another type of payload could be a file...

o In all cases we might require that each payload have a run of text 
in the message body to act as the name, anchor and access point for 
the payload.  payload names might be styled to look like HTML links; 
and clicking on them might invoke the payload (opening a link in a 
new window or tab, opening an email address in the user's mail 
client, downloading a file to the user's computer, etc.).

o It should thus be possible for a message to have an arbitrary 
number of payloads.

o If we require that every payload have some "name" in the body of 
the message, then it should be easy to copy and paste these names 
(and thus the payloads) between messages.  This would be useful, for 
example, when you want to share a file or a link from a message you 
received with another user; or to include it in a new message you are 
sending out.

o My first idea about payloads that are files is that the ESME server 
should act like a file store and forward server.  As long as there is 
any message in any pool that references a given file we should store 
it, and allow it to be retrieved whenever a user asks for it.

o Privacy and security is, to a first approximation, enforced by the 
group pools:  if a file is attached to a message, it can only be 
accessed by users in that pool.  If a user chooses to copy the name 
of (and hence a link to) a file and to paste it into a message posted 
to another pool, then it becomes available to that pool too; at the 
responsibility of the user.

o One might ask how to remove a file from circulation once it's been 
attached to an intial message and uploaded to the ESME server.  Once 
might ask if the file has an owner, who can get a list of his/her 
files and force them to be deleted.  Reasonable questions.  But I'm 
wondering if it makes sense to treat files the same way we do URLs. 
Each URL points to a resource (typically a file of SOME sort) on the 
internet, and once published there's no way to retract the 
distribution of a URL string.  So maybe we could treat files the same 

o Or we might say that when the original message either expires 
(because the sender put an expiration timepoint on it) or is purged 
from the system (because there's a server policy in place that purges 
all messages older than a certain age), that only then does the file 
get deleted from the file store.

o It's easy for a payload reference (in essence, perhaps just a URL) 
to be embedded in the message string itself if the message string is 
HTML.  If so, then we embed URLs as part of anchor tags and the web 
browser automagically displays just the name of the anchor while 
carrying with it the full URL string.

o On the other hand, what happens when we forward a message to a 
non-HTML-based system; such as cellphone text messaging (SMS)?  In 
these cases we'd have to strip out the payload, and perhaps just 
insert a payload stub -- a special symbol that means "there used to 
be a payload attached to this word", etc.


o I'd also like us to nail down the at least the basic set of special 
characters and commands that can be embedded into messages.

o We use @username to make sure a user sees a message (subject to 
access rules and depending on what timeline he/she's viewing).

o We also use #tagname to mark embedded tags.

o It also occurs to me that we could use $servicename to direct a 
message to a service (a machine that will respond to our message)...

o Twitter has a number of letter commands, such as "d" at the 
beginning of a message to make it private. I remember seeing a long 
list of special commands recognized by Twitter, but they seemed: (a) 
fragile, and (b) hard to extend the command set.

o I wonder about a command language with a syntax like "p: @fred @joe 
here's my message" to cause a private message to be sent to only fred 
and joe?  Does anyone have a handle on the set of commands we 
envision for our command-line language?



Bill Fernandez  *  User Interface Architect  *  Bill Fernandez Design

(505) 346-3080  *  *

View raw message