esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ethan Jewett <esjew...@gmail.com>
Subject Re: Scalability was Re: What are pools? was Re: [VOTE] Approve the release of apache-esme-1.3
Date Thu, 04 Aug 2011 08:37:33 GMT
Hi Jack,

Here is an example of setting up a Lift app with PostgreSQL. The same
approach would work for MySQL:
http://www.assembla.com/spaces/liftweb/wiki/Set_Up_Jetty_and_PostgreSQL

ESME currently works in terms of large-ish memory models and load
balancers, if you really need it. But you should be able to get a
single application server up to at least 10s of thousands of users, so
the need for a load-balancer setup is purely theoretical at this
point.

Regarding your scenario, the point about ESME is that it is designed
so that very few operations will result in database reads. The history
of timelines is not easily accessible, and new messages are inserted
into every subscriber's mailbox at the time of message creation
(rather than on read). Yes, if someone goes back in history via search
or another mechanism, we'll need to read the database once, but that
is a read on the single key field (the message ID) so it should be
very fast. And the next time that message pops up in search on the
same application server, it is already cached so no DB read is
necessary.

We do have a bit of an in-depth explanation of why avoiding the
database is a good idea for messaging applications on our wiki. You
can read it here.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=9373341

Cheers,
Ethan



On Wed, Aug 3, 2011 at 9:21 PM, Jack Park <jackpark@topicquests.org> wrote:
> Thanks, Ethan.
> I'd like to know more about using MySQL.
> Indeed, I'd like to know more about other classes of databases, e.g. graph
> databases.
>
> Still, while hit rate is one aspect of scalability, when you talk about
> "safely cached", I wonder about the combinatoric aspect of scalability, say,
> a million users each with timelines in the hundreds to thousands long, and
> all the entailed message objects. Are you thinking in terms of, say,
> external memcached solutions, or just large memory models and load
> balancers?
>
> Curious minds...
> Thanks
> Jack
>
> On 8/3/2011 4:56 AM, Ethan Jewett wrote:
>>
>> Hi Jack,
>>
>> It should be able to run on any Java application server with any
>> database via JNDI or with any JDBC database by providing JDBC
>> connection information in a .props file. I believe that on Stax (now
>> CloudBees) we are using a MySQL database.
>>
>> Regarding scalability, the design of ESME is such that it doesn't rely
>> much on the database for scalability. Operations should generally
>> happen on data in memory, and there shouldn't be access of the
>> database on most operations. We can do this because the data model has
>> messages as immutable objects (you can't change or delete them), so
>> they can be safely cached and we can rest assured that they will never
>> be out of sync with the database.
>>
>> I'm fairly sure we did some performance tests a while ago that showed
>> ESME was able to support thousands of concurrent users on a fairly
>> small server without trouble, but I can't currently find that
>> documentation. Maybe someone else knows where it ran off to? Seems
>> like something we should have on a our website (and a test that we
>> should redo at some point).
>>
>> Cheers,
>> Ethan
>>
>

Mime
View raw message