Mailing-List: contact dev-help@esme.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@esme.apache.org
Received-SPF: pass (nike.apache.org: domain of hirsch.dick@gmail.com
 designates 209.85.216.47 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAEaXKzqy2+k-4xOKVBnYnfPpyze=xyNuDEuTz=yVLSZt4spjZQ@mail.gmail.com>
References: 
 <CAN5nAX5SkNdqoNJvwKYwaniPQ-63uHwWE=PYAq0EhD+X=PwKvQ@mail.gmail.com>
	<CAN2rV4GVoet_+Cbu7OUxMzk4Lu5q668=TbRDw=5jGXXafCnMVQ@mail.gmail.com>
	<4E319ECC.2000301@topicquests.org>
	<4E31A39D.6090901@topicquests.org>
	<4E31A51E.3020806@topicquests.org>
	<CAN5nAX4L--w2u_-o7Chz8C+w8hUyO8Qm+Mn8qtSoKxT2Kk0t3A@mail.gmail.com>
	<4E31A8AE.9080708@topicquests.org>
	<CAN5nAX4PLBN3++v0ijLOTHyrk2zHXaENi79t=rQNFWRs=-BgTA@mail.gmail.com>
	<4E321D4A.1030702@topicquests.org>
	<CAN5nAX4-DfNJ09B9oYST3MAc4+LR8e3JuxK8yg-X_zbiqYanMQ@mail.gmail.com>
	<4E322105.3090406@topicquests.org>
	<4E322466.7010904@topicquests.org>
	<CAN5nAX7NPQW26Q+xkdKDCC_xhvs+P=LLu6Mm+F7XXw_PoW5crg@mail.gmail.com>
	<4E322C18.8050907@topicquests.org>
	<CAEaXKzpJpSXfn8psJyB_Lm_nDp_khBuykDQHrL+DURaVmMgc4g@mail.gmail.com>
	<4E32D091.90400@topicquests.org>
	<CAEaXKzpCGtDAph2rQzRs4MJunEnDXHsCaLXH9H3pSuaOHnLzRw@mail.gmail.com>
	<4E382D7C.6010902@topicquests.org>
	<CAEaXKzrYQh1k761G0_Mhr7F4VN56YpTUkOghvEfoocQ1Pp4W-w@mail.gmail.com>
	<4E399FAB.9050106@topicquests.org>
	<CAEaXKzqy2+k-4xOKVBnYnfPpyze=xyNuDEuTz=yVLSZt4spjZQ@mail.gmail.com>
Date: Thu, 18 Aug 2011 14:55:17 +0200
Message-ID: 
 <CAN5nAX6sRFKjkcBSwy=nfp3=ig7XewjNiSLKn92GN47GDGp79g@mail.gmail.com>
Subject: Re: Scalability was Re: What are pools? was Re: [VOTE] Approve the
 release of apache-esme-1.3
From: Richard Hirsch <hirsch.dick@gmail.com>
To: dev@esme.apache.org
Content-Type: text/plain; charset=ISO-8859-1

We use MySQL for the CloudBees instance.  It is no problem since ESME
uses the J2EE container to find its javax.sql.DataSource.

D.

On Thu, Aug 4, 2011 at 10:37 AM, Ethan Jewett <esjewett@gmail.com> wrote:
> Hi Jack,
>
> Here is an example of setting up a Lift app with PostgreSQL. The same
> approach would work for MySQL:
> http://www.assembla.com/spaces/liftweb/wiki/Set_Up_Jetty_and_PostgreSQL
>
> ESME currently works in terms of large-ish memory models and load
> balancers, if you really need it. But you should be able to get a
> single application server up to at least 10s of thousands of users, so
> the need for a load-balancer setup is purely theoretical at this
> point.
>
> Regarding your scenario, the point about ESME is that it is designed
> so that very few operations will result in database reads. The history
> of timelines is not easily accessible, and new messages are inserted
> into every subscriber's mailbox at the time of message creation
> (rather than on read). Yes, if someone goes back in history via search
> or another mechanism, we'll need to read the database once, but that
> is a read on the single key field (the message ID) so it should be
> very fast. And the next time that message pops up in search on the
> same application server, it is already cached so no DB read is
> necessary.
>
> We do have a bit of an in-depth explanation of why avoiding the
> database is a good idea for messaging applications on our wiki. You
> can read it here.
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=9373341
>
> Cheers,
> Ethan
>
>
>
> On Wed, Aug 3, 2011 at 9:21 PM, Jack Park <jackpark@topicquests.org> wrote:
>> Thanks, Ethan.
>> I'd like to know more about using MySQL.
>> Indeed, I'd like to know more about other classes of databases, e.g. graph
>> databases.
>>
>> Still, while hit rate is one aspect of scalability, when you talk about
>> "safely cached", I wonder about the combinatoric aspect of scalability, say,
>> a million users each with timelines in the hundreds to thousands long, and
>> all the entailed message objects. Are you thinking in terms of, say,
>> external memcached solutions, or just large memory models and load
>> balancers?
>>
>> Curious minds...
>> Thanks
>> Jack
>>
>> On 8/3/2011 4:56 AM, Ethan Jewett wrote:
>>>
>>> Hi Jack,
>>>
>>> It should be able to run on any Java application server with any
>>> database via JNDI or with any JDBC database by providing JDBC
>>> connection information in a .props file. I believe that on Stax (now
>>> CloudBees) we are using a MySQL database.
>>>
>>> Regarding scalability, the design of ESME is such that it doesn't rely
>>> much on the database for scalability. Operations should generally
>>> happen on data in memory, and there shouldn't be access of the
>>> database on most operations. We can do this because the data model has
>>> messages as immutable objects (you can't change or delete them), so
>>> they can be safely cached and we can rest assured that they will never
>>> be out of sync with the database.
>>>
>>> I'm fairly sure we did some performance tests a while ago that showed
>>> ESME was able to support thousands of concurrent users on a fairly
>>> small server without trouble, but I can't currently find that
>>> documentation. Maybe someone else knows where it ran off to? Seems
>>> like something we should have on a our website (and a test that we
>>> should redo at some point).
>>>
>>> Cheers,
>>> Ethan
>>>
>>
>