esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Hirsch <>
Subject Re: Further analysis of the GC issue
Date Thu, 26 Nov 2009 07:06:53 GMT
Love this.

"a popular micro-blogging site" who shall remain anonmyous .->


On Thu, Nov 26, 2009 at 5:00 AM, David Pollak
<> wrote:
> On Wed, Nov 25, 2009 at 7:16 PM, Bechauf, Michael
> <>wrote:
>> Wasn't this exactly the kind of stuff that the Eclipse Memory Analyzer -
>> donated by SAP - was supposed to fix ? A heap of that size for a still
>> moderate number of 300 users is crazy, so either there is stuff like
>> circular references that hog memory, or the design model is fundamentally
>> flawed. I don't understand why ESME needs "sessions" ? How can a scaleable
>> server be created if each user will allocate memory until some timeout. In a
>> world of stateless browser-based UIs that's not going to work.
> You're actually dead wrong about this.  "Stateless" is not... it's just
> pushing state and cache someplace else (the RDBMS, memcached, etc.).
> "Stateless" will lead to radical performance problems.  "Stateless" merely
> moves the caching decisions into code you don't control.  I dealt with this
> issue first-hand while helping a popular micro-blogging site migrate from a
> "stateless" to a Scala-based backend.  I'm dealing with this issue
> first-hand helping another popular site that's experiencing exponential
> growth migrate away from "push everything back to the RDBMS and hope for the
> best."
> My original design for ESME is stateful.  My original design for ESME is
> based on lessoned learned in this very space and was oriented to have things
> intelligently cached so that the caching is not based on RDBMS indexes.  I'm
> not sure what happened to cause the particular issues, but it seems like
> folks are loading messages from the RDBMS rather than asking the message
> cache for them.
>> Time for me to look at that code ...
>> -Michael
>> ----- Original Message -----
>> From: Markus Kohler <>
>> To: <>
>> Sent: Wed Nov 25 12:14:58 2009
>> Subject: Further analysis of the GC issue
>> Hi all,
>> the Garbage Collector issue I was talking about is reproducible.
>> I've uploaded an annotated GC graph to
>> I think the "LOGON" phase where I logon all the 300 users looks ok (given
>> that probably textile formatting is involved) but the phase where just one
>> user sends one message is certainly not looking good.
>> I took the profiler and the result is a bit shocking. For that one message,
>> 881.000.000 objects weighting  23,2 Gbyte where allocated (and reclaimed
>> afterwards). My former record was 2Gbyte ;-)
>> Fortunately I have a theory what happens, without looking into the
>> code,yet,
>> so take it with a grain of salt. It seems that the public time line for all
>> users is re-rendered, because 99% of the allocations come
>> from org.apache.esme.comet.PublicTimeline.render(). I guess all the actors
>> for all the users are sitting there, not knowing that the user has closed
>> the browser, because the user session has not yet expired.
>> I wonder how we get around this with a real "push" model. If the browser
>> would ask for updates this rendering could be done lazily. Or can we "ping"
>> the browser and check whether it responds?
>> On the other side. It should also not be necessary the re-render the
>> message
>> again and again because the result will be the same.
>> I will send Richard some attachments. Not sure whether you will need them,
>> they look very similar to the ones we already have.
>> BTW, we should definitely check the use
>> of scala.xml.XML$.loadString(java.lang.String)
>> It's creating a new Parser each time, which is a bit costly because it
>> allocates a new Buffer each time and also hits the disk, when searching for
>> the name of the Java class.
>> Greetings,
>> Markus
>> "The best way to predict the future is to invent it" -- Alan Kay
> --
> Lift, the simply functional web framework
> Beginning Scala
> Follow me:
> Surf the harmonics

View raw message