esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Pollak <>
Subject Re: Further analysis of the GC issue
Date Thu, 26 Nov 2009 04:00:20 GMT
On Wed, Nov 25, 2009 at 7:16 PM, Bechauf, Michael

> Wasn't this exactly the kind of stuff that the Eclipse Memory Analyzer -
> donated by SAP - was supposed to fix ? A heap of that size for a still
> moderate number of 300 users is crazy, so either there is stuff like
> circular references that hog memory, or the design model is fundamentally
> flawed. I don't understand why ESME needs "sessions" ? How can a scaleable
> server be created if each user will allocate memory until some timeout. In a
> world of stateless browser-based UIs that's not going to work.

You're actually dead wrong about this.  "Stateless" is not... it's just
pushing state and cache someplace else (the RDBMS, memcached, etc.).
"Stateless" will lead to radical performance problems.  "Stateless" merely
moves the caching decisions into code you don't control.  I dealt with this
issue first-hand while helping a popular micro-blogging site migrate from a
"stateless" to a Scala-based backend.  I'm dealing with this issue
first-hand helping another popular site that's experiencing exponential
growth migrate away from "push everything back to the RDBMS and hope for the

My original design for ESME is stateful.  My original design for ESME is
based on lessoned learned in this very space and was oriented to have things
intelligently cached so that the caching is not based on RDBMS indexes.  I'm
not sure what happened to cause the particular issues, but it seems like
folks are loading messages from the RDBMS rather than asking the message
cache for them.

> Time for me to look at that code ...
> -Michael
> ----- Original Message -----
> From: Markus Kohler <>
> To: <>
> Sent: Wed Nov 25 12:14:58 2009
> Subject: Further analysis of the GC issue
> Hi all,
> the Garbage Collector issue I was talking about is reproducible.
> I've uploaded an annotated GC graph to
> I think the "LOGON" phase where I logon all the 300 users looks ok (given
> that probably textile formatting is involved) but the phase where just one
> user sends one message is certainly not looking good.
> I took the profiler and the result is a bit shocking. For that one message,
> 881.000.000 objects weighting  23,2 Gbyte where allocated (and reclaimed
> afterwards). My former record was 2Gbyte ;-)
> Fortunately I have a theory what happens, without looking into the
> code,yet,
> so take it with a grain of salt. It seems that the public time line for all
> users is re-rendered, because 99% of the allocations come
> from org.apache.esme.comet.PublicTimeline.render(). I guess all the actors
> for all the users are sitting there, not knowing that the user has closed
> the browser, because the user session has not yet expired.
> I wonder how we get around this with a real "push" model. If the browser
> would ask for updates this rendering could be done lazily. Or can we "ping"
> the browser and check whether it responds?
> On the other side. It should also not be necessary the re-render the
> message
> again and again because the result will be the same.
> I will send Richard some attachments. Not sure whether you will need them,
> they look very similar to the ones we already have.
> BTW, we should definitely check the use
> of scala.xml.XML$.loadString(java.lang.String)
> It's creating a new Parser each time, which is a bit costly because it
> allocates a new Buffer each time and also hits the disk, when searching for
> the name of the Java class.
> Greetings,
> Markus
> "The best way to predict the future is to invent it" -- Alan Kay

Lift, the simply functional web framework
Beginning Scala
Follow me:
Surf the harmonics

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message