Mailing-List: contact esme-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: esme-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of markus.kohler@gmail.com
 designates 209.85.211.188 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=sUI72VUcTbPLYNc7eE6sjseyMx5sEXB7/7iu7kjBAMdKWp/vQlWNJIe853s8M8L9Ua
         HpfN7KG6nXsVOhBY1nb/BI9qHSC4/jsGyiRxN7UF8ua5kPT8pT/iSs1qi9ktIuZYSntH
         7DG49TGVDbyKmPw4ABvdInnZFWha3vJUk1Zzk=
MIME-Version: 1.0
In-Reply-To: <cdbebedf0911250935n1d4f6414ia87ef4610cdf14f2@mail.gmail.com>
References: <771905290911250914q5ee704a4r1cd2c723f25b0dc7@mail.gmail.com>
	 <cdbebedf0911250935n1d4f6414ia87ef4610cdf14f2@mail.gmail.com>
Date: Wed, 25 Nov 2009 19:10:43 +0100
Message-ID: <771905290911251010r73f05962t3c55bf3afe522a69@mail.gmail.com>
Subject: Re: Further analysis of the GC issue
From: Markus Kohler <markus.kohler@gmail.com>
To: esme-dev@incubator.apache.org
Content-Type: multipart/alternative; boundary=001636283f3827caa5047935fa12

--001636283f3827caa5047935fa12
Content-Type: text/plain; charset=ISO-8859-1

BTW, you don't need 10 user , 2 or 3 would be enough.
You just have to take care that the session does not expire.
The issue disappears if you wait long enough, I verified that already.


Regards,
Markus

"The best way to predict the future is to invent it" -- Alan Kay


On Wed, Nov 25, 2009 at 6:35 PM, David Pollak <feeder.of.the.bears@gmail.com
> wrote:

> Markus,
>
> I'll have to look into how the code is currently implemented.  The original
> design called for a message to hang out for quite a while (not be reloaded
> from RDBMS for each user) and lazily render different pieces of itself.
>  Put
> another way, for each message, no matter where it's viewed in the system,
> there should be a single, cached instance of the message that contains the
> rendered Textile markup, etc.
>
> Put another way, a message that appears in 300 timelines should be the same
> message instance and should have its complex values calculated once.  Sure,
> if you go back and look at a message that's a week old, it'll have fallen
> out of memory and need to be re-rendered, but in the normal case, a message
> that's viewed by 300 people as part of their timeline or the public
> timeline
> should only be materialized/rendered once.
>
> Can you point me to a place (maybe even a VM Ware instance) where I can
> reproduce your tests?  I'd love to cycle on making 23GB -> 100MB (1/300th
> the size).
>
> Thanks,
>
> David
>
> On Wed, Nov 25, 2009 at 9:14 AM, Markus Kohler <markus.kohler@gmail.com
> >wrote:
>
> > Hi all,
> > the Garbage Collector issue I was talking about is reproducible.
> > I've uploaded an annotated GC graph to
> >
> >
> http://picasaweb.google.com/lh/photo/wB-RRtb0wIVfpxJkTJPNuw?authkey=Gv1sRgCOve7LThpfvXsQE&feat=directlink
> >
> > I think the "LOGON" phase where I logon all the 300 users looks ok (given
> > that probably textile formatting is involved) but the phase where just
> one
> > user sends one message is certainly not looking good.
> >
> > I took the profiler and the result is a bit shocking. For that one
> message,
> > 881.000.000 objects weighting  23,2 Gbyte where allocated (and reclaimed
> > afterwards). My former record was 2Gbyte ;-)
> >
> > Fortunately I have a theory what happens, without looking into the
> > code,yet,
> > so take it with a grain of salt. It seems that the public time line for
> all
> > users is re-rendered, because 99% of the allocations come
> > from org.apache.esme.comet.PublicTimeline.render(). I guess all the
> actors
> > for all the users are sitting there, not knowing that the user has closed
> > the browser, because the user session has not yet expired.
> >
> > I wonder how we get around this with a real "push" model. If the browser
> > would ask for updates this rendering could be done lazily. Or can we
> "ping"
> > the browser and check whether it responds?
> > On the other side. It should also not be necessary the re-render the
> > message
> > again and again because the result will be the same.
> >
> > I will send Richard some attachments. Not sure whether you will need
> them,
> > they look very similar to the ones we already have.
> >
> > BTW, we should definitely check the use
> > of scala.xml.XML$.loadString(java.lang.String)
> > It's creating a new Parser each time, which is a bit costly because it
> > allocates a new Buffer each time and also hits the disk, when searching
> for
> > the name of the Java class.
> >
> > Greetings,
> > Markus
> >
> >
> >
> > "The best way to predict the future is to invent it" -- Alan Kay
> >
>
>
>
> --
> Lift, the simply functional web framework http://liftweb.net
> Beginning Scala http://www.apress.com/book/view/1430219890
> Follow me: http://twitter.com/dpp
> Surf the harmonics
>

--001636283f3827caa5047935fa12--