Mailing-List: contact esme-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: esme-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of hirsch.dick@gmail.com
 designates 209.85.220.216 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=XiJb4e2orDjEv+DhJyqxk5OfG03cSTcfDsUedX99ak4BT9v/aFZJcc9JvM17W33kAG
         tU/AbJdsjvozXdkzCrfvGK0Ga9KzlTOeUYkU8lg1AULZYtCYt0FgLdAnreORRyiJZLBz
         R1aJ5kRe8w3VPHn3oHZyz04mfZbexOrXleyKA=
MIME-Version: 1.0
In-Reply-To: <771905290912071603y73cd4bbevd2ec7b2fd7284e54@mail.gmail.com>
References: <771905290912071603y73cd4bbevd2ec7b2fd7284e54@mail.gmail.com>
Date: Tue, 8 Dec 2009 11:30:38 +0100
Message-ID: <fa2d9f450912080230x57a94f61n2aeb0c6ede96f26d@mail.gmail.com>
Subject: Re: Performance update: Message size in memory
From: Richard Hirsch <hirsch.dick@gmail.com>
To: esme-dev@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

org.tartarus.snowball.ext.PorterStemmer is from the compass search.
Maybe we can configure it, so that it is not retained after usage.

D.

On Tue, Dec 8, 2009 at 1:03 AM, Markus Kohler <markus.kohler@gmail.com> wro=
te:
> Hi all,
> I've been busy otherwise, and therefore didn't find much time for ESME la=
st
> week.
> I tried a few things with regards to performance.
> As you all noticed the performance on the performance instance is current=
ly
> excellent.
> I tried various approaches to measure it, but most failed due to the come=
nt
> requests, which the tools I usually use don't like.
> The best I could get are some numbers from the Firebug Firefox plugin. =
=A0It
> seems that the response time for entering a message until it appears in t=
he
> users timeline is around 350ms, which is really excellent. It will be eve=
n
> harder to measure (using the browser) how long it takes for a message fro=
m
> one user to the user. I'm not sure how to do that yet. I tested manually
> sending messages from chrome to firefox and it' s really fast.
>
> I also let one of the 300+x Users send 1000 messages and did some heap
> dumps.
> I'm not yet fully through it but it's already clear that messages take up
> too much space.
> Around 1400 messages would need =A09,3 Million bytes which means that in
> average one messages needs 6Kbyte!
> Ok there were probably also a lot of relatively long update status messag=
es,
> but still I think this is too much.
>
> The reason seems to be that The messages still retain an instance to the
> Stemmer (org.tartarus.snowball.ext.PorterStemmer) which alone takes 2 Kby=
te.
> Do we really need this Stemmer after we ran it?
>
> Another reason is that scala.xml.Elem is referenced in the toXML field. I
> guess this is the result of parsing XML. Not sure whether this is still
> needed after it's done, but storing DOM like structures is for sure not
> memory efficient. originialXML looks similiar.
>
> It would be important to get these numbers down, otherwise we will be kil=
led
> by memory usage as soon as we get a lot of messages send.
>
> I also asked on the Scala list about the loadXML function accessing the
> filesystem, but someone claimed this would not be the case in trunk and
> asked for the version. So maybe they can backport =A0a fix for this.
> I seem to remember during some profiling that this function is still used=
.
>
> Haven't had any time to draft a blog, but I hope I can start with that on
> Wednesday or Thursday.
>
> Regards,
> Markus
>
>
> "The best way to predict the future is to invent it" -- Alan Kay
>