Mailing-List: contact dev-help@geronimo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@geronimo.apache.org
Received-SPF: neutral (asf.osuosl.org: local policy)
Message-ID: <44369237.8070906@apache.org>
Date: Fri, 07 Apr 2006 10:24:23 -0600
From: Jeff Genender <jgenender@apache.org>
Reply-To: jgenender@apache.org
Organization: Apache Geronimo
User-Agent: Thunderbird 1.5 (Macintosh/20051201)
MIME-Version: 1.0
To: dev@geronimo.apache.org
Subject: Re: Geronimo peak performance
References: <eac5ea1a0604050619i2778bb93q4bb5d4f013e8291a@mail.gmail.com>
	 <44340026.6080600@hogstrom.org>
 <eac5ea1a0604060633v2732a729n5eac66d86eeb0725@mail.gmail.com>
 <443689DB.7010005@hogstrom.org>
In-Reply-To: <443689DB.7010005@hogstrom.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

I will have some numbers soon too...

We are running on some Sun T2000 4 core machines.  The numbers should be
interesting on these boxes ;-)

Jeff

Matt Hogstrom wrote:
> I'm going to start externalizing the data as it evolves and Adopt
> Aaron's progresive disclosure method of publishing.
> 
> As a first attempt to get us n the same page here is a drawing of my
> configuration:
> 
> http://people.apache.org/~hogstrom/performance/PerformanceTestBed.pdf
> 
> There is a png as well if that's better for you but it takes longer to
> download.  What does your test bed look like?
> 
> Also, in my testing I'm not receiving any errors (non reported any
> way).  I think we need to be careful with results that are failing.  I'm
> assuming your running on a 4-way.  I'd like to work on resolving the
> problems your having so we'll be comparing similar data.
> 
> Matt
> 
> Maxim Berkultsev wrote:
>> Matt,
>>
>> first of all sorry for confusion - all the throughput values should be
>> divided by 60:).
>>
>> I've rerun both tests.
>>
>> The parameters for the run were the ones you've mentioned - 100 users
>> with 10 ms delay for each request from a user. Also for the second
>> scenario I've changed configuration to Direct Mode with 5000 Users and
>> 10000 quotes.
>>
>> Here is the data I've got:
>>
>> ----------------------------------
>>
>> Example 1: PingServlet2SessionEJB
>> Your data: 100 168 582
>> Mine: 100 472 145
>>
>> About 6600 samples (66 requests per thread) were used - about 10
>> percents of responses had "Non HTTP response code".
>>
>> The resulting throughput was calculated with 8754/60=145.
>>
>> -----------------------------------
>>
>> Example 2: PingServlet2TwoPhase
>> Your data: 100 2096 46
>> Mine: 100 3324 29
>>
>> 4000 samples were provided.
>>
>> But only ~10 percents of queries return 200OK, the rest - 500 Error.
>>
>> Here is a typical error log I was receiving from the server:
>>
>> *******************************
>>
>> java.lang.NullPointerException
>>         at
>> org.apache.jsp.error_jsp._jspService(Ljavax.servlet.http.HttpServletR
>> equest;Ljavax.servlet.http.HttpServletResponse;)V(org.apache.jsp.error_jsp:96)
>>
>>         at
>> org.apache.jasper.runtime.HttpJspBase.service(Ljavax.servlet.http.Htt
>> pServletRequest;Ljavax.servlet.http.HttpServletResponse;)V(HttpJspBase.java:97)
>>
>>         at
>> javax.servlet.http.HttpServlet.service(Ljavax.servlet.ServletRequest;
>> Ljavax.servlet.ServletResponse;)V(Optimized Method)
>>         at
>> org.apache.jasper.servlet.JspServletWrapper.service(Ljavax.servlet.ht
>> tp.HttpServletRequest;Ljavax.servlet.http.HttpServletResponse;Z)V(JspServletWrap
>>
>> per.java:322)
>>         at
>> org.apache.jasper.servlet.JspServlet.serviceJspFile(Ljavax.servlet.ht
>> tp.HttpServletRequest;Ljavax.servlet.http.HttpServletResponse;Ljava.lang.String;
>>
>> Ljava.lang.Throwable;Z)V(JspServlet.java:314)
>>         at
>> org.apache.jasper.servlet.JspServlet.service(Ljavax.servlet.http.Http
>> ServletRequest;Ljavax.servlet.http.HttpServletResponse;)V(JspServlet.java:264)
>>
>>         at
>> javax.servlet.http.HttpServlet.service(Ljavax.servlet.ServletRequest;
>> Ljavax.servlet.ServletResponse;)V(Optimized Method)
>>         at
>> org.mortbay.jetty.servlet.ServletHolder.handle(Ljavax.servlet.Servlet
>> Request;Ljavax.servlet.ServletResponse;)V(ServletHolder.java:428)
>>         at
>> org.apache.geronimo.jetty.JettyServletHolder.handle(Ljavax.servlet.Se
>> rvletRequest;Ljavax.servlet.ServletResponse;)V(JettyServletHolder.java:99)
>>
>>         at
>> org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(
>> Ljavax.servlet.ServletRequest;Ljavax.servlet.ServletResponse;)V(WebApplicationHa
>>
>> ndler.java:830)
>>         at
>> org.mortbay.jetty.servlet.JSR154Filter.doFilter(Ljavax.servlet.Servle
>> tRequest;Ljavax.servlet.ServletResponse;Ljavax.servlet.FilterChain;)V(JSR154Filt
>>
>> er.java:170)
>>         at
>> org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(
>> Ljavax.servlet.ServletRequest;Ljavax.servlet.ServletResponse;)V(WebApplicationHa
>>
>> ndler.java:821)
>>         at
>> org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(Ljava.lang.S
>> tring;Ljavax.servlet.http.HttpServletRequest;Ljavax.servlet.http.HttpServletResp
>>
>> onse;Lorg.mortbay.jetty.servlet.ServletHolder;I)V(WebApplicationHandler.java:471
>>
>> )
>>         at
>> org.mortbay.jetty.servlet.Dispatcher.dispatch(Ljavax.servlet.ServletR
>> equest;Ljavax.servlet.ServletResponse;I)V(Dispatcher.java:283)
>>         at
>> org.mortbay.jetty.servlet.Dispatcher.error(Ljavax.servlet.ServletRequ
>> est;Ljavax.servlet.ServletResponse;)V(Dispatcher.java:179)
>>         at
>> org.mortbay.jetty.servlet.ServletHttpResponse.sendError(ILjava.lang.S
>> tring;)V(ServletHttpResponse.java:415)
>>         at
>> javax.servlet.http.HttpServletResponseWrapper.sendError(ILjava.lang.S
>> tring;)V(HttpServletResponseWrapper.java:107)
>>         at
>> org.apache.geronimo.samples.daytrader.web.prims.PingServlet2TwoPhase.
>> doGet(Ljavax.servlet.http.HttpServletRequest;Ljavax.servlet.http.HttpServletResp
>>
>> onse;)V(Optimized Method)
>>         at
>> javax.servlet.http.HttpServlet.service(Ljavax.servlet.http.HttpServle
>> tRequest;Ljavax.servlet.http.HttpServletResponse;)V(Optimized Method)
>>         at
>> javax.servlet.http.HttpServlet.service(Ljavax.servlet.ServletRequest;
>> Ljavax.servlet.ServletResponse;)V(Optimized Method)
>>         at
>> org.mortbay.jetty.servlet.ServletHolder.handle(Ljavax.servlet.Servlet
>> Request;Ljavax.servlet.ServletResponse;)V(ServletHolder.java:428)
>>         at
>> org.apache.geronimo.jetty.JettyServletHolder.handle(Ljavax.servlet.Se
>> rvletRequest;Ljavax.servlet.ServletResponse;)V(JettyServletHolder.java:99)
>>
>>         at
>> org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(
>> Ljavax.servlet.ServletRequest;Ljavax.servlet.ServletResponse;)V(WebApplicationHa
>>
>> ndler.java:830)
>>         at
>> org.mortbay.jetty.servlet.JSR154Filter.doFilter(Ljavax.servlet.Servle
>> tRequest;Ljavax.servlet.ServletResponse;Ljavax.servlet.FilterChain;)V(JSR154Filt
>>
>> er.java:170)
>>         at
>> org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(
>> Ljavax.servlet.ServletRequest;Ljavax.servlet.ServletResponse;)V(WebApplicationHa
>>
>> ndler.java:821)
>>         at
>> org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(Ljava.lang.S
>> tring;Ljavax.servlet.http.HttpServletRequest;Ljavax.servlet.http.HttpServletResp
>>
>> onse;Lorg.mortbay.jetty.servlet.ServletHolder;I)V(WebApplicationHandler.java:471
>>
>> )
>>         at
>> org.mortbay.jetty.servlet.ServletHandler.handle(Ljava.lang.String;Lja
>> va.lang.String;Lorg.mortbay.http.HttpRequest;Lorg.mortbay.http.HttpResponse;)V(O
>>
>> ptimized Method)
>>         at
>> org.mortbay.jetty.servlet.WebApplicationContext.handle(Ljava.lang.Str
>> ing;Ljava.lang.String;Lorg.mortbay.http.HttpRequest;Lorg.mortbay.http.HttpRespon
>>
>> se;)V(Optimized Method)
>>         at
>> org.mortbay.http.HttpContext.handle(Lorg.mortbay.http.HttpRequest;Lor
>> g.mortbay.http.HttpResponse;)V(Optimized Method)
>>         at
>> org.mortbay.http.HttpServer.service(Lorg.mortbay.http.HttpRequest;Lor
>> g.mortbay.http.HttpResponse;)Lorg.mortbay.http.HttpContext;(Optimized
>> Method)
>>         at
>> org.mortbay.http.HttpConnection.service(Lorg.mortbay.http.HttpRequest
>> ;Lorg.mortbay.http.HttpResponse;)Lorg.mortbay.http.HttpContext;(Optimized
>> Method
>> )
>>         at org.mortbay.http.HttpConnection.handleNext()Z(Optimized
>> Method)
>>
>> *******************************
>> -----------------------------------
>>
>> To run experiments I was using my desktop with default Geronimo
>> installation + jRockit 1.4.2_04.
>>
>> How one can avoid the problems with illegal response code when
>> multiple threads are in work? Have I omitted some issue in Daytrader
>> configuration before starting the experiment?
>>
>> Thank you for your help.
>>
>> -- 
>> Best regards,
>> Maxim Berkultsev, Intel Middleware Products Division
>>
>>
>>
>> 2006/4/5, Matt Hogstrom wrote:
>>
>>> Maxim,
>>>
>>> Thanks for sharing your results.  I have a whole set of numbers that
>>> I've been sitting on.  My tests
>>> don't use JMeter so if you would like to share your setup for that
>>> I'll incorporate it into the
>>> DayTrader tree.
>>>
>>> All the tests I've run were with a fixed number of users of 100 with
>>> 10ms think time.  My goal was
>>> to stress the server and see if it would stay up and how it would
>>> perform.  The system I'm testing
>>> on is an Intel 2 x 3.0Ghz Potomac System.  Each processor has 8MB L3
>>> Cache.  My tests were conducted
>>> with the Sun JDK (1.4.2_b09).  The Database system is on a separate
>>> box.  I'm using DB2 as Oracle
>>> has some clause in their license that does not allow publish of
>>> benchmark results without their
>>> express permission.
>>>
>>> See inline
>>>
>>> I'm rerunning some tests this afternoon as it looks like we're not
>>> comparing teh same things.  I am
>>> using an internal Load Generator and would like to move something
>>> Open Source so we can all compare
>>> the same numbers.
>>>
>>> Matt
>>>
>>> Maxim Berkultsev wrote:
>>>
>>>> Hi, all!
>>>>
>>>> Geronimo peak performance is under test and let me share some results.
>>>>
>>>> I was using JMeter and Daytrader web primitives to measure throughput
>>>> for a fixed number of simultaneously woking virtual users.
>>>>
>>>> However I've realized that the results for the equal numbers of users
>>>> in each scenario do not look valuable and tried to find some peak
>>>> values for throughput depending on the number of users. It looks as if
>>>> such peak values are reached when the number of users are minimal as
>>>> well as the peak exists at some 'optimal' number of users.
>>>>
>>>> I've used two scenarios.
>>>>
>>>> Example 1: For scenario PingServlet2SessionEJB from Daytrader web
>>>> primitives I've got max througput (~14670) for a minimal number of
>>>> users - 5 with average time per single request equals to 17. The table
>>>> below contains triples (number of users, average request time,
>>>> throughput) for different number of users.
>>>>
>>>> ----------------
>>>> 5 17 14670
>>>> ----------------
>>>> 10 40 13037
>>>> ----------------
>>>> 50 188 12646
>>>> ----------------
>>>> 100 447 11028
>>>
>>>  100 168 582   << I'm confused by this.  Actually all these numbers
>>> are way higher than I'm
>>> achieving.  Can you shed some light on your configuration?  Also, I
>>> assume your not getting 404's or
>>> something?
>>>
>>>> ----------------
>>>> 150 588 10770
>>>> ----------------
>>>> 200 634 10444
>>>> ----------------
>>>>
>>>> It looks as if the peak is reached when a number of users is minimal.
>>>>
>>>> Example 2: In scenario PingServlet2TwoPhase the throughput grows to
>>>> some saturation value and then begins to decrease. The maximal values
>>>> (~1300-1350) for throughput covers a wide interval between >10 and 150
>>>> virtual users. Here are the triples' table (number of users, average
>>>> request time, throughput):
>>>>
>>>> ----------------
>>>> 5, 390, 764
>>>> ----------------
>>>> 10, 492, 1207
>>>> ----------------
>>>> 50, 2250, 1314
>>>> ----------------
>>>> 100, 4380, 1356
>>>
>>>  100  2096    46  << Again...something is out of sorts.  Can you run
>>> Direct Mode with 5000 Users
>>> and 10000 quotes?
>>>
>>>> ----------------
>>>> 150, 6580, 1350
>>>> ----------------
>>>> 200, 9050, 1260
>>>> ----------------
>>>>
>>>> All values do not pretend to a significant:) accuracy but to some
>>>> general trend.
>>>>
>>>> Somehow usually there is some 'common sense' number of users to be
>>>> used in performance estimations. Can someone provide an idea how to
>>>> find this value for Geronimo?
>>>>
>>>> Thank you.
>>>>
>>>> -- 
>>>> Best regards,
>>>> Maxim Berkultsev, Intel Middleware Products Division
>>>>
>>>>
>>>>
>>>
>>
>>
>>