incubator-olio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harold Lim <rold...@yahoo.com>
Subject Re: Question about Faban/workload driver + multiple Glassfish servers
Date Tue, 24 Nov 2009 20:37:23 GMT
Hi Shanti,

I looked and tried nginx. The imbalance is even worse. To have sticky session (for logging
in and stuff), we need to use the ip_hash directive of nginx's load balancer.

"This directive causes requests to be distributed between upstreams based on the IP-address
of the client.
The key for the hash is the class-C network address of the client. This method guarantees
that the client request will always be transferred to the same server."

Since I only have 1 workload driver (hence the http requests are always from the same IP address),
nginx always uses the same glassfish server. 



Apache mod_proxy, on the other hand, uses JSESSIONID/cookies for its sticky session. What
I don't understand is even if I explicitly add calls to doLogout() in OlioDriver, there is
still a big skew in the load distribution of glassfish servers. When I run it I have 1 server
running with 70+% cpu utilization and 2 other servers with 40+% utilization.

Simply specifying the 3 glassfish servers in OlioDriver, I get around 55-60% for each of my
3 glassfish servers.

I am curious as to see whether the HttpTransport creates a new http connection or it reuses
the same connection. I also read the URLConnection reuses the socket connections. "HTTPUrlConnection
(not directly, but in the underlying classes) keeps a small pool of open keep-alive sockets"


If HttpTransport does properly closes the connections/creates new instances, I wonder if that
might explain why I'm still getting a skew when using apache mod_proxy.

The thing is when I manually try accessing olio (through apache) on my browser, I do get sticky
session. However, when I delete my cookies and try to access olio again, I am redirected to
a different back-end server. I assume that calling HttpTransport's newInstance should clear
all cookies.



Thanks,
Harold


--- On Tue, 11/24/09, Shanti Subramanyam <shanti.subramanyam@gmail.com> wrote:

> From: Shanti Subramanyam <shanti.subramanyam@gmail.com>
> Subject: Re: Question about Faban/workload driver + multiple Glassfish servers
> To: olio-user@incubator.apache.org
> Cc: "Shanti Subramanyam" <Shanti.Subramanyam@Sun.COM>
> Date: Tuesday, November 24, 2009, 1:14 PM
> Harold - you may also want to try
> nginx. mod_proxy performance is supposed to be pretty bad.
> The rails folk typically use nginx for proxying.
> 
> Shanti
> 
> Akara Sucharitakul wrote:
> > Unfortunately not. If there is any accounting
> information available through mod_proxy, this would be most
> helpful. We have to determine at this level whether requests
> are distributed evenly. If not, why? And can we do anything
> about it, short of fixing mod_proxy. All this information is
> missing. Thanks.
> > 
> > -Akara
> > 
> > Shanti Subramanyam wrote:
> >> I assume mod_proxy will do a round-robin
> scheduling of the initial request. Even if subsequent
> requests in the same session are tied to the same appserver,
> I would think that you should see an even distribution as
> each session (on avg) should be placing the same load.
> >> Let's see if Akara has better insights.
> >> 
> >> Shanti
> >> 
> >> Harold Lim wrote:
> >>> Hi Akara,
> >>> 
> >>> Currently, what I have is an apache front-end
> using mod_proxy as a load balancer for multiple glassfish
> servers. In Faban, I just enter the address of the apache
> server. It works fine but I have to turn on sticky session.
> Without sticky session, it's not allowing the pages to
> log-in (i think because it's returning with different
> glassfish server each time).
> >>> 
> >>> The problem I have now is that when I run my
> benchmark run, the load is not balanced. It is skewed
> towards one of the glassfish server (i.e, 1 glassfish server
> has 70+% utilization, the other 2 only has 40+%).
> >>> 
> >>> I was looking at the UIDriver of Olio and saw
> that the httpconnection instance is being reused. It only
> gets "a new fresh state" in the doLogout. But, doLogout is
> only being called by doLogin() and in doAddPerson().
> >>> 
> >>> What is the best way to address this problem?
> Do I call doLogout() at the end of each benchmark operation
> function (i.e, in doTagSearch(), etc)? I feel that if I do
> that, it breaks the realism of the simulation/benchmark
> run.
> >>> 
> >>> In Faban, how does it determine or what
> constitutes a "session". I would think a session is
> something like a user going to a website, browsing a couple
> of the pages within that site and then leaving. Ideally,
> what we want is having something like that, with the leaving
> part corresponding to calling the doLogout method.
> >>> 
> >>> 
> >>> 
> >>> 
> >>> 
> >>> Thanks,
> >>> -Harold
> >>> 
> >>> 
> >>> 
> >>> --- On Fri, 11/6/09, Akara Sucharitakul <Akara.Sucharitakul@Sun.COM>
> wrote:
> >>> 
> >>>  
> >>>> From: Akara Sucharitakul <Akara.Sucharitakul@Sun.COM>
> >>>> Subject: Re: Question about Faban/workload
> driver + multiple Glassfish servers
> >>>> To: "Harold Lim" <rold_50@yahoo.com>
> >>>> Date: Friday, November 6, 2009, 4:13 PM
> >>>> OK, with that version it is probably
> >>>> harder to achieve what you're trying to do
> (go through switches). Check out the new
> >>>> benchmark class.
> >>>> 
> >>>> -Akara
> >>>> 
> >>>> Harold Lim wrote:
> >>>>   
> >>>>> Hi Akara,
> >>>>> 
> >>>>> Yes. I built my olio package from the
> Java source (a
> >>>>>       
> >>>> few months back).
> >>>>   
> >>>>> -Harold
> >>>>> --- On Fri, 11/6/09, Akara
> Sucharitakul <Akara.Sucharitakul@Sun.COM>
> >>>>>       
> >>>> wrote:
> >>>>   
> >>>>>> From: Akara Sucharitakul <Akara.Sucharitakul@Sun.COM>
> >>>>>> Subject: Re: Question about
> Faban/workload driver
> >>>>>>     
>    
> >>>> + multiple Glassfish servers
> >>>>   
> >>>>>> To: "Harold Lim" <rold_50@yahoo.com>
> >>>>>> Date: Friday, November 6, 2009,
> 3:11 PM
> >>>>>> Sorry, I meant the Olio kit. It
> is
> >>>>>> the Olio benchmarking process that
> causes this issue.
> >>>>>> 
> >>>>>> -Akara

> >>>>>> 
> >>>>>> Harold Lim wrote:
> >>>>>>       
> >>>>>>> Hi Akara,
> >>>>>>> 
> >>>>>>> I'm using the faban binary you
> gave me last
> >>>>>>>       
>    
> >>>> September
> >>>>   
> >>>>>> 19 with the load variation fix.
> >>>>>>       
> >>>>>>> -Harold
> >>>>>>> 
> >>>>>>> --- On Fri, 11/6/09, Akara
> Sucharitakul <Akara.Sucharitakul@Sun.COM>
> >>>>>>>       
>    
> >>>>>> wrote:
> >>>>>>       
> >>>>>>>> From: Akara Sucharitakul
> <Akara.Sucharitakul@Sun.COM>
> >>>>>>>> Subject: Re: Question
> about Faban/workload
> >>>>>>>>       
>      
> >>>> driver
> >>>>   
> >>>>>> + multiple Glassfish servers
> >>>>>>       
> >>>>>>>> To: olio-user@incubator.apache.org
> >>>>>>>> Date: Friday, November 6,
> 2009, 2:47 PM
> >>>>>>>> Hmmm, based on previous
> questions I
> >>>>>>>> thought you were on the
> latest and pretty much in-sync with the repository.
> >>>>>>>>       
>      
> >>>>>> Apparently not.   
>    
> >>>>>>>> OlioBenchmark.configure no
> longer makes a
> >>>>>>>>       
>      
> >>>> call to
> >>>>   
> >>>>>> getFile.
> >>>>>>       
> >>>>>>>> This has been eliminated
> in favor of the Faban
> >>>>>>>>       
>      
> >>>> services/tools
> >>>>   
> >>>>>>>> infrastructure we had in
> place lately.
> >>>>>>>> 
> >>>>>>>> Did you build the kit
> yourself? Or is this
> >>>>>>>>       
>      
> >>>> based
> >>>>   
> >>>>>> on a
> >>>>>>       
> >>>>>>>> released kit. Thanks.
> >>>>>>>> 
> >>>>>>>> -Akara
> >>>>>>>> 
> >>>>>>>> Harold Lim wrote:
> >>>>>>>>       
>    
> >>>>>>>>> Hi Akara,
> >>>>>>>>> 
> >>>>>>>>> Here is the stack
> trace:
> >>>>>>>>> 
> >>>>>>>>> Logger:
> >>>>>>>>>     
>          
> >>>>>>
> com.sun.faban.harness.engine.GenericBenchmark
> >>>>>>       
> >>>>>>>>> Thread 
>    Class         
>      
> >>>>>>         
> 
> >>>>>>>> Method
> >>>>>>>>       
>    
> >>>>>>>>> 11     
>             
> >>>>
> com.sun.faban.harness.engine.GenericBenchmark
> >>>>   
> >>>>>>>>   
>    start
> >>>>>>>>       
>    
> >>>>>>>>> Exception:
> >>>>>>>>> Message:
> >>>>>>>>> 
> >>>>>>>>>     
>          
> >>>>>>
> com.sun.faban.harness.FabanHostUnknownException:
> >>>>>>     
>    
> >>>> Host
> >>>>   
> >>>>>>>> xxx.xxx.xxx.xxx not
> found!
> >>>>>>>>       
>    
> >>>>>>>>> Stack Trace:
> >>>>>>>>> Class 
>    Method         
>      
> >>>>>>         
> 
> >>>>>>>> Line
> >>>>>>>>       
>      
> >>>> com.sun.faban.harness.engine.CmdService
> >>>>   
> >>>>>>>>   
>    get         
>    
> >>>>>>     1276
> >>>>>>     
>    
> >>>> com.sun.faban.harness.RunContext 
>    
> >>>>>>>> getFile 
>    346
> >>>>>>>>       
>      
> >>>>>>
> org.apache.olio.workload.harness.OlioBenchmark
> >>>>>>       
> >>>>>>>>   
>    configure       
>      
> >>>>>>     107
> >>>>>> 
> >>>>>>     
>    
> >>>>
> com.sun.faban.harness.engine.InterfaceBenchmarkWrapper
> >>>>   
> >>>>>>>>   
>    configure       
>      
> >>>>>>     56
> >>>>>>     
>    
> >>>>
> com.sun.faban.harness.engine.GenericBenchmark
> >>>>   
> >>>>>>>>   
>    start         
>    
> >>>>>>     264
> >>>>>>     
>    
> >>>> com.sun.faban.harness.engine.RunDaemon
> >>>>   
> >>>>>>>>   
>    run         
>    
> >>>>>>     338
> >>>>>>       
> >>>>>>>>> java.lang.Thread 
>              
> >>>>    run
> >>>>   
> >>>>>>>>   
>    619
> >>>>>>>>       
>    
> >>>>>>>>> Thanks!
> >>>>>>>>> -Harold
> >>>>>>>>> 
> >>>>>>>>> --- On Thu, 11/5/09,
> Akara
> >>>>>>>>>     
>          
> >>>> Sucharitakul <Akara.Sucharitakul@Sun.COM>
> >>>>   
> >>>>>>>> wrote:
> >>>>>>>>       
>    
> >>>>>>>>>> From: Akara
> Sucharitakul <Akara.Sucharitakul@Sun.COM>
> >>>>>>>>>> Subject: Re:
> Question about
> >>>>>>>>>>   
>              
> >>>> Faban/workload
> >>>>   
> >>>>>> driver
> >>>>>>       
> >>>>>>>> + multiple Glassfish
> servers
> >>>>>>>>       
>    
> >>>>>>>>>> To: olio-user@incubator.apache.org
> >>>>>>>>>> Date: Thursday,
> November 5, 2009,
> >>>>>>>>>>   
>              
> >>>> 4:37 PM
> >>>>   
> >>>>>>>>>> Can you get me the
> stack trace?
> >>>>>>>>>> Thanks.
> >>>>>>>>>> 
> >>>>>>>>>> -Akara
> >>>>>>>>>> 
> >>>>>>>>>> Harold Lim wrote:
> >>>>>>>>>>   
>            
> >>>>>>>>>>> Hi Akara,
> >>>>>>>>>>> 
> >>>>>>>>>>> When I set
> hostConfig/enabled
> >>>>>>>>>>>   
>            
>    
> >>>> to
> >>>>   
> >>>>>> false, I am
> >>>>>>       
> >>>>>>>> getting
> >>>>>>>>       
>    
> >>>>>>>>>>
> FabanUnknownHostException pointing
> >>>>>>>>>>   
>              
> >>>> to the
> >>>>   
> >>>>>> address
> >>>>>>       
> >>>>>>>> of the
> >>>>>>>>       
>    
> >>>>>>>>>> switch. But If I
> set it to true,
> >>>>>>>>>>   
>              
> >>>> it fails
> >>>>   
> >>>>>> because
> >>>>>>       
> >>>>>>>> it is not
> >>>>>>>>       
>    
> >>>>>>>>>> able to connect to
> the agent
> >>>>>>>>>>   
>              
> >>>> (which I
> >>>>   
> >>>>>> don't have)
> >>>>>>       
> >>>>>>>> in the
> >>>>>>>>       
>    
> >>>>>>>>>> machine that has
> the switch.
> >>>>>>>>>>   
>            
> >>>>>>>>>>> -Harold
> >>>>>>>>>>> 
> >>>>>>>>>>> --- On Fri,
> 10/30/09, Akara
> >>>>>>>>>>>   
>            
>    
> >>>>>> Sucharitakul
> >>>>>>       
> >>>>>>>> <Akara.Sucharitakul@Sun.COM>
> >>>>>>>>       
>    
> >>>>>>>>>> wrote:
> >>>>>>>>>>   
>            
> >>>>>>>>>>>> From:
> Akara Sucharitakul
> >>>>>>>>>>>> 
>                
>    
> >>>> <Akara.Sucharitakul@Sun.COM>
> >>>>   
> >>>>>>>>>>>> Subject:
> Re: Question
> >>>>>>>>>>>> 
>                
>    
> >>>> about
> >>>>   
> >>>>>> Faban/workload
> >>>>>>       
> >>>>>>>> driver
> >>>>>>>>       
>    
> >>>>>>>>>> + multiple
> Glassfish servers
> >>>>>>>>>>   
>            
> >>>>>>>>>>>> To: olio-user@incubator.apache.org
> >>>>>>>>>>>> Date:
> Friday, October 30,
> >>>>>>>>>>>> 
>                
>    
> >>>> 2009,
> >>>>   
> >>>>>> 7:56 PM
> >>>>>>       
> >>>>>>>>>>>> Lets say
> it has not been
> >>>>>>>>>>>> 
>                
>    
> >>>> tested
> >>>>   
> >>>>>> this
> >>>>>>       
> >>>>>>>>>>>> way. It
> certainly cannot
> >>>>>>>>>>>> 
>                
>    
> >>>>>> dynamically add
> >>>>>>       
> >>>>>>>> servers
> >>>>>>>>       
>    
> >>>>>>>>>> into the
> >>>>>>>>>>   
>            
> >>>>>>>>>>>> driver.
> But you can use a
> >>>>>>>>>>>> 
>                
>    
> >>>> switch
> >>>>   
> >>>>>> that
> >>>>>>       
> >>>>>>>> allows
> >>>>>>>>       
>    
> >>>>>>>>>> dynamic
> >>>>>>>>>>   
>            
> >>>>>>>>>>>> addition
> of the servers.
> >>>>>>>>>>>> 
>                
>    
> >>>> If you
> >>>>   
> >>>>>> want
> >>>>>>       
> >>>>>>>> stats
> >>>>>>>>       
>    
> >>>>>>>>>> collection for
> >>>>>>>>>>   
>            
> >>>>>>>>>>>> all the
> servers behind
> >>>>>>>>>>>> 
>                
>    
> >>>> that
> >>>>   
> >>>>>> switch, we may
> >>>>>>       
> >>>>>>>> need to
> >>>>>>>>       
>    
> >>>>>>>>>> add an
> >>>>>>>>>>   
>            
> >>>>>>>>>>>> explicit
> URL field in the
> >>>>>>>>>>>> 
>                
>    
> >>>>>> configuration
> >>>>>>       
> >>>>>>>> (which
> >>>>>>>>       
>    
> >>>>>>>>>> will default
> >>>>>>>>>>   
>            
> >>>>>>>>>>>> to the web
> servers if left
> >>>>>>>>>>>> 
>                
>    
> >>>> blank.
> >>>>   
> >>>>>> If
> >>>>>>       
> >>>>>>>> that's not
> >>>>>>>>       
>    
> >>>>>>>>>> needed, just
> >>>>>>>>>>   
>            
> >>>>>>>>>>>> specify
> the single server
> >>>>>>>>>>>> 
>                
>    
> >>>>>> representing the
> >>>>>>       
> >>>>>>>> switch
> >>>>>>>>       
>    
> >>>>>>>>>> and set
> >>>>>>>>>>   
>            
> >>>>>>>>>>>>
> hostConfig/enabled to
> >>>>>>>>>>>> 
>                
>    
> >>>> false to
> >>>>   
> >>>>>> prevent
> >>>>>>       
> >>>>>>>> Faban
> >>>>>>>>       
>    
> >>>>>>>>>> starting an
> >>>>>>>>>>   
>            
> >>>>>>>>>>>> agent
> there. Thanks.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> -Akara
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Harold Lim
> wrote:
> >>>>>>>>>>>> 
>              
>    
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Does
> the workload
> >>>>>>>>>>>>> 
>                
>      
> >>>> driver
> >>>>   
> >>>>>> supports
> >>>>>>       
> >>>>>>>> dynamically
> >>>>>>>>       
>    
> >>>>>>>>>> adding a
> >>>>>>>>>>   
>            
> >>>>>>>>>>>> glassfish
> server, while a
> >>>>>>>>>>>> 
>                
>    
> >>>>>> benchmark test
> >>>>>>       
> >>>>>>>> is
> >>>>>>>>       
>    
> >>>>>>>>>> running?
> >>>>>>>>>>   
>            
> >>>>>>>>>>>>>
> Currently, I specify
> >>>>>>>>>>>>> 
>                
>      
> >>>> all the
> >>>>   
> >>>>>> glassfish
> >>>>>>       
> >>>>>>>> servers
> >>>>>>>>       
>    
> >>>>>>>>>> in the
> >>>>>>>>>>   
>            
> >>>>>>>>>>>> host:pair
> field of the web
> >>>>>>>>>>>> 
>                
>    
> >>>> server
> >>>>   
> >>>>>> tab.
> >>>>>>       
> >>>>>>>>>>>>> If
> not, is there any
> >>>>>>>>>>>>> 
>                
>      
> >>>> way
> >>>>   
> >>>>>> around it? Do
> >>>>>>       
> >>>>>>>> you
> >>>>>>>>       
>    
> >>>>>>>>>> have any
> >>>>>>>>>>   
>            
> >>>>>>>>>>>> ideas of
> how to deal with
> >>>>>>>>>>>> 
>                
>    
> >>>> adding
> >>>>   
> >>>>>> or
> >>>>>>       
> >>>>>>>> removing
> >>>>>>>>       
>    
> >>>>>>>>>> glassfish
> >>>>>>>>>>   
>            
> >>>>>>>>>>>> servers
> dynamically?
> >>>>>>>>>>>> 
>              
>    
> >>>>>>>>>>>>>
> -Harold
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> 
>                
>         
> >>>>      
> >>>>>>>>       
>           
> >>>>>>> 
> >>>>>>>       
>    
> >>>>> 
> >>>>>       
> >>>>     
> >>> 
> >>> 
> >>> 
> >>>   
> > 
> 
>




      

Mime
View raw message