oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (388J)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: Question about xmlrpc
Date Thu, 16 Feb 2012 17:14:12 GMT
Hi Cecilia,

Oops, I didn't even realize you brought this on list.

Awesome, here we go then, happy to reply :) I just wanted
others in the community to benefit from the answers.

So, here we go. Some background folks: ACOS and OCO2
and NPP Sounder PEATE have been seeing some interesting issues
with the XML-RPC connection pool under heavy process, ingest, and 
resource manager load. This is due to an old version of the XML-RPC
library we are using (2.0.1) and its hard coded thread limitation of 100
threads. 

In certain projects, we've seen them address this by adding more file managers
or workflow managers, and running them on multiple ports and scaling out
horizontally that way. Cecilia's projects, however, decided to take the 
initiative and modify the XML-RPC library to up the thread limit to 2000.

I suggested that Cecilia, in response to her questions below if this is
a good number or not, bring this conversation on list, so that others could
benefit, so here we are.

OK, so my thought on this is the following:

1. file a JIRA issue to make XML-RPC thread limit in FM, WM and RM (and 
crawler and pushpull) take a config property, akin to the other XML-RPC
config properties, e.g., in file manager, etc. We could call it:

org.apache.oodt.cas.[filemgr|workflow|resmgr|crawler|pushpull].xmlrpc.maxThreads

2. We could pass this in via the *.properties files for the file manager, workflow manager,
resource manager, crawler and push pull as a system property the same way we 
have other configs for those servers

This way the property is configurable. A sensible default value is 256, as this will 
have a lot to do (as the value is raised) with sysadmins who have the ability to increase
the default ulimit. In addition, whether you are on Windows or Linux, the cost of creating
a thread compared to a process is significantly different, so high values of this may affect
the target deployment environment. We shouldn't set it in the upstream core Apache OODT
services to be much larger (by default) than 256, but then in downstream projects they can
configure it however they want.

3. the patch in #1 should simply include only the portions of the XML-RPC library that required
modification, and they should probably live in parallel to the src/main/java/org/apache/oodt

tree (perhaps in src/main/java/org/apache/xmlrpc).

Thoughts?

Cheers,
Chris

On Feb 15, 2012, at 10:42 PM, Cheng, Cecilia S (388K) wrote:

> 
> Hi Chris,
> 
> Sure we can discuss this in dev@oodt.apache.org.
> 
> If you feel comfortable w/ the 2000 number, of course I can push the patch
> upstream into Apache OODT. But what kind of tests, if any, should we do
> before we deliver the patch? Our projects are concerned that if we
> arbitrarily set a number, we don't know what other problems it might cause.
> 
> Thanks,
> Cecilia
> 
> On 2/15/12 10:07 PM, "Mattmann, Chris A (388J)"
> <chris.a.mattmann@jpl.nasa.gov> wrote:
> 
>> Hi Cecilia,
>> 
>> This is really good news!
>> 
>> A couple questions:
>> 
>> 1. Do you think you would be willing to push your XML-RPC patches upstream
>> into Apache OODT so others in the
>> community could benefit? This would involve filing corresponding JIRA issue(s)
>> [1], and then letting the dev@oodt.apache.org
>> know.
>> 
>> 2. Can we move this conversation onto dev@oodt.apache.org? I think others
>> could benefit from the answers below.
>> 
>> Thanks and let me know. If you'd like to discuss more, that's fine too, but
>> I'd urge us to move this onto the public Apache OODT
>> lists. 
>> 
>> Cheers,
>> Chris
>> 
>> [1] http://issues.apache.org/jira/browse/OODT
>> 
>> On Feb 15, 2012, at 2:31 PM, Cheng, Cecilia S (388K) wrote:
>> 
>>> Hi Chris and Paul,
>>> 
>>> Just want to fill you in on where we are w/ the xmlrpc problem that we see on
>>> ACOS and PEATE and get your advice.
>>> 
>>> As you might recall, on both projects, and in all 3 components (FM, RM, and
>>> WEngine), we will periodically see the following message in the console:
>>> 
>>> java.lang.RuntimeException: System overload: Maximum number of concurrent
>>> requests (100) exceeded
>>> 
>>> when the system is very busy. Since upgrading to the newer version of xmlrpc
>>> seems to be quite involved, we thought that we will just download the source
>>> code and change the hardcoded number of 100 to something bigger, recompile
>>> the jar file and use that in our system.
>>> 
>>> So I set the number to 2000 and have Lan, Michael and Irina try again. All 3
>>> of them said that it solved their problems, but now that this works, we have
>>> other concerns:
>>> 
>>> [1] Will setting this number so high (2000 vs. 100)  create other problems?
>>> [2] How can we find out what is a “good” number to use?
>>> [3] What are some ways I can monitor these concurrent requests as they run?
>>> netstat?
>>> 
>>> Would you please share your thought on this?
>>> 
>>> Thanks,
>>> Cecilia
>>> 
>> 
>> 
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW:   http://sunset.usc.edu/~mattmann/
>> Phone: +1 (818) 354-8810
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 
> 


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Mime
View raw message