ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jaimin Jetly" <jai...@hortonworks.com>
Subject Re: Review Request 41997: Upgrading Ambari Jetty Server from Jetty 8 to Jetty 9
Date Wed, 13 Jan 2016 02:13:58 GMT


> On Jan. 12, 2016, 11:42 p.m., Jonathan Hurley wrote:
> > OK, this looked a lot better. Still 1 issue; we can't use Jetty's calculation for
selectors because of how it behaves on very large systems. Consider that we've seen problems
with hosts which have 64 or 128 processors.

One of the way Starvation occurs is when (max QTP threads - (Acceptor threads + Selector threads))
is very less value.
For 128 processors 
Default Acceptors as calculated by Jetty = Math.max(1, Math.min(4, (cores available to JVM)/8))
= 4 (http://grepcode.com/file/repo1.maven.org/maven2/org.eclipse.jetty/jetty-server/9.2.2.v20140723/org/eclipse/jetty/server/AbstractConnector.java#189)
Ambari code changes this to Desired Acceptors = Math.max(2, Default Acceptors/ 2) = 2
Default Selectors as calculated by Jetty = Math.max(1, Math.min(4, (cores available to JVM)/2)))
= 4 (http://grepcode.com/file/repo1.maven.org/maven2/org.eclipse.jetty/jetty-server/9.2.2.v20140723/org/eclipse/jetty/server/ServerConnector.java#209)

Default max QTP threads as initiated by Jetty = 200. (http://grepcode.com/file/repo1.maven.org/maven2/org.eclipse.jetty/jetty-util/9.2.11.v20150529/org/eclipse/jetty/util/thread/QueuedThreadPool.java#67)
This by default makes (max QTP threads - (Acceptor threads + Selector threads)) = 194 
But Ambari in configureJettyThreadPool method changes max QTP threads to Max(25, 20 + Desired
Acceptors*2) = Max(25, 24) = 25.
This makes (max QTP threads - (Acceptor threads + Selector threads)) = 19

If you can help I wanted to understand  how is ambari achieving better performance by loweing
the value of (max QTP threads - (Acceptor threads + Selector threads)) in configureJettyThreadPool
method

Is there any other starvation reson that you are referring to different from what I am perceiving
?


> On Jan. 12, 2016, 11:42 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java,
line 420
> > <https://reviews.apache.org/r/41997/diff/3/?file=1194891#file1194891line420>
> >
> >     This is a problem - using Jetty's calculation on systems with 64+ CPUs can cause
thread starvation issues. We should use the calculation we had before.

Even before, we did not have any calculation for setting selector count and relied on Jetty's
calculation. If I am missing something over here please point me to a github link of the present
ambari code that sets selector count. 
I will be happy to fix the selector count issue irrespective of it being an existing issue
or being introduced as part of this patch. 
I just need to know what is the logic to calculate the desired selector count


- Jaimin


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41997/#review114107
-----------------------------------------------------------


On Jan. 13, 2016, 1:54 a.m., Jaimin Jetly wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41997/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2016, 1:54 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Yusaku Sako.
> 
> 
> Bugs: AMBARI-14231
>     https://issues.apache.org/jira/browse/AMBARI-14231
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> The reviewboard addresses the concerns raised at https://reviews.apache.org/r/41018/
and fixes the unit test failures that happened due to the original patch uploaded at https://reviews.apache.org/r/41018/
> 
> 
> Diffs
> -----
> 
>   ambari-funtest/pom.xml d84b1c8 
>   ambari-project/pom.xml 68bdb94 
>   ambari-server/pom.xml 9c0ba26 
>   ambari-server/src/main/java/org/apache/ambari/server/api/AmbariErrorHandler.java c4a80f2

>   ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java 353a972

>   ambari-server/src/test/java/org/apache/ambari/server/api/AmbariErrorHandlerTest.java
8b4f99c 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariHandlerListTest.java
2e2cc45 
>   ambari-web/pom.xml d160e37 
> 
> Diff: https://reviews.apache.org/r/41997/diff/
> 
> 
> Testing
> -------
> 
> 1. Ambari Jetty server is being upgraded to 9.2.11 version. Verifed that there is no
known critcial vulnerability with this version of Jetty (http://www.eclipse.org/jetty/documentation/current/security-reports.html)
> 2. ambari-web resources files are no longer explicitly compressed in the maven build
process with the changes in the patch. Verified manually by using browser developer tools
that ambari-server does this dynamically for any *.js and *.css files.
> 3. Verified manually that HDP cluster and views can be created with the changes made
in the patch.
> 4. Verified that all functional tests written in ambari-funtest project passes with the
changes done in the patch.
> 5. Verified that all unit tests passes with the changes done in the patch.
> 
> #mvn clean test:#
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 41
licence.
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Ambari Main ........................................ SUCCESS [  6.024 s]
> [INFO] Apache Ambari Project POM .......................... SUCCESS [  0.053 s]
> [INFO] Ambari Web ......................................... SUCCESS [01:01 min]
> [INFO] Ambari Views ....................................... SUCCESS [  4.496 s]
> [INFO] Ambari Admin View .................................. SUCCESS [ 13.839 s]
> [INFO] ambari-metrics ..................................... SUCCESS [  0.439 s]
> [INFO] Ambari Metrics Common .............................. SUCCESS [  3.800 s]
> [INFO] Ambari Metrics Hadoop Sink ......................... SUCCESS [  6.071 s]
> [INFO] Ambari Metrics Flume Sink .......................... SUCCESS [  4.089 s]
> [INFO] Ambari Metrics Kafka Sink .......................... SUCCESS [  5.205 s]
> [INFO] Ambari Metrics Storm Sink .......................... SUCCESS [  1.802 s]
> [INFO] Ambari Metrics Collector ........................... SUCCESS [01:31 min]
> [INFO] Ambari Metrics Monitor ............................. SUCCESS [  4.857 s]
> [INFO] Ambari Metrics Assembly ............................ SUCCESS [  6.001 s]
> [INFO] Ambari Server ...................................... SUCCESS [  01:19 h]
> [INFO] Ambari Functional Tests ............................ SUCCESS [01:17 min]
> [INFO] Ambari Agent ....................................... SUCCESS [ 16.568 s]
> [INFO] Ambari Client ...................................... SUCCESS [  0.041 s]
> [INFO] Ambari Python Client ............................... SUCCESS [  1.307 s]
> [INFO] Ambari Groovy Client ............................... SUCCESS [ 13.300 s]
> [INFO] Ambari Shell ....................................... SUCCESS [  0.046 s]
> [INFO] Ambari Python Shell ................................ SUCCESS [  0.086 s]
> [INFO] Ambari Groovy Shell ................................ SUCCESS [ 10.767 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:25 h
> [INFO] Finished at: 2016-01-06T14:49:30-08:00
> [INFO] Final Memory: 90M/606M
> 
> 
> Thanks,
> 
> Jaimin Jetly
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message