ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitro Lisnichenko" <dlysniche...@hortonworks.com>
Subject Re: Review Request 41774: BE: Install packages call takes more than 3 minutes on 900 node cluster
Date Wed, 30 Dec 2015 10:46:40 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41774/#review112302
-----------------------------------------------------------



ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java

<https://reviews.apache.org/r/41774/#comment172738>

    This code was obsolete. We set clusterHostInfo on stage level, and don't need it on command
level



ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
(line 371)
<https://reviews.apache.org/r/41774/#comment172732>

    Now we generate clusterHostInfo once instead of doing that 900 times. It's a costy operation,
it was the slowest place when processing request



ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
(line 466)
<https://reviews.apache.org/r/41774/#comment172734>

    Avoid parsing json multiple times, minor optimization



ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java (line
1193)
<https://reviews.apache.org/r/41774/#comment172731>

    Here we avoid opening/closing transaction on every host version update, gives considerable
performance boost on big cluster



ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java (line 863)
<https://reviews.apache.org/r/41774/#comment172736>

    Avoid parsing json multiple times, minor optimization


- Dmitro Lisnichenko


On Dec. 30, 2015, 12:41 p.m., Dmitro Lisnichenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41774/
> -----------------------------------------------------------
> 
> (Updated Dec. 30, 2015, 12:41 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Jonathan Hurley, Jayush
Luniya, Nate Cole, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-14520
>     https://issues.apache.org/jira/browse/AMBARI-14520
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> On 900 host cluster the call to install packages took like 3 minutes 20 seconds which
is kind of long.
> {noformat}
> POST http://perf-a-1:8080/api/v1/clusters/c1/stack_versions
> 
> {"ClusterStackVersions":{"stack":"HDP","version":"2.2","repository_version":"2.2.9.0-3393"}}:
> {noformat}
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java
6e7e06b 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
b6ac291 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostStackVersionResourceProvider.java
43a4423 
>   ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
3358e8c 
>   ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java c56d7ce

> 
> Diff: https://reviews.apache.org/r/41774/diff/
> 
> 
> Testing
> -------
> 
> mvn clean test
> 
> Current request execution time: 0m22.678s
> 
> With current patch, most time (about 40%) is consumed by findConfigurationTagsWithOverrides(cluster,
hostname), but it seems to be inevitable even with batching addExecutionCommandsToStage()
calls into a single invocation. I've tried inserting logging statements to verify that number
of invocations equals to number of hosts.
> Rest of time is mostly consumed by org.apache.ambari.server.actionmanager.Stage.addHostRoleExecutionCommand()
(20% of time) and by N invocations of ClustersImpl.getHostsForCluster() (19% of time). The
latter calls can not be cached in method, so I can only rewrite method signature AmbariActionExecutionHelper.addExecutionCommandsToStage(ActionExecutionContext,
Stage) and cache it in ClusterStackVersionResourceProvider. Seems to be a bad deal considering
1-3 seconds (on large clusters ) of performance boost for a rare call vs readability/maintainability
of code.
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message