ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Toader <stoa...@hortonworks.com>
Subject Re: Review Request 43621: Decrease the load on ambari database after cluster creation
Date Wed, 17 Feb 2016 14:29:07 GMT


> On Feb. 17, 2016, 12:41 a.m., Sid Wagle wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
line 358
> > <https://reviews.apache.org/r/43621/diff/1/?file=1251919#file1251919line358>
> >
> >     Could you provide some guidance of size of the DTO object as a *comment* here
to help gugage the size of the cache relative to memory available.

The values in the cache are HostRoleCommandStatusSummaryDTO objects grouped by stage id. The
approx size of an entry in the cache is: 480 * nr_of_stages_in_request + 4 * nr_of_tasks.
(I used this http://java-performance.info/memory-consumption-of-java-data-types-2/ as reference
to estimate the footprint of HashMaps)

Let's assume that there is 15000 node cluster that is being provisioned using Blueprint (this
is one provision cluster request). That will create 15000 stages. Let's assume that we install
all HDP services on each host (23 service * 5 avg components * 2 install_and_start tasks =>
roughly 200 tasks). I think we can consider the provision cluster request via Blueprints being
the largest request that will yield the largest entry into the cache: 480 * 15000 + 4 * (200
* 15000) => 18 MB one entry in the cache. Most of the requests (like restart all components
on a host, restart all components of a service etc) will result in a much smaller entry in
the cache ~10 KB

Hope this makes sense and my estimation is correct.


> On Feb. 17, 2016, 12:41 a.m., Sid Wagle wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java,
line 2433
> > <https://reviews.apache.org/r/43621/diff/1/?file=1251923#file1251923line2433>
> >
> >     Why not store state in the DB and query active only?

I don't know the reason why the active state state was never stored in the db. I'm just guessing
that since this active flag is not used by the server it just needed by the client (eg. UI)
of the server this was never stored in the db. The state is in the db already in a way (version
and create time stamp) and the most recent records are considered the active ones. Having
a separate active column is kind of redundant.


> On Feb. 17, 2016, 12:41 a.m., Sid Wagle wrote:
> > ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql, line 656
> > <https://reviews.apache.org/r/43621/diff/1/?file=1251925#file1251925line656>
> >
> >     Status and role are not unique enough for an index, what sort of gain does this
buy us ?

The most often executed queries against host_role_command table are:
SELECT DISTINCT task_id FROM host_role_command WHERE ((role = @P0) AND (status = @P1)) ORDER
BY task_id
SELECT COUNT(task_id) FROM host_role_command WHERE (status IN (@P0,@P1,@P2,@P3,@P4,@P5))

This index will cover these two queries. Also many of the other queries against this table
filter by status and role so these two fields seems to be common accros the quries.


- Sebastian


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43621/#review119375
-----------------------------------------------------------


On Feb. 16, 2016, 10:39 p.m., Sebastian Toader wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43621/
> -----------------------------------------------------------
> 
> (Updated Feb. 16, 2016, 10:39 p.m.)
> 
> 
> Review request for Ambari, Myroslav Papirkovskyy, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-15011
>     https://issues.apache.org/jira/browse/AMBARI-15011
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> 1. Created new index on host_role_command table on (status, role) fields as this table
is being queries quite often by status and role
> 2. Removed uneccessary querying of ambari server actions by hostname
> 3. Cache HostRoleCommandStatusSummaryDTO computed objects using guava cache
> 4. The method returning service configs made one call to the database to get the active
service config entities. Then retrieved from database all service config entities and while
building the response it was using the first collection to determine which reponse item should
be marked as active/inactive. This has been modified to make only one roundtrip to the database
and determine the active flag using plain java code.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
91f2d30 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
d2fe4fc 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java
bd5fb5a 
>   ambari-server/src/main/java/org/apache/ambari/server/serveraction/ServerActionExecutor.java
20cf5bb 
>   ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
d3fdc65 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
88b3151 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql eded221 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 780e81a 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql d4a1ddb 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 64f2a4e 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql c557cf6 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql adda30c 
>   ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java
bc4d397 
>   ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java 510e1fb

>   ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
1f90813 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
862776f 
>   ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java 0cdf50a

>   ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog222Test.java
aa4090e 
>   ambari-server/src/test/java/org/apache/ambari/server/utils/StageUtilsTest.java 2d663d9

> 
> Diff: https://reviews.apache.org/r/43621/diff/
> 
> 
> Testing
> -------
> 
> Manual testing:
> 
> Created a cluster of 3 nodes waited 30 mins with UI running and collected postgres sql
statement stats, than left the cluster running for another 30 mins wiht UI closed and getting
new sql statement stats. Verified that the number of execution of  queries mentioned in the
JIRA has decreased.
> 
> Unit tests:
> 
> Total run:884
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Sebastian Toader
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message