ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nate Cole" <nc...@hortonworks.com>
Subject Re: Review Request 31239: Ambari Exhausts Connection Pool When Using MySQL Due To Invalid Connections
Date Sat, 21 Feb 2015 12:47:41 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31239/#review73403
-----------------------------------------------------------

Ship it!


Ship It!

- Nate Cole


On Feb. 20, 2015, 4:05 p.m., Jonathan Hurley wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31239/
> -----------------------------------------------------------
> 
> (Updated Feb. 20, 2015, 4:05 p.m.)
> 
> 
> Review request for Ambari, Nate Cole and Tom Beerbower.
> 
> 
> Bugs: AMBARI-9736
>     https://issues.apache.org/jira/browse/AMBARI-9736
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> When using MySQL, Ambari appears to try to use invalid connections about about 8 hours
of use.
> 
> EclipseLink does not appear to correctly handle connections that have been invalidated
by MySQL after the default timeout time of 8 hours.
> 
> STR:
> Set MySQL's timeouts via /etc/my.cnf
> [mysqld]
> interactive_timeout=120
> wait_timeout=120
> 
> Verify the connection timeouts are lowered
> mysql> SHOW VARIABLES LIKE "%timeout%";
> +----------------------------+----------+
> | Variable_name              | Value    |
> +----------------------------+----------+
> | connect_timeout            | 10       |
> | delayed_insert_timeout     | 300      |
> | innodb_lock_wait_timeout   | 50       |
> | innodb_rollback_on_timeout | OFF      |
> | interactive_timeout        | 120      |
> | lock_wait_timeout          | 31536000 |
> | net_read_timeout           | 30       |
> | net_write_timeout          | 60       |
> | slave_net_timeout          | 3600     |
> | wait_timeout               | 120      |
> +----------------------------+----------+
> 10 rows in set (0.00 sec)
> 
> Launch Ambari and wait a few minutes. The exceptions will begin and Ambari will stop
working properly.
> 
> The fix here was to use c3p0 (already included via Quartz) instead of EclipseLink's internal
pool management. Since we're so close to release, I felt that only changing mysql was the
right option, with the ability to instruct Ambari to use c3p0 for other databases if needed.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
495c1e4 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
3b59fed 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
85ae85b 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/EclipseLinkSessionCustomizer.java
PRE-CREATION 
>   ambari-server/src/main/java/org/apache/ambari/server/scheduler/ExecutionSchedulerImpl.java
324ee27 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalog.java
71049b2 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog150.java
2ccf16b 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog151.java
72305c5 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog160.java
cd01779 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog161.java
f5afb46 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog170.java
47f4587 
>   ambari-server/src/main/resources/META-INF/persistence.xml d89f5d5 
>   ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
0f7de36 
> 
> Diff: https://reviews.apache.org/r/31239/diff/
> 
> 
> Testing
> -------
> 
> Verification that the issue has been resolved in a reproduction environment. Also, verification
that Ambari's pool stays a consistent size:
> 
> mysql> show processlist;
> +-----+--------+--------------------+--------+---------+------+-------+------------------+
> | Id  | User   | Host               | db     | Command | Time | State | Info        
    |
> +-----+--------+--------------------+--------+---------+------+-------+------------------+
> | 340 | ambari | 192.168.64.1:53549 | ambari | Sleep   |   66 |       | NULL        
    |
> | 341 | ambari | 192.168.64.1:53557 | ambari | Sleep   |   30 |       | NULL        
    |
> | 346 | ambari | 192.168.64.1:53675 | ambari | Sleep   |    2 |       | NULL        
    |
> | 347 | ambari | 192.168.64.1:53676 | ambari | Sleep   |   26 |       | NULL        
    |
> | 348 | ambari | 192.168.64.1:53677 | ambari | Sleep   |    1 |       | NULL        
    |
> | 349 | ambari | 192.168.64.1:53727 | ambari | Sleep   |    2 |       | NULL        
    |
> | 350 | root   | localhost          | NULL   | Query   |    0 | NULL  | show processlist
|
> +-----+--------+--------------------+--------+---------+------+-------+------------------+
> 7 rows in set (0.00 sec)
> 
> New tests written to cover configuration options.
> 
> Tests run: 2721, Failures: 0, Errors: 0, Skipped: 15
> 
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 14:40 min
> [INFO] Finished at: 2015-02-20T16:01:43-05:00
> [INFO] Final Memory: 34M/361M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message