ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hurley" <jhur...@hortonworks.com>
Subject Review Request 31239: Ambari Exhausts Connection Pool When Using MySQL Due To Invalid Connections
Date Fri, 20 Feb 2015 20:51:17 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31239/
-----------------------------------------------------------

Review request for Ambari, Nate Cole and Tom Beerbower.


Bugs: AMBARI-9736
    https://issues.apache.org/jira/browse/AMBARI-9736


Repository: ambari


Description
-------

When using MySQL, Ambari appears to try to use invalid connections about about 8 hours of
use.

EclipseLink does not appear to correctly handle connections that have been invalidated by
MySQL after the default timeout time of 8 hours.

STR:
Set MySQL's timeouts via /etc/my.cnf
[mysqld]
interactive_timeout=120
wait_timeout=120

Verify the connection timeouts are lowered
mysql> SHOW VARIABLES LIKE "%timeout%";
+----------------------------+----------+
| Variable_name              | Value    |
+----------------------------+----------+
| connect_timeout            | 10       |
| delayed_insert_timeout     | 300      |
| innodb_lock_wait_timeout   | 50       |
| innodb_rollback_on_timeout | OFF      |
| interactive_timeout        | 120      |
| lock_wait_timeout          | 31536000 |
| net_read_timeout           | 30       |
| net_write_timeout          | 60       |
| slave_net_timeout          | 3600     |
| wait_timeout               | 120      |
+----------------------------+----------+
10 rows in set (0.00 sec)

Launch Ambari and wait a few minutes. The exceptions will begin and Ambari will stop working
properly.

The fix here was to use c3p0 (already included via Quartz) instead of EclipseLink's internal
pool management. Since we're so close to release, I felt that only changing mysql was the
right option, with the ability to instruct Ambari to use c3p0 for other databases if needed.


Diffs
-----

  ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java 495c1e4

  ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
3b59fed 
  ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java 85ae85b

  ambari-server/src/main/java/org/apache/ambari/server/orm/EclipseLinkSessionCustomizer.java
PRE-CREATION 
  ambari-server/src/main/java/org/apache/ambari/server/scheduler/ExecutionSchedulerImpl.java
324ee27 
  ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalog.java
71049b2 
  ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog150.java 2ccf16b

  ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog151.java 72305c5

  ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog160.java cd01779

  ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog161.java f5afb46

  ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog170.java 47f4587

  ambari-server/src/main/resources/META-INF/persistence.xml d89f5d5 
  ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
0f7de36 

Diff: https://reviews.apache.org/r/31239/diff/


Testing
-------

Verification that the issue has been resolved in a reproduction environment. Also, verification
that Ambari's pool stays a consistent size:

mysql> show processlist;
+-----+--------+--------------------+--------+---------+------+-------+------------------+
| Id  | User   | Host               | db     | Command | Time | State | Info             |
+-----+--------+--------------------+--------+---------+------+-------+------------------+
| 340 | ambari | 192.168.64.1:53549 | ambari | Sleep   |   66 |       | NULL             |
| 341 | ambari | 192.168.64.1:53557 | ambari | Sleep   |   30 |       | NULL             |
| 346 | ambari | 192.168.64.1:53675 | ambari | Sleep   |    2 |       | NULL             |
| 347 | ambari | 192.168.64.1:53676 | ambari | Sleep   |   26 |       | NULL             |
| 348 | ambari | 192.168.64.1:53677 | ambari | Sleep   |    1 |       | NULL             |
| 349 | ambari | 192.168.64.1:53727 | ambari | Sleep   |    2 |       | NULL             |
| 350 | root   | localhost          | NULL   | Query   |    0 | NULL  | show processlist |
+-----+--------+--------------------+--------+---------+------+-------+------------------+
7 rows in set (0.00 sec)

New tests written to cover configuration options.


Thanks,

Jonathan Hurley


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message