hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ning Zhang" <nzh...@fb.com>
Subject Review Request: HIVE-2127. Improve stats gathering reliability by retries on failures
Date Mon, 25 Apr 2011 18:25:40 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/664/
-----------------------------------------------------------

Review request for hive.


Summary
-------

The major changes are:

 0) 2 parameters are introduced: hive.stats.retries.max (default 0) to be the maximum # of
retries on SQLException failures, and hive.stats.retries.wait (default 3 sec) to be the base
time window (explained below) to wait before the next retry. 

 1) introduced a couple of Utilities function to execute SQL queries with retries on failures.
One Utilities function is to determine the wait time based on the number of failures and a
base wait window (same as the one introduced in HDFS-767 for DFSClient to retry on BlockMissingExceptions).
The actual wait time is determined by baseWindow * failues + baseWindow * (failure + 1) *
(random number between [0.0,1.0]).

 2) changed the JDBCStatsAggregator.java to use PreparedStatement to be able to use executeWithRetries().


 3) change the JDBCStatsPublisher.java and JDBCStasAggregator.java to use retries on SQL connections
and SQL executions. 


Diffs
-----

  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1095959 
  trunk/conf/hive-default.xml 1095959 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1095959 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 1095959

  trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 1095959 

Diff: https://reviews.apache.org/r/664/diff


Testing
-------

Running unit tests. 


Thanks,

Ning


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message