hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas Nair" <>
Subject Re: Review Request 11029: HIVE-4513 - disable hivehistory logs by default
Date Sat, 10 Aug 2013 16:24:16 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Aug. 10, 2013, 4:24 p.m.)

Review request for hive.


HIVE-4513.6.patch - addresses review comments. Fixes race condition that was causing the TestHiveServerSessions.testSessionVars
test failure.

Bugs: HIVE-4513

Repository: hive-git


HiveHistory log files (hive_job_log_hive_*.txt files) store information about hive query such
as query string, plan , counters and MR job progress information.

There is no mechanism to delete these files and as a result they get accumulated over time,
using up lot of disk space. 
I don't think this is used by most people, so I think it would better to turn this off by
default. Jobtracker logs already capture most of this information, though it is not as structured
as history logs.

The change :
A new config parameter hive.session.history.enabled controls if the history-log is enabled.
By default it is set to false.
SessionState initializes the HiveHIstory object. When this config is set to false, it creates
a Proxy object that does not do anything. I did this instead of having SessionState return
null, because that would add null checks in too many places. This keeps the code cleaner and
avoids possibility of NPE.
As the proxy only works against interfaces, i created a HiveHistory interface, moved the implementation
to HiveHistoryImpl. static functions were moved to HiveHistoryUtil .

Diffs (updated)

  common/src/java/org/apache/hadoop/hive/conf/ 83f337b 
  conf/hive-default.xml.template 0a6e433 
  hbase-handler/src/test/templates/TestHBaseCliDriver.vm c59e882 
  ql/src/java/org/apache/hadoop/hive/ql/history/ 97436c5 
  ql/src/java/org/apache/hadoop/hive/ql/history/ PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/history/ PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/history/ PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/history/ fdd56db 
  ql/src/java/org/apache/hadoop/hive/ql/session/ ab369f0 
  ql/src/test/org/apache/hadoop/hive/ql/history/ a783303 
  ql/src/test/templates/TestCliDriver.vm a6ae6c3 




Thejas Nair

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message