hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aihua Xu <...@cloudera.com>
Subject Re: Review Request 61010: HIVE-17128 Operation Logging leaks file descriptors as the log4j Appender is never closed
Date Fri, 21 Jul 2017 23:39:30 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61010/#review181158
-----------------------------------------------------------


Ship it!




Ship It!

- Aihua Xu


On July 20, 2017, 10:45 p.m., Andrew Sherman wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61010/
> -----------------------------------------------------------
> 
> (Updated July 20, 2017, 10:45 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Previously HIVE-16061 and HIVE-16400 changed Operation Logging to use the Log4j2
> RoutingAppender to automatically output the log for each query into an
> individual operation log file.  As log4j does not know when a query is finished
> it keeps the OutputStream in the Appender open even when the query completes.
> The stream holds a file descriptor and so we leak file descriptors. Note that we
> are already careful to close any streams reading from the operation log file.
> To fix this we use a technique described in the comments of LOG4J2-510 which
> uses reflection to close the appender. The test in TestOperationLoggingLayout is
> extended to check that the Appender is closed.
> 
> 
> Diffs
> -----
> 
>   itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java
1a8337f574bb753e8c3c48a6b477b17700b05256 
>   ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java e697b545984555414e27bafe92d7f22829a22687

>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 8d453d5d9153c2ec86c4adc7a68bd3b5dd249743

> 
> 
> Diff: https://reviews.apache.org/r/61010/diff/1/
> 
> 
> Testing
> -------
> 
> Hand testing to show leak has gone.
> The test in TestOperationLoggingLayout is extended to check that the Appender is closed.
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message