hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <>
Subject [jira] [Commented] (HIVE-17128) Operation Logging leaks file descriptors as the log4j Appender is never closed
Date Thu, 20 Jul 2017 20:18:01 GMT


Hive QA commented on HIVE-17128:

Here are the results of testing the latest attachment:

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11088 tests executed
*Failed tests:*
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar] (batchId=153)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=179)

Test results:
Console output:
Test logs:

Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed

This message is automatically generated.

ATTACHMENT ID: 12878232 - PreCommit-HIVE-Build

> Operation Logging leaks file descriptors as the log4j Appender is never closed
> ------------------------------------------------------------------------------
>                 Key: HIVE-17128
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Logging
>            Reporter: Andrew Sherman
>            Assignee: Andrew Sherman
>         Attachments: HIVE-17128.1.patch
> [HIVE-16061] and [HIVE-16400] changed operation logging to use the Log4j2 RoutingAppender
to automatically output the log for each query into each individual operation log file. As
log4j does not know when a query is finished it keeps the OutputStream in the Appender open
even when the query completes. The stream holds a file descriptor and so we leak file descriptors.
Note that we are already careful to close any streams reading from the operation log file.
> h2. Fix
> To fix this we use a technique described in the comments of [LOG4J2-510] which uses reflection
to close the appender. The test in TestOperationLoggingLayout will be extended to check that
the Appender is closed.

This message was sent by Atlassian JIRA

View raw message