hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vihang Karajgaonkar (JIRA)" <>
Subject [jira] [Updated] (HIVE-16109) TestDbTxnManager generates a huge hive.log
Date Sat, 04 Mar 2017 02:21:46 GMT


Vihang Karajgaonkar updated HIVE-16109:
    Attachment: HIVE-16109.01.patch

Attaching the patch to fix the test. We don't really need to change the debug message in TxnManager
to log lockInfo to trace. Currently the log file generated is 80 MB and the runtime of the
test is ~110 sec. Both the disk space and runtime are reasonable.

> TestDbTxnManager generates a huge hive.log
> ------------------------------------------
>                 Key: HIVE-16109
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Vihang Karajgaonkar
>            Assignee: Vihang Karajgaonkar
>         Attachments: HIVE-16109.01.patch
> Pre-commit jobs are failing currently due to out of disk space. The issue is happening
due to huge size of hive.log when TestDbTxnManager test fails or times-out. When this test
fails or times-out Ptest tries to persist these logs for debugging. Since this test has been
timing out frequently, this accumulates a lot of these log files and eventually Ptest server
runs of disk space. Each run of TestDbTxnManager is generating ~30 GB of hive.log. I tried
to run it locally and it quickly reached 7 GB until I had to cancel it.
> The issue seems to be coming from this code block in
> {noformat}
> if(LOG.isDebugEnabled()) {
>         LOG.debug("Locks to check(full): ");
>         for(LockInfo info : locks) {
>           LOG.debug("  " + info);
>         }
>       }
> {noformat}
> We should either change it trace or change the log mode of this test to INFO so that
it generates smaller log files.

This message was sent by Atlassian JIRA

View raw message