hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <>
Subject [jira] [Commented] (HIVE-17411) LLAP IO may incorrectly release a refcount in some rare cases
Date Wed, 30 Aug 2017 07:17:00 GMT


Hive QA commented on HIVE-17411:

Here are the results of testing the latest attachment:

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11014 tests executed
*Failed tests:*
TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out) (batchId=280)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_windowing] (batchId=84)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=143)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234)

Test results:
Console output:
Test logs:

Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed

This message is automatically generated.

ATTACHMENT ID: 12884379 - PreCommit-HIVE-Build

> LLAP IO may incorrectly release a refcount in some rare cases
> -------------------------------------------------------------
>                 Key: HIVE-17411
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>         Attachments: HIVE-17411.patch
> In a large stream whose buffers are not reused, and that is separated into many CB (e.g.
due to a small ORC compression buffer size), it may happen that some, but not all, buffers
that are read together as a unit are evicted from cache.
> If CacheBuffer follows BufferChunk in the buffer list when a stream like this is read,
the latter will be converted to ProcCacheChunk;  it is possible for early refcount release
logic from the former to release the refcount (for a dictionary stream, the initial refCount
is always released early), and then backtrack to the latter to see if we can unlock more buffers.
It would then to decref an uninitialized MemoryBuffer in ProcCacheChunk because ProcCacheChunk
looks like a CacheChunk. PCC initial refcounts are released separately after the data is uncompressed.
> I'm assuming this would almost never happen with non-stripe-level streams because one
would need a large RG to span 2+ CBs, no overlap with next/previous RGs in 2+ buffers for
the early release to kick in, and an unfortunate eviction order. However it's possible with
large-ish dictionaries.

This message was sent by Atlassian JIRA

View raw message