hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-19490) Locking on Insert into for non native and managed tables.
Date Wed, 16 May 2018 06:16:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476899#comment-16476899
] 

Hive QA commented on HIVE-19490:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923587/HIVE-19490.5.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 14407 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_8] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_struct_type_vectorization] (batchId=71)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=183)
org.apache.hadoop.hive.ql.plan.mapping.TestReOptimization.testStatCachingMetaStore (batchId=299)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/10980/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10980/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10980/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12923587 - PreCommit-HIVE-Build

> Locking on Insert into for non native and managed tables.
> ---------------------------------------------------------
>
>                 Key: HIVE-19490
>                 URL: https://issues.apache.org/jira/browse/HIVE-19490
>             Project: Hive
>          Issue Type: Improvement
>          Components: Druid integration
>            Reporter: slim bouguerra
>            Assignee: slim bouguerra
>            Priority: Major
>              Labels: druid, locking
>         Attachments: HIVE-19490.2.patch, HIVE-19490.3.patch, HIVE-19490.4.patch, HIVE-19490.5.patch,
HIVE-19490.patch
>
>
> Current state of the art: 
> Managed non native table like Druid Tables, will need to get a Lock on Insert into or
insert Over write. The nature of this lock is set to Exclusive by default for any non native
table.
> This implies that Inserts into Druid table will Lock any read query as well during the
execution of the insert into. IMO this lock (on insert into) is  not needed since the insert
statement is appending data and the state of loading it is managed partially by Hive Storage
handler hook and part of it by Druid. 
> What i am proposing is to relax the lock level to shared for all non native tables on
insert into operations and keep it as Exclusive Write for insert Overwrite for now.
>  
> Any feedback is welcome.
> cc [~ekoifman] / [~ashutoshc] / [~jdere] / [~hagleitn]
> Also am not sure what is the best way to unit test this currently am using debugger to
check if locks are what i except, please let me know if there is a better way to do this. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message