hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-17608) REPL LOAD should overwrite the data files if exists instead of duplicating it
Date Wed, 04 Oct 2017 16:23:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-17608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191516#comment-16191516
] 

Hive QA commented on HIVE-17608:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890319/HIVE-17608.02.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11200 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=232)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=232)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=157)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] (batchId=240)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7117/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7117/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7117/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12890319 - PreCommit-HIVE-Build

> REPL LOAD should overwrite the data files if exists instead of duplicating it
> -----------------------------------------------------------------------------
>
>                 Key: HIVE-17608
>                 URL: https://issues.apache.org/jira/browse/HIVE-17608
>             Project: Hive
>          Issue Type: Sub-task
>          Components: HiveServer2, repl
>    Affects Versions: 3.0.0
>            Reporter: Sankar Hariappan
>            Assignee: Sankar Hariappan
>              Labels: DR, pull-request-available, replication
>             Fix For: 3.0.0
>
>         Attachments: HIVE-17608.01.patch, HIVE-17608.02.patch
>
>
> This is to make insert event idempotent.
> Currently, MoveTask would create a new file if the destination folder contains a file
of the same name. This is wrong if we have the same file in both bootstrap dump and incremental
dump (by design, duplicate file in incremental dump will be ignored for idempotent reason),
we will get duplicate files eventually. Also it is wrong to just retain the filename in the
staging folder. Suppose we get the same insert event twice, the first time we get the file
from source table folder, the second time we get the file from cm, we still end up with duplicate
copy. The right solution is to keep the same file name as the source table folder.
> To do that, we can put the original filename in MoveWork, and in MoveTask, if original
filename is set, don't generate a new name, simply overwrite. We need to do it in both bootstrap
and incremental load.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message