hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-21197) Hive replication can add duplicate data during migration to a target with hive.strict.managed.tables enabled
Date Tue, 26 Feb 2019 09:54:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777777#comment-16777777
] 

Hive QA commented on HIVE-21197:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12960156/HIVE-21197.05.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15818 tests passed

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/16246/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16246/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16246/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12960156 - PreCommit-HIVE-Build

> Hive replication can add duplicate data during migration to a target with hive.strict.managed.tables
enabled
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-21197
>                 URL: https://issues.apache.org/jira/browse/HIVE-21197
>             Project: Hive
>          Issue Type: Task
>          Components: repl
>            Reporter: mahesh kumar behera
>            Assignee: mahesh kumar behera
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: HIVE-21197.01.patch, HIVE-21197.02.patch, HIVE-21197.03.patch, HIVE-21197.04.patch,
HIVE-21197.05.patch
>
>          Time Spent: 22h 20m
>  Remaining Estimate: 0h
>
> During bootstrap phase it may happen that the files copied to target are created by events
which are not part of the bootstrap. This is because of the fact that, bootstrap first gets
the last event id and then the file list. During this period if some event are added, then
bootstrap will include files created by these events also.The same files will be copied again
during the first incremental replication just after the bootstrap. In normal scenario, the
duplicate copy does not cause any issue as hive allows the use of target database only after
the first incremental. But in case of migration, the file at source and target are copied
to different location (based on the write id at target) and thus this may lead to duplicate
data at target. This can be avoided by having at check at load time for duplicate file. This
check can be done only for the first incremental and the search can be done in the bootstrap
directory (with write id 1). if the file is already present then just ignore the copy.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message