hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20968) Support conversion of managed to external where location set was not owned by hive
Date Wed, 10 Apr 2019 04:09:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16814025#comment-16814025
] 

Hive QA commented on HIVE-20968:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 41s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 10s{color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 51s{color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 19s{color}
| {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 41s{color} | {color:blue}
standalone-metastore/metastore-common in master has 29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 36s{color} | {color:blue}
common in master has 63 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m 10s{color} | {color:blue}
standalone-metastore/metastore-server in master has 179 extant Findbugs warnings. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  8s{color} | {color:blue}
ql in master has 2258 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 29s{color} | {color:blue}
hcatalog/server-extensions in master has 3 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 28s{color} | {color:blue}
hcatalog/webhcat/java-client in master has 3 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 42s{color} | {color:blue}
itests/hive-unit in master has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 30s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 36s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 21s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 49s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  3m 49s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 11s{color}
| {color:green} The patch metastore-common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 16s{color}
| {color:green} The patch common passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 19s{color} | {color:red}
standalone-metastore/metastore-server: The patch generated 41 new + 97 unchanged - 32 fixed
= 138 total (was 129) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 43s{color} | {color:red}
ql: The patch generated 2 new + 190 unchanged - 0 fixed = 192 total (was 190) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 13s{color} | {color:red}
hcatalog/server-extensions: The patch generated 9 new + 118 unchanged - 6 fixed = 127 total
(was 124) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 12s{color}
| {color:green} hcatalog/webhcat/java-client: The patch generated 0 new + 62 unchanged - 1
fixed = 62 total (was 63) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 23s{color} | {color:red}
itests/hive-unit: The patch generated 5 new + 633 unchanged - 0 fixed = 638 total (was 633)
{color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m  4s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 25s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 14s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 31s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19)
x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-16904/dev-support/hive-personality.sh
|
| git revision | master / 928f3d6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-16904/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-16904/yetus/diff-checkstyle-ql.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-16904/yetus/diff-checkstyle-hcatalog_server-extensions.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-16904/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: standalone-metastore/metastore-common common standalone-metastore/metastore-server
ql hcatalog/server-extensions hcatalog/webhcat/java-client itests/hive-unit U: . |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-16904/yetus.txt |
| Powered by | Apache Yetus    http://yetus.apache.org |


This message was automatically generated.



> Support conversion of managed to external where location set was not owned by hive
> ----------------------------------------------------------------------------------
>
>                 Key: HIVE-20968
>                 URL: https://issues.apache.org/jira/browse/HIVE-20968
>             Project: Hive
>          Issue Type: Sub-task
>          Components: repl
>    Affects Versions: 4.0.0
>            Reporter: mahesh kumar behera
>            Assignee: mahesh kumar behera
>            Priority: Major
>              Labels: DR, pull-request-available
>         Attachments: HIVE-20968.01.patch
>
>          Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> As per migration rule, if a location is outside the default managed table directory and
the location is not owned by "hive" user, then it should be converted to external table after
upgrade.
>  So, the same rule is applicable for Hive replication where the data of source managed
table is residing outside the default warehouse directory and is not owned by "hive" user.
>  During this conversion, the path should be preserved in target as well so that failover
works seamlessly.
>  # If the table location is out side hive warehouse and is not owned by hive, then the
table at target will be converted to external table. But the location can not be retained
, it will be retained relative to hive external warehouse directory. 
>  #  As the table is not an external table at source, only those data which are added
using events will be replicated.
>  # The ownership of the location will be stored in the create table event and will be
used to compare it with strict.managed.tables.migration.owner to decide if the flag in replication
scope can be set. This flag is used to convert the managed table to external table at target.
> Some of the scenarios needs to be blocked if the database is set for replication from
a cluster with non strict managed table setting to strict managed table.
> 1. Block alter table / partition set location for database with source of replication
set for managed tables
> 2. If user manually changes the ownership of the location, hive replication may go to
a non recoverable state.
> 3. Block add partition if the location ownership is different than table location for
managed tables.
> 4. User needs to set strict.managed.tables.migration.owner along with dump command (default
to hive user). This value will be used during dump to decide the ownership which will be used
during load to decide the table type. The location owner information can be stored in the
events during create table. The flag can be stored in replication spec. Check other such configs
used in upgrade tool.
> 5. Replication flow also set additional parameter "external.table.purge"="true" ..only
for migration to external table
> 6. Block conversion from managed to external and vice versa. Pass some flag in upgrade
flow to allow this conversion during upgrade flow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message