Return-Path: X-Original-To: apmail-hive-issues-archive@minotaur.apache.org Delivered-To: apmail-hive-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 561DD1745F for ; Sun, 12 Apr 2015 04:57:12 +0000 (UTC) Received: (qmail 94399 invoked by uid 500); 12 Apr 2015 04:57:12 -0000 Delivered-To: apmail-hive-issues-archive@hive.apache.org Received: (qmail 94381 invoked by uid 500); 12 Apr 2015 04:57:12 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 94371 invoked by uid 99); 12 Apr 2015 04:57:12 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Apr 2015 04:57:12 +0000 Date: Sun, 12 Apr 2015 04:57:12 +0000 (UTC) From: "Hive QA (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-10228) Changes to Hive Export/Import/DropTable/DropPartition to support replication semantics MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-10228?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1449= 1328#comment-14491328 ]=20 Hive QA commented on HIVE-10228: -------------------------------- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12724677/HIVE-10228.patch {color:red}ERROR:{color} -1 due to 102 failed/errored test(s), 8675 tests e= xecuted *Failed tests:* {noformat} TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_s= ort_dyn_part.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_buck= et_sort_merge.q-auto_sortmerge_join_16.q-and-1-more - did not produce a TES= T-*.xml file TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputf= ormat.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_= external.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin= 7.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucke= ting.q-uber_reduce.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-sc= riptfile1_win.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name= .q-root_dir_external_table.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-an= d-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx= _cbo_1.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbto= idx_cbo_2.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce = a TEST-*.xml file TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - = did not produce a TEST-*.xml file org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testAlterPartiti= on org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testAlterTable org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testAlterViewPar= ititon org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testColumnStatis= tics org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testComplexTable org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testConcurrentMe= tastores org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDropTable org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testFilterLastPa= rtition org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testFilterSingle= Partition org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testListPartitio= nNames org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testListPartitio= ns org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testPartition org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testPartitionFil= ter org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testRenamePartit= ion org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testSimpleTable org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTableDatabas= e org.apache.hadoop.hive.metastore.TestHiveMetaTool.testExecuteJDOQL org.apache.hadoop.hive.metastore.TestHiveMetaTool.testListFSRoot org.apache.hadoop.hive.metastore.TestHiveMetaTool.testUpdateFSRootLocation org.apache.hadoop.hive.metastore.TestMetastoreExpr.testPartitionExpr org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testAlterPartition org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testAlterTable org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testAlterViewParit= iton org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testColumnStatisti= cs org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testComplexTable org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testConcurrentMeta= stores org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testDropTable org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testFilterLastPart= ition org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testFilterSinglePa= rtition org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testListPartitionN= ames org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testListPartitions org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testPartition org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testPartitionFilte= r org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testRenamePartitio= n org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testSimpleTable org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testTableDatabase org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHand= ler org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterPart= ition org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterTabl= e org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterView= Parititon org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testColumnSta= tistics org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testComplexTa= ble org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testConcurren= tMetastores org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDropTable org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testFilterLas= tPartition org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testFilterSin= glePartition org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testListParti= tionNames org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testListParti= tions org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartition org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartition= Filter org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testRenamePar= tition org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSimpleTab= le org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testTableData= base org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testAlterPartition org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testAlterTable org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testAlterViewPariti= ton org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testColumnStatistic= s org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testComplexTable org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testConcurrentMetas= tores org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testDropTable org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testFilterLastParti= tion org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testFilterSinglePar= tition org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testListPartitionNa= mes org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testListPartitions org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testPartition org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testPartitionFilter org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testRenamePartition org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testSimpleTable org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testTableDatabase org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testAlterPartition org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testAlterTable org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testAlterViewPariti= ton org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testColumnStatistic= s org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testComplexTable org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testConcurrentMetas= tores org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testDropTable org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testFilterLastParti= tion org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testFilterSinglePar= tition org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testListPartitionNa= mes org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testListPartitions org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testPartition org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testPartitionFilter org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testRenamePartition org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testSimpleTable org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testTableDatabase org.apache.hive.hcatalog.cli.TestPermsGrp.testCustomPerms org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat.org.apache.hiv= e.hcatalog.mapreduce.TestHCatMultiOutputFormat org.apache.hive.jdbc.TestJdbcWithMiniHS2.testNewConnectionConfiguration org.apache.hive.jdbc.TestSSL.testSSLFetchHttp {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job= /PreCommit-HIVE-TRUNK-Build/3389/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/j= ob/PreCommit-HIVE-TRUNK-Build/3389/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit= -HIVE-TRUNK-Build-3389/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 102 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12724677 - PreCommit-HIVE-TRUNK-Build > Changes to Hive Export/Import/DropTable/DropPartition to support replicat= ion semantics > -------------------------------------------------------------------------= ------------- > > Key: HIVE-10228 > URL: https://issues.apache.org/jira/browse/HIVE-10228 > Project: Hive > Issue Type: Sub-task > Components: Import/Export > Affects Versions: 1.2.0 > Reporter: Sushanth Sowmyan > Assignee: Sushanth Sowmyan > Attachments: HIVE-10228.patch > > > We need to update a couple of hive commands to support replication semant= ics. To wit, we need the following: > EXPORT ... [FOR [METADATA] REPLICATION(=E2=80=9Ccomment=E2=80=9D)] > Export will now support an extra optional clause to tell it that this exp= ort is being prepared for the purpose of replication. There is also an addi= tional optional clause here, that allows for the export to be a metadata-on= ly export, to handle cases of capturing the diff for alter statements, for = example. > Also, if done for replication, the non-presence of a table, or a table be= ing a view/offline table/non-native table is not considered an error, and i= nstead, will result in a successful no-op. > IMPORT ... (as normal) =E2=80=93 but handles new semantics=20 > No syntax changes for import, but import will have to change to be able t= o handle all the permutations of export dumps possible. Also, import will h= ave to ensure that it should update the object only if the update being imp= orted is not older than the state of the object. > DROP TABLE ... FOR REPLICATION('eventid') > Drop Table now has an additional clause, to specify that this drop table = is being done for replication purposes, and that the dop should not actuall= y drop the table if the table is newer than that event id specified. > ALTER TABLE ... DROP PARTITION (...) FOR REPLICATION('eventid') > Similarly, Drop Partition also has an equivalent change to Drop Table. > =3D > In addition, we introduce a new property "repl.last.id", which when tagge= d on to table properties or partition properties on a replication-destinati= on, holds the effective "state identifier" of the object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)