hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14864) Distcp is not called from MoveTask when src is a directory
Date Fri, 14 Oct 2016 02:51:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573955#comment-15573955
] 

Hive QA commented on HIVE-14864:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833234/HIVE-14864.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 89 failed/errored test(s), 10534 tests executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_7]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[crtseltbl_serdeprops]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_3]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_4]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_4]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_5]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_00_nonpart_empty]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_01_nonpart]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_02_00_part_empty]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_02_part]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_03_nonpart_over_compat]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_04_all_part]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_04_evolved_parts]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_05_some_part]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_06_one_part]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_07_all_part_over_nonoverlap]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_08_nonpart_rename]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_09_part_spec_nonoverlap]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_10_external_managed]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_11_managed_external]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_12_external_location]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_13_managed_location]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_14_managed_location_over_existing]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_15_external_part]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_16_part_external]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_17_part_managed]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_18_part_external]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_19_00_part_external_location]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_19_part_external_location]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_20_part_managed_location]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_21_export_authsuccess]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_22_import_exist_authsuccess]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_23_import_part_authsuccess]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_24_import_nonexist_authsuccess]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_25_export_parentpath_has_inaccessible_children]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_hidden_files]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[foldts]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_9]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_describe]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reloadJar]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[repl_2_exim_basic]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[repl_3_exim_metadata]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_partial_size]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_gb1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_join1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_subquery1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_tablesample_rows]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_load_data_to_encrypted_tables]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[import_exported_table]
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[import_exported_table]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_import]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[ctas_noemptyfolder]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_01_nonpart_over_loaded]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_02_all_part_over_overlap]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_03_nonpart_noncompat_colschema]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_04_nonpart_noncompat_colnumber]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_05_nonpart_noncompat_coltype]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_06_nonpart_noncompat_storage]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_07_nonpart_noncompat_ifof]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_08_nonpart_noncompat_serde]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_09_nonpart_noncompat_serdeparam]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_10_nonpart_noncompat_bucketing]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_11_nonpart_noncompat_sorting]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_13_nonnative_import]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_14_nonpart_part]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_15_part_nonpart]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_16_part_noncompat_schema]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_17_part_spec_underspec]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_18_part_spec_missing]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_19_external_over_existing]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_20_managed_location_over_existing]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_21_part_managed_external]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_23_import_exist_authfail]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_24_import_part_authfail]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_25_import_nonexist_authfail]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[temp_table]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[temp_table_gb1]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[temp_table_join1]
org.apache.hadoop.hive.ql.history.TestHiveHistory.testHiveHistoryConfigDisabled
org.apache.hadoop.hive.ql.history.TestHiveHistory.testHiveHistoryConfigEnabled
org.apache.hadoop.hive.ql.history.TestHiveHistory.testQueryloglocParentDirNotExist
org.apache.hadoop.hive.ql.history.TestHiveHistory.testSimpleQuery
org.apache.hadoop.hive.ql.security.TestExtendedAcls.testExim
org.apache.hadoop.hive.ql.security.TestFolderPermissions.testExim
org.apache.hive.hcatalog.api.repl.commands.TestCommands.testBasicReplEximCommands
org.apache.hive.hcatalog.api.repl.commands.TestCommands.testMetadataReplEximCommands
org.apache.hive.hcatalog.api.repl.commands.TestCommands.testNoopReplEximCommands
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1546/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1546/console
Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1546/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 89 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12833234 - PreCommit-HIVE-Build

> Distcp is not called from MoveTask when src is a directory
> ----------------------------------------------------------
>
>                 Key: HIVE-14864
>                 URL: https://issues.apache.org/jira/browse/HIVE-14864
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Vihang Karajgaonkar
>            Assignee: Sahil Takiar
>         Attachments: HIVE-14864.patch
>
>
> In FileUtils.java the following code does not get executed even when src directory size
is greater than HIVE_EXEC_COPYFILE_MAXSIZE because 
> srcFS.getFileStatus(src).getLen() returns 0 when src is a directory. We should use srcFS.getContentSummary(src).getLength()
instead.
> {noformat}
>     /* Run distcp if source file/dir is too big */
>     if (srcFS.getUri().getScheme().equals("hdfs") &&
>         srcFS.getFileStatus(src).getLen() > conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE))
{
>       LOG.info("Source is " + srcFS.getFileStatus(src).getLen() + " bytes. (MAX: " +
conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE) + ")");
>       LOG.info("Launch distributed copy (distcp) job.");
>       HiveConfUtil.updateJobCredentialProviders(conf);
>       copied = shims.runDistCp(src, dst, conf);
>       if (copied && deleteSource) {
>         srcFS.delete(src, true);
>       }
>     }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message