hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio Peña (JIRA) <j...@apache.org>
Subject [jira] [Commented] (HIVE-14864) Distcp is not called from MoveTask when src is a directory
Date Mon, 17 Oct 2016 14:50:58 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582459#comment-15582459
] 

Sergio Peña commented on HIVE-14864:
------------------------------------

A couple of comments:

* Will srcFS.getContentSummary(src) cause extra time when source is on S3? If so, maybe we
want to put this line inside the if() statement.

* Once we're here, could you fix the message from HIVE_EXEC_COPYFILE_MAXSIZE to say (in Bytes)
instead of (in Mb) ?

> Distcp is not called from MoveTask when src is a directory
> ----------------------------------------------------------
>
>                 Key: HIVE-14864
>                 URL: https://issues.apache.org/jira/browse/HIVE-14864
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Vihang Karajgaonkar
>            Assignee: Sahil Takiar
>         Attachments: HIVE-14864.1.patch, HIVE-14864.2.patch, HIVE-14864.patch
>
>
> In FileUtils.java the following code does not get executed even when src directory size
is greater than HIVE_EXEC_COPYFILE_MAXSIZE because 
> srcFS.getFileStatus(src).getLen() returns 0 when src is a directory. We should use srcFS.getContentSummary(src).getLength()
instead.
> {noformat}
>     /* Run distcp if source file/dir is too big */
>     if (srcFS.getUri().getScheme().equals("hdfs") &&
>         srcFS.getFileStatus(src).getLen() > conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE))
{
>       LOG.info("Source is " + srcFS.getFileStatus(src).getLen() + " bytes. (MAX: " +
conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE) + ")");
>       LOG.info("Launch distributed copy (distcp) job.");
>       HiveConfUtil.updateJobCredentialProviders(conf);
>       copied = shims.runDistCp(src, dst, conf);
>       if (copied && deleteSource) {
>         srcFS.delete(src, true);
>       }
>     }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message