hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13572) Redundant setting full file status in Hive::copyFiles
Date Thu, 21 Apr 2016 06:03:25 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15251327#comment-15251327
] 

Rui Li commented on HIVE-13572:
-------------------------------

Thanks Ashutosh for the review! I just thought maybe another way to solve this is to set the
full status on the single file rather than the destination folder, which can be done concurrently
in the threads. I'll do some test to see which is better for performance.

> Redundant setting full file status in Hive::copyFiles
> -----------------------------------------------------
>
>                 Key: HIVE-13572
>                 URL: https://issues.apache.org/jira/browse/HIVE-13572
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Rui Li
>            Assignee: Rui Li
>         Attachments: HIVE-13572.1.patch
>
>
> We set full file status in each copy-file thread. I think it's redundant and hurts performance
when we have multiple files to copy.
> {code}
>             if (inheritPerms) {
>               ShimLoader.getHadoopShims().setFullFileStatus(conf, fullDestStatus, destFs,
destf);
>             }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message