hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ning Zhang (JIRA)" <>
Subject [jira] Commented: (HIVE-1852) Reduce unnecessary DFSClient.rename() calls
Date Mon, 20 Dec 2010 19:35:09 GMT


Ning Zhang commented on HIVE-1852:

Yes, it's because the oldPath could be from a different file system. In this patch, a new
file system is got from oldPath and it is used to delete oldPath. It should be equivalent
to FsShell. Hairong mentioned that FsShell is relatively slow and more expensive that calling

Yongqiang, is there a test case for HIVE-1707?

> Reduce unnecessary DFSClient.rename() calls
> -------------------------------------------
>                 Key: HIVE-1852
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Ning Zhang
>            Assignee: Ning Zhang
>         Attachments: HIVE-1852.2.patch, HIVE-1852.3.patch, HIVE-1852.patch
> In Hive client side (MoveTask etc), DFSCleint.rename() is called for every file inside
a directory. It is very expensive for a large directory in a busy DFS namenode. We should
replace it with a single rename() call on the whole directory. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message