hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1068) Reduce NameNode GC by reusing HdfsFileStatus objects in RPC handlers
Date Wed, 06 Sep 2017 11:46:02 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16155242#comment-16155242

Steve Loughran commented on HDFS-1068:

S3Guard/HADOOP-13345 is really for caching shared state in dynamodb, rather than talking to
the (very slow) S3 store. But you can run a metadata store in-VM, which we do purely for testing...it
gets inconsistent as soon as anyone changes the store.

One thing we have done to try and improve perf is (a) deprecate FileSystem.exists(), .isDir()
and .isFile(), pushing people to using getFileStatus instead. If you look through downstream
code to see how those probes get used, you see a lot of things like
if (fs.exists(path)) fs.delete(path)

if (fs.exists(path)) fs.mkdirs(path)

or worst of all
if (fs.exists(path)) {
  FileStatus stat = fs.getFileStatus(path)

essentially: calling getFilestatus, discarding the result and then calling an operation which
does the checks & downgrades if fails, or even duplicating the work.

Every IPC call is sacred, especially if blobstores convert that to an HTTP request

> Reduce NameNode GC by reusing HdfsFileStatus objects in RPC handlers
> --------------------------------------------------------------------
>                 Key: HDFS-1068
>                 URL: https://issues.apache.org/jira/browse/HDFS-1068
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Hairong Kuang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-1068.00.patch, Screen Shot 2017-08-31 at 3.58.15 PM.png
> In our production clusters, getFileInfo is the most frequent operation that hit NameNode,
and its frequency is highly correlated to the GC behavior. HDFS-946 has already reduced the
amount of heap/cpu and the number of temporary objects for each getFileInfo call. Yet another
improvement is to avoid creation of a HdfsFileStatus object for each getFileInfo call. Instead
each RPC handler can have a thread local HdfsFileStatus object. Each getFileInfo call simply
sets values for all fields of the thread local HdfsFileStatus object. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message