hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6930) Improve replica eviction from RAM disk
Date Tue, 02 Sep 2014 20:09:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118657#comment-14118657

Colin Patrick McCabe commented on HDFS-6930:

bq. Eviction is done when we have Less than 10% free space or Insufficient space for 3 default
length blocks.

One thing that might be suboptimal here is that we're using the {{dfs.blocksize}} configuration
key on the DataNode and assuming that will be the same value used by the client.  Clearly,
the client could use 256 MB blocks, whereas the DN could use 128 MB blocks.  Etc.

Also, we don't really know how big the ramdisks are going to be.  I can easily see a 300 GB
ramdisk being used in a few years.  Just defaulting to keeping 10% free seems like too much.

So, why not just have a minimum free space configuration key.  It could be specified as a
number of bytes, rather than as a percentage.  So we could default it to 128 MB * 3 to get
your current default of leaving space for 3 blocks.  This would work better for bigger ramdisks
(unlike a percentage-based scheme) and wouldn't make assumptions about the client's and DN's
block size configuration being the same.

> Improve replica eviction from RAM disk
> --------------------------------------
>                 Key: HDFS-6930
>                 URL: https://issues.apache.org/jira/browse/HDFS-6930
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: HDFS-6581
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>         Attachments: HDFS-6930.01.patch
> The current replica eviction scheme is inefficient since it performs multiple file operations
in the context of block allocation.
> A better implementation would be asynchronous eviction when free space on RAM disk falls
below a low watermark to make block allocation faster.

This message was sent by Atlassian JIRA

View raw message