hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6988) Add configurable limit for percentage-based eviction threshold
Date Fri, 03 Oct 2014 05:28:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157687#comment-14157687
] 

Arpit Agarwal commented on HDFS-6988:
-------------------------------------

bq. Why is dfs.datanode.ram.disk.low.watermark.replicas specified in terms of number of replicas?
Block size is a per-replica property-- 
Ok, we can make them byte counts.

bq. Block size is a per-replica property-- I could easily have a client that writes 256 MB
or 1 GB replicas, while the DataNode is configured with dfs.blocksize at 64MB. It's pretty
common for formats like ORCFile and Apache Parquet to use large blocks and seek around within
them.
This also favors a limit of at least 1GB + some delta so we can keep space around at least
1 replica and its meta file.

If we are going with a single value can you suggest a default.

> Add configurable limit for percentage-based eviction threshold
> --------------------------------------------------------------
>
>                 Key: HDFS-6988
>                 URL: https://issues.apache.org/jira/browse/HDFS-6988
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: HDFS-6581
>            Reporter: Arpit Agarwal
>             Fix For: HDFS-6581
>
>         Attachments: HDFS-6988.01.patch, HDFS-6988.02.patch
>
>
> Per feedback from [~cmccabe] on HDFS-6930, we can make the eviction thresholds configurable.
The hard-coded thresholds may not be appropriate for very large RAM disks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message