hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6988) Add configurable limit for percentage-based eviction threshold
Date Fri, 03 Oct 2014 05:28:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157687#comment-14157687

Arpit Agarwal commented on HDFS-6988:

bq. Why is dfs.datanode.ram.disk.low.watermark.replicas specified in terms of number of replicas?
Block size is a per-replica property-- 
Ok, we can make them byte counts.

bq. Block size is a per-replica property-- I could easily have a client that writes 256 MB
or 1 GB replicas, while the DataNode is configured with dfs.blocksize at 64MB. It's pretty
common for formats like ORCFile and Apache Parquet to use large blocks and seek around within
This also favors a limit of at least 1GB + some delta so we can keep space around at least
1 replica and its meta file.

If we are going with a single value can you suggest a default.

> Add configurable limit for percentage-based eviction threshold
> --------------------------------------------------------------
>                 Key: HDFS-6988
>                 URL: https://issues.apache.org/jira/browse/HDFS-6988
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: HDFS-6581
>            Reporter: Arpit Agarwal
>             Fix For: HDFS-6581
>         Attachments: HDFS-6988.01.patch, HDFS-6988.02.patch
> Per feedback from [~cmccabe] on HDFS-6930, we can make the eviction thresholds configurable.
The hard-coded thresholds may not be appropriate for very large RAM disks.

This message was sent by Atlassian JIRA

View raw message