hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6581) Write to single replica in memory
Date Fri, 22 Aug 2014 07:35:15 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106593#comment-14106593

Arpit Agarwal commented on HDFS-6581:

bq. Memory pressure is worse with mmaped files because we cannot control the timing of when
the pages will be freed. We can evict pages from memory via unmap faster than the memory manager
can write them to disk. tmpfs has better characteristics, once we run into the configured
limit we can just stop allocating more blocks in memory. A related optimization I'd really
like to have is to use unbuffered IO when writing to block files on disk so we don't churn
buffer cache.
Also our initial proposal on HDFS-5851 was to use mmapped files. However using a RAM disk
allows fairly good control over memory usage on the write path with the least effort.

> Write to single replica in memory
> ---------------------------------
>                 Key: HDFS-6581
>                 URL: https://issues.apache.org/jira/browse/HDFS-6581
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>         Attachments: HDFSWriteableReplicasInMemory.pdf
> Per discussion with the community on HDFS-5851, we will implement writing to a single
replica in DN memory via DataTransferProtocol.
> This avoids some of the issues with short-circuit writes, which we can revisit at a later

This message was sent by Atlassian JIRA

View raw message