Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Fri, 17 May 2013 17:23:17 +0000 (UTC)
From: "Hari Mankude (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12647143.1368232581718.340127.1368811397166@arcas>
In-Reply-To: <JIRA.12647143.1368232581718@arcas>
References: <JIRA.12647143.1368232581718@arcas>
Subject: [jira] [Commented] (HDFS-4817) make HDFS advisory caching
 configurable on a per-file basis
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13660880#comment-13660880 ] 

Hari Mankude commented on HDFS-4817:
------------------------------------

I would look at the patch as an ability for the user to provide "hints" to DN regarding the access patterns (random reads/sequential read/write once only/multiple access etc). It is incidental that these hints are currently used to manage pagecache. The same hints or similar hints can be used for moving blocks to different storage tiers at DN. 

Another suggestion that I had is to provide a fadvise() like interface on the iostream that a user can use to send hints.

I am aware of hfds-4672. It is a complicated and correct way of managing storage pools.
                
> make HDFS advisory caching configurable on a per-file basis
> -----------------------------------------------------------
>
>                 Key: HDFS-4817
>                 URL: https://issues.apache.org/jira/browse/HDFS-4817
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 3.0.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-4817.001.patch
>
>
> HADOOP-7753 and related JIRAs introduced some performance optimizations for the DataNode.  One of them was readahead.  When readahead is enabled, the DataNode starts reading the next bytes it thinks it will need in the block file, before the client requests them.  This helps hide the latency of rotational media and send larger reads down to the device.  Another optimization was "drop-behind."  Using this optimization, we could remove files from the Linux page cache after they were no longer needed.
> Using {{dfs.datanode.drop.cache.behind.writes}} and {{dfs.datanode.drop.cache.behind.reads}} can improve performance  substantially on many MapReduce jobs.  In our internal benchmarks, we have seen speedups of 40% on certain workloads.  The reason is because if we know the block data will not be read again any time soon, keeping it out of memory allows more memory to be used by the other processes on the system.  See HADOOP-7714 for more benchmarks.
> We would like to turn on these configurations on a per-file or per-client basis, rather than on the DataNode as a whole.  This will allow more users to actually make use of them.  It would also be good to add unit tests for the drop-cache code path, to ensure that it is functioning as we expect.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira