hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5394) fix race conditions in DN caching and uncaching
Date Wed, 06 Nov 2013 00:24:17 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13814437#comment-13814437
] 

Colin Patrick McCabe commented on HDFS-5394:
--------------------------------------------

OK, I figured out the test failure.  It seems that when computing how much data we can mlock,
we must round up mmap'ed regions to the operating system page size.  In the case of Linux,
that is almost always 4096.  The reason behind this is because the OS manages memory in units
of 4096 bytes.  It is simply impossible to lock at a finer granularity than that.  So we should
take this into account in our statistics.  I adjusted the test to take this into account,
and also added a skip if we don't have enough lockable memory available.

> fix race conditions in DN caching and uncaching
> -----------------------------------------------
>
>                 Key: HDFS-5394
>                 URL: https://issues.apache.org/jira/browse/HDFS-5394
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>    Affects Versions: 3.0.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-5394-caching.001.patch, HDFS-5394-caching.002.patch, HDFS-5394-caching.003.patch,
HDFS-5394-caching.004.patch, HDFS-5394.005.patch, HDFS-5394.006.patch
>
>
> The DN needs to handle situations where it is asked to cache the same replica more than
once.  (Currently, it can actually do two mmaps and mlocks.)  It also needs to handle the
situation where caching a replica is cancelled before said caching completes.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message