hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5957) Provide support for different mmap cache retention policies in ShortCircuitCache.
Date Tue, 25 Feb 2014 22:47:22 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912186#comment-13912186

Colin Patrick McCabe commented on HDFS-5957:

Thanks for this data point, Gopal.  I agree that allocating your own direct {{ByteBuffer}}
objects will probably be slower than {{mmap}}.  It certainly will cause issues with the Java
heap.  I was suggesting it as a temporary workaround until YARN-1747 is fixed, not as a permanent

It seems like a better interim solution would be simply to tweak {{dfs.client.mmap.cache.size}}
(and perhaps some job-specific parameters that you're using) to ensure that you don't exceed
the limits that YARN is placing.

As I mentioned earlier, HDFS's behavior here is completely deterministic and synchronous...
we will immediately {{munmap}} old segments in {{releaseBuffer}} (according to the LRU policy)
when there is no more space for them.  It is not "an eventual {{munmap}} on a background thread."
 If you set the cache size too big, that's a problem, but not one that HDFS can solve.  Only
the administrator can solve it.

Why don't we close this JIRA and move the discussion to YARN-1747?

> Provide support for different mmap cache retention policies in ShortCircuitCache.
> ---------------------------------------------------------------------------------
>                 Key: HDFS-5957
>                 URL: https://issues.apache.org/jira/browse/HDFS-5957
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 2.3.0
>            Reporter: Chris Nauroth
> Currently, the {{ShortCircuitCache}} retains {{mmap}} regions for reuse by multiple reads
of the same block or by multiple threads.  The eventual {{munmap}} executes on a background
thread after an expiration period.  Some client usage patterns would prefer strict bounds
on this cache and deterministic cleanup by calling {{munmap}}.  This issue proposes additional
support for different caching policies that better fit these usage patterns.

This message was sent by Atlassian JIRA

View raw message