hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5810) Unify mmap cache and short-circuit file descriptor cache
Date Mon, 10 Feb 2014 22:53:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897159#comment-13897159
] 

Colin Patrick McCabe commented on HDFS-5810:
--------------------------------------------

I uploaded a new version which is rebased on trunk.  It changes the "caller strings" for dumping
stack traces, uses {{dfs.client.read.shortcircuit.streams.cache.size}} as an upper bound on
the size of both mmapped and non-mmapped replicas, and uses {{TimeUnit}} for time conversions.

I changed the handling of {{outstandingMmapCount}} a little bit.  Although we still track
this stat, we don't try to cap the number of outstanding mmaps.  That is up to the caller
code, not to us.  This is similar to how we handle opening new FDs in general... we do it
on request, no matter how many existing FDs there are.  Only when something is returned to
the cache do we apply the limits.

> Unify mmap cache and short-circuit file descriptor cache
> --------------------------------------------------------
>
>                 Key: HDFS-5810
>                 URL: https://issues.apache.org/jira/browse/HDFS-5810
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>    Affects Versions: 2.3.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-5810.001.patch, HDFS-5810.004.patch, HDFS-5810.006.patch, HDFS-5810.008.patch,
HDFS-5810.015.patch, HDFS-5810.016.patch, HDFS-5810.018.patch, HDFS-5810.019.patch
>
>
> We should unify the client mmap cache and the client file descriptor cache.  Since mmaps
are granted corresponding to file descriptors in the cache (currently FileInputStreamCache),
they have to be tracked together to do "smarter" things like HDFS-5182.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message