hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-5810) Unify mmap cache and short-circuit file descriptor cache
Date Tue, 04 Feb 2014 02:19:08 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Colin Patrick McCabe updated HDFS-5810:

    Attachment: HDFS-5810.015.patch

Some fixes:
* Don't move tcpReadsDisabledForTesting to BlockReaderFactory, to avoid code churn.  For now,
leave it in DFSInputStream.
* Add TestBlockReaderFactory to test a few corner cases surrounding the BlockReaderFactory.
* Add new configuration keys to hdfs-default.xml.
* Straighten out a few issues surrounding purging replicas from the cache.  Formerly, there
were some bugs that could happen if a replica was purged while others held on to references
to it.
* Rework the way eviction is handled somewhat.  When we run out of mmap slots, we now bump
the replicas down into the standard eviction map, rather than purging them immediately.  This
also motivated me to use eviction maps rather than eviction    lists, since otherwise this
bumping would be an O(n^2) operation.
* Use a HashMap rather than TreeMap to store replicas.
* Rework ShortCircuitCache#getOrCreateClientMmap and ShortCircuitCache#fetchOrCreate

> Unify mmap cache and short-circuit file descriptor cache
> --------------------------------------------------------
>                 Key: HDFS-5810
>                 URL: https://issues.apache.org/jira/browse/HDFS-5810
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>    Affects Versions: 2.3.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-5810.001.patch, HDFS-5810.004.patch, HDFS-5810.006.patch, HDFS-5810.008.patch,
> We should unify the client mmap cache and the client file descriptor cache.  Since mmaps
are granted corresponding to file descriptors in the cache (currently FileInputStreamCache),
they have to be tracked together to do "smarter" things like HDFS-5182.

This message was sent by Atlassian JIRA

View raw message