hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4960) Unnecessary .meta seeks even when skip checksum is true
Date Wed, 10 Jul 2013 20:11:49 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705000#comment-13705000

Colin Patrick McCabe commented on HDFS-4960:

bq. Would that cache the meta header Colin?

No.  We would still read it each time.

bq. Varun, Colin gave me crash-course offline on his option #1 above caching the meta data
header for files in FileInputStreamCache; I can hack up patch when you want something to try...

This seems like the way to go for addressing the concerns in this JIRA.  Not a huge optimization
but it's easy to do.
> Unnecessary .meta seeks even when skip checksum is true
> -------------------------------------------------------
>                 Key: HDFS-4960
>                 URL: https://issues.apache.org/jira/browse/HDFS-4960
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 3.0.0, 2.1.0-beta
>            Reporter: Varun Sharma
>            Assignee: Varun Sharma
>         Attachments: 4960-branch2.patch, 4960-trunk.patch
> While attempting to benchmark an HBase + Hadoop 2.0 setup on SSDs, we found unnecessary
seeks into .meta files, each seek was a 7 byte read at the head of the file - this attempts
to validate the version #. Since the client is requesting no-checksum, we should not be needing
to touch the .meta file at all.
> Since the purpose of skip checksum is to also avoid the performance penalty of the extra
seek, we should not be seeking into .meta if skip checksum is true

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message