hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2699) Store data and checksums together in block file
Date Sun, 18 Dec 2011 22:56:31 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171951#comment-13171951

dhruba borthakur commented on HDFS-2699:

Thanks for your comments Scott, Andrew, Todd and Allen.

Scott: most of our our hbase production clusters have io.bytes.per.checksum to 4096 (instead
of 512)

Allen: One can put crcs on a logging device, e.g. bookkeeper perhaps? But at the end of day,
each random io from an hdfs file will consume two disk iops (one on the hdfs block storage
and one from the loogging device), is it not? Won't it be optimal to inline crc and data.

If we decide to implement inline crc, can we make the hdfs support two different data formats
and not do any automatic data format upgrade for exisiting data? pre-existing data can remain
in the older format while newly created files will have data in the new -inline-data-and-crc
format. What to do people think about this idea?

> Store data and checksums together in block file
> -----------------------------------------------
>                 Key: HDFS-2699
>                 URL: https://issues.apache.org/jira/browse/HDFS-2699
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
> The current implementation of HDFS stores the data in one block file and the metadata(checksum)
in another block file. This means that every read from HDFS actually consumes two disk iops,
one to the datafile and one to the checksum file. This is a major problem for scaling HBase,
because HBase is usually  bottlenecked on the number of random disk iops that the storage-hardware

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message