hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Thomas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6561) Byte array native checksumming on client side
Date Thu, 19 Jun 2014 18:58:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037675#comment-14037675

James Thomas commented on HDFS-6561:

It seems like the best approach here would be to add some buffering to FSOutputSummer. I did
some testing and it seems like going to native code is strictly faster than using the incremental
summer as long as flushes happen no less often than every 100 bytes. More specifically, if
we use the native code, which has no option for incremental checksumming (it could be added
in, but I don't think it's necessary), if we compute a checksum of a partial chunk, we will
need to recompute the entire checksum (rather than just the checksum of the newly added bytes)
when the chunk either 1) fills out or 2) is incremented a bit and then a flush occurs. But
the native checksum on 512 bytes (the chunk size and the maximum possible size of a partial
chunk) takes at most 7 microseconds on my machine, and the incremental Java checksum requires
around 7 microseconds for 100 bytes. So as long as it's reasonable to assume flushes won't
happen more often than every 100 bytes, I propose that we eliminate the incremental summer
from FSOutputSummer and always use the native code for byte arrays.

Buffering FSOutputSummer will mean that writes on the wire are more bursty and data is kept
in client memory longer, but Todd mentioned that many clients already wrap DFSOutputStream
in a BufferedWriter anyway. To achieve maximum performance in the native code, we need the
buffer to be at least a few thousand bytes. The larger the buffer, the better, since we can
amortize away the cost of crossing the JNI boundary (which seems to be quite small actually,
on the order of a microsecond at most).

What do people think about this?

> Byte array native checksumming on client side
> ---------------------------------------------
>                 Key: HDFS-6561
>                 URL: https://issues.apache.org/jira/browse/HDFS-6561
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, hdfs-client, performance
>            Reporter: James Thomas
>            Assignee: James Thomas

This message was sent by Atlassian JIRA

View raw message