hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin McCabe <cmcc...@alumni.cmu.edu>
Subject Re: [jira] [Created] (HDFS-8722) Optimize datanode writes for small writes and flushes
Date Wed, 05 Aug 2015 16:52:17 GMT
Hi Jean-Marc,

Short-circuit covers reads, but this performance improvement covers writes.

best,
Colin

On Wed, Aug 5, 2015 at 7:17 AM, Jean-Marc Spaggiari
<jean-marc@spaggiari.org> wrote:
> Hi Nick,
>
> If we are doing short-circuit, we skip Hadoop CRC, right? So this should
> impact us only in case we are not doing short-circuit? Or wall doesn't
> bypass it?
>
> JM
>
> 2015-08-03 19:04 GMT-04:00 Nick Dimiduk <ndimiduk@apache.org>:
>
>> FYI, this looks like it would impact small WAL writes.
>>
>> On Tue, Jul 7, 2015 at 10:44 AM, Kihwal Lee (JIRA) <jira@apache.org>
>> wrote:
>>
>> > Kihwal Lee created HDFS-8722:
>> > --------------------------------
>> >
>> >              Summary: Optimize datanode writes for small writes and
>> flushes
>> >                  Key: HDFS-8722
>> >                  URL: https://issues.apache.org/jira/browse/HDFS-8722
>> >              Project: Hadoop HDFS
>> >           Issue Type: Improvement
>> >             Reporter: Kihwal Lee
>> >             Priority: Critical
>> >
>> >
>> > After the data corruption fix by HDFS-4660, the CRC recalculation for
>> > partial chunk is executed more frequently, if the client repeats writing
>> > few bytes and calling hflush/hsync.  This is because the generic logic
>> > forces CRC recalculation if on-disk data is not CRC chunk aligned. Prior
>> to
>> > HDFS-4660, datanode blindly accepted whatever CRC client provided, if the
>> > incoming data is chunk-aligned. This was the source of the corruption.
>> >
>> > We can still optimize for the most common case where a client is
>> > repeatedly writing small number of bytes followed by hflush/hsync with no
>> > pipeline recovery or append, by allowing the previous behavior for this
>> > specific case.  If the incoming data has a duplicate portion and that is
>> at
>> > the last chunk-boundary before the partial chunk on disk, datanode can
>> use
>> > the checksum supplied by the client without redoing the checksum on its
>> > own.  This reduces disk reads as well as CPU load for the checksum
>> > calculation.
>> >
>> > If the incoming packet data goes back further than the last on-disk chunk
>> > boundary, datanode will still do a recalculation, but this occurs rarely
>> > during pipeline recoveries. Thus the optimization for this specific case
>> > should be sufficient to speed up the vast majority of cases.
>> >
>> >
>> >
>> > --
>> > This message was sent by Atlassian JIRA
>> > (v6.3.4#6332)
>> >
>>

Mime
View raw message