hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li Bo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files
Date Fri, 27 Mar 2015 03:26:53 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383282#comment-14383282

Li Bo commented on HDFS-7889:

Thanks Zhe’s careful review.
bq. Isn’t currentPacket always null after super.writeChunk?

{{super.writeChunk}} just writes a chunk(typically 512 bytes) to {{currentPacket}}, a packet
is typically about 64K bytes, so {{currentPacket}} may not be null if it is not full.

bq. Shouldn't we call super.writeChunkImpl instead? We definitely don't want to handle the
tracing scope twice

You’re right. I will replace it with {{super.writeChunkImpl()}} in next patch.

bq. Maybe streamer() should be getStreamer(), and leadingStreamer() should be getLeadingStreamer()?

Agree. {{getStreamer()}} and {{getLeadingStreamer()}} are better. I will fix it in next patch.

bq.Can we get rid of sizeOfCellInBuffer, by using cellBuffers[i].array().length?

You mean {{ cellBuffers[i].position()}}? I will add a new function {{getSizeOfCellInBuffer()}}
to get rid of {{sizeOfCellInBuffer}}

bq.hookAfterRetrievingBlock looks complex and I'm not sure if it's the best way to allocate
new blocks. Can we move it to DataStreamer?

I think DataStreamer should not be aware of the existence of other streamers,  so I add a
hook to DataStreamer. We can further discuss about a better implementation of {{hookAfterRetrievingBlock}}

> Subclass DFSOutputStream to support writing striping layout files
> -----------------------------------------------------------------
>                 Key: HDFS-7889
>                 URL: https://issues.apache.org/jira/browse/HDFS-7889
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Li Bo
>            Assignee: Li Bo
>         Attachments: HDFS-7889-001.patch
> After HDFS-7888, we can subclass  {{DFSOutputStream}} to support writing striping layout

This message was sent by Atlassian JIRA

View raw message