hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7729) Add logic to DFSOutputStream to support writing a file in striping layout
Date Fri, 13 Feb 2015 02:21:12 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319468#comment-14319468

Kai Zheng commented on HDFS-7729:

Thanks [~jingzhao] and [~szetszwo] for the great thoughts, and [~zhz] for the meaningful analysis
at this phase ! It's never late and it's good to experiment in this direction as well, so
we will be more clear about which way is better.
bq.whether we can/should now define DataStreamer as a static class and move it out of DFSOutputStream....It
may be better and more clear to make DataStreamer as a standalone class only handling the
logic about transferring packets that is assigned to it from outside.
It's surely doable, sounds much elegant and clean. I discussed with [~libo-intel], actually
initially when he worked on this, he wanted to go this way. As he's not sure if the community
would like that big refactoring he first experimented to do small changes as current patch,
as we thought it's important to have an initial workable patch for the community experts'
review. It's great we have [~zhz]'s review and also your feedback now. We're open and would
like to adapt the patch as you suggested. As [~libo-intel] has worked on this part for quite
much time, he's familiar with it now and has much confident in the bigger refactoring. He
will provide a new revision according to this thought. Thanks !
bq.After this refactoring, we can extend the current DFSOutputStream to DFSOutputStreamStriped,
which contains the striping logic
Again, very good idea and design thoughts. [~libo-intel], please consider this as well in
your new update. Thanks.
bq.Hope the following analysis helps exploring the DataStreamer refactor.
[~zhz] you made good points about what's doing in current codes. I thought you would also
agree with the refactoring. Glad we have a strong basis !

As [~libo-intel] is working on the new revision in this approach, please kindly let we know
if you have other concerns. Thanks.

> Add logic to DFSOutputStream to support writing a file in striping layout 
> --------------------------------------------------------------------------
>                 Key: HDFS-7729
>                 URL: https://issues.apache.org/jira/browse/HDFS-7729
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Li Bo
>            Assignee: Li Bo
>         Attachments: Codec-tmp.patch, HDFS-7729-001.patch, HDFS-7729-002.patch, HDFS-7729-003.patch,
HDFS-7729-004.patch, HDFS-7729-005.patch, HDFS-7729-006.patch, HDFS-7729-007.patch, HDFS-7729-008.patch
> If client wants to directly write a file striping layout, we need to add some logic to
DFSOutputStream.  DFSOutputStream needs multiple DataStreamers to write each cell of a stripe
to a remote datanode. 

This message was sent by Atlassian JIRA

View raw message