hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files
Date Fri, 03 Apr 2015 22:13:53 GMT

     [ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zhe Zhang updated HDFS-7889:
----------------------------
    Attachment: HDFS-7889-009.patch

Thanks Bo for updating the patch! It looks good overall.

# The current {{stripeBlocks}} is not type safe. I think we can use an ArrayList here.
# {{blockGroupSize}} and {{blockGroupDataBlocks}} should come from the constructor. We can
change it later, when EC policy is returned as part of file status.
# In {{encode}}, can we directly use the given buffers, instead of making a copy first?
# Is this really the last block? Shouldn't it be the block from the current streamer?
{code}
      // get last block before destroying the streamer
      ExtendedBlock lastBlock = streamers.get(0).getBlock();
{code}
# It seems {{writeParityCellsForLastStripe}} uses stale data in the cellBuffers? In the below
code, shouldn't we clear cellBuffers beyond {{index}}?
{code}
      if (i >= index) {
        cellBuffers[i].position(0);
        cellBuffers[i].limit(parityCellSize);
      }
{code}
# You already have the hook to open a file in striped format, maybe need to change the below
in the test?
{code}
      DFSStripedOutputStream dos = (DFSStripedOutputStream) Whitebox.getInternalState(os,
          "wrappedStream");
{code}
# Could you also make a pass and split long lines (each line should be <= 80 chars)?

I made some minor changes in 009 patch. Since HDFS-7839 moved EC policy to XAttr, we don't
have a way to check the EC policy from file status yet. The current code checks if replication
factor is 0, which works but if not ideal. Will address after HDFS-7859.

> Subclass DFSOutputStream to support writing striping layout files
> -----------------------------------------------------------------
>
>                 Key: HDFS-7889
>                 URL: https://issues.apache.org/jira/browse/HDFS-7889
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Li Bo
>            Assignee: Li Bo
>         Attachments: HDFS-7889-001.patch, HDFS-7889-002.patch, HDFS-7889-003.patch, HDFS-7889-004.patch,
HDFS-7889-005.patch, HDFS-7889-006.patch, HDFS-7889-007.patch, HDFS-7889-008.patch, HDFS-7889-009.patch
>
>
> After HDFS-7888, we can subclass  {{DFSOutputStream}} to support writing striping layout
files. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message