hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12612) DFSStripedOutputStream#close will throw if called a second time with a failed streamer
Date Tue, 10 Oct 2017 23:01:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16199530#comment-16199530
] 

Andrew Wang commented on HDFS-12612:
------------------------------------

Change looks pretty simple :) Did you dig up any info on the history of this code? I find
it strange that it goes to all this work to explicitly re-throw the error like this. Even
logging again is unnecessary, as long as it logs the first time.

> DFSStripedOutputStream#close will throw if called a second time with a failed streamer
> --------------------------------------------------------------------------------------
>
>                 Key: HDFS-12612
>                 URL: https://issues.apache.org/jira/browse/HDFS-12612
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: erasure-coding
>    Affects Versions: 3.0.0-beta1
>            Reporter: Andrew Wang
>            Assignee: Lei (Eddy) Xu
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: HDFS-12612.00.patch
>
>
> Found while testing with Hive. We have a cluster with 2 DNs and the XOR-2-1 policy. If
you write a file and call close() twice, it throws this exception:
> {noformat}
> 17/10/04 16:02:14 WARN hdfs.DFSOutputStream: Cannot allocate parity block(index=2, policy=XOR-2-1-1024k).
Not enough datanodes? Exclude nodes=[]
> ...
> Caused by: java.io.IOException: Failed to get parity block, index=2
> at org.apache.hadoop.hdfs.DFSStripedOutputStream.allocateNewBlock(DFSStripedOutputStream.java:500)
~[hadoop-hdfs-client-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
> at org.apache.hadoop.hdfs.DFSStripedOutputStream.writeChunk(DFSStripedOutputStream.java:524)
~[hadoop-hdfs-client-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
> {noformat}
> This is because in DFSStripedOutputStream#closeImpl, if the stream is closed, we throw
an exception if any of the striped streamers had an exception:
> {code}
>   protected synchronized void closeImpl() throws IOException {
>     if (isClosed()) {
>       final MultipleIOException.Builder b = new MultipleIOException.Builder();
>       for(int i = 0; i < streamers.size(); i++) {
>         final StripedDataStreamer si = getStripedDataStreamer(i);
>         try {
>           si.getLastException().check(true);
>         } catch (IOException e) {
>           b.add(e);
>         }
>       }
>       final IOException ioe = b.build();
>       if (ioe != null) {
>         throw ioe;
>       }
>       return;
>     }
> {code}
> I think this is incorrect, since we only need to throw in this situation if we have too
many failed streamers. close should also be idempotent, so it should throw the first time
we call close if it's going to throw at all.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message