hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
Date Wed, 15 Nov 2017 20:50:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254152#comment-16254152

Anu Engineer commented on HDFS-12794:

[~shashikant] Thanks for working on this. It is a nice improvement to have. Overall the code
looks nice.

I have some minor comments:

*  {code}
      //make sure all the data in the ChunkoutputStreams is written to the
      //  container
          semaphore.availablePermits() == getMaxOutstandingChunks());
This will throw an illegal argument exception. What we should do is to wait until the responses
come back in the close path, with a max time out. 
* nit: unused imports in ChunkOutputStream.java
* I am worried there is a latent bug here, which our tests are not exposing.
throw new CompletionException(
              "Unexpected Storage Container Exception: " + e.toString(), e);
I think we should return the a completeableFuture object and then complete it exceptionally.
If you throw this here, then the caller thread might be already gone. Please correct me if
I am wrong.

> Ozone: Parallelize ChunkOutputSream Writes to container
> -------------------------------------------------------
>                 Key: HDFS-12794
>                 URL: https://issues.apache.org/jira/browse/HDFS-12794
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>    Affects Versions: HDFS-7240
>            Reporter: Shashikant Banerjee
>            Assignee: Shashikant Banerjee
>             Fix For: HDFS-7240
>         Attachments: HDFS-12794-HDFS-7240.001.patch, HDFS-12794-HDFS-7240.002.patch
> The chunkOutPutStream Write are sync in nature .Once one chunk of data gets written,
the next chunk write is blocked until the previous chunk is written to the container.
> The ChunkOutputWrite Stream writes should be made async and Close on the OutputStream
should ensure flushing of all dirty buffers to the container.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message