hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
Date Wed, 15 Nov 2017 20:50:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254152#comment-16254152
] 

Anu Engineer commented on HDFS-12794:
-------------------------------------

[~shashikant] Thanks for working on this. It is a nice improvement to have. Overall the code
looks nice.

I have some minor comments:

*  {code}
      //make sure all the data in the ChunkoutputStreams is written to the
      //  container
      Preconditions.checkArgument(
          semaphore.availablePermits() == getMaxOutstandingChunks());
{code}
This will throw an illegal argument exception. What we should do is to wait until the responses
come back in the close path, with a max time out. 
* nit: unused imports in ChunkOutputStream.java
* I am worried there is a latent bug here, which our tests are not exposing.
{code}
throw new CompletionException(
              "Unexpected Storage Container Exception: " + e.toString(), e);
{code}
I think we should return the a completeableFuture object and then complete it exceptionally.
If you throw this here, then the caller thread might be already gone. Please correct me if
I am wrong.


> Ozone: Parallelize ChunkOutputSream Writes to container
> -------------------------------------------------------
>
>                 Key: HDFS-12794
>                 URL: https://issues.apache.org/jira/browse/HDFS-12794
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>    Affects Versions: HDFS-7240
>            Reporter: Shashikant Banerjee
>            Assignee: Shashikant Banerjee
>             Fix For: HDFS-7240
>
>         Attachments: HDFS-12794-HDFS-7240.001.patch, HDFS-12794-HDFS-7240.002.patch
>
>
> The chunkOutPutStream Write are sync in nature .Once one chunk of data gets written,
the next chunk write is blocked until the previous chunk is written to the container.
> The ChunkOutputWrite Stream writes should be made async and Close on the OutputStream
should ensure flushing of all dirty buffers to the container.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message