hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andras Bokor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10429) DataStreamer interrupted warning always appears when using CLI upload file
Date Tue, 08 Aug 2017 11:53:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118235#comment-16118235

Andras Bokor commented on HDFS-10429:

I am not confident in this area but I tried out the patch on a cluster with 4 datanodes.
I have tested with 12000 random files between 1-10 MB. Before the patch I got exceptions.
After the patch I run the copy 3 times (which means 36000 files) without exception so it seems
the patch fixes the issue.
Related the code of patch:
* We should consider introducing some checks in finally to not to call close methods twice.
Something like {{if(!getStreamer().isSocketClosed() || !getStreamer().isAlive())}} and call
forced close if something is not closed properly. What do you think?
* I would not hide the InterruptedExceptions where it's not expected. It can show when our
thread handling is not clean like in this case.

Other than the two bullet points I'd give a non-binding +1 since my test shows that the exceptions
disappeared but and all the files were copied to HDFS.

> DataStreamer interrupted warning  always appears when using CLI upload file
> ---------------------------------------------------------------------------
>                 Key: HDFS-10429
>                 URL: https://issues.apache.org/jira/browse/HDFS-10429
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.7.3
>            Reporter: Zhiyuan Yang
>            Assignee: Wei-Chiu Chuang
>            Priority: Minor
>         Attachments: HDFS-10429.1.patch, HDFS-10429.2.patch, HDFS-10429.3.patch
> Every time I use 'hdfs dfs -put' upload file, this warning is printed:
> {code:java}
> 16/05/18 20:57:56 WARN hdfs.DataStreamer: Caught exception
> java.lang.InterruptedException
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Thread.join(Thread.java:1245)
> 	at java.lang.Thread.join(Thread.java:1319)
> 	at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:871)
> 	at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:519)
> 	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:696)
> {code}
> The reason is this: originally, DataStreamer::closeResponder always prints a warning
about InterruptedException; since HDFS-9812, DFSOutputStream::closeImpl  always forces threads
to close, which causes InterruptedException.
> A simple fix is to use debug level log instead of warning level.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message