hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mingxi Wu <Mingxi...@turn.com>
Subject upload hang at DFSClient$DFSOutputStream.close(3488)
Date Mon, 16 Apr 2012 01:55:13 GMT
Hi, 

I use hadoop cloudera 0.20.2-cdh3u0.

I have a program which uploads local files to HDFS every hour.

Basically, I open a gzip input stream by in= new GZIPInputStream(fin); And write to HDFS file.
After less than two days, it will hang. It hangs at FSDataOutputStream.close(86).
Here is the stack:

State: WAITING Running 16660 ms (user 13770 ms) blocked 11276 times for <> ms waiting
11209 times for <> ms
LockName: java.util.LinkedList@f1ca0de LockOwnerId: -1
java.lang.Object.wait(-2)
java.lang.Object.wait(485)
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(3468)
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal(3457)
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(3549)
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(3488)
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(61)
org.apache.hadoop.fs.FSDataOutputStream.close(86)
org.apache.hadoop.io.IOUtils.copyBytes(59)
org.apache.hadoop.io.IOUtils.copyBytes(74)

Any suggestion to avoid this issue? It seems this is a bug in hadoop. I found this issue is
less severe when my upload server do one upload at a time, instead of using multiple concurrent
uploads.

Thanks,

Mingxi

Mime
View raw message