hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard Lee (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-3156) DFSClient should timeout on long writes
Date Wed, 02 Apr 2008 20:53:33 GMT
DFSClient should timeout on long writes

                 Key: HADOOP-3156
                 URL: https://issues.apache.org/jira/browse/HADOOP-3156
             Project: Hadoop Core
          Issue Type: Bug
          Components: dfs
    Affects Versions: 0.16.1
         Environment: CentOS release 5, jdk1.6.0_02
            Reporter: Richard Lee

I was experiencing what LOOKED like a deadlock on dfs writes... After digging a little deeper,
i came across what i think is the root cause.

DFSClient on createBlockOutputStream() at line 2005 sets the SoTimeout and specifies a timeout
on socket connect which successfully takes care of both connection and read timeouts, but
it seems as though socket writes do not honor any particular timeout value.  When you trace
the SocketOutputStream.flush() to where we hang at socketWrite0(), there are no timeout values
passed... and i'm doubtful that the native method reads system properties for any other timeout

I feel as though DFSClient should be changed to wrap writes in some form of delegate to a
method that takes a timeout  which uses the value  specified as "dfs.socket.timeout"

Including my thread dump in case others are searching for similar traces:

"DataStreamer for file /tmp/AtomSeqImportProvider-119106c44fd block blk_672010209009706845"
prio=10 tid=0x00002aaae00d9800 nid=0x6e37 runnable [0x000000004141b000..0x000000004141ba80]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        - locked <0x00002aaab46fda10> (a java.io.BufferedOutputStream)
        at java.io.DataOutputStream.flush(DataOutputStream.java:106)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1627)
        - locked <0x00002aaab411fff8> (a java.util.LinkedList)

"Thread-0" prio=10 tid=0x00002aaae009f000 nid=0x6e29 waiting for monitor entry [0x0000000040f16000..0x0000000040f16a00]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:2117)
        - waiting to lock <0x00002aaab411fff8> (a java.util.LinkedList)
        at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:141)
        at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:124)
        - locked <0x00002aaab41202c8> (a org.apache.hadoop.dfs.DFSClient$DFSOutputStream)
        at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
        at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
        - locked <0x00002aaab41202c8> (a org.apache.hadoop.dfs.DFSClient$DFSOutputStream)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:41)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        - locked <0x00002aaab406e898> (a org.apache.hadoop.fs.FSDataOutputStream)
        at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
        at com.imeem.talbot.tools.DataSourceImporter.performImport(DataSourceImporter.java:100)
        at com.imeem.talbot.tools.DataSourceImporter.access$000(DataSourceImporter.java:46)
        at com.imeem.talbot.tools.DataSourceImporter$1.run(DataSourceImporter.java:219)
        at java.lang.Thread.run(Thread.java:619)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message