hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sriramadasu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3965) DFSInputStream should not eat up exceptions if file is under construction
Date Fri, 21 Sep 2012 08:51:08 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460364#comment-13460364
] 

Amareshwari Sriramadasu commented on HDFS-3965:
-----------------------------------------------

We saw a scenario where datanode is unresponsive and the client silently exits with following
log:

{noformat}
21 Sep 2012 00:00:00,050 [IPC Client (47) connection to /a.b.c.d:50020
from blk_5626968076878217612_215416] DEBUG org.apache.hadoop.ipc.Client  -
closing ipc connection to /a.b.c.d:50020: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/a.b.c.d:55558
remote=/a.b.c.d:50020]
java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel
to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/a.b.c.d:55558 remote=/a.b.c.d:50020]
        at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
        at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
        at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
        at java.io.FilterInputStream.read(FilterInputStream.java:116)
        at
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:375)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:815)
        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:724)
21 Sep 2012 00:00:00,055 DEBUG
org.apache.hadoop.hdfs.DFSClient  - DFSClient file
/myfile
is being concurrently append to but datanode a.b.c.d probably does not
have block blk_5626968076878217612_215416
{noformat}

Instead these exceptions should be thrown out so that the read fails and users can retry later
                
> DFSInputStream should not eat up exceptions if file is under construction
> -------------------------------------------------------------------------
>
>                 Key: HDFS-3965
>                 URL: https://issues.apache.org/jira/browse/HDFS-3965
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client
>    Affects Versions: 1.0.3
>            Reporter: Amareshwari Sriramadasu
>
> Currently DFSclient eats up exceptions if it could not open the file under construction:
> {noformat}
> 1881       try {
> 1882         primary = createClientDatanodeProtocolProxy(primaryNode, conf,
> 1883             last.getBlock(), last.getBlockToken(), socketTimeout);
> 1884         Block newBlock = primary.getBlockInfo(last.getBlock());
> 1885         long newBlockSize = newBlock.getNumBytes();
> 1886         long delta = newBlockSize - last.getBlockSize();
> 1887         // if the size of the block on the datanode is different
> 1888         // from what the NN knows about, the datanode wins!
> 1889         last.getBlock().setNumBytes(newBlockSize);
> 1890         long newlength = newInfo.getFileLength() + delta;
> 1891         newInfo.setFileLength(newlength);
> 1892         LOG.debug("DFSClient setting last block " + last + " to length "
> 1893             + newBlockSize + " filesize is now " + newInfo.getFileLength());
> 1894       } catch (IOException e) {
> 1895         if (e.getMessage().startsWith(
> 1896             "java.io.IOException: java.lang.NoSuchMethodException: "
> 1897                 + "org.apache.hadoop.hdfs.protocol"
> 1898                 + ".ClientDatanodeProtocol.getBlockInfo")) {
> 1899           // We're talking to a server that doesn't implement HDFS-200.
> 1900           serverSupportsHdfs200 = false;
> 1901         } else {
> 1902           LOG.debug("DFSClient file " + src
> 1903               + " is being concurrently append to" + " but datanode "
> 1904               + primaryNode.getHostName() + " probably does not have block "
> 1905               + last.getBlock());
> 1906         }
> 1907       }
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message