TSocket timed out reading 4 bytes error against HadoopThriftServer from Hadoop-0.20.2 in Perl
---------------------------------------------------------------------------------------------
Key: HDFS-2550
URL: https://issues.apache.org/jira/browse/HDFS-2550
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.20.2
Environment: Cloudera Hadoop 0.20.2 distribution running on CentOS 5.5 x64, Java
SE 1.6.0_23-b05, Perl 5.14.0
Reporter: Jason Crickmer
For the past few weeks I have randomly been receiving errors via the Perl binding of ThriftFS
TSocket cannot read 4 bytes. I thought that it was a symptom of too many clients against
the server (say 16 doing a mix of file read and write, as well as listStatus), but in the
past couple of days, I have started getting them all the time, even with only 1 client trying
to read. The Perl client error is:
{noformat}
$VAR1 = bless( {
'code' => 0,
'message' => 'TSocket: timed out reading 4 bytes from bigwws001:9090'
}, 'Thrift::TException' );
{noformat}
This typically happens in conjunction with errors in other clients, thus initially leading
me to believe that it was really a timeout issue. But after it started occurring with only
1 client running and well within the 10 sec timeout I had set on the TSocket within my Perl
client, I checked on the exceptions in the ThriftFS server log:
{noformat}
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.checkBounds(String.java:397)
at java.lang.String.<init>(String.java:442)
at org.apache.hadoop.thriftfs.HadoopThriftServer$HadoopThriftHandler.read(HadoopThriftServer.java:307)
at org.apache.hadoop.thriftfs.api.ThriftHadoopFileSystem$Processor$read.process(Unknown
Source)
at org.apache.hadoop.thriftfs.api.ThriftHadoopFileSystem$Processor.process(Unknown
Source)
at com.facebook.thrift.server.TThreadPoolServer$WorkerProcess.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
This change seems to have "solved" the problem, but this is very much a hack since I do not
know the code (maybe throwing an IOException is the right thing to do, and then let it turn
into a ThriftIOException?).
{noformat}
--- src/contrib/thriftfs/src/java/org/apache/hadoop/thriftfs/HadoopThriftServer.java-orig
2011-11-11 09:18:44.000000000 -0600
+++ src/contrib/thriftfs/src/java/org/apache/hadoop/thriftfs/HadoopThriftServer.java 2011-11-11
09:00:47.000000000 -0600
@@ -303,8 +303,9 @@
}
byte[] tmp = new byte[length];
int numbytes = in.read(offset, tmp, 0, length);
- HadoopThriftHandler.LOG.debug("read done: " + tout.id);
- return new String(tmp, 0, numbytes, "UTF-8");
+ HadoopThriftHandler.LOG.debug("read done: " + tout.id +
+ " numbytes: " + numbytes);
+ return new String(tmp, 0, numbytes > 0 ? numbytes : 0, "UTF-8");
} catch (IOException e) {
throw new ThriftIOException(e.getMessage());
}
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
|