hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Crickmer (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-2550) TSocket timed out reading 4 bytes error against HadoopThriftServer from Hadoop-0.20.2 in Perl
Date Fri, 11 Nov 2011 20:38:52 GMT
TSocket timed out reading 4 bytes error against HadoopThriftServer from Hadoop-0.20.2 in Perl
---------------------------------------------------------------------------------------------

                 Key: HDFS-2550
                 URL: https://issues.apache.org/jira/browse/HDFS-2550
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: hdfs client
    Affects Versions: 0.20.2
         Environment: Cloudera Hadoop 0.20.2 distribution running on CentOS 5.5 x64, Java
SE 1.6.0_23-b05, Perl 5.14.0
            Reporter: Jason Crickmer


For the past few weeks I have randomly been receiving errors via the Perl binding of ThriftFS
TSocket cannot read 4 bytes.  I thought that it was a symptom of too many clients against
the server (say 16 doing a mix of file read and write, as well as listStatus), but in the
past couple of days, I have started getting them all the time, even with only 1 client trying
to read.  The Perl client error is:

{noformat}
$VAR1 = bless( {
                 'code' => 0,
                 'message' => 'TSocket: timed out reading 4 bytes from bigwws001:9090'
               }, 'Thrift::TException' );
{noformat}

This typically happens in conjunction with errors in other clients, thus initially leading
me to believe that it was really a timeout issue.  But after it started occurring with only
1 client running and well within the 10 sec timeout I had set on the TSocket within my Perl
client, I checked on the exceptions in the ThriftFS server log:

{noformat}
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
        at java.lang.String.checkBounds(String.java:397)
        at java.lang.String.<init>(String.java:442)
        at org.apache.hadoop.thriftfs.HadoopThriftServer$HadoopThriftHandler.read(HadoopThriftServer.java:307)
        at org.apache.hadoop.thriftfs.api.ThriftHadoopFileSystem$Processor$read.process(Unknown
Source)
        at org.apache.hadoop.thriftfs.api.ThriftHadoopFileSystem$Processor.process(Unknown
Source)
        at com.facebook.thrift.server.TThreadPoolServer$WorkerProcess.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
{noformat}


This change seems to have "solved" the problem, but this is very much a hack since I do not
know the code (maybe throwing an IOException is the right thing to do, and then let it turn
into a ThriftIOException?).

{noformat}
--- src/contrib/thriftfs/src/java/org/apache/hadoop/thriftfs/HadoopThriftServer.java-orig
      2011-11-11 09:18:44.000000000 -0600
+++ src/contrib/thriftfs/src/java/org/apache/hadoop/thriftfs/HadoopThriftServer.java    2011-11-11
09:00:47.000000000 -0600
@@ -303,8 +303,9 @@
         }
         byte[] tmp = new byte[length];
         int numbytes = in.read(offset, tmp, 0, length);
-        HadoopThriftHandler.LOG.debug("read done: " + tout.id);
-        return new String(tmp, 0, numbytes, "UTF-8");
+        HadoopThriftHandler.LOG.debug("read done: " + tout.id +
+                                     " numbytes: " + numbytes);
+        return new String(tmp, 0, numbytes > 0 ? numbytes : 0, "UTF-8");
       } catch (IOException e) {
         throw new ThriftIOException(e.getMessage());
       }

{noformat}


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message