hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (Resolved) (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-2550) TSocket timed out reading 4 bytes error against HadoopThriftServer from Hadoop-0.20.2 in Perl
Date Fri, 11 Nov 2011 21:51:51 GMT

     [ https://issues.apache.org/jira/browse/HDFS-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aaron T. Myers resolved HDFS-2550.
----------------------------------

    Resolution: Invalid

Hi Jason, I would recommend either emailing cdh-user@cloudera.org or filing a JIRA at https://issues.cloudera.org/browse/DISTRO.
If you do that, please be sure to include which version of CDH you're using.
                
> TSocket timed out reading 4 bytes error against HadoopThriftServer from Hadoop-0.20.2
in Perl
> ---------------------------------------------------------------------------------------------
>
>                 Key: HDFS-2550
>                 URL: https://issues.apache.org/jira/browse/HDFS-2550
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client
>    Affects Versions: 0.20.2
>         Environment: Cloudera Hadoop 0.20.2 distribution running on CentOS 5.5 x64, Java
SE 1.6.0_23-b05, Perl 5.14.0
>            Reporter: Jason Crickmer
>
> For the past few weeks I have randomly been receiving errors via the Perl binding of
ThriftFS TSocket cannot read 4 bytes.  I thought that it was a symptom of too many clients
against the server (say 16 doing a mix of file read and write, as well as listStatus), but
in the past couple of days, I have started getting them all the time, even with only 1 client
trying to read.  The Perl client error is:
> {noformat}
> $VAR1 = bless( {
>                  'code' => 0,
>                  'message' => 'TSocket: timed out reading 4 bytes from bigwws001:9090'
>                }, 'Thrift::TException' );
> {noformat}
> This typically happens in conjunction with errors in other clients, thus initially leading
me to believe that it was really a timeout issue.  But after it started occurring with only
1 client running and well within the 10 sec timeout I had set on the TSocket within my Perl
client, I checked on the exceptions in the ThriftFS server log:
> {noformat}
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
>         at java.lang.String.checkBounds(String.java:397)
>         at java.lang.String.<init>(String.java:442)
>         at org.apache.hadoop.thriftfs.HadoopThriftServer$HadoopThriftHandler.read(HadoopThriftServer.java:307)
>         at org.apache.hadoop.thriftfs.api.ThriftHadoopFileSystem$Processor$read.process(Unknown
Source)
>         at org.apache.hadoop.thriftfs.api.ThriftHadoopFileSystem$Processor.process(Unknown
Source)
>         at com.facebook.thrift.server.TThreadPoolServer$WorkerProcess.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> {noformat}
> This change seems to have "solved" the problem, but this is very much a hack since I
do not know the code (maybe throwing an IOException is the right thing to do, and then let
it turn into a ThriftIOException?).
> {noformat}
> --- src/contrib/thriftfs/src/java/org/apache/hadoop/thriftfs/HadoopThriftServer.java-orig
      2011-11-11 09:18:44.000000000 -0600
> +++ src/contrib/thriftfs/src/java/org/apache/hadoop/thriftfs/HadoopThriftServer.java
   2011-11-11 09:00:47.000000000 -0600
> @@ -303,8 +303,9 @@
>          }
>          byte[] tmp = new byte[length];
>          int numbytes = in.read(offset, tmp, 0, length);
> -        HadoopThriftHandler.LOG.debug("read done: " + tout.id);
> -        return new String(tmp, 0, numbytes, "UTF-8");
> +        HadoopThriftHandler.LOG.debug("read done: " + tout.id +
> +                                     " numbytes: " + numbytes);
> +        return new String(tmp, 0, numbytes > 0 ? numbytes : 0, "UTF-8");
>        } catch (IOException e) {
>          throw new ThriftIOException(e.getMessage());
>        }
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message