Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0723D108CA for ; Mon, 14 Apr 2014 16:51:53 +0000 (UTC) Received: (qmail 53830 invoked by uid 500); 14 Apr 2014 16:51:48 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 53745 invoked by uid 500); 14 Apr 2014 16:51:48 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Delivered-To: moderator for user@hbase.apache.org Received: (qmail 7979 invoked by uid 99); 14 Apr 2014 16:36:19 -0000 X-ASF-Spam-Status: No, hits=2.4 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of siddharthajana24@gmail.com designates 209.85.192.45 as permitted sender) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=Q3MuYVN2ud001m17107aX7fOA1ck+nnvgD6kufFwP1s=; b=s/wFgfVpDgmkZ/lbOuGt8plb/yLJA2mc/wdqlxlDiQkX5IBZaddLywlW9rZCwZyvkJ YKizodCavTI5KYExBapC+GUXUAUFxIvwSH7pzemAOoo66gvwiXM/GzMcA9eLaaAhHCbX 4NlIlXFjIEGQlaMVhMxkU39c8aMSypqbRfrGrS3rP2tI2hBYONJi+QZu7nh4sblYj3+s 6j8fnjMBhRH57JJ+EIbRde1ARMqs02SMeZvnoWBILA4R/F7HriB6AwDgGOdBcfvQL5lU Gvhgfi6sYaGGIVtKvazBBlKjfQH8LVesbN4vWnzeXhTotsEruOLJDxJLfp+Xyudxx/yF VZKA== X-Received: by 10.229.54.201 with SMTP id r9mr52794804qcg.6.1397493353229; Mon, 14 Apr 2014 09:35:53 -0700 (PDT) MIME-Version: 1.0 From: Siddhartha Jana Date: Mon, 14 Apr 2014 11:35:13 -0500 Message-ID: Subject: RPC Timeout - DoNotRetryIOException To: user@hbase.apache.org Content-Type: multipart/alternative; boundary=001a1135ef48e534b304f703464e X-Virus-Checked: Checked by ClamAV on apache.org --001a1135ef48e534b304f703464e Content-Type: text/plain; charset=UTF-8 Hi, We have a 20-node cluster configured with HBase 0.98.1 While running some basic applications across our large HBase tables, I observe the DoNotRetryIOException being thrown repeatedly. This is observed after our Map-Reduce job makes a decent amount of progress. I see some RPC related bug-fixes mentioned in the release notes of this version and also see a similar bug-fix note in this link. Could some one please direct us to a solution / alternative HBase version? I have appended the stack trace below: Thanks, Siddhartha Jana University of Houston ====================== Error: org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of OutOfOrderScannerNextException: was there a rpc timeout? at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:384) at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:221) at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:138) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532) at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157) ====================== --001a1135ef48e534b304f703464e--