Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B5FDF200C3F for ; Tue, 7 Mar 2017 20:31:43 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id B4E11160B82; Tue, 7 Mar 2017 19:31:43 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id E1044160B74 for ; Tue, 7 Mar 2017 20:31:42 +0100 (CET) Received: (qmail 35351 invoked by uid 500); 7 Mar 2017 19:31:35 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 35325 invoked by uid 99); 7 Mar 2017 19:31:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Mar 2017 19:31:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id EE037C0C0D for ; Tue, 7 Mar 2017 19:31:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.399 X-Spam-Level: ** X-Spam-Status: No, score=2.399 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id kDxPNqRyPGYo for ; Tue, 7 Mar 2017 19:31:33 +0000 (UTC) Received: from mail-io0-f180.google.com (mail-io0-f180.google.com [209.85.223.180]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 733415F297 for ; Tue, 7 Mar 2017 19:31:33 +0000 (UTC) Received: by mail-io0-f180.google.com with SMTP id f84so11633004ioj.0 for ; Tue, 07 Mar 2017 11:31:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=g75+fPzIVYCLn2eGeDl2rn3691See7lTkE1f0CRRvMY=; b=NWtlZm1AAcOJKy0FUBLmVmieclkIiJzC1F7wcLwSv43RKEFMXnbJxNvlUD2UGriLNa AiDwcZlZZOF0Jrb2mS7p/L7TP/txkSmrZ/95PJMjYzSPWniaYL8XvGJqarbxZWiPcBl6 om17idiqoUvCFBLiEKgO9GyEXAUua6JoyrHFabPlpbrx0UEEHimdusxOqL34k7BHUxgf e4XNesvwh3VXCd9Avo7gLl5HBdW0t5I3XIW7QQFsbjiVok/Y2IF3FtK/oj9JGUwfz/nC lJ/y3v9Q8cAPuqAqP6Fyx/ygbaCUoGQpdLn/mdPVb+GWSNHRekXj88l7/8MjjflvGYeK KkmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=g75+fPzIVYCLn2eGeDl2rn3691See7lTkE1f0CRRvMY=; b=MBhor1JBznRmONtiOeqcmxDDxAG3ypNFgdZGpbJYvJXsHXQ4zgOdnW0WCrBGYhwJlZ 1vOzZGyKC+ANvGAVxTPIkr/MFCMrH648vdVO2qcoEheeU1f01z149lFlRR3Vms8B5ThO BB7qzp1ne0axlhzu95L5qk28II2UaglEkEa+cUVOLDlPxas3we67RUFwaXUhz2UYafTJ kudEY1vkSIqwfFzZ2zrZRptuhoqO9mIh2wVYwsC5P6Q0iXVnGPL3GYjm3asiAyUqYgU1 gOqvS+OblKemLRyPHx/TuRfEMO0c8d6+qCzJH3MERZ/NndYYlbrxGxPa64m9BoUfxl2v OauQ== X-Gm-Message-State: AMke39lRyTxQXXxuyHWed180MJaQsHxKHIxuTk5dl0ytTbTSlRkpVIg7Jp7zytv6lO4kmd7zunZi8RaUv6b71g== X-Received: by 10.107.200.151 with SMTP id y145mr2498330iof.171.1488915092090; Tue, 07 Mar 2017 11:31:32 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.8.99 with HTTP; Tue, 7 Mar 2017 11:31:31 -0800 (PST) In-Reply-To: <58b85e47.15dc630a.947c2.567c@mx.google.com> References: <58b85e47.15dc630a.947c2.567c@mx.google.com> From: Ravi Prakash Date: Tue, 7 Mar 2017 11:31:31 -0800 Message-ID: Subject: Re: last exception: java.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on local exception To: Motty Cruz Cc: user Content-Type: multipart/alternative; boundary=94eb2c0b8cdc2a3411054a290f92 archived-at: Tue, 07 Mar 2017 19:31:43 -0000 --94eb2c0b8cdc2a3411054a290f92 Content-Type: text/plain; charset=UTF-8 You should probably email Hbase mailing lists rather than Hadoop : https://hbase.apache.org/mail-lists.html On Thu, Mar 2, 2017 at 10:02 AM, Motty Cruz wrote: > Hello, in the past two weeks, I see the following error on HBase Thrift > servers, we have total of about 10 Thrift servers and randomly get the > following errors: > > > > 2017-02-28 10:45:56,541 INFO org.apache.hadoop.hbase.client.AsyncProcess: > #4087940, table=MBData, attempt=11/35 failed=2ops, last exception: ja > > va.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on > local exception: org.apache.hadoop.hbase.ipc.CallTimeoutExcepti > > on: Call id=36009061, waitTime=180001, operationTimeout=180000 expired. on > e26-node.fqdn.com,60020,1487800633128, tracking started null, > > retrying after=10060ms, replay=2ops > > 2017-02-28 10:45:57,207 INFO org.apache.hadoop.hbase.client.AsyncProcess: > #4084675, waiting for some tasks to finish. Expected max=0, tasksInPro > > gress=22 > > > > Restarting the Thrift server resolves the issue. Any ideas where should I > be looking for? > > > > Thanks, > Motty > --94eb2c0b8cdc2a3411054a290f92 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
You should probably email Hbase mailing lists rather than = Hadoop : https://hbase= .apache.org/mail-lists.html

On Thu, Mar 2, 2017 at 10:02 AM, Motty Cruz <mo= tty.cruz@gmail.com> wrote:
=

Hello, in the past two wee= ks, I see the following error on HBase Thrift servers, we have total of abo= ut 10 Thrift servers and randomly get the following errors:

=C2=A0

201= 7-02-28 10:45:56,541 INFO org.apache.hadoop.hbase.client.AsyncProcess:= #4087940, table=3DMBData, attempt=3D11/35 failed=3D2ops, last exception: j= a

va.io.IOException: Call to e26-node= .fqdn.com/10.12.1.209:60020 failed on local exception: org.apache.= hadoop.hbase.ipc.CallTimeoutExcepti

on: Call id=3D36009061, waitTime=3D180001, operationTimeout=3D180000 = expired. on e26-node= .fqdn.com,60020,1487800633128, tracking started null,

retrying after=3D10060ms, replay=3D2ops=

2017-02-28 10:45:57,207 INFO org.apache.h= adoop.hbase.client.AsyncProcess: #4084675, waiting for some tasks to f= inish. Expected max=3D0, tasksInPro

gress=3D22

=C2=A0

Restarting the Thrift server resolves the issue. An= y ideas where should I be looking for?

=C2=A0

Thanks,
Motty


--94eb2c0b8cdc2a3411054a290f92--