Return-Path: 
 <user-return-34883-apmail-hbase-user-archive=hbase.apache.org@hbase.apache.org>
X-Original-To: apmail-hbase-user-archive@www.apache.org
Delivered-To: apmail-hbase-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0ABBCF49C
	for <apmail-hbase-user-archive@www.apache.org>;
 Wed, 20 Mar 2013 08:01:58 +0000 (UTC)
Received: (qmail 54644 invoked by uid 500); 20 Mar 2013 08:01:55 -0000
Delivered-To: apmail-hbase-user-archive@hbase.apache.org
Received: (qmail 54112 invoked by uid 500); 20 Mar 2013 08:01:54 -0000
Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hbase.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hbase.apache.org>
List-Post: <mailto:user@hbase.apache.org>
List-Id: <user.hbase.apache.org>
Reply-To: user@hbase.apache.org
Delivered-To: mailing list user@hbase.apache.org
Received: (qmail 54044 invoked by uid 99); 20 Mar 2013 08:01:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Mar 2013 08:01:51 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of
 ramkrishna.s.vasudevan@gmail.com designates 209.85.216.181 as permitted
 sender)
Received: from [209.85.216.181] (HELO mail-qc0-f181.google.com)
 (209.85.216.181)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Mar 2013 08:01:29 +0000
Received: by mail-qc0-f181.google.com with SMTP id a22so683304qcs.12
        for <user@hbase.apache.org>; Wed, 20 Mar 2013 01:01:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:x-received:in-reply-to:references:date:message-id
         :subject:from:to:content-type;
        bh=9hq6ebnehkQmkGqeqUSGNryu4+NLGK8bovwT5G47N64=;
        b=0MvnuM9rxFQZwWnORhLULVgGTbcvF1Mb/CSkaRaqL4oLbSqWniPQGd+wPPL7eE23/G
         MnrHovUXLeuuKSk3arLuwvMjM6nAsEYZeqIe09R3SmBZCw0b3yqxcYvG8rYteAlnWAnz
         wF3qJcBR4mL5nGjVaqxVOsx+Wk43K/htvQBuWlhGT/D7cMwkc7HWAGHNp4LKBQAzc1jU
         /6Sjqhuv9DktGEwc9WIc9V8roQJIYi5NvZVKnR4j2kHZpImO8FPEOE6A7K1itofzre7z
         puB2CIy7kFr1iINiwQ9CKO6jNRdymeweZaJRUKovaVFg8RI+xJ+EellkmYex++GeUOHk
         uBNg==
MIME-Version: 1.0
X-Received: by 10.224.183.197 with SMTP id ch5mr5383217qab.27.1363766468248;
 Wed, 20 Mar 2013 01:01:08 -0700 (PDT)
Received: by 10.49.71.114 with HTTP; Wed, 20 Mar 2013 01:01:08 -0700 (PDT)
In-Reply-To: 
 <1FA690FF1F80B649BA7B8DFACF3070062DA4A6ED@EXTXMB19.nam.nsroot.net>
References: 
 <CAFLnt_pN_iyoUgBRsMGL3Qs26fgfbVPVxdBZNHwdZihPFM2-Xg@mail.gmail.com>
	<1FA690FF1F80B649BA7B8DFACF3070062330B7B3@EXTXMB19.nam.nsroot.net>
	<CALte62wV2Hs5miUE39Kwwh73kE8tMo5Q3GpGYdBMPNLKgHEg9w@mail.gmail.com>
	<CAFLnt_q3zse7eqELkFxT7oZ5WWnP9sihtk53o2mdR5LQ4Kc=-w@mail.gmail.com>
	<0CE69E9126D0344088798A3B7F7F80863AEE3740@SZXEML553-MBX.china.huawei.com>
	<1FA690FF1F80B649BA7B8DFACF3070062DA4A6ED@EXTXMB19.nam.nsroot.net>
Date: Wed, 20 Mar 2013 13:31:08 +0530
Message-ID: 
 <CAAT7MkqmSiA0+Z4+8_HRJY-Jkpuj-E0POTPaLBOsOdwu8Gopag@mail.gmail.com>
Subject: Re: Regionserver goes down while endpoint execution
From: ramkrishna vasudevan <ramkrishna.s.vasudevan@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=20cf303b4181e588d904d8569f7a
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf303b4181e588d904d8569f7a
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Deepak

"this.table.put(putInIndexTable);""

I think this one should be the problem.

Your table is at the instance level.   See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."

So if you try creating a new HTable every time this problem should be
avoided.

Regards
Ram

On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <deepak8.kumar@citi.com>wro=
te:

> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying t=
o
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocesso=
r:
> Counter: 2408
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocesso=
r:
> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:391=
3805817870658:
> vm-ab1f-dd21.nam.nsroot.net:
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocesso=
r:
> Counter: 2410
>
> 2013-03-20 02:40:42,122 ERROR
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnviron=
ment@2f86780d'
> from environment because it threw:  java.lang.IndexOutOfBoundsException:
> Index: 1, Size: 1
>
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>
>         at java.util.ArrayList.remove(ArrayList.java:387)
>
>         at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:960)
>
>         at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826)
>
>         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801)
>
>         at
> org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.jav=
a:394)
>
>         at
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocesso=
r.postPut(SecondaryIndexCoprocessor.java:91)
>
>
>
> If I synchronize on current object in the postPut then it works fine? If =
I
> use some counter valriable in the postput then it seems the counter misse=
s
> some value just before this exception. As you can see above 2409 is
> missing. But if I synchronize the postPut block then it executes fine & t=
he
> counter is also in serial order. But I  think using synchronize would slo=
w
> down the insertion in secondary index. Could you guide me the exact reaso=
n
> for this?
>
>
>
> Here is the complete code for postPut
>
>
>
> public void postPut(final ObserverContext<RegionCoprocessorEnvironment> e=
,
>
>                   final Put put, final WALEdit edit, final boolean
> writeToWAL) throws IOException {
>
>
>
>             List<KeyValue> hostName =3D put.get(Bytes.toBytes("sysInfo"),
> Bytes.toBytes("hostName"));
>
>             List<KeyValue> logFilePath =3D put.get(Bytes.toBytes("content=
"),
> Bytes.toBytes("logFilePath"));
>
>             List<KeyValue> logFileName =3D put.get(Bytes.toBytes("content=
"),
> Bytes.toBytes("logFileName"));
>
>
>
>             if(hostName.size() > 0 && logFilePath.size() > 0 &&
> logFileName.size() >0){
>
>                   byte[] hostNameVal =3D hostName.get(0).getValue();
>
>                   byte[] logFilePathVal =3D logFilePath.get(0).getValue()=
;
>
>                   byte[] logFileNameVal =3D logFileName.get(0).getValue()=
;
>
>                   byte[] rowKeyBody =3D hostName.get(0).getRow();
>
>                   synchronized(this){
>
>
>
>
>
>                   byte[] rowKey =3D (
>
>                               Bytes.toString(hostNameVal) + ":" +
> Bytes.toString(logFilePathVal) + Bytes.toString(logFileNameVal)
>
>                               + LOG_INDEX_DELIM +
> Bytes.toString(rowKeyBody)).getBytes();
>
>
>
>                   logger.debug("Row Key Secondary Index:
> "+Bytes.toString(rowKey));
>
>                   Put putInIndexTable =3D new Put(rowKey);
>
>
>
>                   putInIndexTable.add(LOG_INDEX_FAMILY_CORE.getBytes(),
> LOG_INDEX_ELFLOG_RK.getBytes(), Bytes.toBytes(dummyRowkey));
>
>                   dummyRowkey++;
>
>
>
>                   this.table.put(putInIndexTable);
>
>                   logger.info("Added in "+tableName+":
> "+Bytes.toString(rowKey));
>
>                   logger.info("Counter: "+dummyRowkey);
>
>                   }
>
>
>
>
>
>
>
>             }else{
>
>                   logger.error("Size of hostName"+hostName.size());
>
>                   logger.error("Size of logFilePath"+logFilePath.size());
>
>                   logger.error("Size of logFileName"+logFileName.size());
>
>                   logger.error("Size of logFileBody"+hostName.size());
>
>             }
>
>       }
>
>
>
>
>
> Regards,
>
> Deepak
>
> -----Original Message-----
> From: Anoop Sam John [mailto:anoopsj@huawei.com]
> Sent: Friday, March 15, 2013 2:56 AM
> To: user@hbase.apache.org
> Subject: RE: Regionserver goes down while endpoint execution
>
>
>
> Himanshu told it clearly. To make to more clear I am adding :)
>
>
>
> When the range of rowkeys that you are looking for spread across 5
> regions, at the client side there will be 5 exec requests created and
> submitted to a thread pool.[HBase client side thread pool associated with
> HTable] Now as per the availability of slots in this pool the operations
> will be requested to regions in RSs. So in different RS there can be
> parallel execution.  Within one RS itself parallel execution on different
> regions also possible.
>
>
>
> It is upto the client app to deal with the results what the Endpoint has
> returned. It will return a Map right? region name vs result from region.
> Client app need to take care about the ordering
>
>
>
> -Anoop-
>
>
>
> ________________________________________
>
> From: hv.csuoa@gmail.com<mailto:hv.csuoa@gmail.com> [hv.csuoa@gmail.com]
> on behalf of Himanshu Vashishtha [hvashish@cs.ualberta.ca]
>
> Sent: Thursday, March 14, 2013 11:15 PM
>
> To: user@hbase.apache.org<mailto:user@hbase.apache.org>
>
> Subject: Re: Regionserver goes down while endpoint execution
>
>
>
> There is no ordering guarantee for the endpoint execution, other than tha=
t
> calls will be executed in parallel across all the regions.
>
> In case you have 5 regions, then there will be 5 separate calls to these
> regions. Then, you get 5 results from these regions at your client, where
> you use the Callback class to aggregate the results. You can define your
> ordering this aggregate class for sure.
>
>
>
>
>
> On Thu, Mar 14, 2013 at 10:15 AM, Ted Yu <yuzhihong@gmail.com<mailto:
> yuzhihong@gmail.com>> wrote:
>
> > bq. provide the rowkey range as rowkey1 to rowkey100 in endpoint RPC
>
> > client
>
> >
>
> > If I understand you correctly, you perform batching at the client as
>
> > described above.
>
> > The order would be as you expected.
>
> >
>
> > Cheers
>
> >
>
> > On Thu, Mar 14, 2013 at 10:09 AM, Kumar, Deepak8 <deepak8.kumar@citi.co=
m
> <mailto:deepak8.kumar@citi.com>>wrote:
>
> >
>
> >>  Hi,****
>
> >>
>
> >> It seems due to huge data the RegionServer is getting down. Now I am
>
> >> trying to fetch the data in parts & is running fine. I need some more
>
> >> info about the Endpoint execution:****
>
> >>
>
> >> ** **
>
> >>
>
> >> My use case is to fetch the data from HBase as per some rowkey range
>
> >> & to render it at UI. Since endpoints are executed in parallel so I
>
> >> am looking to use it. ****
>
> >>
>
> >> ** **
>
> >>
>
> >> **=D8  **Suppose I provide the rowkey range as rowkey1 to rowkey100 in
>
> >> endpoint RPC client & these rowkeys are distributed at 5 regions
>
> >> across 4 region servers. If I fetch 10  records at a time, do we have
>
> >> any way to guarantee that it would come in serial order like first
>
> >> result would of
>
> >> rowkey1 to rowkey10, next time I set the start rowkey as rowkey11 &
>
> >> the fetch would be from rowkey11 to rowkey20, irrespective of the
>
> >> region & region servers?****
>
> >>
>
> >> ** **
>
> >>
>
> >> Regards,****
>
> >>
>
> >> Deepak****
>
> >>
>
> >> ** **
>
> >>
>
> >> ** **
>
> >>
>
> >> -----Original Message-----
>
> >> From: hv.csuoa@gmail.com<mailto:hv.csuoa@gmail.com> [mailto:
> hv.csuoa@gmail.com] On Behalf Of
>
> >> Himanshu Vashishtha
>
> >> Sent: Wednesday, March 13, 2013 12:09 PM
>
> >> To: user@hbase.apache.org<mailto:user@hbase.apache.org>
>
> >> Cc: Gary Helmling; yuzhihong@gmail.com<mailto:yuzhihong@gmail.com>;
> lars hofhansl
>
> >> Subject: Re: Regionserver goes down while endpoint execution
>
> >>
>
> >> ** **
>
> >>
>
> >> On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8
>
> >> <deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com>>
>
> >> wrote:****
>
> >>
>
> >> > Thanks guys for assisting. I am getting OOM exception yet. I have
>
> >> > one
>
> >> query about Endpoints. As endpoint executes in parallel, so if I have
>
> >> a table which is distributed at 101 regions across 5 regionserver.
>
> >> Would it be 101 threads of endpoint executing in parallel?****
>
> >>
>
> >> ** **
>
> >>
>
> >> No and Yes.****
>
> >>
>
> >> ** **
>
> >>
>
> >> The endpoints are not processed as separate threads, they are
>
> >> processed as just another request (via regionserver handlers). Yes,
>
> >> the execution will be in parallel in the sense that a separate client
>
> >> side call will be used for each of the regions that are in the range
>
> >> you specify.****
>
> >>
>
> >> ** **
>
> >>
>
> >> >** **
>
> >>
>
> >> > Regards,****
>
> >>
>
> >> > Deepak****
>
> >>
>
> >> >** **
>
> >>
>
> >> > From: Gary Helmling [mailto:ghelmling@gmail.com
>
> >> > <ghelmling@gmail.com<mailto:ghelmling@gmail.com>>]**
>
> >> **
>
> >>
>
> >> > Sent: Tuesday, March 12, 2013 2:14 PM****
>
> >>
>
> >> > To: user@hbase.apache.org****<mailto:user@hbase.apache.org****>
>
> >>
>
> >> > Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]****
>
> >>
>
> >> > Subject: Re: Regionserver goes down while endpoint execution****
>
> >>
>
> >> >** **
>
> >>
>
> >> > To expand on what Himanshu said, your endpoint is doing an
>
> >> > unbounded
>
> >> scan on the region, so with a region with a lot of rows it's taking
>
> >> more than 60 seconds to run to the region end, which is why the
>
> >> client side of the call is timing out.  In addition you're building
>
> >> up an in memory list of all the values for that qualifier in that
>
> >> region, which could cause you to bump into OOM issues, depending on
>
> >> how big your values are and how sparse the given column qualifier is.
>
> >> If you trigger an OOMException, then the region server would
>
> >> abort.****
>
> >>
>
> >> >** **
>
> >>
>
> >> > For this usage specifically, though -- scanning through a single
>
> >> > column
>
> >> qualifier for all rows -- you would be better off just doing a normal
>
> >> client side scan, ie. HTable.getScanner().  Then you will avoid the
>
> >> client timeout and potential server-side memory issues.****
>
> >>
>
> >> >** **
>
> >>
>
> >> > On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <
>
> >> yuzhihong@gmail.com<mailto:yuzhihong@gmail.com<mailto:
> yuzhihong@gmail.com%3cmailto:yuzhihong@gmail.com>>> wrote:****
>
> >>
>
> >> > From region server log:****
>
> >>
>
> >> >** **
>
> >>
>
> >> > 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient:
>
> >> > Error **
>
> >> **
>
> >>
>
> >> > making BlockReader. Closing stale ****
>
> >>
>
> >> > Socket[addr=3D/10.42.105.112<http://10.42.105.112>,port=3D50010,loca=
lpo
>
> >> > rt=3D**
>
> >> **
>
> >>
>
> >> > 54114]****
>
> >>
>
> >> > java.io.EOFException: Premature EOF: no length prefix available****
>
> >>
>
> >> >         at
>
> >> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoU
>
> >> til.java:162)
>
> >> ****
>
> >>
>
> >> >         at ****
>
> >>
>
> >> > org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBloc
>
> >> > kRe**
>
> >> **
>
> >>
>
> >> > ader2.java:407)****
>
> >>
>
> >> >** **
>
> >>
>
> >> > What version of HBase and hadoop are you using ?****
>
> >>
>
> >> > Do versions of hadoop on Eclipse machine and in your cluster match
>
> >> > ?****
>
> >>
>
> >> >** **
>
> >>
>
> >> > Cheers****
>
> >>
>
> >> >** **
>
> >>
>
> >> > On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <
>
> >> deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com
> >>wrote:****<mailto:deepak8.kumar@citi.com%3cmailto:deepak8.kumar@citi.co=
m
> %3e%3ewrote:****>
>
> >>
>
> >> >>  Lars,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> I am getting following errors at datanode & region
>
> >> >> servers.********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Regards,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Deepak********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> *From:* Kumar, Deepak8 [CCC-OT_IT NE]****
>
> >>
>
> >> >> *Sent:* Tuesday, March 12, 2013 3:00 AM****
>
> >>
>
> >> >> *To:* Kumar, Deepak8 [CCC-OT_IT NE]; ****
>
> >>
>
> >> >> 'user@hbase.apache.org<mailto:user@hbase.apache.org<user@hbase.apa
>
> >> >> che.org>>';
>
> >> 'lars hofhansl'****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> *Subject:* RE: Regionserver goes down while endpoint
>
> >> >> execution********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>  ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Lars,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> It is having following errors when I execute the Endpoint RPC
>
> >> >> client **
>
> >> **
>
> >>
>
> >> >> from eclipse. It seems some of the regions at regionserver ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net<http://vm-8aa9-fe74.nam.nsroot.net> is
>
> >> >> ****
>
> >>
>
> >> >> taking more time to reponse.********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Could you guide how to fix it. I don't find any option to set ****
>
> >>
>
> >> >> hbase.rpc.timeout from hbase configuration menu in CDH4 CM server
>
> >> >> for *
>
> >> ***
>
> >>
>
> >> >> hbase configuration.******
>
> >>
>
> >> >> ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Regards,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Deepak********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment
>
> >> >> ****
>
> >>
>
> >> >> complete on server ****
>
> >>
>
> >> >> vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181<http://vm-15c2-3bbf.
>
> >> >> nam**
>
> >> **
>
> >>
>
> >> >> .nsroot.net/10.96.172.44:2181>, sessionid =3D 0x53d591b77090026,
>
> >> >> ****
>
> >>
>
> >> >> negotiated timeout =3D 60000********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration****
>
> >>
>
> >> >> warnOnceIfDeprecated********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> WARNING: hadoop.native.lib is deprecated. Instead, use****
>
> >>
>
> >> >> io.native.lib.available********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Mar 12, 2013 2:44:00 AM****
>
> >>
>
> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >> >> men**
>
> >> **
>
> >>
>
> >> >> tation****
>
> >>
>
> >> >> processExecs********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> WARNING: Error executing for row
> 153299:1362780381523:2932572079500658:
>
> >> ****
>
> >>
>
> >> >> vm-ab1f-dd21.nam.nsroot.net:********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> *java.util.concurrent.ExecutionException*: *****
>
> >>
>
> >> >> org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed
>
> >> >> ****
>
> >>
>
> >> >> after attempts=3D10, exceptions:********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:34:15 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2271remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:35:16 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2403remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:36:18 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2465remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:37:20 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2500remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:38:22 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2538remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:39:25 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2572remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:40:30 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2606remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:41:34 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2640remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:42:43 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2677remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:44:00 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2842remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.util.concurrent.FutureTask$Sync.innerGet(Unknown
>
> >> >> ****
>
> >>
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.util.concurrent.FutureTask.get(Unknown
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >> >> men**
>
> >> **
>
> >>
>
> >> >> tation.processExecs(****
>
> >>
>
> >> >> *HConnectionManager.java:1466*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at
>
> >> >> org.apache.hadoop.hbase.client.HTable.coprocessorExec(*****
>
> >>
>
> >> >> HTable.java:1577*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at
>
> >> >> org.apache.hadoop.hbase.client.HTable.coprocessorExec(*****
>
> >>
>
> >> >> HTable.java:1557*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at ****
>
> >>
>
> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog.main(*
>
> >> >> ***
>
> >>
>
> >> >> *HBaseEndPointClientForElfLog.java:33*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Caused by:
>
> >> >> *org.apache.hadoop.hbase.client.RetriesExhaustedException*:*
>
> >> ***
>
> >>
>
> >> >> Failed after attempts=3D10, exceptions:********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:34:15 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2271remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:35:16 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2403remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:36:18 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2465remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:37:20 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2500remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:38:22 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2538remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:39:25 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2572remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:40:30 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2606remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:41:34 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2640remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:42:43 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2677remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:44:00 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2842remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >> >> men**
>
> >> **
>
> >>
>
> >> >> tation.getRegionServerWithRetries(****
>
> >>
>
> >> >> *HConnectionManager.java:1345*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(*****
>
> >>
>
> >> >> ExecRPCInvoker.java:79*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at $Proxy8.getValues(Unknown Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call
>
> >> >> (****
>
> >> *
>
> >>
>
> >> >> HBaseEndPointClientForElfLog.java:38*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call
>
> >> >> (****
>
> >> *
>
> >>
>
> >> >> HBaseEndPointClientForElfLog.java:1*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >> >> men**
>
> >> **
>
> >>
>
> >> >> tation$4.call(****
>
> >>
>
> >> >> *HConnectionManager.java:1454*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.util.concurrent.FutureTask$Sync.innerRun(Unknown
>
> >> >> ****
>
> >>
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.util.concurrent.FutureTask.run(Unknown
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at ****
>
> >>
>
> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown****
>
> >>
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at
>
> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown ***
>
> >> *
>
> >>
>
> >> >> Source)****
>
> >>
>
> >> >> ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.lang.Thread.run(Unknown Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> *org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed
>
> >> >> ****
>
> >>
>
> >> >> after attempts=3D10, exceptions:********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:34:15 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2271remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:35:16 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2403remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:36:18 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2465remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:37:20 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2500remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:38:22 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2538remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:39:25 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2572remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:40:30 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2606remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:41:34 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2640remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:42:43 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2677remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:44:00 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=3D/150.110.96.212:2842remote=3D ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >> >> men**
>
> >> **
>
> >>
>
> >> >> tation.getRegionServerWithRetries(****
>
> >>
>
> >> >> *HConnectionManager.java:1345*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(*****
>
> >>
>
> >> >> ExecRPCInvoker.java:79*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at $Proxy8.getValues(Unknown Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call
>
> >> >> (****
>
> >> *
>
> >>
>
> >> >> HBaseEndPointClientForElfLog.java:38*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call
>
> >> >> (****
>
> >> *
>
> >>
>
> >> >> HBaseEndPointClientForElfLog.java:1*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >> >> men**
>
> >> **
>
> >>
>
> >> >> tation$4.call(****
>
> >>
>
> >> >> *HConnectionManager.java:1454*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.util.concurrent.FutureTask$Sync.innerRun(Unknown
>
> >> >> ****
>
> >>
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.util.concurrent.FutureTask.run(Unknown
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at ****
>
> >>
>
> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown****
>
> >>
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at
>
> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown ***
>
> >> *
>
> >>
>
> >> >> Source)****
>
> >>
>
> >> >> ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.lang.Thread.run(Unknown Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> *From:* Kumar, Deepak8 [CCC-OT_IT NE]****
>
> >>
>
> >> >> *Sent:* Tuesday, March 12, 2013 2:27 AM****
>
> >>
>
> >> >> *To:*
>
> >> >> 'user@hbase.apache.org<mailto:user@hbase.apache.org<user@hbase.apa
>
> >> >> che.org>>';
>
> >> 'lars hofhansl'****
>
> >>
>
> >> >> *Subject:* RE: Regionserver goes down while endpoint
>
> >> >> execution********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Lars,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Thanks for your quick response.There is not much info in region
>
> >> >> ****
>
> >>
>
> >> >> server log. I am again executing it with DEBUG log level in region
>
> >> >> ****
>
> >>
>
> >> >> servers.********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> *Here is the endpoint code*****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> public class ColumnAggregationEndpoint extends ****
>
> >>
>
> >> >> BaseEndpointCoprocessor********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> implements  ColumnAggregationProtocol {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       @Override********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>         public List<String> getValues(byte[] family, byte[] ****
>
> >>
>
> >> >> qualifier, int batchSize, int cacheSize)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>         throws IOException {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           // aggregate at each region********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           Scan scan =3D new Scan();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           scan.addColumn(family, qualifier);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           scan.setCaching(cacheSize);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           scan.setBatch(batchSize);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           List<String> values =3D new ArrayList<String>();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           RegionCoprocessorEnvironment environment =3D********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   (RegionCoprocessorEnvironment) ** **
>
> >>
>
> >> >> getEnvironment();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           InternalScanner scanner =3D****
>
> >>
>
> >> >> environment.getRegion().getScanner(scan);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           try {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>             List<KeyValue> curVals =3D new
>
> >> >> ArrayList<KeyValue>();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>             boolean hasMore =3D false;********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>             do {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           curVals.clear();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           hasMore =3D scanner.next(curVals);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           KeyValue kv =3D curVals.get(0);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           values.add(Bytes.toString(kv.getValue()));********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>             } while (hasMore);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           } finally {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>               scanner.close();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           return values;********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>         }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> *The RPC client to invoke the Endpoint is as follows:*****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> public class HBaseEndPointClientForElfLog {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       public static void main(String[] args) {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>             try {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   Configuration conf =3D ****
>
> >>
>
> >> >> HBaseConfiguration.create();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   conf.set(********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                               "hbase.zookeeper.quorum",********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                               ****
>
> >>
>
> >> >> "vm-ab1f-dd21.nam.nsroot.net<http://vm-ab1f-dd21.nam.nsroot.net>,*
>
> >> >> ***
>
> >>
>
> >> >> vm-cb03-2277.nam.nsroot.net<http://vm-cb03-2277.nam.nsroot.net>,vm
>
> >> >> -15**
>
> >> **
>
> >>
>
> >> >> c2-3bbf.nam.nsroot.net<http://vm-15c2-3bbf.nam.nsroot.net>");*****
>
> >> >> ***
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   String tableName =3D "elf_log";********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   final String columnFamily =3D "content";********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   final String columnQualifier =3D
>
> >> >> "logFileName";********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   final String startRowKey =3D****
>
> >>
>
> >> >> "153299:1362780381523:2932572079500658:vm-ab1f-dd21.nam.nsroot.net
>
> >> >> <ht**
>
> >> **
>
> >>
>
> >> >> tp://vm-ab1f-dd21.nam.nsroot.net>:";********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   final String endRowKey =3D ****
>
> >>
>
> >> >> "153299:1362953388000";********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   HTableInterface table =3D new HTable(conf, ****
>
> >>
>
> >> >> tableName);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   Scan scan;********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   Map<byte[], List<String>> results;********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   // scan: for all regions********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   scan =3D new Scan();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   results =3D****
>
> >>
>
> >> >> table.coprocessorExec(ColumnAggregationProtocol.class,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                               startRowKey.getBytes(), ** **
>
> >>
>
> >> >> endRowKey.getBytes(),****
>
> >>
>
> >> >> ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                               new ****
>
> >>
>
> >> >> Batch.Call<ColumnAggregationProtocol,****
>
> >>
>
> >> >> List<String>>() {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                                     public List<String> ****
>
> >>
>
> >> >> call(ColumnAggregationProtocol instance)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                                                 throws IOException
>
> >> >> ****
>
> >>
>
> >> >> {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                                           return****
>
> >>
>
> >> >> instance.getValues(columnFamily.getBytes(),********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> columnQualifier.getBytes(),2,5);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                                     }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                               });********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   for (Map.Entry<byte[], List<String>> e :****
>
> >>
>
> >> >> results.entrySet()) {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                         System.out.println("Size of list
>
> >> >> returned:****
>
> >>
>
> >> >> "+e.getValue().size());********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                         for(String singleVal:
>
> >> >> e.getValue()){********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>
>
> >> >> System.out.println(singleVal);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                         }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                         ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                         ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>             } catch (Throwable throwable) {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   throwable.printStackTrace();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>             }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Regards,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Deepak********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> -----Original Message-----****
>
> >>
>
> >> >> From: lars hofhansl
>
> >> >> [mailto:larsh@apache.org<mailto:larsh@apache.org> *
>
> >> ***
>
> >>
>
> >> >> <larsh@apache.org<mailto:larsh@apache.org<mailto:larsh@apache.org%
> 3cmailto:larsh@apache.org>>>]****
>
> >>
>
> >> >> Sent: Tuesday, March 12, 2013 2:01 AM****
>
> >>
>
> >> >> To: user@hbase.apache.org<mailto:user@hbase.apache.org>****<mailto:
> user@hbase.apache.org%3cmailto:user@hbase.apache.org%3e****>
>
> >>
>
> >> >> Subject: Re: Regionserver goes down while endpoint
>
> >> >> execution********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> What does the region server log say?********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Endpoints do not run in a sandbox. You can call System.exit(...)
>
> >> >> and **
>
> >> **
>
> >>
>
> >> >> your RegionServer will happily exit.********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> If you can, please show us your endpoint code.********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> -- Lars********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ________________________________********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> From: "Kumar, Deepak8 " ****
>
> >>
>
> >> >> <deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com<mailto:
> deepak8.kumar@citi.com%3cmailto:deepak8.kumar@citi.com>>>********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> To: "'user@hbase.apache.org<mailto:user@hbase.apache.org<
> user@hbase.apache.org<mailto:user@hbase.apache.org%3cuser@hbase.apache.or=
g
> >>>'"
>
> >> ****
>
> >>
>
> >> >> <user@hbase.apache.org<mailto:user@hbase.apache.org<mailto:
> user@hbase.apache.org%3cmailto:user@hbase.apache.org>>> ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Sent: Monday, March 11, 2013 10:51 PM********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Subject: Regionserver goes down while endpoint execution********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Hi,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> I have a table in hbase which has more than 5GB of data, it is
>
> >> >> ****
>
> >>
>
> >> >> distributed at 101 regions at 5 regionservers.********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> When I execute an endpoint which is supposed to fetch a column
>
> >> >> ****
>
> >>
>
> >> >> qualifier value using an endpoint RPC client, the region server
>
> >> >> goes **
>
> >> **
>
> >>
>
> >> >> down. The hbase master log says "Can't connect to region,
>
> >> >> retrying.." *
>
> >> ***
>
> >>
>
> >> >> The same endpoint works fine for tables which has 300
>
> >> >> records.********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Could you please guide me the reason for being regionserver
>
> >> >> down?******
>
> >> **
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Regards,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Deepak********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >** **
>
> >>
>

--20cf303b4181e588d904d8569f7a--