hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kumar, Deepak8 " <deepak8.ku...@citi.com>
Subject RE: Regionserver goes down while endpoint execution
Date Wed, 20 Mar 2013 12:44:00 GMT
Hi,

Even when I moved table = HbaseTableAccess.getInstance().connectTable(tableName); to postPut method of coprocessor, I am getting the same exception.



Regards,

Deepak



-----Original Message-----
From: Anoop Sam John [mailto:anoopsj@huawei.com]
Sent: Wednesday, March 20, 2013 4:37 AM
To: user@hbase.apache.org
Subject: RE: Regionserver goes down while endpoint execution



Hi Deepak

                      Yes Ram told it correctly regarding the thread safety issue with HTable.

I can see you issue HTable#put() to write to index table and this call you are making from a CP.

Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.



In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl.  If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.



-Anoop-

________________________________________

From: ramkrishna vasudevan [ramkrishna.s.vasudevan@gmail.com]

Sent: Wednesday, March 20, 2013 1:31 PM

To: user@hbase.apache.org<mailto:user@hbase.apache.org>

Subject: Re: Regionserver goes down while endpoint execution



Hi Deepak



"this.table.put(putInIndexTable);""



I think this one should be the problem.



Your table is at the instance level.   See the documentation of HTable.

"<p>This class is not thread safe for reads nor write."



So if you try creating a new HTable every time this problem should be

avoided.



Regards

Ram



On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com>>wrote:



> Hi Anoop,

>

> Quite inspired by your coprocessor secondary indexing document & trying to

> implement one, for better response :)

>

>

>

> The coprocessor executes for some time, but later on (say after 400-500

> inserts) it gives IndexOutofBounds exception.

>

>

>

> The stack trace is

>

> 2013-03-20 02:40:42,074 INFO

> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:

> Counter: 2408

>

> 2013-03-20 02:40:42,098 INFO

> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:

> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:

> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:

> vm-ab1f-dd21.nam.nsroot.net:

>

> 2013-03-20 02:40:42,098 INFO

> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:

> Counter: 2410

>

> 2013-03-20 02:40:42,122 ERROR

> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor

> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d'

> from environment because it threw:  java.lang.IndexOutOfBoundsException:

> Index: 1, Size: 1

>

> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1

>

>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)

>

>         at java.util.ArrayList.remove(ArrayList.java:387)

>

>         at

> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:960)

>

>         at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826)

>

>         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801)

>

>         at

> org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:394)

>

>         at

> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor.postPut(SecondaryIndexCoprocessor.java:91)

>

>

>

> If I synchronize on current object in the postPut then it works fine? If I

> use some counter valriable in the postput then it seems the counter misses

> some value just before this exception. As you can see above 2409 is

> missing. But if I synchronize the postPut block then it executes fine & the

> counter is also in serial order. But I  think using synchronize would slow

> down the insertion in secondary index. Could you guide me the exact reason

> for this?

>

>

>

> Here is the complete code for postPut

>

>

>

> public void postPut(final ObserverContext<RegionCoprocessorEnvironment> e,

>

>                   final Put put, final WALEdit edit, final boolean

> writeToWAL) throws IOException {

>

>

>

>             List<KeyValue> hostName = put.get(Bytes.toBytes("sysInfo"),

> Bytes.toBytes("hostName"));

>

>             List<KeyValue> logFilePath = put.get(Bytes.toBytes("content"),

> Bytes.toBytes("logFilePath"));

>

>             List<KeyValue> logFileName = put.get(Bytes.toBytes("content"),

> Bytes.toBytes("logFileName"));

>

>

>

>             if(hostName.size() > 0 && logFilePath.size() > 0 &&

> logFileName.size() >0){

>

>                   byte[] hostNameVal = hostName.get(0).getValue();

>

>                   byte[] logFilePathVal = logFilePath.get(0).getValue();

>

>                   byte[] logFileNameVal = logFileName.get(0).getValue();

>

>                   byte[] rowKeyBody = hostName.get(0).getRow();

>

>                   synchronized(this){

>

>

>

>

>

>                   byte[] rowKey = (

>

>                               Bytes.toString(hostNameVal) + ":" +

> Bytes.toString(logFilePathVal) + Bytes.toString(logFileNameVal)

>

>                               + LOG_INDEX_DELIM +

> Bytes.toString(rowKeyBody)).getBytes();

>

>

>

>                   logger.debug("Row Key Secondary Index:

> "+Bytes.toString(rowKey));

>

>                   Put putInIndexTable = new Put(rowKey);

>

>

>

>                   putInIndexTable.add(LOG_INDEX_FAMILY_CORE.getBytes(),

> LOG_INDEX_ELFLOG_RK.getBytes(), Bytes.toBytes(dummyRowkey));

>

>                   dummyRowkey++;

>

>

>

>                   this.table.put(putInIndexTable);

>

>                   logger.info("Added in "+tableName+":

> "+Bytes.toString(rowKey));

>

>                   logger.info("Counter: "+dummyRowkey);

>

>                   }

>

>

>

>

>

>

>

>             }else{

>

>                   logger.error("Size of hostName"+hostName.size());

>

>                   logger.error("Size of logFilePath"+logFilePath.size());

>

>                   logger.error("Size of logFileName"+logFileName.size());

>

>                   logger.error("Size of logFileBody"+hostName.size());

>

>             }

>

>       }

>

>

>

>

>

> Regards,

>

> Deepak

>

> -----Original Message-----

> From: Anoop Sam John [mailto:anoopsj@huawei.com]

> Sent: Friday, March 15, 2013 2:56 AM

> To: user@hbase.apache.org<mailto:user@hbase.apache.org>

> Subject: RE: Regionserver goes down while endpoint execution

>

>

>

> Himanshu told it clearly. To make to more clear I am adding :)

>

>

>

> When the range of rowkeys that you are looking for spread across 5

> regions, at the client side there will be 5 exec requests created and

> submitted to a thread pool.[HBase client side thread pool associated with

> HTable] Now as per the availability of slots in this pool the operations

> will be requested to regions in RSs. So in different RS there can be

> parallel execution.  Within one RS itself parallel execution on different

> regions also possible.

>

>

>

> It is upto the client app to deal with the results what the Endpoint has

> returned. It will return a Map right? region name vs result from region.

> Client app need to take care about the ordering

>

>

>

> -Anoop-

>

>

>

> ________________________________________

>

> From: hv.csuoa@gmail.com<mailto:hv.csuoa@gmail.com<mailto:hv.csuoa@gmail.com%3cmailto:hv.csuoa@gmail.com>> [hv.csuoa@gmail.com]

> on behalf of Himanshu Vashishtha [hvashish@cs.ualberta.ca]

>

> Sent: Thursday, March 14, 2013 11:15 PM

>

> To: user@hbase.apache.org<mailto:user@hbase.apache.org<mailto:user@hbase.apache.org%3cmailto:user@hbase.apache.org>>

>

> Subject: Re: Regionserver goes down while endpoint execution

>

>

>

> There is no ordering guarantee for the endpoint execution, other than that

> calls will be executed in parallel across all the regions.

>

> In case you have 5 regions, then there will be 5 separate calls to these

> regions. Then, you get 5 results from these regions at your client, where

> you use the Callback class to aggregate the results. You can define your

> ordering this aggregate class for sure.

>

>

>

>

>

> On Thu, Mar 14, 2013 at 10:15 AM, Ted Yu <yuzhihong@gmail.com<mailto:

> yuzhihong@gmail.com<mailto:yuzhihong@gmail.com>>> wrote:

>

> > bq. provide the rowkey range as rowkey1 to rowkey100 in endpoint RPC

>

> > client

>

> >

>

> > If I understand you correctly, you perform batching at the client as

>

> > described above.

>

> > The order would be as you expected.

>

> >

>

> > Cheers

>

> >

>

> > On Thu, Mar 14, 2013 at 10:09 AM, Kumar, Deepak8 <deepak8.kumar@citi.com

> <mailto:deepak8.kumar@citi.com>>wrote:

>

> >

>

> >>  Hi,****

>

> >>

>

> >> It seems due to huge data the RegionServer is getting down. Now I am

>

> >> trying to fetch the data in parts & is running fine. I need some more

>

> >> info about the Endpoint execution:****

>

> >>

>

> >> ** **

>

> >>

>

> >> My use case is to fetch the data from HBase as per some rowkey range

>

> >> & to render it at UI. Since endpoints are executed in parallel so I

>

> >> am looking to use it. ****

>

> >>

>

> >> ** **

>

> >>

>

> >> **Ø  **Suppose I provide the rowkey range as rowkey1 to rowkey100 in

>

> >> endpoint RPC client & these rowkeys are distributed at 5 regions

>

> >> across 4 region servers. If I fetch 10  records at a time, do we have

>

> >> any way to guarantee that it would come in serial order like first

>

> >> result would of

>

> >> rowkey1 to rowkey10, next time I set the start rowkey as rowkey11 &

>

> >> the fetch would be from rowkey11 to rowkey20, irrespective of the

>

> >> region & region servers?****

>

> >>

>

> >> ** **

>

> >>

>

> >> Regards,****

>

> >>

>

> >> Deepak****

>

> >>

>

> >> ** **

>

> >>

>

> >> ** **

>

> >>

>

> >> -----Original Message-----

>

> >> From: hv.csuoa@gmail.com<mailto:hv.csuoa@gmail.com<mailto:hv.csuoa@gmail.com%3cmailto:hv.csuoa@gmail.com>> [mailto:

> hv.csuoa@gmail.com<mailto:hv.csuoa@gmail.com>] On Behalf Of

>

> >> Himanshu Vashishtha

>

> >> Sent: Wednesday, March 13, 2013 12:09 PM

>

> >> To: user@hbase.apache.org<mailto:user@hbase.apache.org<mailto:user@hbase.apache.org%3cmailto:user@hbase.apache.org>>

>

> >> Cc: Gary Helmling; yuzhihong@gmail.com<mailto:yuzhihong@gmail.com<mailto:yuzhihong@gmail.com%3cmailto:yuzhihong@gmail.com>>;

> lars hofhansl

>

> >> Subject: Re: Regionserver goes down while endpoint execution

>

> >>

>

> >> ** **

>

> >>

>

> >> On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8

>

> >> <deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com%3cmailto:deepak8.kumar@citi.com>>>

>

> >> wrote:****

>

> >>

>

> >> > Thanks guys for assisting. I am getting OOM exception yet. I have

>

> >> > one

>

> >> query about Endpoints. As endpoint executes in parallel, so if I have

>

> >> a table which is distributed at 101 regions across 5 regionserver.

>

> >> Would it be 101 threads of endpoint executing in parallel?****

>

> >>

>

> >> ** **

>

> >>

>

> >> No and Yes.****

>

> >>

>

> >> ** **

>

> >>

>

> >> The endpoints are not processed as separate threads, they are

>

> >> processed as just another request (via regionserver handlers). Yes,

>

> >> the execution will be in parallel in the sense that a separate client

>

> >> side call will be used for each of the regions that are in the range

>

> >> you specify.****

>

> >>

>

> >> ** **

>

> >>

>

> >> >** **

>

> >>

>

> >> > Regards,****

>

> >>

>

> >> > Deepak****

>

> >>

>

> >> >** **

>

> >>

>

> >> > From: Gary Helmling [mailto:ghelmling@gmail.com

>

> >> > <ghelmling@gmail.com<mailto:ghelmling@gmail.com<mailto:ghelmling@gmail.com%3cmailto:ghelmling@gmail.com>>>]**

>

> >> **

>

> >>

>

> >> > Sent: Tuesday, March 12, 2013 2:14 PM****

>

> >>

>

> >> > To: user@hbase.apache.org****<mailto:user@hbase.apache.org****<mailto:user@hbase.apache.org****%3cmailto:user@hbase.apache.org****>>

>

> >>

>

> >> > Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]****

>

> >>

>

> >> > Subject: Re: Regionserver goes down while endpoint execution****

>

> >>

>

> >> >** **

>

> >>

>

> >> > To expand on what Himanshu said, your endpoint is doing an

>

> >> > unbounded

>

> >> scan on the region, so with a region with a lot of rows it's taking

>

> >> more than 60 seconds to run to the region end, which is why the

>

> >> client side of the call is timing out.  In addition you're building

>

> >> up an in memory list of all the values for that qualifier in that

>

> >> region, which could cause you to bump into OOM issues, depending on

>

> >> how big your values are and how sparse the given column qualifier is.

>

> >> If you trigger an OOMException, then the region server would

>

> >> abort.****

>

> >>

>

> >> >** **

>

> >>

>

> >> > For this usage specifically, though -- scanning through a single

>

> >> > column

>

> >> qualifier for all rows -- you would be better off just doing a normal

>

> >> client side scan, ie. HTable.getScanner().  Then you will avoid the

>

> >> client timeout and potential server-side memory issues.****

>

> >>

>

> >> >** **

>

> >>

>

> >> > On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <

>

> >> yuzhihong@gmail.com<mailto:yuzhihong@gmail.com<mailto<mailto:yuzhihong@gmail.com%3cmailto:yuzhihong@gmail.com%3cmailto>:

> yuzhihong@gmail.com%3cmailto:yuzhihong@gmail.com<mailto:yuzhihong@gmail.com%3cmailto:yuzhihong@gmail.com>>>> wrote:****

>

> >>

>

> >> > From region server log:****

>

> >>

>

> >> >** **

>

> >>

>

> >> > 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient:

>

> >> > Error **

>

> >> **

>

> >>

>

> >> > making BlockReader. Closing stale ****

>

> >>

>

> >> > Socket[addr=/10.42.105.112<http://10.42.105.112>,port=50010,localpo

>

> >> > rt=**

>

> >> **

>

> >>

>

> >> > 54114]****

>

> >>

>

> >> > java.io.EOFException: Premature EOF: no length prefix available****

>

> >>

>

> >> >         at

>

> >> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoU

>

> >> til.java:162)

>

> >> ****

>

> >>

>

> >> >         at ****

>

> >>

>

> >> > org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBloc

>

> >> > kRe**

>

> >> **

>

> >>

>

> >> > ader2.java:407)****

>

> >>

>

> >> >** **

>

> >>

>

> >> > What version of HBase and hadoop are you using ?****

>

> >>

>

> >> > Do versions of hadoop on Eclipse machine and in your cluster match

>

> >> > ?****

>

> >>

>

> >> >** **

>

> >>

>

> >> > Cheers****

>

> >>

>

> >> >** **

>

> >>

>

> >> > On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <

>

> >> deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com%3cmailto:deepak8.kumar@citi.com>

> >>wrote:****<mailto:deepak8.kumar@citi.com%3cmailto:deepak8.kumar@citi.com

> %3e%3ewrote:****>

>

> >>

>

> >> >>  Lars,********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> I am getting following errors at datanode & region

>

> >> >> servers.********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Regards,********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Deepak********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> *From:* Kumar, Deepak8 [CCC-OT_IT NE]****

>

> >>

>

> >> >> *Sent:* Tuesday, March 12, 2013 3:00 AM****

>

> >>

>

> >> >> *To:* Kumar, Deepak8 [CCC-OT_IT NE]; ****

>

> >>

>

> >> >> 'user@hbase.apache.org<mailto:user@hbase.apache.org<user@hbase.apa

>

> >> >> che.org>>';

>

> >> 'lars hofhansl'****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> *Subject:* RE: Regionserver goes down while endpoint

>

> >> >> execution********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>  ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Lars,********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> It is having following errors when I execute the Endpoint RPC

>

> >> >> client **

>

> >> **

>

> >>

>

> >> >> from eclipse. It seems some of the regions at regionserver ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net<http://vm-8aa9-fe74.nam.nsroot.net> is

>

> >> >> ****

>

> >>

>

> >> >> taking more time to reponse.********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Could you guide how to fix it. I don't find any option to set ****

>

> >>

>

> >> >> hbase.rpc.timeout from hbase configuration menu in CDH4 CM server

>

> >> >> for *

>

> >> ***

>

> >>

>

> >> >> hbase configuration.******

>

> >>

>

> >> >> ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Regards,********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Deepak********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment

>

> >> >> ****

>

> >>

>

> >> >> complete on server ****

>

> >>

>

> >> >> vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181<http://vm-15c2-3bbf.

>

> >> >> nam**

>

> >> **

>

> >>

>

> >> >> .nsroot.net/10.96.172.44:2181>, sessionid = 0x53d591b77090026,

>

> >> >> ****

>

> >>

>

> >> >> negotiated timeout = 60000********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration****

>

> >>

>

> >> >> warnOnceIfDeprecated********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> WARNING: hadoop.native.lib is deprecated. Instead, use****

>

> >>

>

> >> >> io.native.lib.available********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Mar 12, 2013 2:44:00 AM****

>

> >>

>

> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple

>

> >> >> men**

>

> >> **

>

> >>

>

> >> >> tation****

>

> >>

>

> >> >> processExecs********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> WARNING: Error executing for row

> 153299:1362780381523:2932572079500658:

>

> >> ****

>

> >>

>

> >> >> vm-ab1f-dd21.nam.nsroot.net:********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> *java.util.concurrent.ExecutionException*: *****

>

> >>

>

> >> >> org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed

>

> >> >> ****

>

> >>

>

> >> >> after attempts=10, exceptions:********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:34:15 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2271remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:35:16 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2403remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:36:18 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2465remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:37:20 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2500remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:38:22 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2538remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:39:25 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2572remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:40:30 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2606remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:41:34 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2640remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:42:43 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2677remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:44:00 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2842remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at java.util.concurrent.FutureTask$Sync.innerGet(Unknown

>

> >> >> ****

>

> >>

>

> >> >> Source)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at java.util.concurrent.FutureTask.get(Unknown

>

> >> >> Source)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at****

>

> >>

>

> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple

>

> >> >> men**

>

> >> **

>

> >>

>

> >> >> tation.processExecs(****

>

> >>

>

> >> >> *HConnectionManager.java:1466*)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at

>

> >> >> org.apache.hadoop.hbase.client.HTable.coprocessorExec(*****

>

> >>

>

> >> >> HTable.java:1577*)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at

>

> >> >> org.apache.hadoop.hbase.client.HTable.coprocessorExec(*****

>

> >>

>

> >> >> HTable.java:1557*)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at ****

>

> >>

>

> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog.main(*

>

> >> >> ***

>

> >>

>

> >> >> *HBaseEndPointClientForElfLog.java:33*)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Caused by:

>

> >> >> *org.apache.hadoop.hbase.client.RetriesExhaustedException*:*

>

> >> ***

>

> >>

>

> >> >> Failed after attempts=10, exceptions:********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:34:15 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2271remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:35:16 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2403remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:36:18 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2465remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:37:20 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2500remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:38:22 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2538remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:39:25 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2572remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:40:30 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2606remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:41:34 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2640remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:42:43 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2677remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:44:00 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2842remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at****

>

> >>

>

> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple

>

> >> >> men**

>

> >> **

>

> >>

>

> >> >> tation.getRegionServerWithRetries(****

>

> >>

>

> >> >> *HConnectionManager.java:1345*)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(*****

>

> >>

>

> >> >> ExecRPCInvoker.java:79*)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at $Proxy8.getValues(Unknown Source)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at****

>

> >>

>

> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call

>

> >> >> (****

>

> >> *

>

> >>

>

> >> >> HBaseEndPointClientForElfLog.java:38*)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at****

>

> >>

>

> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call

>

> >> >> (****

>

> >> *

>

> >>

>

> >> >> HBaseEndPointClientForElfLog.java:1*)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at****

>

> >>

>

> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple

>

> >> >> men**

>

> >> **

>

> >>

>

> >> >> tation$4.call(****

>

> >>

>

> >> >> *HConnectionManager.java:1454*)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at java.util.concurrent.FutureTask$Sync.innerRun(Unknown

>

> >> >> ****

>

> >>

>

> >> >> Source)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at java.util.concurrent.FutureTask.run(Unknown

>

> >> >> Source)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at ****

>

> >>

>

> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown****

>

> >>

>

> >> >> Source)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at

>

> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown ***

>

> >> *

>

> >>

>

> >> >> Source)****

>

> >>

>

> >> >> ********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at java.lang.Thread.run(Unknown Source)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> *org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed

>

> >> >> ****

>

> >>

>

> >> >> after attempts=10, exceptions:********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:34:15 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2271remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:35:16 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2403remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:36:18 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2465remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:37:20 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2500remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:38:22 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2538remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:39:25 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto:org.apa>

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2572remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:40:30 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2606remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:41:34 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2640remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:42:43 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2677remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Tue Mar 12 02:44:00 EDT 2013,****

>

> >>

>

> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa

> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f

> %3cmailto:org.apa>

>

> >> >> che**

>

> >> **

>

> >>

>

> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:

> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****

>

> >>

>

> >> >> java.net.SocketTimeoutException*: Call to ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74

>

> >> >> .na**

>

> >> **

>

> >>

>

> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****

>

> >>

>

> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout

>

> >> >> ****

>

> >>

>

> >> >> while waiting for channel to be ready for read. ch :****

>

> >>

>

> >> >> java.nio.channels.SocketChannel[connected ****

>

> >>

>

> >> >> local=/150.110.96.212:2842remote= ****

>

> >>

>

> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9

>

> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>

>

> >> <http://vm-8aa9-fe****

>

> >>

>

> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at****

>

> >>

>

> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple

>

> >> >> men**

>

> >> **

>

> >>

>

> >> >> tation.getRegionServerWithRetries(****

>

> >>

>

> >> >> *HConnectionManager.java:1345*)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(*****

>

> >>

>

> >> >> ExecRPCInvoker.java:79*)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at $Proxy8.getValues(Unknown Source)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at****

>

> >>

>

> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call

>

> >> >> (****

>

> >> *

>

> >>

>

> >> >> HBaseEndPointClientForElfLog.java:38*)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at****

>

> >>

>

> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call

>

> >> >> (****

>

> >> *

>

> >>

>

> >> >> HBaseEndPointClientForElfLog.java:1*)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at****

>

> >>

>

> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple

>

> >> >> men**

>

> >> **

>

> >>

>

> >> >> tation$4.call(****

>

> >>

>

> >> >> *HConnectionManager.java:1454*)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at java.util.concurrent.FutureTask$Sync.innerRun(Unknown

>

> >> >> ****

>

> >>

>

> >> >> Source)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at java.util.concurrent.FutureTask.run(Unknown

>

> >> >> Source)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at ****

>

> >>

>

> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown****

>

> >>

>

> >> >> Source)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at

>

> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown ***

>

> >> *

>

> >>

>

> >> >> Source)****

>

> >>

>

> >> >> ********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       at java.lang.Thread.run(Unknown Source)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> *From:* Kumar, Deepak8 [CCC-OT_IT NE]****

>

> >>

>

> >> >> *Sent:* Tuesday, March 12, 2013 2:27 AM****

>

> >>

>

> >> >> *To:*

>

> >> >> 'user@hbase.apache.org<mailto:user@hbase.apache.org<user@hbase.apa

>

> >> >> che.org>>';

>

> >> 'lars hofhansl'****

>

> >>

>

> >> >> *Subject:* RE: Regionserver goes down while endpoint

>

> >> >> execution********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Lars,********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Thanks for your quick response.There is not much info in region

>

> >> >> ****

>

> >>

>

> >> >> server log. I am again executing it with DEBUG log level in region

>

> >> >> ****

>

> >>

>

> >> >> servers.********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> *Here is the endpoint code*****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> public class ColumnAggregationEndpoint extends ****

>

> >>

>

> >> >> BaseEndpointCoprocessor********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> implements  ColumnAggregationProtocol {********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       ********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       @Override********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>         public List<String> getValues(byte[] family, byte[] ****

>

> >>

>

> >> >> qualifier, int batchSize, int cacheSize)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>         throws IOException {********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           // aggregate at each region********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           Scan scan = new Scan();********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           scan.addColumn(family, qualifier);********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           scan.setCaching(cacheSize);********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           scan.setBatch(batchSize);********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           List<String> values = new ArrayList<String>();********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           RegionCoprocessorEnvironment environment =********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   (RegionCoprocessorEnvironment) ** **

>

> >>

>

> >> >> getEnvironment();********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           ********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           InternalScanner scanner =****

>

> >>

>

> >> >> environment.getRegion().getScanner(scan);********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           try {********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>             List<KeyValue> curVals = new

>

> >> >> ArrayList<KeyValue>();********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>             boolean hasMore = false;********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>             do {********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           curVals.clear();********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           hasMore = scanner.next(curVals);********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           KeyValue kv = curVals.get(0);********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           values.add(Bytes.toString(kv.getValue()));********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>             } while (hasMore);********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           } finally {********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>               scanner.close();********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           }********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>           return values;********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>         }********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> }********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> *The RPC client to invoke the Endpoint is as follows:*****

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> public class HBaseEndPointClientForElfLog {********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       public static void main(String[] args) {********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>             try {********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   Configuration conf = ****

>

> >>

>

> >> >> HBaseConfiguration.create();********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   conf.set(********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                               "hbase.zookeeper.quorum",********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                               ****

>

> >>

>

> >> >> "vm-ab1f-dd21.nam.nsroot.net<http://vm-ab1f-dd21.nam.nsroot.net>,*

>

> >> >> ***

>

> >>

>

> >> >> vm-cb03-2277.nam.nsroot.net<http://vm-cb03-2277.nam.nsroot.net>,vm

>

> >> >> -15**

>

> >> **

>

> >>

>

> >> >> c2-3bbf.nam.nsroot.net<http://vm-15c2-3bbf.nam.nsroot.net>");*****

>

> >> >> ***

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   String tableName = "elf_log";********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   final String columnFamily = "content";********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   final String columnQualifier =

>

> >> >> "logFileName";********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   final String startRowKey =****

>

> >>

>

> >> >> "153299:1362780381523:2932572079500658:vm-ab1f-dd21.nam.nsroot.net

>

> >> >> <ht**

>

> >> **

>

> >>

>

> >> >> tp://vm-ab1f-dd21.nam.nsroot.net>:";********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   final String endRowKey = ****

>

> >>

>

> >> >> "153299:1362953388000";********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   HTableInterface table = new HTable(conf, ****

>

> >>

>

> >> >> tableName);********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   Scan scan;********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   Map<byte[], List<String>> results;********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   ********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   // scan: for all regions********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   scan = new Scan();********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   ********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   results =****

>

> >>

>

> >> >> table.coprocessorExec(ColumnAggregationProtocol.class,********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                               startRowKey.getBytes(), ** **

>

> >>

>

> >> >> endRowKey.getBytes(),****

>

> >>

>

> >> >> ********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                               new ****

>

> >>

>

> >> >> Batch.Call<ColumnAggregationProtocol,****

>

> >>

>

> >> >> List<String>>() {********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                                     public List<String> ****

>

> >>

>

> >> >> call(ColumnAggregationProtocol instance)********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                                                 throws IOException

>

> >> >> ****

>

> >>

>

> >> >> {********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                                           return****

>

> >>

>

> >> >> instance.getValues(columnFamily.getBytes(),********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> columnQualifier.getBytes(),2,5);********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                                     }********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                               });********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   ********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   for (Map.Entry<byte[], List<String>> e :****

>

> >>

>

> >> >> results.entrySet()) {********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                         System.out.println("Size of list

>

> >> >> returned:****

>

> >>

>

> >> >> "+e.getValue().size());********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                         for(String singleVal:

>

> >> >> e.getValue()){********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>

>

> >> >> System.out.println(singleVal);********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                         }********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                         ********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                         ********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   }********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>             } catch (Throwable throwable) {********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>                   throwable.printStackTrace();********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>             }********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >>       }********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> }********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Regards,********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Deepak********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> -----Original Message-----****

>

> >>

>

> >> >> From: lars hofhansl

>

> >> >> [mailto:larsh@apache.org<mailto:larsh@apache.org> *

>

> >> ***

>

> >>

>

> >> >> <larsh@apache.org<mailto:larsh@apache.org<mailto:larsh@apache.org%

> 3cmailto:larsh@apache.org>>>]****

>

> >>

>

> >> >> Sent: Tuesday, March 12, 2013 2:01 AM****

>

> >>

>

> >> >> To: user@hbase.apache.org<mailto:user@hbase.apache.org>****<mailto:

> user@hbase.apache.org%3cmailto:user@hbase.apache.org%3e****>

>

> >>

>

> >> >> Subject: Re: Regionserver goes down while endpoint

>

> >> >> execution********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> What does the region server log say?********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Endpoints do not run in a sandbox. You can call System.exit(...)

>

> >> >> and **

>

> >> **

>

> >>

>

> >> >> your RegionServer will happily exit.********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> If you can, please show us your endpoint code.********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> -- Lars********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ________________________________********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> From: "Kumar, Deepak8 " ****

>

> >>

>

> >> >> <deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com<mailto:

> deepak8.kumar@citi.com%3cmailto:deepak8.kumar@citi.com>>>********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> To: "'user@hbase.apache.org<mailto:user@hbase.apache.org<

> user@hbase.apache.org<mailto:user@hbase.apache.org%3cuser@hbase.apache.org

> >>>'"

>

> >> ****

>

> >>

>

> >> >> <user@hbase.apache.org<mailto:user@hbase.apache.org<mailto:

> user@hbase.apache.org%3cmailto:user@hbase.apache.org>>> ********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Sent: Monday, March 11, 2013 10:51 PM********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Subject: Regionserver goes down while endpoint execution********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Hi,********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> I have a table in hbase which has more than 5GB of data, it is

>

> >> >> ****

>

> >>

>

> >> >> distributed at 101 regions at 5 regionservers.********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> When I execute an endpoint which is supposed to fetch a column

>

> >> >> ****

>

> >>

>

> >> >> qualifier value using an endpoint RPC client, the region server

>

> >> >> goes **

>

> >> **

>

> >>

>

> >> >> down. The hbase master log says "Can't connect to region,

>

> >> >> retrying.." *

>

> >> ***

>

> >>

>

> >> >> The same endpoint works fine for tables which has 300

>

> >> >> records.********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Could you please guide me the reason for being regionserver

>

> >> >> down?******

>

> >> **

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> ** ******

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Regards,********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >> Deepak********

>

> >>

>

> >> >>** **

>

> >>

>

> >> >** **

>

> >>

>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message