hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com>
Subject Re: Regionserver goes down while endpoint execution
Date Wed, 20 Mar 2013 08:01:08 GMT
Hi Deepak

"this.table.put(putInIndexTable);""

I think this one should be the problem.

Your table is at the instance level.   See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."

So if you try creating a new HTable every time this problem should be
avoided.

Regards
Ram

On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <deepak8.kumar@citi.com>wrote:

> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2408
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:
> vm-ab1f-dd21.nam.nsroot.net:
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2410
>
> 2013-03-20 02:40:42,122 ERROR
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d'
> from environment because it threw:  java.lang.IndexOutOfBoundsException:
> Index: 1, Size: 1
>
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>
>         at java.util.ArrayList.remove(ArrayList.java:387)
>
>         at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:960)
>
>         at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826)
>
>         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801)
>
>         at
> org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:394)
>
>         at
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor.postPut(SecondaryIndexCoprocessor.java:91)
>
>
>
> If I synchronize on current object in the postPut then it works fine? If I
> use some counter valriable in the postput then it seems the counter misses
> some value just before this exception. As you can see above 2409 is
> missing. But if I synchronize the postPut block then it executes fine & the
> counter is also in serial order. But I  think using synchronize would slow
> down the insertion in secondary index. Could you guide me the exact reason
> for this?
>
>
>
> Here is the complete code for postPut
>
>
>
> public void postPut(final ObserverContext<RegionCoprocessorEnvironment> e,
>
>                   final Put put, final WALEdit edit, final boolean
> writeToWAL) throws IOException {
>
>
>
>             List<KeyValue> hostName = put.get(Bytes.toBytes("sysInfo"),
> Bytes.toBytes("hostName"));
>
>             List<KeyValue> logFilePath = put.get(Bytes.toBytes("content"),
> Bytes.toBytes("logFilePath"));
>
>             List<KeyValue> logFileName = put.get(Bytes.toBytes("content"),
> Bytes.toBytes("logFileName"));
>
>
>
>             if(hostName.size() > 0 && logFilePath.size() > 0 &&
> logFileName.size() >0){
>
>                   byte[] hostNameVal = hostName.get(0).getValue();
>
>                   byte[] logFilePathVal = logFilePath.get(0).getValue();
>
>                   byte[] logFileNameVal = logFileName.get(0).getValue();
>
>                   byte[] rowKeyBody = hostName.get(0).getRow();
>
>                   synchronized(this){
>
>
>
>
>
>                   byte[] rowKey = (
>
>                               Bytes.toString(hostNameVal) + ":" +
> Bytes.toString(logFilePathVal) + Bytes.toString(logFileNameVal)
>
>                               + LOG_INDEX_DELIM +
> Bytes.toString(rowKeyBody)).getBytes();
>
>
>
>                   logger.debug("Row Key Secondary Index:
> "+Bytes.toString(rowKey));
>
>                   Put putInIndexTable = new Put(rowKey);
>
>
>
>                   putInIndexTable.add(LOG_INDEX_FAMILY_CORE.getBytes(),
> LOG_INDEX_ELFLOG_RK.getBytes(), Bytes.toBytes(dummyRowkey));
>
>                   dummyRowkey++;
>
>
>
>                   this.table.put(putInIndexTable);
>
>                   logger.info("Added in "+tableName+":
> "+Bytes.toString(rowKey));
>
>                   logger.info("Counter: "+dummyRowkey);
>
>                   }
>
>
>
>
>
>
>
>             }else{
>
>                   logger.error("Size of hostName"+hostName.size());
>
>                   logger.error("Size of logFilePath"+logFilePath.size());
>
>                   logger.error("Size of logFileName"+logFileName.size());
>
>                   logger.error("Size of logFileBody"+hostName.size());
>
>             }
>
>       }
>
>
>
>
>
> Regards,
>
> Deepak
>
> -----Original Message-----
> From: Anoop Sam John [mailto:anoopsj@huawei.com]
> Sent: Friday, March 15, 2013 2:56 AM
> To: user@hbase.apache.org
> Subject: RE: Regionserver goes down while endpoint execution
>
>
>
> Himanshu told it clearly. To make to more clear I am adding :)
>
>
>
> When the range of rowkeys that you are looking for spread across 5
> regions, at the client side there will be 5 exec requests created and
> submitted to a thread pool.[HBase client side thread pool associated with
> HTable] Now as per the availability of slots in this pool the operations
> will be requested to regions in RSs. So in different RS there can be
> parallel execution.  Within one RS itself parallel execution on different
> regions also possible.
>
>
>
> It is upto the client app to deal with the results what the Endpoint has
> returned. It will return a Map right? region name vs result from region.
> Client app need to take care about the ordering
>
>
>
> -Anoop-
>
>
>
> ________________________________________
>
> From: hv.csuoa@gmail.com<mailto:hv.csuoa@gmail.com> [hv.csuoa@gmail.com]
> on behalf of Himanshu Vashishtha [hvashish@cs.ualberta.ca]
>
> Sent: Thursday, March 14, 2013 11:15 PM
>
> To: user@hbase.apache.org<mailto:user@hbase.apache.org>
>
> Subject: Re: Regionserver goes down while endpoint execution
>
>
>
> There is no ordering guarantee for the endpoint execution, other than that
> calls will be executed in parallel across all the regions.
>
> In case you have 5 regions, then there will be 5 separate calls to these
> regions. Then, you get 5 results from these regions at your client, where
> you use the Callback class to aggregate the results. You can define your
> ordering this aggregate class for sure.
>
>
>
>
>
> On Thu, Mar 14, 2013 at 10:15 AM, Ted Yu <yuzhihong@gmail.com<mailto:
> yuzhihong@gmail.com>> wrote:
>
> > bq. provide the rowkey range as rowkey1 to rowkey100 in endpoint RPC
>
> > client
>
> >
>
> > If I understand you correctly, you perform batching at the client as
>
> > described above.
>
> > The order would be as you expected.
>
> >
>
> > Cheers
>
> >
>
> > On Thu, Mar 14, 2013 at 10:09 AM, Kumar, Deepak8 <deepak8.kumar@citi.com
> <mailto:deepak8.kumar@citi.com>>wrote:
>
> >
>
> >>  Hi,****
>
> >>
>
> >> It seems due to huge data the RegionServer is getting down. Now I am
>
> >> trying to fetch the data in parts & is running fine. I need some more
>
> >> info about the Endpoint execution:****
>
> >>
>
> >> ** **
>
> >>
>
> >> My use case is to fetch the data from HBase as per some rowkey range
>
> >> & to render it at UI. Since endpoints are executed in parallel so I
>
> >> am looking to use it. ****
>
> >>
>
> >> ** **
>
> >>
>
> >> **Ø  **Suppose I provide the rowkey range as rowkey1 to rowkey100 in
>
> >> endpoint RPC client & these rowkeys are distributed at 5 regions
>
> >> across 4 region servers. If I fetch 10  records at a time, do we have
>
> >> any way to guarantee that it would come in serial order like first
>
> >> result would of
>
> >> rowkey1 to rowkey10, next time I set the start rowkey as rowkey11 &
>
> >> the fetch would be from rowkey11 to rowkey20, irrespective of the
>
> >> region & region servers?****
>
> >>
>
> >> ** **
>
> >>
>
> >> Regards,****
>
> >>
>
> >> Deepak****
>
> >>
>
> >> ** **
>
> >>
>
> >> ** **
>
> >>
>
> >> -----Original Message-----
>
> >> From: hv.csuoa@gmail.com<mailto:hv.csuoa@gmail.com> [mailto:
> hv.csuoa@gmail.com] On Behalf Of
>
> >> Himanshu Vashishtha
>
> >> Sent: Wednesday, March 13, 2013 12:09 PM
>
> >> To: user@hbase.apache.org<mailto:user@hbase.apache.org>
>
> >> Cc: Gary Helmling; yuzhihong@gmail.com<mailto:yuzhihong@gmail.com>;
> lars hofhansl
>
> >> Subject: Re: Regionserver goes down while endpoint execution
>
> >>
>
> >> ** **
>
> >>
>
> >> On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8
>
> >> <deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com>>
>
> >> wrote:****
>
> >>
>
> >> > Thanks guys for assisting. I am getting OOM exception yet. I have
>
> >> > one
>
> >> query about Endpoints. As endpoint executes in parallel, so if I have
>
> >> a table which is distributed at 101 regions across 5 regionserver.
>
> >> Would it be 101 threads of endpoint executing in parallel?****
>
> >>
>
> >> ** **
>
> >>
>
> >> No and Yes.****
>
> >>
>
> >> ** **
>
> >>
>
> >> The endpoints are not processed as separate threads, they are
>
> >> processed as just another request (via regionserver handlers). Yes,
>
> >> the execution will be in parallel in the sense that a separate client
>
> >> side call will be used for each of the regions that are in the range
>
> >> you specify.****
>
> >>
>
> >> ** **
>
> >>
>
> >> >** **
>
> >>
>
> >> > Regards,****
>
> >>
>
> >> > Deepak****
>
> >>
>
> >> >** **
>
> >>
>
> >> > From: Gary Helmling [mailto:ghelmling@gmail.com
>
> >> > <ghelmling@gmail.com<mailto:ghelmling@gmail.com>>]**
>
> >> **
>
> >>
>
> >> > Sent: Tuesday, March 12, 2013 2:14 PM****
>
> >>
>
> >> > To: user@hbase.apache.org****<mailto:user@hbase.apache.org****>
>
> >>
>
> >> > Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]****
>
> >>
>
> >> > Subject: Re: Regionserver goes down while endpoint execution****
>
> >>
>
> >> >** **
>
> >>
>
> >> > To expand on what Himanshu said, your endpoint is doing an
>
> >> > unbounded
>
> >> scan on the region, so with a region with a lot of rows it's taking
>
> >> more than 60 seconds to run to the region end, which is why the
>
> >> client side of the call is timing out.  In addition you're building
>
> >> up an in memory list of all the values for that qualifier in that
>
> >> region, which could cause you to bump into OOM issues, depending on
>
> >> how big your values are and how sparse the given column qualifier is.
>
> >> If you trigger an OOMException, then the region server would
>
> >> abort.****
>
> >>
>
> >> >** **
>
> >>
>
> >> > For this usage specifically, though -- scanning through a single
>
> >> > column
>
> >> qualifier for all rows -- you would be better off just doing a normal
>
> >> client side scan, ie. HTable.getScanner().  Then you will avoid the
>
> >> client timeout and potential server-side memory issues.****
>
> >>
>
> >> >** **
>
> >>
>
> >> > On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <
>
> >> yuzhihong@gmail.com<mailto:yuzhihong@gmail.com<mailto:
> yuzhihong@gmail.com%3cmailto:yuzhihong@gmail.com>>> wrote:****
>
> >>
>
> >> > From region server log:****
>
> >>
>
> >> >** **
>
> >>
>
> >> > 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient:
>
> >> > Error **
>
> >> **
>
> >>
>
> >> > making BlockReader. Closing stale ****
>
> >>
>
> >> > Socket[addr=/10.42.105.112<http://10.42.105.112>,port=50010,localpo
>
> >> > rt=**
>
> >> **
>
> >>
>
> >> > 54114]****
>
> >>
>
> >> > java.io.EOFException: Premature EOF: no length prefix available****
>
> >>
>
> >> >         at
>
> >> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoU
>
> >> til.java:162)
>
> >> ****
>
> >>
>
> >> >         at ****
>
> >>
>
> >> > org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBloc
>
> >> > kRe**
>
> >> **
>
> >>
>
> >> > ader2.java:407)****
>
> >>
>
> >> >** **
>
> >>
>
> >> > What version of HBase and hadoop are you using ?****
>
> >>
>
> >> > Do versions of hadoop on Eclipse machine and in your cluster match
>
> >> > ?****
>
> >>
>
> >> >** **
>
> >>
>
> >> > Cheers****
>
> >>
>
> >> >** **
>
> >>
>
> >> > On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <
>
> >> deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com
> >>wrote:****<mailto:deepak8.kumar@citi.com%3cmailto:deepak8.kumar@citi.com
> %3e%3ewrote:****>
>
> >>
>
> >> >>  Lars,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> I am getting following errors at datanode & region
>
> >> >> servers.********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Regards,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Deepak********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> *From:* Kumar, Deepak8 [CCC-OT_IT NE]****
>
> >>
>
> >> >> *Sent:* Tuesday, March 12, 2013 3:00 AM****
>
> >>
>
> >> >> *To:* Kumar, Deepak8 [CCC-OT_IT NE]; ****
>
> >>
>
> >> >> 'user@hbase.apache.org<mailto:user@hbase.apache.org<user@hbase.apa
>
> >> >> che.org>>';
>
> >> 'lars hofhansl'****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> *Subject:* RE: Regionserver goes down while endpoint
>
> >> >> execution********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>  ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Lars,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> It is having following errors when I execute the Endpoint RPC
>
> >> >> client **
>
> >> **
>
> >>
>
> >> >> from eclipse. It seems some of the regions at regionserver ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net<http://vm-8aa9-fe74.nam.nsroot.net> is
>
> >> >> ****
>
> >>
>
> >> >> taking more time to reponse.********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Could you guide how to fix it. I don't find any option to set ****
>
> >>
>
> >> >> hbase.rpc.timeout from hbase configuration menu in CDH4 CM server
>
> >> >> for *
>
> >> ***
>
> >>
>
> >> >> hbase configuration.******
>
> >>
>
> >> >> ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Regards,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Deepak********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment
>
> >> >> ****
>
> >>
>
> >> >> complete on server ****
>
> >>
>
> >> >> vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181<http://vm-15c2-3bbf.
>
> >> >> nam**
>
> >> **
>
> >>
>
> >> >> .nsroot.net/10.96.172.44:2181>, sessionid = 0x53d591b77090026,
>
> >> >> ****
>
> >>
>
> >> >> negotiated timeout = 60000********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration****
>
> >>
>
> >> >> warnOnceIfDeprecated********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> WARNING: hadoop.native.lib is deprecated. Instead, use****
>
> >>
>
> >> >> io.native.lib.available********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Mar 12, 2013 2:44:00 AM****
>
> >>
>
> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >> >> men**
>
> >> **
>
> >>
>
> >> >> tation****
>
> >>
>
> >> >> processExecs********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> WARNING: Error executing for row
> 153299:1362780381523:2932572079500658:
>
> >> ****
>
> >>
>
> >> >> vm-ab1f-dd21.nam.nsroot.net:********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> *java.util.concurrent.ExecutionException*: *****
>
> >>
>
> >> >> org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed
>
> >> >> ****
>
> >>
>
> >> >> after attempts=10, exceptions:********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:34:15 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2271remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:35:16 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2403remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:36:18 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2465remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:37:20 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2500remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:38:22 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2538remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:39:25 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2572remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:40:30 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2606remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:41:34 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2640remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:42:43 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2677remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:44:00 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2842remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.util.concurrent.FutureTask$Sync.innerGet(Unknown
>
> >> >> ****
>
> >>
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.util.concurrent.FutureTask.get(Unknown
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >> >> men**
>
> >> **
>
> >>
>
> >> >> tation.processExecs(****
>
> >>
>
> >> >> *HConnectionManager.java:1466*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at
>
> >> >> org.apache.hadoop.hbase.client.HTable.coprocessorExec(*****
>
> >>
>
> >> >> HTable.java:1577*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at
>
> >> >> org.apache.hadoop.hbase.client.HTable.coprocessorExec(*****
>
> >>
>
> >> >> HTable.java:1557*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at ****
>
> >>
>
> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog.main(*
>
> >> >> ***
>
> >>
>
> >> >> *HBaseEndPointClientForElfLog.java:33*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Caused by:
>
> >> >> *org.apache.hadoop.hbase.client.RetriesExhaustedException*:*
>
> >> ***
>
> >>
>
> >> >> Failed after attempts=10, exceptions:********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:34:15 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2271remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:35:16 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2403remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:36:18 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2465remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:37:20 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2500remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:38:22 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2538remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:39:25 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2572remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:40:30 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2606remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:41:34 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2640remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:42:43 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2677remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:44:00 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2842remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >> >> men**
>
> >> **
>
> >>
>
> >> >> tation.getRegionServerWithRetries(****
>
> >>
>
> >> >> *HConnectionManager.java:1345*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(*****
>
> >>
>
> >> >> ExecRPCInvoker.java:79*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at $Proxy8.getValues(Unknown Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call
>
> >> >> (****
>
> >> *
>
> >>
>
> >> >> HBaseEndPointClientForElfLog.java:38*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call
>
> >> >> (****
>
> >> *
>
> >>
>
> >> >> HBaseEndPointClientForElfLog.java:1*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >> >> men**
>
> >> **
>
> >>
>
> >> >> tation$4.call(****
>
> >>
>
> >> >> *HConnectionManager.java:1454*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.util.concurrent.FutureTask$Sync.innerRun(Unknown
>
> >> >> ****
>
> >>
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.util.concurrent.FutureTask.run(Unknown
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at ****
>
> >>
>
> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown****
>
> >>
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at
>
> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown ***
>
> >> *
>
> >>
>
> >> >> Source)****
>
> >>
>
> >> >> ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.lang.Thread.run(Unknown Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> *org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed
>
> >> >> ****
>
> >>
>
> >> >> after attempts=10, exceptions:********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:34:15 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2271remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:35:16 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2403remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:36:18 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2465remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:37:20 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2500remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:38:22 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2538remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:39:25 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2572remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:40:30 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2606remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:41:34 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2640remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:42:43 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2677remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Tue Mar 12 02:44:00 EDT 2013,****
>
> >>
>
> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> >> >> che**
>
> >> **
>
> >>
>
> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >>
>
> >> >> java.net.SocketTimeoutException*: Call to ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >> >> .na**
>
> >> **
>
> >>
>
> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >>
>
> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >> >> ****
>
> >>
>
> >> >> while waiting for channel to be ready for read. ch :****
>
> >>
>
> >> >> java.nio.channels.SocketChannel[connected ****
>
> >>
>
> >> >> local=/150.110.96.212:2842remote= ****
>
> >>
>
> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >> <http://vm-8aa9-fe****
>
> >>
>
> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >> >> men**
>
> >> **
>
> >>
>
> >> >> tation.getRegionServerWithRetries(****
>
> >>
>
> >> >> *HConnectionManager.java:1345*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(*****
>
> >>
>
> >> >> ExecRPCInvoker.java:79*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at $Proxy8.getValues(Unknown Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call
>
> >> >> (****
>
> >> *
>
> >>
>
> >> >> HBaseEndPointClientForElfLog.java:38*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call
>
> >> >> (****
>
> >> *
>
> >>
>
> >> >> HBaseEndPointClientForElfLog.java:1*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at****
>
> >>
>
> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >> >> men**
>
> >> **
>
> >>
>
> >> >> tation$4.call(****
>
> >>
>
> >> >> *HConnectionManager.java:1454*)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.util.concurrent.FutureTask$Sync.innerRun(Unknown
>
> >> >> ****
>
> >>
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.util.concurrent.FutureTask.run(Unknown
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at ****
>
> >>
>
> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown****
>
> >>
>
> >> >> Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at
>
> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown ***
>
> >> *
>
> >>
>
> >> >> Source)****
>
> >>
>
> >> >> ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       at java.lang.Thread.run(Unknown Source)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> *From:* Kumar, Deepak8 [CCC-OT_IT NE]****
>
> >>
>
> >> >> *Sent:* Tuesday, March 12, 2013 2:27 AM****
>
> >>
>
> >> >> *To:*
>
> >> >> 'user@hbase.apache.org<mailto:user@hbase.apache.org<user@hbase.apa
>
> >> >> che.org>>';
>
> >> 'lars hofhansl'****
>
> >>
>
> >> >> *Subject:* RE: Regionserver goes down while endpoint
>
> >> >> execution********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Lars,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Thanks for your quick response.There is not much info in region
>
> >> >> ****
>
> >>
>
> >> >> server log. I am again executing it with DEBUG log level in region
>
> >> >> ****
>
> >>
>
> >> >> servers.********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> *Here is the endpoint code*****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> public class ColumnAggregationEndpoint extends ****
>
> >>
>
> >> >> BaseEndpointCoprocessor********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> implements  ColumnAggregationProtocol {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       @Override********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>         public List<String> getValues(byte[] family, byte[] ****
>
> >>
>
> >> >> qualifier, int batchSize, int cacheSize)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>         throws IOException {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           // aggregate at each region********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           Scan scan = new Scan();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           scan.addColumn(family, qualifier);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           scan.setCaching(cacheSize);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           scan.setBatch(batchSize);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           List<String> values = new ArrayList<String>();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           RegionCoprocessorEnvironment environment =********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   (RegionCoprocessorEnvironment) ** **
>
> >>
>
> >> >> getEnvironment();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           InternalScanner scanner =****
>
> >>
>
> >> >> environment.getRegion().getScanner(scan);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           try {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>             List<KeyValue> curVals = new
>
> >> >> ArrayList<KeyValue>();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>             boolean hasMore = false;********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>             do {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           curVals.clear();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           hasMore = scanner.next(curVals);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           KeyValue kv = curVals.get(0);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           values.add(Bytes.toString(kv.getValue()));********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>             } while (hasMore);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           } finally {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>               scanner.close();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>           return values;********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>         }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> *The RPC client to invoke the Endpoint is as follows:*****
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> public class HBaseEndPointClientForElfLog {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       public static void main(String[] args) {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>             try {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   Configuration conf = ****
>
> >>
>
> >> >> HBaseConfiguration.create();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   conf.set(********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                               "hbase.zookeeper.quorum",********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                               ****
>
> >>
>
> >> >> "vm-ab1f-dd21.nam.nsroot.net<http://vm-ab1f-dd21.nam.nsroot.net>,*
>
> >> >> ***
>
> >>
>
> >> >> vm-cb03-2277.nam.nsroot.net<http://vm-cb03-2277.nam.nsroot.net>,vm
>
> >> >> -15**
>
> >> **
>
> >>
>
> >> >> c2-3bbf.nam.nsroot.net<http://vm-15c2-3bbf.nam.nsroot.net>");*****
>
> >> >> ***
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   String tableName = "elf_log";********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   final String columnFamily = "content";********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   final String columnQualifier =
>
> >> >> "logFileName";********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   final String startRowKey =****
>
> >>
>
> >> >> "153299:1362780381523:2932572079500658:vm-ab1f-dd21.nam.nsroot.net
>
> >> >> <ht**
>
> >> **
>
> >>
>
> >> >> tp://vm-ab1f-dd21.nam.nsroot.net>:";********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   final String endRowKey = ****
>
> >>
>
> >> >> "153299:1362953388000";********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   HTableInterface table = new HTable(conf, ****
>
> >>
>
> >> >> tableName);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   Scan scan;********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   Map<byte[], List<String>> results;********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   // scan: for all regions********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   scan = new Scan();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   results =****
>
> >>
>
> >> >> table.coprocessorExec(ColumnAggregationProtocol.class,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                               startRowKey.getBytes(), ** **
>
> >>
>
> >> >> endRowKey.getBytes(),****
>
> >>
>
> >> >> ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                               new ****
>
> >>
>
> >> >> Batch.Call<ColumnAggregationProtocol,****
>
> >>
>
> >> >> List<String>>() {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                                     public List<String> ****
>
> >>
>
> >> >> call(ColumnAggregationProtocol instance)********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                                                 throws IOException
>
> >> >> ****
>
> >>
>
> >> >> {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                                           return****
>
> >>
>
> >> >> instance.getValues(columnFamily.getBytes(),********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> columnQualifier.getBytes(),2,5);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                                     }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                               });********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   for (Map.Entry<byte[], List<String>> e :****
>
> >>
>
> >> >> results.entrySet()) {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                         System.out.println("Size of list
>
> >> >> returned:****
>
> >>
>
> >> >> "+e.getValue().size());********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                         for(String singleVal:
>
> >> >> e.getValue()){********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>
>
> >> >> System.out.println(singleVal);********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                         }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                         ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                         ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>             } catch (Throwable throwable) {********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>                   throwable.printStackTrace();********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>             }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >>       }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> }********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Regards,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Deepak********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> -----Original Message-----****
>
> >>
>
> >> >> From: lars hofhansl
>
> >> >> [mailto:larsh@apache.org<mailto:larsh@apache.org> *
>
> >> ***
>
> >>
>
> >> >> <larsh@apache.org<mailto:larsh@apache.org<mailto:larsh@apache.org%
> 3cmailto:larsh@apache.org>>>]****
>
> >>
>
> >> >> Sent: Tuesday, March 12, 2013 2:01 AM****
>
> >>
>
> >> >> To: user@hbase.apache.org<mailto:user@hbase.apache.org>****<mailto:
> user@hbase.apache.org%3cmailto:user@hbase.apache.org%3e****>
>
> >>
>
> >> >> Subject: Re: Regionserver goes down while endpoint
>
> >> >> execution********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> What does the region server log say?********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Endpoints do not run in a sandbox. You can call System.exit(...)
>
> >> >> and **
>
> >> **
>
> >>
>
> >> >> your RegionServer will happily exit.********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> If you can, please show us your endpoint code.********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> -- Lars********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ________________________________********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> From: "Kumar, Deepak8 " ****
>
> >>
>
> >> >> <deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com<mailto:
> deepak8.kumar@citi.com%3cmailto:deepak8.kumar@citi.com>>>********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> To: "'user@hbase.apache.org<mailto:user@hbase.apache.org<
> user@hbase.apache.org<mailto:user@hbase.apache.org%3cuser@hbase.apache.org
> >>>'"
>
> >> ****
>
> >>
>
> >> >> <user@hbase.apache.org<mailto:user@hbase.apache.org<mailto:
> user@hbase.apache.org%3cmailto:user@hbase.apache.org>>> ********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Sent: Monday, March 11, 2013 10:51 PM********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Subject: Regionserver goes down while endpoint execution********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Hi,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> I have a table in hbase which has more than 5GB of data, it is
>
> >> >> ****
>
> >>
>
> >> >> distributed at 101 regions at 5 regionservers.********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> When I execute an endpoint which is supposed to fetch a column
>
> >> >> ****
>
> >>
>
> >> >> qualifier value using an endpoint RPC client, the region server
>
> >> >> goes **
>
> >> **
>
> >>
>
> >> >> down. The hbase master log says "Can't connect to region,
>
> >> >> retrying.." *
>
> >> ***
>
> >>
>
> >> >> The same endpoint works fine for tables which has 300
>
> >> >> records.********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Could you please guide me the reason for being regionserver
>
> >> >> down?******
>
> >> **
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> ** ******
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Regards,********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >> Deepak********
>
> >>
>
> >> >>** **
>
> >>
>
> >> >** **
>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message