hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrien Mogenet <adrien.moge...@gmail.com>
Subject Re: Regionserver goes down while endpoint execution
Date Tue, 26 Mar 2013 07:42:21 GMT
Sure, and it might be costly if your CP is building secondary index through
`Put' to external tables/regions. Each of your RPC call will generate
another RPC call. Also, take note that your client could get an unexpected
behavior; it might be blocked waiting for RPC answer just because of your
CP.

As Anoop said, rethink on doing such things from your CP hook. Otherwise,
think about implementing asynchronous `Put' using a thread but it might
lead in some headach (ensure - eventually - consistency, handle
reconections, etc.)


On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <deepak8.kumar@citi.com>wrote:

> Thanks Anoop for assisting.
>
> I don't have Put(List). All those calls are independent of each other. Let
> me put it this way. Suppose I have 1000 insertions/seconds on a table,
> (each insert is independent & not coming in a list). Would it be a
> performance impact to use CP to update secondary index?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Anoop Sam John [mailto:anoopsj@huawei.com]
> Sent: Tuesday, March 26, 2013 2:20 AM
> To: Kumar, Deepak8 [CCC-OT_IT NE]; 'user@hbase.apache.org'; '
> ramkrishna.s.vasudevan@gmail.com'
> Subject: RE: Regionserver goes down while endpoint execution
>
> Deepak
>      When you say 1000 invocations, you mean you are having a put(List)
> with 1000 items?  In the region level when it deals with a batch of Puts,
> it calls postPut() for each of the Put one after the other. In the same
> handler thread which is doing the write. So yes one need to wait for the
> other. Otherwise you need to handle within the implementation of having a
> thread pool or so.
>
> I would say rethink on doing HTable#put() from a CP hook.
>
> -Anoop-
>
> ________________________________________
> From: Kumar, Deepak8  [deepak8.kumar@citi.com]
> Sent: Monday, March 25, 2013 10:23 PM
> To: 'user@hbase.apache.org'; 'ramkrishna.s.vasudevan@gmail.com'; Anoop
> Sam John
> Subject: RE: Regionserver goes down while endpoint execution
>
> Hi Anoop,
>
> Could you please provide some pointers. If we have a regionbase
> coprocessor which has some set of operation implemented in its postPut
> implementation.
>
> If I have say 1000 invocations of postPut per seconds, are those calls
> would be blocking? I mean, the second call will only start  after the
> completion of first call of postPut, or it would execute independently?
>
> Regards,
> Deepak
> -----Original Message-----
> From: Anoop Sam John [mailto:anoopsj@huawei.com]
> Sent: Wednesday, March 20, 2013 8:59 AM
> To: Kumar, Deepak8 [CCC-OT_IT NE]; 'user@hbase.apache.org'; '
> ramkrishna.s.vasudevan@gmail.com'
> Subject: RE: Regionserver goes down while endpoint execution
>
> What is this doing?
>
> HbaseTableAccess.getInstance().connectTable(tableName);
>
>
>
> Can you take a look at HTablePool?
>
>
>
> -Anoop-
>
>
>
> ________________________________
> From: Kumar, Deepak8 [deepak8.kumar@citi.com]
> Sent: Wednesday, March 20, 2013 6:14 PM
> To: 'user@hbase.apache.org'; Anoop Sam John; '
> ramkrishna.s.vasudevan@gmail.com'
> Subject: RE: Regionserver goes down while endpoint execution
>
>
> Hi,
>
> Even when I moved table =
> HbaseTableAccess.getInstance().connectTable(tableName); to postPut method
> of coprocessor, I am getting the same exception.
>
>
>
> Regards,
>
> Deepak
>
>
>
> -----Original Message-----
> From: Anoop Sam John [mailto:anoopsj@huawei.com]
> Sent: Wednesday, March 20, 2013 4:37 AM
> To: user@hbase.apache.org
> Subject: RE: Regionserver goes down while endpoint execution
>
>
>
> Hi Deepak
>
>                       Yes Ram told it correctly regarding the thread
> safety issue with HTable.
>
> I can see you issue HTable#put() to write to index table and this call you
> are making from a CP.
>
> Our design was around making the actual table region and corresponding
> index table region collocated in the same RS. In our impl we are not
> calling HTable#put() from CP. We get the index region and directly call put
> on that.
>
>
>
> In some mail thread Andrew Purtell explained it nicely that doing a client
> based call like this from CP is an anti pattern. Think of avoiding this if
> you are going with CP based impl.  If you are doing with the current way or
> you need a global ordered indexing, IMHO think of managing the index from
> the client side only rather than CP.
>
>
>
> -Anoop-
>
> ________________________________________
>
> From: ramkrishna vasudevan [ramkrishna.s.vasudevan@gmail.com]
>
> Sent: Wednesday, March 20, 2013 1:31 PM
>
> To: user@hbase.apache.org<mailto:user@hbase.apache.org>
>
> Subject: Re: Regionserver goes down while endpoint execution
>
>
>
> Hi Deepak
>
>
>
> "this.table.put(putInIndexTable);""
>
>
>
> I think this one should be the problem.
>
>
>
> Your table is at the instance level.   See the documentation of HTable.
>
> "<p>This class is not thread safe for reads nor write."
>
>
>
> So if you try creating a new HTable every time this problem should be
>
> avoided.
>
>
>
> Regards
>
> Ram
>
>
>
> On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <deepak8.kumar@citi.com
> <mailto:deepak8.kumar@citi.com>>wrote:
>
>
>
> > Hi Anoop,
>
> >
>
> > Quite inspired by your coprocessor secondary indexing document & trying
> to
>
> > implement one, for better response :)
>
> >
>
> >
>
> >
>
> > The coprocessor executes for some time, but later on (say after 400-500
>
> > inserts) it gives IndexOutofBounds exception.
>
> >
>
> >
>
> >
>
> > The stack trace is
>
> >
>
> > 2013-03-20 02:40:42,074 INFO
>
> >
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
>
> > Counter: 2408
>
> >
>
> > 2013-03-20 02:40:42,098 INFO
>
> >
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
>
> > Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
>
> >
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:
>
> > vm-ab1f-dd21.nam.nsroot.net:
>
> >
>
> > 2013-03-20 02:40:42,098 INFO
>
> >
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
>
> > Counter: 2410
>
> >
>
> > 2013-03-20 02:40:42,122 ERROR
>
> > org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
>
> >
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d
> '
>
> > from environment because it threw:  java.lang.IndexOutOfBoundsException:
>
> > Index: 1, Size: 1
>
> >
>
> > java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>
> >
>
> >         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>
> >
>
> >         at java.util.ArrayList.remove(ArrayList.java:387)
>
> >
>
> >         at
>
> > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:960)
>
> >
>
> >         at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826)
>
> >
>
> >         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801)
>
> >
>
> >         at
>
> >
> org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:394)
>
> >
>
> >         at
>
> >
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor.postPut(SecondaryIndexCoprocessor.java:91)
>
> >
>
> >
>
> >
>
> > If I synchronize on current object in the postPut then it works fine? If
> I
>
> > use some counter valriable in the postput then it seems the counter
> misses
>
> > some value just before this exception. As you can see above 2409 is
>
> > missing. But if I synchronize the postPut block then it executes fine &
> the
>
> > counter is also in serial order. But I  think using synchronize would
> slow
>
> > down the insertion in secondary index. Could you guide me the exact
> reason
>
> > for this?
>
> >
>
> >
>
> >
>
> > Here is the complete code for postPut
>
> >
>
> >
>
> >
>
> > public void postPut(final ObserverContext<RegionCoprocessorEnvironment>
> e,
>
> >
>
> >                   final Put put, final WALEdit edit, final boolean
>
> > writeToWAL) throws IOException {
>
> >
>
> >
>
> >
>
> >             List<KeyValue> hostName = put.get(Bytes.toBytes("sysInfo"),
>
> > Bytes.toBytes("hostName"));
>
> >
>
> >             List<KeyValue> logFilePath =
> put.get(Bytes.toBytes("content"),
>
> > Bytes.toBytes("logFilePath"));
>
> >
>
> >             List<KeyValue> logFileName =
> put.get(Bytes.toBytes("content"),
>
> > Bytes.toBytes("logFileName"));
>
> >
>
> >
>
> >
>
> >             if(hostName.size() > 0 && logFilePath.size() > 0 &&
>
> > logFileName.size() >0){
>
> >
>
> >                   byte[] hostNameVal = hostName.get(0).getValue();
>
> >
>
> >                   byte[] logFilePathVal = logFilePath.get(0).getValue();
>
> >
>
> >                   byte[] logFileNameVal = logFileName.get(0).getValue();
>
> >
>
> >                   byte[] rowKeyBody = hostName.get(0).getRow();
>
> >
>
> >                   synchronized(this){
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >                   byte[] rowKey = (
>
> >
>
> >                               Bytes.toString(hostNameVal) + ":" +
>
> > Bytes.toString(logFilePathVal) + Bytes.toString(logFileNameVal)
>
> >
>
> >                               + LOG_INDEX_DELIM +
>
> > Bytes.toString(rowKeyBody)).getBytes();
>
> >
>
> >
>
> >
>
> >                   logger.debug("Row Key Secondary Index:
>
> > "+Bytes.toString(rowKey));
>
> >
>
> >                   Put putInIndexTable = new Put(rowKey);
>
> >
>
> >
>
> >
>
> >                   putInIndexTable.add(LOG_INDEX_FAMILY_CORE.getBytes(),
>
> > LOG_INDEX_ELFLOG_RK.getBytes(), Bytes.toBytes(dummyRowkey));
>
> >
>
> >                   dummyRowkey++;
>
> >
>
> >
>
> >
>
> >                   this.table.put(putInIndexTable);
>
> >
>
> >                   logger.info("Added in "+tableName+":
>
> > "+Bytes.toString(rowKey));
>
> >
>
> >                   logger.info("Counter: "+dummyRowkey);
>
> >
>
> >                   }
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >             }else{
>
> >
>
> >                   logger.error("Size of hostName"+hostName.size());
>
> >
>
> >                   logger.error("Size of logFilePath"+logFilePath.size());
>
> >
>
> >                   logger.error("Size of logFileName"+logFileName.size());
>
> >
>
> >                   logger.error("Size of logFileBody"+hostName.size());
>
> >
>
> >             }
>
> >
>
> >       }
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > Regards,
>
> >
>
> > Deepak
>
> >
>
> > -----Original Message-----
>
> > From: Anoop Sam John [mailto:anoopsj@huawei.com]
>
> > Sent: Friday, March 15, 2013 2:56 AM
>
> > To: user@hbase.apache.org<mailto:user@hbase.apache.org>
>
> > Subject: RE: Regionserver goes down while endpoint execution
>
> >
>
> >
>
> >
>
> > Himanshu told it clearly. To make to more clear I am adding :)
>
> >
>
> >
>
> >
>
> > When the range of rowkeys that you are looking for spread across 5
>
> > regions, at the client side there will be 5 exec requests created and
>
> > submitted to a thread pool.[HBase client side thread pool associated with
>
> > HTable] Now as per the availability of slots in this pool the operations
>
> > will be requested to regions in RSs. So in different RS there can be
>
> > parallel execution.  Within one RS itself parallel execution on different
>
> > regions also possible.
>
> >
>
> >
>
> >
>
> > It is upto the client app to deal with the results what the Endpoint has
>
> > returned. It will return a Map right? region name vs result from region.
>
> > Client app need to take care about the ordering
>
> >
>
> >
>
> >
>
> > -Anoop-
>
> >
>
> >
>
> >
>
> > ________________________________________
>
> >
>
> > From: hv.csuoa@gmail.com<mailto:hv.csuoa@gmail.com<mailto:
> hv.csuoa@gmail.com%3cmailto:hv.csuoa@gmail.com>> [hv.csuoa@gmail.com]
>
> > on behalf of Himanshu Vashishtha [hvashish@cs.ualberta.ca]
>
> >
>
> > Sent: Thursday, March 14, 2013 11:15 PM
>
> >
>
> > To: user@hbase.apache.org<mailto:user@hbase.apache.org<mailto:
> user@hbase.apache.org%3cmailto:user@hbase.apache.org>>
>
> >
>
> > Subject: Re: Regionserver goes down while endpoint execution
>
> >
>
> >
>
> >
>
> > There is no ordering guarantee for the endpoint execution, other than
> that
>
> > calls will be executed in parallel across all the regions.
>
> >
>
> > In case you have 5 regions, then there will be 5 separate calls to these
>
> > regions. Then, you get 5 results from these regions at your client, where
>
> > you use the Callback class to aggregate the results. You can define your
>
> > ordering this aggregate class for sure.
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > On Thu, Mar 14, 2013 at 10:15 AM, Ted Yu <yuzhihong@gmail.com<mailto:
>
> > yuzhihong@gmail.com<mailto:yuzhihong@gmail.com>>> wrote:
>
> >
>
> > > bq. provide the rowkey range as rowkey1 to rowkey100 in endpoint RPC
>
> >
>
> > > client
>
> >
>
> > >
>
> >
>
> > > If I understand you correctly, you perform batching at the client as
>
> >
>
> > > described above.
>
> >
>
> > > The order would be as you expected.
>
> >
>
> > >
>
> >
>
> > > Cheers
>
> >
>
> > >
>
> >
>
> > > On Thu, Mar 14, 2013 at 10:09 AM, Kumar, Deepak8 <
> deepak8.kumar@citi.com
>
> > <mailto:deepak8.kumar@citi.com>>wrote:
>
> >
>
> > >
>
> >
>
> > >>  Hi,****
>
> >
>
> > >>
>
> >
>
> > >> It seems due to huge data the RegionServer is getting down. Now I am
>
> >
>
> > >> trying to fetch the data in parts & is running fine. I need some more
>
> >
>
> > >> info about the Endpoint execution:****
>
> >
>
> > >>
>
> >
>
> > >> ** **
>
> >
>
> > >>
>
> >
>
> > >> My use case is to fetch the data from HBase as per some rowkey range
>
> >
>
> > >> & to render it at UI. Since endpoints are executed in parallel so I
>
> >
>
> > >> am looking to use it. ****
>
> >
>
> > >>
>
> >
>
> > >> ** **
>
> >
>
> > >>
>
> >
>
> > >> **Ø  **Suppose I provide the rowkey range as rowkey1 to rowkey100 in
>
> >
>
> > >> endpoint RPC client & these rowkeys are distributed at 5 regions
>
> >
>
> > >> across 4 region servers. If I fetch 10  records at a time, do we have
>
> >
>
> > >> any way to guarantee that it would come in serial order like first
>
> >
>
> > >> result would of
>
> >
>
> > >> rowkey1 to rowkey10, next time I set the start rowkey as rowkey11 &
>
> >
>
> > >> the fetch would be from rowkey11 to rowkey20, irrespective of the
>
> >
>
> > >> region & region servers?****
>
> >
>
> > >>
>
> >
>
> > >> ** **
>
> >
>
> > >>
>
> >
>
> > >> Regards,****
>
> >
>
> > >>
>
> >
>
> > >> Deepak****
>
> >
>
> > >>
>
> >
>
> > >> ** **
>
> >
>
> > >>
>
> >
>
> > >> ** **
>
> >
>
> > >>
>
> >
>
> > >> -----Original Message-----
>
> >
>
> > >> From: hv.csuoa@gmail.com<mailto:hv.csuoa@gmail.com<mailto:
> hv.csuoa@gmail.com%3cmailto:hv.csuoa@gmail.com>> [mailto:
>
> > hv.csuoa@gmail.com<mailto:hv.csuoa@gmail.com>] On Behalf Of
>
> >
>
> > >> Himanshu Vashishtha
>
> >
>
> > >> Sent: Wednesday, March 13, 2013 12:09 PM
>
> >
>
> > >> To: user@hbase.apache.org<mailto:user@hbase.apache.org<mailto:
> user@hbase.apache.org%3cmailto:user@hbase.apache.org>>
>
> >
>
> > >> Cc: Gary Helmling; yuzhihong@gmail.com<mailto:yuzhihong@gmail.com
> <mailto:yuzhihong@gmail.com%3cmailto:yuzhihong@gmail.com>>;
>
> > lars hofhansl
>
> >
>
> > >> Subject: Re: Regionserver goes down while endpoint execution
>
> >
>
> > >>
>
> >
>
> > >> ** **
>
> >
>
> > >>
>
> >
>
> > >> On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8
>
> >
>
> > >> <deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com<mailto:
> deepak8.kumar@citi.com%3cmailto:deepak8.kumar@citi.com>>>
>
> >
>
> > >> wrote:****
>
> >
>
> > >>
>
> >
>
> > >> > Thanks guys for assisting. I am getting OOM exception yet. I have
>
> >
>
> > >> > one
>
> >
>
> > >> query about Endpoints. As endpoint executes in parallel, so if I have
>
> >
>
> > >> a table which is distributed at 101 regions across 5 regionserver.
>
> >
>
> > >> Would it be 101 threads of endpoint executing in parallel?****
>
> >
>
> > >>
>
> >
>
> > >> ** **
>
> >
>
> > >>
>
> >
>
> > >> No and Yes.****
>
> >
>
> > >>
>
> >
>
> > >> ** **
>
> >
>
> > >>
>
> >
>
> > >> The endpoints are not processed as separate threads, they are
>
> >
>
> > >> processed as just another request (via regionserver handlers). Yes,
>
> >
>
> > >> the execution will be in parallel in the sense that a separate client
>
> >
>
> > >> side call will be used for each of the regions that are in the range
>
> >
>
> > >> you specify.****
>
> >
>
> > >>
>
> >
>
> > >> ** **
>
> >
>
> > >>
>
> >
>
> > >> >** **
>
> >
>
> > >>
>
> >
>
> > >> > Regards,****
>
> >
>
> > >>
>
> >
>
> > >> > Deepak****
>
> >
>
> > >>
>
> >
>
> > >> >** **
>
> >
>
> > >>
>
> >
>
> > >> > From: Gary Helmling [mailto:ghelmling@gmail.com
>
> >
>
> > >> > <ghelmling@gmail.com<mailto:ghelmling@gmail.com<mailto:
> ghelmling@gmail.com%3cmailto:ghelmling@gmail.com>>>]**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> > Sent: Tuesday, March 12, 2013 2:14 PM****
>
> >
>
> > >>
>
> >
>
> > >> > To: user@hbase.apache.org****<mailto:user@hbase.apache.org
> ****<mailto:user@hbase.apache.org****%3cmailto:user@hbase.apache.org****>>
>
> >
>
> > >>
>
> >
>
> > >> > Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]****
>
> >
>
> > >>
>
> >
>
> > >> > Subject: Re: Regionserver goes down while endpoint execution****
>
> >
>
> > >>
>
> >
>
> > >> >** **
>
> >
>
> > >>
>
> >
>
> > >> > To expand on what Himanshu said, your endpoint is doing an
>
> >
>
> > >> > unbounded
>
> >
>
> > >> scan on the region, so with a region with a lot of rows it's taking
>
> >
>
> > >> more than 60 seconds to run to the region end, which is why the
>
> >
>
> > >> client side of the call is timing out.  In addition you're building
>
> >
>
> > >> up an in memory list of all the values for that qualifier in that
>
> >
>
> > >> region, which could cause you to bump into OOM issues, depending on
>
> >
>
> > >> how big your values are and how sparse the given column qualifier is.
>
> >
>
> > >> If you trigger an OOMException, then the region server would
>
> >
>
> > >> abort.****
>
> >
>
> > >>
>
> >
>
> > >> >** **
>
> >
>
> > >>
>
> >
>
> > >> > For this usage specifically, though -- scanning through a single
>
> >
>
> > >> > column
>
> >
>
> > >> qualifier for all rows -- you would be better off just doing a normal
>
> >
>
> > >> client side scan, ie. HTable.getScanner().  Then you will avoid the
>
> >
>
> > >> client timeout and potential server-side memory issues.****
>
> >
>
> > >>
>
> >
>
> > >> >** **
>
> >
>
> > >>
>
> >
>
> > >> > On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <
>
> >
>
> > >> yuzhihong@gmail.com<mailto:yuzhihong@gmail.com<mailto<mailto:
> yuzhihong@gmail.com%3cmailto:yuzhihong@gmail.com%3cmailto>:
>
> > yuzhihong@gmail.com%3cmailto:yuzhihong@gmail.com<mailto:
> yuzhihong@gmail.com%3cmailto:yuzhihong@gmail.com>>>> wrote:****
>
> >
>
> > >>
>
> >
>
> > >> > From region server log:****
>
> >
>
> > >>
>
> >
>
> > >> >** **
>
> >
>
> > >>
>
> >
>
> > >> > 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient:
>
> >
>
> > >> > Error **
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> > making BlockReader. Closing stale ****
>
> >
>
> > >>
>
> >
>
> > >> > Socket[addr=/10.42.105.112<http://10.42.105.112>,port=50010,localpo
>
> >
>
> > >> > rt=**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> > 54114]****
>
> >
>
> > >>
>
> >
>
> > >> > java.io.EOFException: Premature EOF: no length prefix available****
>
> >
>
> > >>
>
> >
>
> > >> >         at
>
> >
>
> > >> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoU
>
> >
>
> > >> til.java:162)
>
> >
>
> > >> ****
>
> >
>
> > >>
>
> >
>
> > >> >         at ****
>
> >
>
> > >>
>
> >
>
> > >> > org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBloc
>
> >
>
> > >> > kRe**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> > ader2.java:407)****
>
> >
>
> > >>
>
> >
>
> > >> >** **
>
> >
>
> > >>
>
> >
>
> > >> > What version of HBase and hadoop are you using ?****
>
> >
>
> > >>
>
> >
>
> > >> > Do versions of hadoop on Eclipse machine and in your cluster match
>
> >
>
> > >> > ?****
>
> >
>
> > >>
>
> >
>
> > >> >** **
>
> >
>
> > >>
>
> >
>
> > >> > Cheers****
>
> >
>
> > >>
>
> >
>
> > >> >** **
>
> >
>
> > >>
>
> >
>
> > >> > On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <
>
> >
>
> > >> deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com<mailto:
> deepak8.kumar@citi.com%3cmailto:deepak8.kumar@citi.com>
>
> > >>wrote:****<mailto:deepak8.kumar@citi.com%
> 3cmailto:deepak8.kumar@citi.com
>
> > %3e%3ewrote:****>
>
> >
>
> > >>
>
> >
>
> > >> >>  Lars,********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> I am getting following errors at datanode & region
>
> >
>
> > >> >> servers.********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Regards,********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Deepak********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> *From:* Kumar, Deepak8 [CCC-OT_IT NE]****
>
> >
>
> > >>
>
> >
>
> > >> >> *Sent:* Tuesday, March 12, 2013 3:00 AM****
>
> >
>
> > >>
>
> >
>
> > >> >> *To:* Kumar, Deepak8 [CCC-OT_IT NE]; ****
>
> >
>
> > >>
>
> >
>
> > >> >> 'user@hbase.apache.org<mailto:user@hbase.apache.org<user@hbase.apa
>
> >
>
> > >> >> che.org>>';
>
> >
>
> > >> 'lars hofhansl'****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> *Subject:* RE: Regionserver goes down while endpoint
>
> >
>
> > >> >> execution********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>  ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Lars,********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> It is having following errors when I execute the Endpoint RPC
>
> >
>
> > >> >> client **
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> from eclipse. It seems some of the regions at regionserver ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net<http://vm-8aa9-fe74.nam.nsroot.net> is
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> taking more time to reponse.********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Could you guide how to fix it. I don't find any option to set ****
>
> >
>
> > >>
>
> >
>
> > >> >> hbase.rpc.timeout from hbase configuration menu in CDH4 CM server
>
> >
>
> > >> >> for *
>
> >
>
> > >> ***
>
> >
>
> > >>
>
> >
>
> > >> >> hbase configuration.******
>
> >
>
> > >>
>
> >
>
> > >> >> ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Regards,********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Deepak********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> complete on server ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181<http://vm-15c2-3bbf.
>
> >
>
> > >> >> nam**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .nsroot.net/10.96.172.44:2181>, sessionid = 0x53d591b77090026,
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> negotiated timeout = 60000********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration****
>
> >
>
> > >>
>
> >
>
> > >> >> warnOnceIfDeprecated********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> WARNING: hadoop.native.lib is deprecated. Instead, use****
>
> >
>
> > >>
>
> >
>
> > >> >> io.native.lib.available********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Mar 12, 2013 2:44:00 AM****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >
>
> > >> >> men**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> tation****
>
> >
>
> > >>
>
> >
>
> > >> >> processExecs********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> WARNING: Error executing for row
>
> > 153299:1362780381523:2932572079500658:
>
> >
>
> > >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-ab1f-dd21.nam.nsroot.net:********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> *java.util.concurrent.ExecutionException*: *****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> after attempts=10, exceptions:********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:34:15 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2271remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:35:16 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2403remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:36:18 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2465remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:37:20 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2500remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:38:22 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2538remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:39:25 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2572remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:40:30 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2606remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:41:34 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2640remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:42:43 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2677remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:44:00 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2842remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at java.util.concurrent.FutureTask$Sync.innerGet(Unknown
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> Source)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at java.util.concurrent.FutureTask.get(Unknown
>
> >
>
> > >> >> Source)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >
>
> > >> >> men**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> tation.processExecs(****
>
> >
>
> > >>
>
> >
>
> > >> >> *HConnectionManager.java:1466*)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at
>
> >
>
> > >> >> org.apache.hadoop.hbase.client.HTable.coprocessorExec(*****
>
> >
>
> > >>
>
> >
>
> > >> >> HTable.java:1577*)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at
>
> >
>
> > >> >> org.apache.hadoop.hbase.client.HTable.coprocessorExec(*****
>
> >
>
> > >>
>
> >
>
> > >> >> HTable.java:1557*)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at ****
>
> >
>
> > >>
>
> >
>
> > >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog.main(*
>
> >
>
> > >> >> ***
>
> >
>
> > >>
>
> >
>
> > >> >> *HBaseEndPointClientForElfLog.java:33*)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Caused by:
>
> >
>
> > >> >> *org.apache.hadoop.hbase.client.RetriesExhaustedException*:*
>
> >
>
> > >> ***
>
> >
>
> > >>
>
> >
>
> > >> >> Failed after attempts=10, exceptions:********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:34:15 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2271remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:35:16 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2403remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:36:18 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2465remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:37:20 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2500remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:38:22 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2538remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:39:25 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2572remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:40:30 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2606remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:41:34 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2640remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:42:43 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2677remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:44:00 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2842remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >
>
> > >> >> men**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> tation.getRegionServerWithRetries(****
>
> >
>
> > >>
>
> >
>
> > >> >> *HConnectionManager.java:1345*)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(*****
>
> >
>
> > >>
>
> >
>
> > >> >> ExecRPCInvoker.java:79*)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at $Proxy8.getValues(Unknown Source)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at****
>
> >
>
> > >>
>
> >
>
> > >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call
>
> >
>
> > >> >> (****
>
> >
>
> > >> *
>
> >
>
> > >>
>
> >
>
> > >> >> HBaseEndPointClientForElfLog.java:38*)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at****
>
> >
>
> > >>
>
> >
>
> > >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call
>
> >
>
> > >> >> (****
>
> >
>
> > >> *
>
> >
>
> > >>
>
> >
>
> > >> >> HBaseEndPointClientForElfLog.java:1*)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >
>
> > >> >> men**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> tation$4.call(****
>
> >
>
> > >>
>
> >
>
> > >> >> *HConnectionManager.java:1454*)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at java.util.concurrent.FutureTask$Sync.innerRun(Unknown
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> Source)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at java.util.concurrent.FutureTask.run(Unknown
>
> >
>
> > >> >> Source)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at ****
>
> >
>
> > >>
>
> >
>
> > >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown****
>
> >
>
> > >>
>
> >
>
> > >> >> Source)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at
>
> >
>
> > >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown ***
>
> >
>
> > >> *
>
> >
>
> > >>
>
> >
>
> > >> >> Source)****
>
> >
>
> > >>
>
> >
>
> > >> >> ********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at java.lang.Thread.run(Unknown Source)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> *org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> after attempts=10, exceptions:********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:34:15 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2271remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:35:16 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2403remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:36:18 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2465remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:37:20 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2500remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:38:22 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2538remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:39:25 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
> <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
> %3cmailto:org.apa>
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f%3cmailto>:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2572remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:40:30 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2606remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:41:34 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2640remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:42:43 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2677remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Tue Mar 12 02:44:00 EDT 2013,****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apa
>
> > <mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f
>
> > %3cmailto:org.apa>
>
> >
>
> > >> >> che**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:
>
> > .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>>, *****
>
> >
>
> > >>
>
> >
>
> > >> >> java.net.SocketTimeoutException*: Call to ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74
>
> >
>
> > >> >> .na**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout****
>
> >
>
> > >>
>
> >
>
> > >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> while waiting for channel to be ready for read. ch :****
>
> >
>
> > >>
>
> >
>
> > >> >> java.nio.channels.SocketChannel[connected ****
>
> >
>
> > >>
>
> >
>
> > >> >> local=/150.110.96.212:2842remote= ****
>
> >
>
> > >>
>
> >
>
> > >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9
>
> >
>
> > >> >> -fe74.nam.nsroot.net/10.42.105.91:60020%5D****>
>
> >
>
> > >> <http://vm-8aa9-fe****
>
> >
>
> > >>
>
> >
>
> > >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >
>
> > >> >> men**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> tation.getRegionServerWithRetries(****
>
> >
>
> > >>
>
> >
>
> > >> >> *HConnectionManager.java:1345*)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(*****
>
> >
>
> > >>
>
> >
>
> > >> >> ExecRPCInvoker.java:79*)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at $Proxy8.getValues(Unknown Source)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at****
>
> >
>
> > >>
>
> >
>
> > >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call
>
> >
>
> > >> >> (****
>
> >
>
> > >> *
>
> >
>
> > >>
>
> >
>
> > >> >> HBaseEndPointClientForElfLog.java:38*)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at****
>
> >
>
> > >>
>
> >
>
> > >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call
>
> >
>
> > >> >> (****
>
> >
>
> > >> *
>
> >
>
> > >>
>
> >
>
> > >> >> HBaseEndPointClientForElfLog.java:1*)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at****
>
> >
>
> > >>
>
> >
>
> > >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImple
>
> >
>
> > >> >> men**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> tation$4.call(****
>
> >
>
> > >>
>
> >
>
> > >> >> *HConnectionManager.java:1454*)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at java.util.concurrent.FutureTask$Sync.innerRun(Unknown
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> Source)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at java.util.concurrent.FutureTask.run(Unknown
>
> >
>
> > >> >> Source)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at ****
>
> >
>
> > >>
>
> >
>
> > >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown****
>
> >
>
> > >>
>
> >
>
> > >> >> Source)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at
>
> >
>
> > >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown ***
>
> >
>
> > >> *
>
> >
>
> > >>
>
> >
>
> > >> >> Source)****
>
> >
>
> > >>
>
> >
>
> > >> >> ********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       at java.lang.Thread.run(Unknown Source)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> *From:* Kumar, Deepak8 [CCC-OT_IT NE]****
>
> >
>
> > >>
>
> >
>
> > >> >> *Sent:* Tuesday, March 12, 2013 2:27 AM****
>
> >
>
> > >>
>
> >
>
> > >> >> *To:*
>
> >
>
> > >> >> 'user@hbase.apache.org<mailto:user@hbase.apache.org<user@hbase.apa
>
> >
>
> > >> >> che.org>>';
>
> >
>
> > >> 'lars hofhansl'****
>
> >
>
> > >>
>
> >
>
> > >> >> *Subject:* RE: Regionserver goes down while endpoint
>
> >
>
> > >> >> execution********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Lars,********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Thanks for your quick response.There is not much info in region
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> server log. I am again executing it with DEBUG log level in region
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> servers.********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> *Here is the endpoint code*****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> public class ColumnAggregationEndpoint extends ****
>
> >
>
> > >>
>
> >
>
> > >> >> BaseEndpointCoprocessor********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> implements  ColumnAggregationProtocol {********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       ********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       @Override********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>         public List<String> getValues(byte[] family, byte[] ****
>
> >
>
> > >>
>
> >
>
> > >> >> qualifier, int batchSize, int cacheSize)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>         throws IOException {********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           // aggregate at each region********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           Scan scan = new Scan();********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           scan.addColumn(family, qualifier);********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           scan.setCaching(cacheSize);********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           scan.setBatch(batchSize);********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           List<String> values = new ArrayList<String>();********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           RegionCoprocessorEnvironment environment =********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   (RegionCoprocessorEnvironment) ** **
>
> >
>
> > >>
>
> >
>
> > >> >> getEnvironment();********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           ********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           InternalScanner scanner =****
>
> >
>
> > >>
>
> >
>
> > >> >> environment.getRegion().getScanner(scan);********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           try {********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>             List<KeyValue> curVals = new
>
> >
>
> > >> >> ArrayList<KeyValue>();********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>             boolean hasMore = false;********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>             do {********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           curVals.clear();********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           hasMore = scanner.next(curVals);********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           KeyValue kv = curVals.get(0);********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           values.add(Bytes.toString(kv.getValue()));********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>             } while (hasMore);********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           } finally {********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>               scanner.close();********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           }********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>           return values;********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>         }********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> }********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> *The RPC client to invoke the Endpoint is as follows:*****
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> public class HBaseEndPointClientForElfLog {********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       public static void main(String[] args) {********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>             try {********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   Configuration conf = ****
>
> >
>
> > >>
>
> >
>
> > >> >> HBaseConfiguration.create();********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   conf.set(********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                               "hbase.zookeeper.quorum",********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                               ****
>
> >
>
> > >>
>
> >
>
> > >> >> "vm-ab1f-dd21.nam.nsroot.net<http://vm-ab1f-dd21.nam.nsroot.net>,*
>
> >
>
> > >> >> ***
>
> >
>
> > >>
>
> >
>
> > >> >> vm-cb03-2277.nam.nsroot.net<http://vm-cb03-2277.nam.nsroot.net>,vm
>
> >
>
> > >> >> -15**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> c2-3bbf.nam.nsroot.net<http://vm-15c2-3bbf.nam.nsroot.net>");*****
>
> >
>
> > >> >> ***
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   String tableName = "elf_log";********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   final String columnFamily = "content";********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   final String columnQualifier =
>
> >
>
> > >> >> "logFileName";********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   final String startRowKey =****
>
> >
>
> > >>
>
> >
>
> > >> >> "153299:1362780381523:2932572079500658:vm-ab1f-dd21.nam.nsroot.net
>
> >
>
> > >> >> <ht**
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> tp://vm-ab1f-dd21.nam.nsroot.net>:";********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   final String endRowKey = ****
>
> >
>
> > >>
>
> >
>
> > >> >> "153299:1362953388000";********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   HTableInterface table = new HTable(conf, ****
>
> >
>
> > >>
>
> >
>
> > >> >> tableName);********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   Scan scan;********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   Map<byte[], List<String>> results;********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   ********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   // scan: for all regions********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   scan = new Scan();********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   ********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   results =****
>
> >
>
> > >>
>
> >
>
> > >> >> table.coprocessorExec(ColumnAggregationProtocol.class,********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                               startRowKey.getBytes(), ** **
>
> >
>
> > >>
>
> >
>
> > >> >> endRowKey.getBytes(),****
>
> >
>
> > >>
>
> >
>
> > >> >> ********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                               new ****
>
> >
>
> > >>
>
> >
>
> > >> >> Batch.Call<ColumnAggregationProtocol,****
>
> >
>
> > >>
>
> >
>
> > >> >> List<String>>() {********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                                     public List<String> ****
>
> >
>
> > >>
>
> >
>
> > >> >> call(ColumnAggregationProtocol instance)********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                                                 throws IOException
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> {********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                                           return****
>
> >
>
> > >>
>
> >
>
> > >> >> instance.getValues(columnFamily.getBytes(),********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> columnQualifier.getBytes(),2,5);********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                                     }********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                               });********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   ********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   for (Map.Entry<byte[], List<String>> e :****
>
> >
>
> > >>
>
> >
>
> > >> >> results.entrySet()) {********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                         System.out.println("Size of list
>
> >
>
> > >> >> returned:****
>
> >
>
> > >>
>
> >
>
> > >> >> "+e.getValue().size());********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                         for(String singleVal:
>
> >
>
> > >> >> e.getValue()){********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>
>
> >
>
> > >> >> System.out.println(singleVal);********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                         }********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                         ********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                         ********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   }********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>             } catch (Throwable throwable) {********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>                   throwable.printStackTrace();********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>             }********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >>       }********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> }********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Regards,********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Deepak********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> -----Original Message-----****
>
> >
>
> > >>
>
> >
>
> > >> >> From: lars hofhansl
>
> >
>
> > >> >> [mailto:larsh@apache.org<mailto:larsh@apache.org> *
>
> >
>
> > >> ***
>
> >
>
> > >>
>
> >
>
> > >> >> <larsh@apache.org<mailto:larsh@apache.org<mailto:larsh@apache.org%
>
> > 3cmailto:larsh@apache.org>>>]****
>
> >
>
> > >>
>
> >
>
> > >> >> Sent: Tuesday, March 12, 2013 2:01 AM****
>
> >
>
> > >>
>
> >
>
> > >> >> To: user@hbase.apache.org<mailto:user@hbase.apache.org
> >****<mailto:
>
> > user@hbase.apache.org%3cmailto:user@hbase.apache.org%3e****>
>
> >
>
> > >>
>
> >
>
> > >> >> Subject: Re: Regionserver goes down while endpoint
>
> >
>
> > >> >> execution********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> What does the region server log say?********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Endpoints do not run in a sandbox. You can call System.exit(...)
>
> >
>
> > >> >> and **
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> your RegionServer will happily exit.********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> If you can, please show us your endpoint code.********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> -- Lars********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ________________________________********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> From: "Kumar, Deepak8 " ****
>
> >
>
> > >>
>
> >
>
> > >> >> <deepak8.kumar@citi.com<mailto:deepak8.kumar@citi.com<mailto:
>
> > deepak8.kumar@citi.com%3cmailto:deepak8.kumar@citi.com>>>********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> To: "'user@hbase.apache.org<mailto:user@hbase.apache.org<
>
> > user@hbase.apache.org<mailto:user@hbase.apache.org%
> 3cuser@hbase.apache.org
>
> > >>>'"
>
> >
>
> > >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> <user@hbase.apache.org<mailto:user@hbase.apache.org<mailto:
>
> > user@hbase.apache.org%3cmailto:user@hbase.apache.org>>> ********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Sent: Monday, March 11, 2013 10:51 PM********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Subject: Regionserver goes down while endpoint execution********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Hi,********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> I have a table in hbase which has more than 5GB of data, it is
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> distributed at 101 regions at 5 regionservers.********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> When I execute an endpoint which is supposed to fetch a column
>
> >
>
> > >> >> ****
>
> >
>
> > >>
>
> >
>
> > >> >> qualifier value using an endpoint RPC client, the region server
>
> >
>
> > >> >> goes **
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >> down. The hbase master log says "Can't connect to region,
>
> >
>
> > >> >> retrying.." *
>
> >
>
> > >> ***
>
> >
>
> > >>
>
> >
>
> > >> >> The same endpoint works fine for tables which has 300
>
> >
>
> > >> >> records.********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Could you please guide me the reason for being regionserver
>
> >
>
> > >> >> down?******
>
> >
>
> > >> **
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> ** ******
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Regards,********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >> Deepak********
>
> >
>
> > >>
>
> >
>
> > >> >>** **
>
> >
>
> > >>
>
> >
>
> > >> >** **
>
> >
>
> > >>
>
> >
>



-- 
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message