hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Royston Sellman <royston.sell...@googlemail.com>
Subject RE: Hbase out of memory error
Date Mon, 23 Jan 2012 13:50:20 GMT
Hi Ted,

Finally rebuilt branch/0.92 and applied your patch and rebuilt my code.
Using AggregationClient.sum() on my test table I get the correct result.
Just swapping to AggregationClient.median() I get the following error: 

  [sshexec] org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=10, exceptions:
  [sshexec] Mon Jan 23 13:44:12 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:13 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:14 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:15 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:17 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:19 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:23 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:27 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:35 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:51 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] 
  [sshexec] Result = -1
  [sshexec] 	at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
getRegionServerWithRetries(HConnectionManager.java:1345)
  [sshexec] 	at
org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:
1203)
  [sshexec] 	at
org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1
126)
  [sshexec] 	at
org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:627)
  [sshexec] 	at
org.apache.hadoop.hbase.client.coprocessor.AggregationClient.median(Aggregat
ionClient.java:469)
  [sshexec] 	at
uk.org.cse.aggregation.EDRPAggregator.testSumWithValidRange(EDRPAggregator.j
ava:55)
  [sshexec] 	at
uk.org.cse.aggregation.EDRPAggregator.main(EDRPAggregator.java:85)
  [sshexec] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
  [sshexec] 	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
  [sshexec] 	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
  [sshexec] 	at java.lang.reflect.Method.invoke(Method.java:597)
  [sshexec] 	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

Something wrong with Scan setup? 

Cheers,
Royston



-----Original Message-----
From: yuzhihong@gmail.com [mailto:yuzhihong@gmail.com] 
Sent: 21 January 2012 17:14
To: user@hbase.apache.org
Subject: Re: Hbase out of memory error

Benoit's patches are already in 0.92

Thanks


On Jan 21, 2012, at 9:11 AM, Royston Sellman
<royston.sellman@googlemail.com> wrote:

> So should I try applying Benoit Sigoure's patch for HBASE-5204? Will this
patch be in the 0.92 branch soon?
> 
> Cheers,
> Royston
> 
> 
> 
> On 21 Jan 2012, at 16:58, yuzhihong@gmail.com wrote:
> 
>> That is the correct branch. 
>> 
>> Thanks
>> 
>> 
>> 
>> On Jan 21, 2012, at 8:50 AM, Royston Sellman
<royston.sellman@googlemail.com> wrote:
>> 
>>> Hi Ted,
>>> 
>>> Yes, I am compiling with the same HBase jars. I wasn't aware of
HBASE-5204, thanks, it sounds possible this is my problem. Can you think of
anything else I should check?
>>> 
>>> Just to make sure: I am checking out the code from
svn.apache.org/repos/asf/hbase/branches/0.92  Is this the correct branch?
>>> 
>>> Thanks,
>>> Royston
>>> 
>>> 
>>> On 20 Jan 2012, at 18:45, Ted Yu wrote:
>>> 
>>>> Royston:
>>>> I guess you have seen HBASE-5204. In particular:
>>>>>> when a 0.92 server fails to deserialize a 0.90-style RPC, it 
>>>>>> attempts to
>>>> allocate a large buffer because it doesn't read fields of 
>>>> 0.90-style RPCs properly.
>>>> 
>>>> Were your client code compiled with the same version of HBase as 
>>>> what was running on your cluster ?
>>>> 
>>>> Thanks
>>>> 
>>>> On Fri, Jan 20, 2012 at 9:20 AM, Royston Sellman < 
>>>> royston.sellman@googlemail.com> wrote:
>>>> 
>>>>> Trying to run my code (a test of Aggregation Protocol and an MR 
>>>>> HBase table
>>>>> loader) on latest build of 0.92.0 (r1232715) I get an 'old server' 
>>>>> warning (I've seen this before and it's always been non-fatal) 
>>>>> then an out of memory exception then job hangs:
>>>>> 
>>>>> 
>>>>> 
>>>>> [sshexec] 12/01/20 16:56:48 WARN zookeeper.ClientCnxnSocket: 
>>>>> Connected to an old server; r-o mode will be unavailable
>>>>> 
>>>>> [sshexec] 12/01/20 16:56:48 INFO zookeeper.ClientCnxn: Session 
>>>>> establishment complete on server namenode/10.0.0.235:2181, 
>>>>> sessionid = 0x34cda4e5d000e5, negotiated timeout = 40000
>>>>> 
>>>>> [sshexec] 12/01/20 16:56:49 WARN ipc.HBaseClient: Unexpected 
>>>>> exception receiving call responses
>>>>> 
>>>>> [sshexec] java.lang.OutOfMemoryError: Java heap space
>>>>> 
>>>>> [sshexec]       at java.lang.reflect.Array.newArray(Native Method)
>>>>> 
>>>>> [sshexec]       at java.lang.reflect.Array.newInstance(Array.java:52)
>>>>> 
>>>>> [sshexec]       at
>>>>> 
>>>>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObj
>>>>> ectWritabl
>>>>> e.java:542)
>>>>> 
>>>>> [sshexec]       at
>>>>> 
>>>>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObj
>>>>> ectWritabl
>>>>> e.java:289)
>>>>> 
>>>>> [sshexec]       at
>>>>> 
>>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse
>>>>> (HBaseClie
>>>>> nt.java:593)
>>>>> 
>>>>> [sshexec]       at
>>>>> 
>>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient
>>>>> .java:505)
>>>>> 
>>>>> 
>>>>> 
>>>>> Hbase shell seems to work (I can list and scan my tables).
>>>>> 
>>>>> 
>>>>> 
>>>>> If I svn roll back to 12 Jan 0.92 and rebuild my code works.
>>>>> 
>>>>> 
>>>>> 
>>>>> Tried setting export HBASE_HEAPSIZE=1500 but got same error.
>>>>> 
>>>>> 
>>>>> 
>>>>> Nothing significant in logs.
>>>>> 
>>>>> 
>>>>> 
>>>>> [Note to Ted Yu: I need to fix this so I can carry on testing on 
>>>>> Aggregation Protocol]
>>>>> 
>>>>> 
>>>>> 
>>>>> Best Regards,
>>>>> 
>>>>> Royston
>>>>> 
>>>>> 
>>> 
> 


Mime
View raw message