Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of jta@bristowhill.com
 designates 75.180.132.120 as permitted sender)
Message-ID: <4F4062F8.10407@bristowhill.com>
Date: Sat, 18 Feb 2012 18:48:24 -0800
From: "Jean T. Anderson" <jta@bristowhill.com>
Reply-To: jta@bristowhill.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6;
 rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2
MIME-Version: 1.0
To: Stack <stack@duboce.net>
CC: user@hbase.apache.org
Subject: Re: HBase 0.90.4: YCSB gets NPE with large loads when timeseries
 is set
References: <4F3EEFB0.4090609@bristowhill.com>
 <CADcMMgHWxUC1-y5Hg9Fze2SFj17tgJfbPk0e3q1Q+Z9JCk0t-g@mail.gmail.com>
In-Reply-To: 
 <CADcMMgHWxUC1-y5Hg9Fze2SFj17tgJfbPk0e3q1Q+Z9JCk0t-g@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 2/18/12 12:02 PM, Stack wrote:
> What version are you running?
HBase 0.90.4 / hadoop 0.20.2

> I don't see any explicit NPE mentions in the commit log on
> HConnectionManager.  Maybe its still an issue?
>
> St.Ack

ok, thanks for the sanity check. I'll dig a little deeper and also see 
if the problem reproduces on 0.92.

regards,

  -jean

> On Fri, Feb 17, 2012 at 4:24 PM, Jean T. Anderson<jta@bristowhill.com>  wrote:
>> Hi, everyone,
>>
>> I often get an NPE during the YCSB load process if timeseries is set and I'm
>> loading lots of records. Here's the syntax I'm using:
>>
>>    YCSB Client 0.1
>>    Command line: -load -db com.yahoo.ycsb.db.HBaseClient -P
>> workloads/workloada -p columnfamily=family -p recordcount=10000000 -threads
>> 32 -s -p measurementtype=timeseries -p timeseries.granularity=5000
>>
>> 100 million records load successfully with 32 threads if timeseries is not
>> set. But when I set timeseries, I often get NPEs with stacks that move
>> around depending on the number of records and the timeseries granularity.
>>
>> For example:
>>
>> 1) 10 million records gets an NPE when timeseries is enabled with a
>> granularity of 5000 ms (but 4000 ms loads successfully):
>>
>> java.lang.NullPointerException
>>         at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:916)
>>         at
>> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:109)
>>         at
>> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:101)
>>         at
>> java.util.TreeMap$NavigableSubMap.checkUpperBound(TreeMap.java:2666)
>>         at java.util.TreeMap$NavigableSubMap.isInRange(TreeMap.java:2688)
>>         at java.util.TreeMap$NavigableSubMap.get(TreeMap.java:2502)
>>         at
>> org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:96)
>>         at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:826)
>>
>> 2) 100 million records gets an NPE with 4000 ms (note the changed stack):
>>
>> java.lang.NullPointerException
>>         at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:916)
>>         at
>> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:109)
>>         at
>> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:101)
>>         at java.util.TreeMap$NavigableSubMap.cmp(TreeMap.java:2748)
>>         at
>> java.util.TreeMap$NavigableSubMap.findLowerEntryImpl(TreeMap.java:2881)
>>         at
>> java.util.TreeMap$NavigableSubMap.findLowerEntry(TreeMap.java:3000)
>>         at java.util.TreeMap$NavigableSubMap.smallerEntry(TreeMap.java:2994)
>>         at
>> java.util.TreeMap$AscendingSubMapIterator.getBoundaryNode(TreeMap.java:281)
>>         at java.util.TreeMap$AbstractSubMapIterator.<init>(TreeMap.java:186)
>>         at
>> java.util.TreeMap$AscendingSubMapKeySet.iterator(TreeMap.java:1968)
>>         at java.util.TreeMap$NavigableSubMap.isEmpty(TreeMap.java:2477)
>>         at
>> org.apache.hadoop.hbase.util.SoftValueSortedMap.isEmpty(SoftValueSortedMap.java:151)
>>         at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:826)
>>
>> 3) 100 million records gets an NPE with 3000 ms (and the stack changes):
>>
>> java.lang.NullPointerException
>>         at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:916)
>>         at
>> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:109)
>>         at
>> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:101)
>>         at java.util.TreeMap.cmp(TreeMap.java:4514)
>>         at java.util.TreeMap.get(TreeMap.java:4386)
>>         at
>> org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:96)
>>         at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:830)
>>
>> I found HBASE-3878, but it looks like that was fixed in 0.90.4.
>> I also found HBASE-5088, which is marked fixed in 0.90.6 -- does anyone
>> recognize if this problem fits that profile?
>>
>> many thanks,
>>
>>   -jean
>>
>>