Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates
 209.85.216.169 as permitted sender)
MIME-Version: 1.0
Sender: jdcryans@gmail.com
In-Reply-To: <1345770654.40868.YahooMailNeo@web121702.mail.ne1.yahoo.com>
References: 
 <CAKYwJ9xNdEgkB7BHo=eiG__Kicsnf9b6FH2bNXrLSqye-8-pfA@mail.gmail.com>
	<1345770324.77012.YahooMailNeo@web121704.mail.ne1.yahoo.com>
	<1345770654.40868.YahooMailNeo@web121702.mail.ne1.yahoo.com>
Date: Thu, 23 Aug 2012 18:28:12 -0700
Message-ID: 
 <CAGpTDNfm2CE2rJeP06yTcZMTZX3kJY-a1cJitCEmW1xMJroqog@mail.gmail.com>
Subject: Re: HBase-0.94.2-SNAPSHOT Scanning Bug
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: dev@hbase.apache.org, lars hofhansl <lhofhansl@yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1

We tried reproducing on a local node but it doesn't show up. It did
show as soon as we put it on our dev cluster.

J-D

On Thu, Aug 23, 2012 at 6:10 PM, lars hofhansl <lhofhansl@yahoo.com> wrote:
> It is interesting, though, because I have been running my local perf testing with this change included and have not seen this issue.
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: lars hofhansl <lhofhansl@yahoo.com>
> To: "dev@hbase.apache.org" <dev@hbase.apache.org>
> Cc:
> Sent: Thursday, August 23, 2012 6:05 PM
> Subject: Re: HBase-0.94.2-SNAPSHOT Scanning Bug
>
> This:
>
> "IPC Server handler 43 on 10304" daemon prio=10 tid=0x00007f16b8b1f000 nid=0x6414 runnable [0x00007f16b47c6000]
>    java.lang.Thread.State: RUNNABLE
>     at org.apache.hadoop.hbase.KeyValue.createFirstOnRowColTS(KeyValue.java:1893)
>     at org.apache.hadoop.hbase.regionserver.StoreFileScanner.requestSeek(StoreFileScanner.java:310)
>     at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:297)
>     at org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:256)
>     at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:522)
>     - locked <0x00000006cec5bd88> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
>     at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
>     at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.requestSeek(NonLazyKeyValueScanner.java:38)
>     at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:297)
>     at org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:256)
>     at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRow(HRegion.java:3508)
>
> points to my change: https://issues.apache.org/jira/browse/HBASE-6577
>
> The trace is interesting: RegionScannerImpl.nextRow now seeks to the last KV in the row and then iterates as before.
> However, then the reseek internally seeks to the first KV of the column, and somehow this interaction makes no progress forward.
>
> I'll revert that change.
>
> -- Lars
>
>
> ________________________________
> From: Elliott Clark <eclark@stumbleupon.com>
> To: dev@hbase.apache.org
> Sent: Thursday, August 23, 2012 5:39 PM
> Subject: HBase-0.94.2-SNAPSHOT Scanning Bug
>
> I recently tried to update one of our clusters to a version of 0.94.2
> seen here: https://github.com/stumbleupon/hbase/commits/su_prod_94
>
> When doing that all of the nodes started taking all available cpu
> time.  Not much interesting was in the logs however jstacks looked
> like this: http://pastebin.com/raw.php?i=fw6P5RKE  Everything is
> spinning in scans.  A version of 0.94.1 works perfectly and reverting
> solved all issues.  I don't really have enough data to point at any
> jira as the cause I was just wondering if anyone had some insight into
> the few commits between 0.94.1 release and the head of the above
> github that could cause scans to spin.
>
> Thanks
>