hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Extremely high CPU usage after upgrading to Hbase 1.4.4
Date Thu, 06 Sep 2018 19:24:52 GMT
>From the stack trace, ColumnPrefixFilter is used during scan.

Can you illustrate how various filters are formed thru FilterListWithOR ?
It would be easier for other people to reproduce the problem given your
query pattern.

Cheers

On Thu, Sep 6, 2018 at 11:43 AM Srinidhi Muppalla <srinidhim@trulia.com>
wrote:

> Hi Vlad,
>
> Thank you for the suggestion. I recreated the issue and attached the stack
> traces I took. Let me know if there’s any other info I can provide. We
> narrowed the issue down to occurring when upgrading from 1.3.0 to any 1.4.x
> version.
>
> Thanks,
> Srinidhi
>
> On 9/4/18, 8:19 PM, "Vladimir Rodionov" <vladrodionov@gmail.com> wrote:
>
>     Hi, Srinidhi
>
>     Next time you will see this issue, take jstack of a RS several times
> in a
>     row. W/o stack traces it is hard
>     to tell what was going on with your cluster after upgrade.
>
>     -Vlad
>
>
>
>     On Tue, Sep 4, 2018 at 3:50 PM Srinidhi Muppalla <srinidhim@trulia.com
> >
>     wrote:
>
>     > Hello all,
>     >
>     > We are currently running Hbase 1.3.0 on an EMR cluster running EMR
> 5.5.0.
>     > Recently, we attempted to upgrade our cluster to using Hbase 1.4.4
> (along
>     > with upgrading our EMR cluster to 5.16). After upgrading, the CPU
> usage for
>     > all of our region servers spiked up to 90%. The load_one for all of
> our
>     > servers spiked from roughly 1-2 to 10 threads. After upgrading, the
> number
>     > of operations to the cluster hasn’t increased. After giving the
> cluster a
>     > few hours, we had to revert the upgrade. From the logs, we are
> unable to
>     > tell what is occupying the CPU resources. Is this a known issue with
> 1.4.4?
>     > Any guidance or ideas for debugging the cause would be greatly
>     > appreciated.  What are the best steps for debugging CPU usage?
>     >
>     > Thank you,
>     > Srinidhi
>     >
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message