hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srinidhi Muppalla <srinid...@trulia.com>
Subject Re: Extremely high CPU usage after upgrading to Hbase 1.4.4
Date Thu, 06 Sep 2018 18:26:19 GMT
Hi Vlad,

Thank you for the suggestion. I recreated the issue and attached the stack traces I took.
Let me know if there’s any other info I can provide. We narrowed the issue down to occurring
when upgrading from 1.3.0 to any 1.4.x version.


On 9/4/18, 8:19 PM, "Vladimir Rodionov" <vladrodionov@gmail.com> wrote:

    Hi, Srinidhi
    Next time you will see this issue, take jstack of a RS several times in a
    row. W/o stack traces it is hard
    to tell what was going on with your cluster after upgrade.
    On Tue, Sep 4, 2018 at 3:50 PM Srinidhi Muppalla <srinidhim@trulia.com>
    > Hello all,
    > We are currently running Hbase 1.3.0 on an EMR cluster running EMR 5.5.0.
    > Recently, we attempted to upgrade our cluster to using Hbase 1.4.4 (along
    > with upgrading our EMR cluster to 5.16). After upgrading, the CPU usage for
    > all of our region servers spiked up to 90%. The load_one for all of our
    > servers spiked from roughly 1-2 to 10 threads. After upgrading, the number
    > of operations to the cluster hasn’t increased. After giving the cluster a
    > few hours, we had to revert the upgrade. From the logs, we are unable to
    > tell what is occupying the CPU resources. Is this a known issue with 1.4.4?
    > Any guidance or ideas for debugging the cause would be greatly
    > appreciated.  What are the best steps for debugging CPU usage?
    > Thank you,
    > Srinidhi

View raw message