impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Apple <>
Subject Re: Stress tests stuck - what should I look at?
Date Thu, 17 Nov 2016 01:42:09 GMT
impala.thrift-server.backend.connections-in-use 97 The number of
active Impala Backend client connections to this Impala Daemon. 97 The total number of
Impala Backend client connections made to this Impala Daemon over its
impala.thrift-server.beeswax-frontend.connections-in-use 64 The number
of active Beeswax API connections to this Impala Daemon. 208 The total
number of Beeswax API connections made to this Impala Daemon over its
lifetime. 11 The
total number of HiveServer2 API connections made to this Impala Daemon
over its lifetime.
impala.thrift-server.backend.connection-setup-queue-size 0 The number
of connections to the Impala Backend Server that have been accepted
and are waiting to be setup.
impala.thrift-server.hiveserver2-frontend.connections-in-use 0 The
number of active HiveServer2 API connections to this Impala Daemon.

0 queries in flight
0 waiting to be closed

On Wed, Nov 16, 2016 at 5:32 PM, Alex Behm <> wrote:
> The mini stress has been prone to hangs in the past due to test bugs.
> I'd recommend dumping the impala-server metrics and checking whether
> the impala.thrift-server.beeswax-frontend.connections-in-use
> is close to 64.
> Then look at how many queries are actually still in flight. If there are
> fewer than 64 queries in flight, then it's probably a test bug (because the
> tests did not yet close their connections despite being done).
> You can grab http://localhost:25000/metrics?raw&json
> On Wed, Nov 16, 2016 at 5:29 PM, Tim Armstrong <>
> wrote:
>> It's probably worth looking at the debug pages to see what queries are
>> active. Probably also worth grabbing stack traces with gdb or core dumps
>> with gcore. If it's a hang then it's often possible to diagnose from the
>> backtraces.
>> Could also be worth running perf top to see where it's spending time (if
>> anywhere).
>> On Wed, Nov 16, 2016 at 5:19 PM, Jim Apple <> wrote:
>> > I'm running the EE tests on a machine and it seems to be stuck in the
>> > stress tests. I have access to the machine for now, but my Jenkins
>> > install is going to steal it from me when the job is force-timed-out
>> > in a few hours. What should I look at now to try and understand what
>> > is happening - in particular, what could be useful to me now but not
>> > visible in the logs?
>> >

View raw message