impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Cox <william....@distilnetworks.com>
Subject Re: queries "waiting to be closed" taking up resources - how to close via API?
Date Thu, 26 Jan 2017 16:29:04 GMT
Matthew,

This is hugely helpful, thanks! I wrote up this python 2 script which seems
to do the trick:

import urllib, json
>
> datanodes = ["http://ip1:25000/",
>              "http://ip2:25000/",
>              "http://ip3:25000/"]
>
> for i, datanode in enumerate(datanodes):
>     print("Checking {}: {}".format(i, datanode))
>     try:
>         response = urllib.urlopen(datanode + "queries?json")
>         data = json.loads(response.read())
>
>         if data["num_waiting_queries"] > 0:
>             print(data["num_waiting_queries"])
>             for in_flight_query in data["in_flight_queries"]:
>                 if in_flight_query["waiting"] is True and in_flight_query['state'] ==
"FINISHED":
>                     cancel_url = datanode + "cancel_query?query_id={}".format(in_flight_query['query_id'])
>                     print(cancel_url)
>                     #response = urllib.urlopen(cancel_url)
>
>     except IOError:
>         print("Skipping {}: {}".format(i, datanode))
>
>     except Exception as e:
>         print(e)
>
>
On Wed, Jan 25, 2017 at 5:50 PM, Matthew Jacobs <mj@cloudera.com> wrote:

> Hi William,
>
>
>
> On Wed, Jan 25, 2017 at 9:42 AM, William Cox
> <william.cox@distilnetworks.com> wrote:
> > Hello all,
> >
> > I'm running CDH CDH-5.8.0-1 and Impala =version 2.6.0-cdh5.8.0 RELEASE
> > (build 8d8652f69461f0dd8d5f474573fb5de7ceb0ee6b).
> > We have a major issue where many queries end up in the "Waiting to be
> > closed" state in the `queries` of the Impala Daemon UI.
> >
> > 1) is there an API endpoint I can use to find these queries in the
> "Waiting
> > to be closed" state? I can seem to find that information at all.
>
> Just the web UI. However, there is a json endpoint in addition to the html:
>
> http://impalad:25000/queries?json
>
> where you'll see the queries that remain open in 'in flight queries', e.g.:
>
>     "in_flight_queries": [
>         {
>             "effective_user": "mj",
>             "default_db": "default",
>             "stmt": "select sleep(10000)",
>             "stmt_type": "QUERY",
>             "start_time": "2017-01-25 14:37:15.455218000",
>             "end_time": "00:00:00",
>             "duration": "2s259ms",
>             "progress": "0 / 0 ( 0%)",
>             "state": "FINISHED",
>             "rows_fetched": 0,
>             "query_id": "844d28e79608ef9c:f4b9b30200000000",
>             "last_event": "First row fetched",
>             "waiting": false,
>             "executing": true,
>             "waiting_time": "2s152ms",
>             "resource_pool": "default-pool"
>         }
>     ],
>
> I believe you can check for queries where "waiting" = true, and, for
> those query_ids then cancel over HTTP via:
> http://impalad:25000/cancel_query?query_id=844d28e79608ef9c:
> f4b9b30200000000
>
> I haven't scripted this up myself before but I think that would work.
>
>
> >
> > 2) Why does this happen?
> >
> >
> > It seems these queries are sometimes (not always) created by Hue. They
> will
> > eventually disappear but during the time (sometimes hours) they are in
> this
> > state they prevent other queries from running. The UI says "they must be
> > closed" in order to free the resources but it's not clear how you do that
> > without visiting *each* of the Impala Daemons and checking manually.
> >
> > It seems this *might* be related to
> > https://issues.cloudera.org/browse/IMPALA-1575. If so, i'm at the point
> > where I want to write a service that queries the API and just nukes these
> > hanging queries on some interval.
> >
>
> In general, because Impala cannot just close and unregister a query
> before the client acknowledges the final state. Some clients are
> notorious for leaving queries open (e.g. Hue) as it has an interface
> that explicitly lets the user page through results at their leisure.
> As a result, Hue doesn't close queries until you close the tab
> (relying on a javascript callback) or explicitly through Hue. That's
> why we added a session timeout, so when a session isn't used for some
> amount of time all queries are fully removed and the session closed.
> Many users end up setting this lower than the default to mitigate
> these issues. Set it as lower as long as you won't frustrate users
> that expect to page through query results. This is why the query state
> is left in 'waiting to be closed'. Without further improvements in
> Impala though (IMPALA-1575), this has the secondary effect of
> sometimes holding resources (this is bad when using admission control
> in particular). We do plan to improve that when we can.
>
> I'd also look at where the other (non Hue) sessions are coming from,
> check the /sessions page.
>
>
>
> > -William
>

Mime
View raw message