ambari-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hurley (JIRA)" <>
Subject [jira] [Updated] (AMBARI-16131) Prevent Views From Causing a Loss of Service For Ambari
Date Wed, 27 Apr 2016 18:17:13 GMT


Jonathan Hurley updated AMBARI-16131:
    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

> Prevent Views From Causing a Loss of Service For Ambari
> -------------------------------------------------------
>                 Key: AMBARI-16131
>                 URL:
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-views
>    Affects Versions: 2.0.0
>            Reporter: Jonathan Hurley
>            Assignee: Jonathan Hurley
>            Priority: Critical
>             Fix For: 2.4.0, 2.2-next
>         Attachments: AMBARI-16131.patch
> The underlying problem is that views are accessed off of the REST endpoint ({{/api/v1/views}}).
This means that the Ambari REST API connector is going to handle the request from its own
threadpool. There is no way to configure Jetty to use a different threadpool for the same
connector. As a result, if a request to load a view holds the Jetty thread hostage, eventually
we will see thread starvation and loss of service.
> An example of this situation is a view which makes an innocent request to a remote resource.
If the view's request has a timeout of 60 seconds, then the Jetty thread is going to be held
for that amount of time. With concurrent users and multiple instances of that view deployed,
the Jetty threadpool can becomes exhausted quickly.
> Although there are more graceful ways of handling this situation, they mostly involve
substantial re-architecture and design:
> - The use of a new connector and threadpool would require binding to another port for
view requests. This will cause problems with "local" views and their assumption that if they
run on the Ambari server they can share the same session.
> - The use of a [Continuation|] in
Jetty which can suspend the incoming request. We would need the ability for views to signal
that they have completed their work in order to proceed with the suspended request.
> A quicker and far less invasive fix would be to create a filter which intercepts requests
for views. It will determine how many executing view requests exist and decide if it will
allow the new request through. For example, if configured to allow a maximum of 10 concurrent
view requests, then the 11th request would be denied with an {{HTTP 503 - Service Unavailable}}.
Although the thread is temporarily used while the filter is processing, it's quickly returned
to the Jetty pool when it's determined there are too many other running view requests.

This message was sent by Atlassian JIRA

View raw message