flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: REST API "broken" on YARN because POST is not allowed via YARN proxy
Date Thu, 05 Apr 2018 18:10:35 GMT
This improvement is unfortunately out of scope for the 1.5 release since
the feature freeze is already quite some time ago. But I hope that this
improvement will make it into the 1.6 release.

Cheers,
Till

On Thu, Apr 5, 2018 at 4:45 PM, Juho Autio <juho.autio@rovio.com> wrote:

> Thanks for the answer. Wrapping with GET sounds good to me. You said next
> version; do you mean that Flink 1.5 would already include this improvement
> when it's released?
>
> On Thu, Apr 5, 2018 at 2:40 PM, Till Rohrmann <trohrmann@apache.org>
> wrote:
>
>> Hi Juho,
>>
>> you are right that due to a limitation in the Yarn proxy [1] we cannot
>> directly contact the cluster through the Yarn proxy.
>>
>> The way it works at the moment is that the Flink client retrieves the
>> AM's hostname through the ApplicationReport and then directly talks to the
>> AM. This of course requires that one can reach the respective Yarn
>> container. This is a limitation, though not a regression, which we want to
>> improve with the next version of Flink. An idea would be to wrap the REST
>> calls in a GET call to make them pass through the Yarn proxy.
>>
>> [1] https://issues.apache.org/jira/browse/YARN-2084
>>
>> Cheers,
>> Till
>>
>> On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <fhueske@gmail.com> wrote:
>>
>>> Hi Juho,
>>>
>>> Thanks for raising this point!
>>>
>>> I'll add Chesnay and Till to the thread who contributed to the REST API.
>>>
>>> Best, Fabian
>>>
>>> 2018-04-04 15:02 GMT+02:00 Juho Autio <juho.autio@rovio.com>:
>>>
>>>> I just learned that Flink savepoints API was refactored to require
>>>> using HTTP POST.
>>>>
>>>> That's fine otherwise, but makes life harder when Flink is run on top
>>>> of YARN.
>>>>
>>>> I've added example calls below to show how POST is declined by
>>>> the hadoop-yarn-server-web-proxy*, which only supports GET and PUT.
>>>>
>>>> Can you think of any solution to this? If I would be able to determine
>>>> the actual host & port for Flink UI, I could use that instead of the
proxy
>>>> address. But that would probably require opening at least one more port,
so
>>>> it's not the optimal solution either. Ideally I would have Flink REST API
>>>> completely accessible with GET and PUT methods.
>>>>
>>>> To me it seems like AWS EMR will also hit this issue as soon as they
>>>> start supporting Flink 1.5, because they seem to run Flink as a YARN app.
>>>>
>>>>
>>>> *) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-p
>>>> roject/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web
>>>> -proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/
>>>> WebAppProxyServlet.java#L296-L306
>>>>
>>>>
>>>> $ http POST http://10.0.10.71:20888/proxy/
>>>> application_1522844153347_0001/jobs/652e207f8578574d4a322e23
>>>> d4f8b908/checkpoints
>>>>
>>>> HTTP/1.1 405 HTTP method POST is not supported by this URL
>>>> Cache-Control: must-revalidate,no-cache,no-store
>>>> Content-Length: 1523
>>>> Content-Type: text/html; charset=iso-8859-1
>>>> Date: Wed, 04 Apr 2018 12:48:05 GMT
>>>> Date: Wed, 04 Apr 2018 12:48:05 GMT
>>>> Pragma: no-cache
>>>> Pragma: no-cache
>>>>
>>>> <html>
>>>> <head>
>>>> <meta http-equiv="Content-Type" content="text/html;
>>>> charset=ISO-8859-1"/>
>>>> <title>Error 405 HTTP method POST is not supported by this URL</title>
>>>> </head>
>>>> <body><h2>HTTP ERROR 405</h2>
>>>> <p>Problem accessing /proxy/application_15228441533
>>>> 47_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason:
>>>> <pre>    HTTP method POST is not supported by this URL</pre></p><hr
>>>> /><i><small>Powered by Jetty://</small></i><br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>> <br/>
>>>>
>>>> </body>
>>>> </html>
>>>>
>>>>
>>>> $ http PUT http://10.0.10.71:20888/proxy/application_1522844153347_0001
>>>> /jobs/652e207f8578574d4a322e23d4f8b908/checkpoints
>>>>
>>>> HTTP/1.1 404 Not Found
>>>> Access-Control-Allow-Origin: *
>>>> Cache-Control: no-cache
>>>> Content-Length: 25
>>>> Content-Type: application/json; charset=UTF-8
>>>> Date: Wed, 04 Apr 2018 12:48:09 GMT
>>>> Date: Wed, 04 Apr 2018 12:48:09 GMT
>>>> Expires: Wed, 04 Apr 2018 12:48:09 GMT
>>>> Expires: Wed, 04 Apr 2018 12:48:09 GMT
>>>> Pragma: no-cache
>>>> Pragma: no-cache
>>>>
>>>> {
>>>>     "errors": [
>>>>         "Not found."
>>>>     ]
>>>> }
>>>>
>>>> ^ expected to get this from flink because there's no PUT /checkpoints.
>>>>
>>>
>>>
>>
>

Mime
View raw message