mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Sallou <olivier.sal...@irisa.fr>
Subject Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed
Date Thu, 08 Nov 2018 09:17:20 GMT


----- Mail original -----
> De: "Olivier Sallou" <olivier.sallou@irisa.fr>
> À: user@mesos.apache.org
> Envoyé: Mercredi 7 Novembre 2018 12:40:24
> Objet: Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed

> On 11/7/18 11:54 AM, Cecile, Adam wrote:
>> Hi,
>>
>> You might be hitting the same bug as I did (no HTTP/2 support in code
>> pulling images for Mesos).
>> https://issues.apache.org/jira/browse/MESOS-9364
> 
> adding some logs as suggested in your issue, error code is different,
> getting "invalid constant string" error from http_parser error:
> 
> 
>      XX(INVALID_CONSTANT, "invalid constant string")


with additional debug, problem is related to http parser vs received answer from dockerhub.
If I remove control on parsed data length vs body length  it works nicely (for test/debug
only....), so it may be a pb of response http compliance...
Anyway, this issue prevents download of docker images when using unified containerizer. As
I am using code from master branch (latest), I can only hope it will be fixed before next
release....



> 
> 
>>
>> My report also include some code you can add in C++ code of the fetcher
>> to retrieve the actual message coming from the http response parser library.
>>
>> Regards, Adam.
>>
>> On 11/7/18 11:42 AM, Adam Cecile wrote:
>>> On 11/7/18 10:48 AM, Olivier Sallou wrote:
>>>> On 11/7/18 10:38 AM, Olivier Sallou wrote:
>>>>> Hi,
>>>>>
>>>>> I installed mesos from source. It works fine with docker containerizer.
>>>>>
>>>>> Howerver it fails with  unified containerizer at container start.
>>>>>
>>>>> It used to work on a previous (older release) install. In the
>>>>> meanwhile,
>>>>> some system libs etc.. have been upgraded.
>>>>>
>>>>> In logs I have the following:
>>>>>
>>>>>
>>>>> I1107 09:32:48.707176 31983 containerizer.cpp:1280] Starting container
>>>>> 28f07a61-676a-4876-aae4-73598de90aae
>>>>> E1107 09:32:49.683372 31986 slave.cpp:6168] Container
>>>>> '28f07a61-676a-4876-aae4-73598de90aae' for executor '1-0' of framework
>>>>> 80fc2079-ba14-454b-8276-79fae090f8b3-0000 failed to start: Failed to
>>>>> decode HTTP responses: Decoding failed
>>>>> HTTP/1.1 200 OK
>>>>> Content-Type: application/json
>>>>> Date: Wed, 07 Nov 2018 08:32:46 GMT
>>>>> Transfer-Encoding: chunked
>>>>> Strict-Transport-Security: max-age=31536000
>>>>>
>>>>> 105c
>>>>> {"token":"eyJhbGciOiJSUzI1NiIsInR5cC.......
>>>>>
>>>>>
>>>>> Logs do not show the destination of the http request (a pull on docker
>>>>> hub ? a request to master ? ...)
>>>> I could increase some slave logging, and HTTP failure occurs with
>>>> pulling of image:
>>>>
>>>>      I1107 10:45:56.689092 31987 registry_puller.cpp:286] Pulling image
>>>> 'library/centos:latest' from
>>>> 'docker-manifest://registry-1.docker.io:443library/centos?latest#https'
>>>> to '/tmp/mesos/store/docker/staging/99WUh3'
>>>>      E1107 10:45:57.634601 31987 slave.cpp:6168] Container
>>>> '48ea5811-3f97-41c1-b1a5-9a4416552545' for executor '6-0' of framework
>>>> 80fc2079-ba14-454b-8276-79fae090f8b3-0000 failed to start: Failed to
>>>> decode HTTP responses: Decoding failed
>>>>      HTTP/1.1 200 OK
>>>>      Content-Type: application/json
>>>>      Date: Wed, 07 Nov 2018 09:45:54 GMT
>>>>      Transfer-Encoding: chunked
>>>>      Strict-Transport-Security: max-age=31536000
>>>>
>>>>      ....
>>>>
>>>>
>>>> so it seems there is an issue with mesos unified management with docker
>>>> hub. Could it be related to libcurl version? (libcurl4 on my system). Is
>>>> specific setup needed?
>>>>
>>>>
>>>>
>>>>> any idea on what could be wrong ? or how to get more debug info ?
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> Olivier
>>>>>
>>>>>
> --
> Olivier Sallou
> Univ Rennes, Inria, CNRS, IRISA
> Irisa, Campus de Beaulieu
> F-35042 RENNES - FRANCE
> Tel: 02.99.84.71.95
> 
> gpg key id: 4096R/326D8438  (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438

Mime
View raw message