mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anand Mazumdar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MESOS-3562) Anomalous bytes in stream from HTTPI Api
Date Thu, 01 Oct 2015 02:36:04 GMT

    [ https://issues.apache.org/jira/browse/MESOS-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14939194#comment-14939194
] 

Anand Mazumdar commented on MESOS-3562:
---------------------------------------

[~BenWhitehead] There seems to be some confusion here. Comments Inline.

> This isn't really standard chunks though, there are chunks within chunks and the configuration
of the client would have to know that.
Can you elaborate a bit more on what do you mean by chunks between chunks here ? We strictly
adhere to the standard chunk encoding format defined in RFC 2616. The only difference here
is that the {{data}} in chunks itself is encoded in {{RecordIO}} format.

> What is the motivation behind using recordio format ?
We wanted a way to delimit two events for JSON/Protobuf responses and RecordIO format allowed
us to do that. We could have gone away with RecordIO for JSON though by just delimiting on
{{\n}} but that would have made it inconsistent in behavior when compared to Protobuf Responses.

>  If standard encoding were used then every HTTP client would already have the necessary
understanding to know how to deal with the chunks.
We use standard chunk encoding as defined in RFC. What do you mean here ?

> Where is the specification for what recordio format is? I have not been able to find
anything online.
We should add more information on this in our docs. For now, till we do that, here is a brief
description on what the format looks like:
{code}
5\n
hello
6\n
world!
{code}

Ideally, whatever client you are using should do the de-chunking for you. You should get this
back from the client i.e. just the {{RecordIO}} encoded data.
{code}
104\n{"subscribed":{"framework_id":{"value":"20150930-103028-16777343-5050-11742-0028"}},"type":"SUBSCRIBED"}
{code}

cc'ing [~bmahler] If I missed anything.

> Anomalous bytes in stream from HTTPI Api
> ----------------------------------------
>
>                 Key: MESOS-3562
>                 URL: https://issues.apache.org/jira/browse/MESOS-3562
>             Project: Mesos
>          Issue Type: Bug
>          Components: HTTP API
>    Affects Versions: 0.24.0
>         Environment: Linux 3.16.7-24-desktop #1 SMP PREEMPT Mon Aug 3 14:37:06 UTC 2015
(ec183cc) x86_64 x86_64 GNU/Linux
> Mesos 0.24.0
> gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch revision 212064]
>            Reporter: Ben Whitehead
>            Priority: Blocker
>              Labels: http, wireprotocol
>         Attachments: app.log, tcpdump.log
>
>
> When connecting to the new HTTP Api and attempting to {{SUBSCRIBE}} there are some anomalous
bytes contained in the chunked stream that appear to be causing problems when I attempting
to integrate.
> Attached are two log files. app.log represents my application trying to connect to mesos
using RxNetty. Netty has been configured to log all data it sends/receives over the wire this
can be seen in the byte blocks in the log. 
> The client is constructing a protobuf in java for the subscribe call  
> {code:java}
>         final Call subscribeCall = Call.newBuilder()
>             .setType(Call.Type.SUBSCRIBE)
>             .setSubscribe(
>                 Call.Subscribe.newBuilder()
>                     .setFrameworkInfo(
>                         Protos.FrameworkInfo.newBuilder()
>                             .setUser("bill")
>                             .setName("testing_this_shit_out")
>                             .build()
>                     )
>             )
>             .build();
> {code}
>  
> lient sends the protobuf to mesos with the following request headers:
> {code}
> POST /api/v1/scheduler HTTP/1.1
> Content-Type: application/x-protobuf
> Accept: application/json
> Content-Length: 35
> Host: localhost:5050
> User-Agent: RxNetty Client
> {code}
> The body is then serialized via protobuf and sent.
> The response from the mesos master has the following headers:
> {code}
> HTTP/1.1 200 OK
> Transfer-Encoding: chunked
> Date: Wed, 30 Sep 2015 21:07:16 GMT
> Content-Type: application/json
> {code}
> followed by 
> {code}
> \r\n\r\n6c\r\n104\n{"subscribed":{"framework_id":{"value":"20150930-103028-16777343-5050-11742-0028"}},"type":"SUBSCRIBED"}
> {code}
> The {{\r\n\r\n}} is expected for standard http bodies, how ever {{6c\r\n}} doesn't appear
to be attached to anything. {{104}} is the correct length of the Subscribe events JSON.
> What is this extra number and why is it there?
> This is not the first time confusion has come up related to the wire format for the event
stream from the new http api see [this|http://mail-archives.apache.org/mod_mbox/mesos-user/201508.mbox/%3C94D2C9E8-2FE8-4C11-B0D3-859DAC654313@me.com%3E]
message from the mailing list.
> In the [Design Doc|https://docs.google.com/document/d/1pnIY_HckimKNvpqhKRhbc9eSItWNFT-priXh_urR-T0/edit#]
there is a statement that said 
> {quote}
> All subsequent events that are relevant to this framework  generated by Mesos are streamed
on this connection. Master encodes each Event in RecordIO format, i.e., string representation
of length of the event followed by JSON or binary Protobuf  (possibly compressed) encoded
event. 
> {quote}
> There is no specification I've been able to find online that actually explains this format.
The only reference I can find to it is some sample go code.
> The attached tcpdump.log contains a tcp dump between the mesos master and my client collected
using the following command {{tcpdump -xx -n -i lo "dst port 5050" or "src port 5050" 2>&1
| tee /tmp/tcpdump.log}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message