storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From churly lin <chury...@gmail.com>
Subject Re: questions about multilang bolt's STDIN&STDOUT
Date Thu, 09 Jan 2014 07:08:10 GMT
Thank you Verardi!
Sorry for my poor English that making my question ambiguously.  Your answer
is so clearly.  Now I know that the byte arrays can be used as "values" for
the field "tuple".
One more question, In my project, I emit byte array tuples in KafakSpout
like this:
    List<Object> tup =
_spoutConfig.scheme.deserialize(Utils.toByteArray(toEmit.msg.payload()));
 // byte array
    collector.emit(tup), new KafkaMessageId(_partition, toEmit.offset));
But when I tried to get tuple in ShellBolt, readMsgs() got an exception.
And I write the emitting JSON message to file, it looks like:
*{"id":"7617035644022584549","stream":"default","comp":"KafkaSpout","tuple":[[B@6d695bcc],"task":1}*
It looks very weird for me. What is the *[[B@6d695bcc]*? Is it a byte
array's object address? Can It be read by Python?
Going even further, If I insist on emitting byte arrays in KafkaSpout, What
should I do to readMsgs in Python Bolt?

Thanks again.


2014/1/9 Antonio Verardi <antonio@yelp.com>

> Hi,
>
> I am extensively using the multilang interface for Python. JSON is the way
> you serialize things for communication. It adds a fairly amount of
> overhead, but it is a reasonable design choice in terms of a multilang
> interface.
>
> If your question is: can I read byte array messages from a bolt (made up
> by command, id, stream, task and tuple), the answer is "that's not that
> easy, you should implement something in order to do that".
>
> If your question is: can I serialize byte arrays in JSON with Python and
> use them as "values" for the field "tuple", the answer is: "yes, even
> though JSON always produce string objects". [
> http://docs.python.org/3.3/library/json.html#basic-usage]. You may want
> to modify storm.py, in order to do that, or simply encode and decode your
> data within your own bolt, it depends on your needs.
>
> This is something I found just googling about encoding binary data in JSON:
> http://bytes.com/topic/python/answers/681314-simplejson-pack-binary-data
>
> I hope it was what you were looking for,
> Antonio Uccio Verardi
>
>
>
>
> On Tue, Jan 7, 2014 at 11:24 PM, churly lin <churylin@gmail.com> wrote:
>
>> Hi all,
>>
>> I am trying to write a topology with a KafkaSpout and a
>> ShellBolt(implemented by python ).
>> According to the Multilang-protocol<https://github.com/nathanmarz/storm/wiki/Multilang-protocol>,
>> multilang uses json messages over stdin/stdout to communicate with the
>> subprocess. Specially, *both ends of this protocol use a line-reading
>> mechanism. *Does it mean that, in multilang, we could not emit message
>> as byte array? If not, how to read a byte array tuple in a python bolt ?
>> the json which was read by python bolt is look like:
>>
>>
>> {
>>         "command": "emit",
>>         // The id for the tuple. Leave this out for an unreliable emit. The id can
>>     // be a string or a number.
>>         "id": "1231231",
>>         // The id of the stream this tuple was emitted to. Leave this empty to emit
to default stream.
>>         "stream": "1",
>>         // If doing an emit direct, indicate the task to send the tuple to
>>         "task": 9,
>>         // All the values in this tuple
>>         "tuple": ["field1", 2, 3]}
>>
>> This example shows that, the "tuple" can be String("field1") and
>> number(2, 3). Could it be a byte array?
>>
>
>

Mime
View raw message