storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ruhollah Farchtchi <ruhollah.farcht...@gmail.com>
Subject Re: Large binary payloads with storm
Date Fri, 10 Jan 2014 13:49:52 GMT
I am using storm to process small (< 100k) image files. I don't have a
real-time requirement as yet, but my bottle neck is more in the image
processing than message passing between bolts. I am using the Clojure DSL
and the python bolt. Everything I've put together right now is very much a
prototype so my next steps are some further processing and integration.
Passing byte arrays didn't seem to work so well so I have had to
encode/decode into base64 binary as it seems the JSON parsers on the python
side didn't like byte arrays. I plan to go back and perhaps re-do the
integration with a native C++ bolt, however I believe that there are other
ways to do this integration as well. I'm As with Wilson, I'm interested if
anyone else is using Storm to process binary payloads and what they have
found works.

Thanks,

Ruhollah

Ruhollah Farchtchi
ruhollah.farchtchi@gmail.com


On Thu, Jan 9, 2014 at 10:24 PM, Lochlainn Wilson <
lochlainn.wilson@gmail.com> wrote:

> Hi all,
>
> I am new to Storm and have been tasked with determining whether it is
> feasible for us to use Apache storm in my company. I have of course
> configured the sample projects and have been poking around. A red flag is
> raised with the "stream processing" style JSON parsing.
>
> I am considering using storm with real time image processing bolts in C++.
> Packaging binary data into a JSON (by escaping it) looks like it will be
> slow and expensive. Is there a better way? Does anyone have experience
> processing large streams of binary data through storm?
>
> How did it go?
>
> Regards,
>
> Lochlainn
>

Mime
View raw message