Mailing-List: contact user-help@storm.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@storm.incubator.apache.org
Received-SPF: pass (nike.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
MIME-Version: 1.0
In-Reply-To: 
 <CAOYa3yh-y9QUka2NB995AamgxyNW0NJixHgSzJOP=oYmRW0wnw@mail.gmail.com>
References: 
 <CANCZ-n19XpjEoiZGZBPBsDL3iH-E-ReGbxJeOe9p-G6MnBHEZQ@mail.gmail.com>
	<CAOYa3yh-y9QUka2NB995AamgxyNW0NJixHgSzJOP=oYmRW0wnw@mail.gmail.com>
Date: Fri, 10 Jan 2014 10:19:33 -0500
Message-ID: 
 <CAP2ryofAG76YYOUB4Jp-7dW_1zi=Amnqa=Pn8sS8+N-8ma9paA@mail.gmail.com>
Subject: Re: Large binary payloads with storm
From: Jon Logan <jmlogan@buffalo.edu>
To: user@storm.incubator.apache.org
Content-Type: multipart/alternative; boundary=089e013d0f48de0a7b04ef9f405a

--089e013d0f48de0a7b04ef9f405a
Content-Type: text/plain; charset=ISO-8859-1

You're going to run into issues if you have large tuples, because they are
buffered in memory. I would suggest moving it to an exterior channel, like
Redis, etc, and only passing meta-data through Storm.

Your other solution is to use quirky things like reflection to prevent your
application from running out of memory when tuples are buffered.


On Fri, Jan 10, 2014 at 8:49 AM, Ruhollah Farchtchi <
ruhollah.farchtchi@gmail.com> wrote:

> I am using storm to process small (< 100k) image files. I don't have a
> real-time requirement as yet, but my bottle neck is more in the image
> processing than message passing between bolts. I am using the Clojure DSL
> and the python bolt. Everything I've put together right now is very much a
> prototype so my next steps are some further processing and integration.
> Passing byte arrays didn't seem to work so well so I have had to
> encode/decode into base64 binary as it seems the JSON parsers on the python
> side didn't like byte arrays. I plan to go back and perhaps re-do the
> integration with a native C++ bolt, however I believe that there are other
> ways to do this integration as well. I'm As with Wilson, I'm interested if
> anyone else is using Storm to process binary payloads and what they have
> found works.
>
> Thanks,
>
> Ruhollah
>
> Ruhollah Farchtchi
> ruhollah.farchtchi@gmail.com
>
>
> On Thu, Jan 9, 2014 at 10:24 PM, Lochlainn Wilson <
> lochlainn.wilson@gmail.com> wrote:
>
>> Hi all,
>>
>> I am new to Storm and have been tasked with determining whether it is
>> feasible for us to use Apache storm in my company. I have of course
>> configured the sample projects and have been poking around. A red flag is
>> raised with the "stream processing" style JSON parsing.
>>
>> I am considering using storm with real time image processing bolts in
>> C++. Packaging binary data into a JSON (by escaping it) looks like it will
>> be slow and expensive. Is there a better way? Does anyone have experience
>> processing large streams of binary data through storm?
>>
>> How did it go?
>>
>> Regards,
>>
>> Lochlainn
>>
>
>

--089e013d0f48de0a7b04ef9f405a
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">You&#39;re going to run into issues if you have large tupl=
es, because they are buffered in memory. I would suggest moving it to an ex=
terior channel, like Redis, etc, and only passing meta-data through Storm.<=
div>
<br></div><div>Your other solution is to use quirky things like reflection =
to prevent your application from running out of memory when tuples are buff=
ered.</div></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quo=
te">
On Fri, Jan 10, 2014 at 8:49 AM, Ruhollah Farchtchi <span dir=3D"ltr">&lt;<=
a href=3D"mailto:ruhollah.farchtchi@gmail.com" target=3D"_blank">ruhollah.f=
archtchi@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quot=
e" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir=3D"ltr">I am using storm to process small (&lt; 100k) image files.=
 I don&#39;t have a real-time requirement as yet, but my bottle neck is mor=
e in the image processing than message passing between bolts. I am using th=
e Clojure DSL and the python bolt. Everything I&#39;ve put together right n=
ow is very much a prototype so my next steps are some further processing an=
d integration. Passing byte arrays didn&#39;t seem to work so well so I hav=
e had to encode/decode into base64 binary as it seems the JSON parsers on t=
he python side didn&#39;t like byte arrays. I plan to go back and perhaps r=
e-do the integration with a native C++ bolt, however I believe that there a=
re other ways to do this integration as well. I&#39;m As with Wilson, I&#39=
;m interested if anyone else is using Storm to process binary payloads and =
what they have found works.<div>

<br></div><div>Thanks,</div><div><br></div><div>Ruhollah</div></div><div cl=
ass=3D"gmail_extra"><span class=3D"HOEnZb"><font color=3D"#888888"><br clea=
r=3D"all"><div>Ruhollah Farchtchi<br><a href=3D"mailto:ruhollah.farchtchi@g=
mail.com" target=3D"_blank">ruhollah.farchtchi@gmail.com</a></div>
</font></span><div><div class=3D"h5">
<br><br><div class=3D"gmail_quote">On Thu, Jan 9, 2014 at 10:24 PM, Lochlai=
nn Wilson <span dir=3D"ltr">&lt;<a href=3D"mailto:lochlainn.wilson@gmail.co=
m" target=3D"_blank">lochlainn.wilson@gmail.com</a>&gt;</span> wrote:<br><b=
lockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px =
#ccc solid;padding-left:1ex">

<div dir=3D"ltr"><div><div>Hi all,<br><br>I am new to Storm and have been t=
asked with=20
determining whether it is feasible for us to use Apache storm in my=20
company. I have of course configured the sample projects and have been=20
poking around. A red flag is raised with the &quot;stream processing&quot; =
style=20
JSON parsing. <br>
<br>I am considering using storm with real time image processing bolts in C=
++. Packaging binary data into a JSON (by escaping it) looks like it will b=
e slow and expensive. Is there a better way? Does anyone have experience pr=
ocessing large streams of binary data through storm?<br>


<br>
</div>How did it go?<br><br></div>Regards,<br><br>Lochlainn</div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>

--089e013d0f48de0a7b04ef9f405a--