incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth <>
Subject RE: S4-0.6.0 and Hadoop Yarn
Date Thu, 04 Apr 2013 05:01:42 GMT
Hi - Can the development team highlight the exact solution/fix that made it
possible for 0.6 release to be so fast compared to the earlier release.


Thanks in advance,




From: Matthieu Morel [] 
Sent: Wednesday, April 03, 2013 3:02 PM
Subject: Re: S4-0.6.0 and Hadoop Yarn


On Apr 2, 2013, at 19:46 , Jeryl Cook wrote:

"handle 200K+ messages per sec"  ,in one instance? or do you mean clustered?


This is for processing small events injected into 1 stream on 1 node. By
using more streams and more nodes the overall throughput can get quite


Note that this is a baseline with a basic PE graph (1 injector and 1 PE
prototype) and performance in practice will be impacted by the complexity of
the application and the nature of the processing, the hardware and allocated
resources, the size and complexity of messages etc..


A benchmarking framework is included in the distribution, so you can
reproduce the experiments.








On Mon, Apr 1, 2013 at 10:42 PM, JiHyoun Park <> wrote:


I am testing the newest release of S4.
It's fantastic that the stream throughput of S4 0.6.0 has been improved to
handle 200K+ messages per sec.!
However, it seems that S4-25 branch - deploying S4 applications with Yarn -
is not included in the 0.6.0 package yet. 
I already built a system to run S4 applications on Yarn and want to migrate
its S4 framework from 0.5.0 to 0.6.0.
How can I use the 'deploying S4 applications with Yarn' feature on S4 0.6.0?

Best Regards

Jeryl Cook
Founder & Chief Executive Officer
VanitySoft, Inc.
A Geo Business Intelligence Technology Consulting Firm <>
Get answers to "who knew what, when, and where"... and everything in

This message contains information which may be confidential and privileged.
Unless you are the addressee (or authorized to receive for the addressee),
you may not use, copy or disclose to anyone the message or any information
contained in the message. If you have received the message in error, please
advise the sender by reply e-mail, and delete the


View raw message