incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: Congradulat for s4 0.6 incubating version released
Date Tue, 04 Jun 2013 13:15:54 GMT

On Jun 4, 2013, at 10:48 , Sky Zhao wrote:

> Thanks Mattieu, see below pls
>  
> From: Matthieu Morel [mailto:mmorel@apache.org] 
> Sent: Tuesday, June 04, 2013 4:32 PM
> To: s4-user@incubator.apache.org
> Subject: Re: Congradulat for s4 0.6 incubating version released
>  
>  
> On Jun 4, 2013, at 04:23 , Sky Zhao wrote:
> 
> 
> Here is one question,
>  
> S4 can send/receive 200,000 events/s, how s4 does it, more memory? more nodes? more machines?
> How get this big figure?
>  
> Have a look at the benchmarks subproject. (you'll need to slightly adapt the code since
it uses the old API though).
> 
> 
> I try an example by read files and send events, only 1000 events/s in one node one machine
2g memory.
>  
> This looks related to your application and the way you inject events. The reference benchmarks
don't do I/O except for passing messages. How is your code like? Are you outputting on the
console? How long is the run? Have you tried generating without reading from the file system?
Are you using a separate injector application?
>  
> Yes, I use new File(filename), then readline, and use adapter put the stream into events(like
getRemoteStream().put(event);) for every line. My PE output to console, also export to csv
by Metrics, 88M data(1237632 events) for 20 minutes sending in only one stream, no try without
reading, no other inject application, only one application, what is the reason do you think?

Outputting to the console from the PE is very likely to impact the performance.

Matthieu


Mime
View raw message