incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sky Zhao <sky.z...@ericsson.com>
Subject RE: Congradulat for s4 0.6 incubating version released
Date Wed, 05 Jun 2013 01:35:44 GMT
Thanks Matthieu and Daniel,

Just like benchmark project, using thread pool to send events with parallelism, I am ready
to test this,
I remember Matthieu said one way of improviment is to use setParallelism in PE, I don't know
whether can improve the events throughput?

/Sky


-----Original Message-----
From: Daniel Gómez Ferro [mailto:danielgf@yahoo-inc.com] 
Sent: Tuesday, June 04, 2013 9:37 PM
To: s4-user@incubator.apache.org
Cc: Matthieu Morel
Subject: Re: Congradulat for s4 0.6 incubating version released



On Tue Jun  4 15:15:54 2013, Matthieu Morel wrote:
>
> On Jun 4, 2013, at 10:48 , Sky Zhao wrote:
>
>> Thanks Mattieu, see below pls
>> *From:*Matthieu Morel [mailto:mmorel@apache.org] *Sent:*Tuesday, June 
>> 04, 2013 4:32 PM *To:*s4-user@incubator.apache.org 
>> <mailto:s4-user@incubator.apache.org>
>> *Subject:*Re: Congradulat for s4 0.6 incubating version released On 
>> Jun 4, 2013, at 04:23 , Sky Zhao wrote:
>>
>>
>> Here is one question,
>> S4 can send/receive 200,000 events/s, how s4 does it, more memory?
>> more nodes? more machines?
>> How get this big figure?
>> Have a look at the benchmarks subproject. (you'll need to slightly 
>> adapt the code since it uses the old API though).
>>
>>
>> I try an example by read files and send events, only 1000 events/s in 
>> one node one machine 2g memory.
>> This looks related to your application and the way you inject events.
>> The reference benchmarks don't do I/O except for passing messages. 
>> How is your code like? Are you outputting on the console?
>> How long is the run? Have you tried generating without reading from 
>> the file system? Are you using a separate injector application?
>> Yes, I use new File(filename), then readline, and use adapter put the 
>> stream into events(like getRemoteStream().put(event);) for every 
>> line. My PE output to console, also export to csv by Metrics, 88M
>> data(1237632 events) for 20 minutes sending in only one stream, no 
>> try without reading, no other inject application, only one 
>> application, what is the reason do you think?
>
> Outputting to the console from the PE is very likely to impact the 
> performance.
>
> Matthieu
>

In addition to removing the writes to the console, I'd suggest you to generate the events
instead of reading them, and maybe use more than one injector in order to saturate the processing
PE.

Cheers,

Daniel
Mime
View raw message