incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: connect db for0.5/0.6
Date Tue, 14 May 2013 09:25:23 GMT

On May 14, 2013, at 10:47 , Sky Zhao wrote:

> Thanks,
>  
> I still think how to design PE as DB output results, you know if too much PE instances
connect DB, cause the connection bottle neck, does it must be singleton PE?
>  
> But if the PE not the final results, how design DB PE to connect DB?


I/O in general, and operations to a database in particular, might indeed induce some latency
to the processing of events: synchronous calls to the database from the PE will delay the
processing of other enqueued events. But it really depends on the nature of the database (some
are write-optimized or tunable), the interface to the database, the rate of output, the latency
constraints and the design of your application.

Possible improvements include: batching messages to the DB (singleton / "aggregator" PE might
help), using periodic outputs (i.e. not for every single event), increasing the parallelism
of a given stream so that multiple events can be processed in parallel (see the "setParallelism"
method when creating a stream), using a database connection pool etc..

Hope this helps,

Matthieu


>  
>  
> /Sky
> 
> From: Matthieu Morel [mailto:mmorel@apache.org] 
> Sent: Saturday, May 11, 2013 2:09 AM
> To: s4-user@incubator.apache.org
> Subject: Re: connect db for0.5/0.6
> 
> There is very little code in common between 0.3 and 0.6, and the feature you mention
is not available off-the shelf with S4 0.6. Does not seem to hard to add though.
> 
> Matthieu
> 
> 
> On May 10, 2013, at 03:44 , Sky Zhao wrote:
> 
>> In 0.3, can use spring container to config clustream-conf.xml to set db settings,
>> in 0.5/0.6 using s4 class how to set db connection as output results?
>>  
>>  
>> /Sky
> 


Mime
View raw message