incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <>
Subject Re: connect db for0.5/0.6
Date Tue, 14 May 2013 09:25:23 GMT

On May 14, 2013, at 10:47 , Sky Zhao wrote:

> Thanks,
> I still think how to design PE as DB output results, you know if too much PE instances
connect DB, cause the connection bottle neck, does it must be singleton PE?
> But if the PE not the final results, how design DB PE to connect DB?

I/O in general, and operations to a database in particular, might indeed induce some latency
to the processing of events: synchronous calls to the database from the PE will delay the
processing of other enqueued events. But it really depends on the nature of the database (some
are write-optimized or tunable), the interface to the database, the rate of output, the latency
constraints and the design of your application.

Possible improvements include: batching messages to the DB (singleton / "aggregator" PE might
help), using periodic outputs (i.e. not for every single event), increasing the parallelism
of a given stream so that multiple events can be processed in parallel (see the "setParallelism"
method when creating a stream), using a database connection pool etc..

Hope this helps,


> /Sky
> From: Matthieu Morel [] 
> Sent: Saturday, May 11, 2013 2:09 AM
> To:
> Subject: Re: connect db for0.5/0.6
> There is very little code in common between 0.3 and 0.6, and the feature you mention
is not available off-the shelf with S4 0.6. Does not seem to hard to add though.
> Matthieu
> On May 10, 2013, at 03:44 , Sky Zhao wrote:
>> In 0.3, can use spring container to config clustream-conf.xml to set db settings,
>> in 0.5/0.6 using s4 class how to set db connection as output results?
>> /Sky

View raw message