incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sky Zhao <>
Subject RE: connect db for0.5/0.6
Date Wed, 15 May 2013 09:13:39 GMT
Yes, it can use different use cases with different ways.


From: Matthieu Morel []
Sent: Tuesday, May 14, 2013 5:25 PM
Subject: Re: connect db for0.5/0.6

On May 14, 2013, at 10:47 , Sky Zhao wrote:


I still think how to design PE as DB output results, you know if too much PE instances connect
DB, cause the connection bottle neck, does it must be singleton PE?

But if the PE not the final results, how design DB PE to connect DB?

I/O in general, and operations to a database in particular, might indeed induce some latency
to the processing of events: synchronous calls to the database from the PE will delay the
processing of other enqueued events. But it really depends on the nature of the database (some
are write-optimized or tunable), the interface to the database, the rate of output, the latency
constraints and the design of your application.

Possible improvements include: batching messages to the DB (singleton / "aggregator" PE might
help), using periodic outputs (i.e. not for every single event), increasing the parallelism
of a given stream so that multiple events can be processed in parallel (see the "setParallelism"
method when creating a stream), using a database connection pool etc..

Hope this helps,



From: Matthieu Morel []
Sent: Saturday, May 11, 2013 2:09 AM
Subject: Re: connect db for0.5/0.6

There is very little code in common between 0.3 and 0.6, and the feature you mention is not
available off-the shelf with S4 0.6. Does not seem to hard to add though.


On May 10, 2013, at 03:44 , Sky Zhao wrote:

In 0.3, can use spring container to config clustream-conf.xml to set db settings,
in 0.5/0.6 using s4 class how to set db connection as output results?


View raw message