incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sky Zhao <sky.z...@ericsson.com>
Subject About 200,000 events/s
Date Mon, 24 Jun 2013 09:19:57 GMT
I try to use Adapter to send s4 events. With metrics report,
20,10462,88.63259092217602,539.6449108859357,18.577650313690874,6.241814566462701
40,36006,417.83633322358764,914.1057643161282,97.55624823196746,33.40088245418529
60,63859,674.1012974987167,1075.2326549158463,176.33878995148274,61.646803531230724
80,97835,953.6282787690939,1232.2934375999696,271.48890371088254,96.56144395108957
100,131535,1162.2060916405578,1323.3704459079934,363.98505627735324,131.98430793014757
120,165282,1327.52314133145,1384.2675551261093,453.5195236495672,167.61679021575551
140,190776,1305.7285112621298,1368.4361242524062,504.7782182758366,191.36049732440895

20,000 events per 20s  => 1000 EVENTS/s

Very slow, I modify the S4_HOME/subprojects/s4-comm/bin/default.s4.comm.properties

s4.comm.emitter.class=org.apache.s4.comm.tcp.TCPEmitter
s4.comm.emitter.remote.class=org.apache.s4.comm.tcp.TCPRemoteEmitter
s4.comm.listener.class=org.apache.s4.comm.tcp.TCPListener

# I/O channel connection timeout, when applicable (e.g. used by netty)
s4.comm.timeout=1000

# NOTE: the following numbers should be tuned according to the application, use case, and
infrastructure

# how many threads to use for the sender stage (i.e. serialization)
#s4.sender.parallelism=1
s4.sender.parallelism=100
# maximum number of events in the buffer of the sender stage
#s4.sender.workQueueSize=10000
s4.sender.workQueueSize=100000
# maximum sending rate from a given node, in events / s (used with throttling sender executors)
s4.sender.maxRate=200000

# how many threads to use for the *remote* sender stage (i.e. serialization)
#s4.remoteSender.parallelism=1
s4.remoteSender.parallelism=100
# maximum number of events in the buffer of the *remote* sender stage
#s4.remoteSender.workQueueSize=10000
s4.remoteSender.workQueueSize=100000
# maximum *remote* sending rate from a given node, in events / s (used with throttling *remote*
sender executors)
s4.remoteSender.maxRate=200000

# maximum number of pending writes to a given comm channel
#s4.emitter.maxPendingWrites=1000
s4.emitter.maxPendingWrites=10000

# maximum number of events in the buffer of the processing stage
#s4.stream.workQueueSize=10000
s4.stream.workQueueSize=100000

only improve from 500 events 1000 events,

I read file 88m only need 8s, but send events cost 620s now for 1,237,632 events, why slow,
s4 can trigger 200,000 events/s, how I can do up to this values, pls give me detail instructions.





Mime
View raw message