incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: socket for adapter
Date Tue, 13 Aug 2013 18:06:26 GMT
That looks like normal behaviour to me: if processing in PEs takes a long time, backpressure
applies and slows down / blocks upstream PEs or adapters.

Still not sure where / how / why you are using sockets directly though.

Matthieu


On Aug 13, 2013, at 03:31 , Sky Zhao <sky.zhao@ericsson.com> wrote:

> Thanks Matthieu, I found socket transmission performance is very slow, adapter has to
wait PE finished then sedn(in my case some algor and complex long process), if too quickly
 send, it will cause the system signal error,(block or refuse etc.), so I tried to add semaphore
to block socket, the speed not ideal.
>  
> /Sky
>  
> From: Matthieu Morel [mailto:mmorel@apache.org] 
> Sent: Monday, August 12, 2013 11:18 PM
> To: s4-user@incubator.apache.org
> Subject: Re: socket for adapter
>  
> If you have a lot of data to send, make sure you are chunking it - i.e. separating it
in multiple pieces. Then you can feed the stream processing system with those chunks. That
is typically the role of the adapter in S4.
>  
> It seems you are doing something else before the adapter but I don't see how that is
specific to S4.
>  
> Regards,
>  
> Matthieu
>  
>  
> On Aug 8, 2013, at 03:50 , Sky Zhao <sky.zhao@ericsson.com> wrote:
> 
> 
> I try to use socket as a socket server to receive the data stream, and use java socket
client program to send data stream(e.g. read big files),
> But in Ubuntu I can’t keep sending the files, always broke during the time, say: broken
pipe or connection reset etc.
>  
> So I change the strategy with interval time sending and increase the data content one
time, it is ok, as a stream feature,
>  
> 1)      How do I keep sending the stream
> 2)      Another issue, socket receiving data is very slower than reading files from adapter.
How to increase the speed?
>  
> Thanks


Mime
View raw message