flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ritesh Kumar Singh <riteshoneinamill...@gmail.com>
Subject Re: Actual byte-streams in multiple-node pipelines
Date Wed, 20 Jan 2016 12:39:13 GMT
I think with sufficient processing power flink can do the above mentioned
task using the stream api
<https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/index.html>
.

Thanks,
*Ritesh Kumar Singh,*
*https://riteshtoday.wordpress.com/* <https://riteshtoday.wordpress.com/>

On Wed, Jan 20, 2016 at 11:18 AM, Tal Maoz <magogo200@gmail.com> wrote:

> Hey,
>
>
>
> I’m a new user to Flink and I’m trying to figure out if I can build a
> pipeline I’m working on using Flink.
>
> I have a data source that sends out a continues data stream at a bandwidth
> of anywhere between 45MB/s to 600MB/s (yes, that’s MiB/s, not Mib/s, and
> NOT a series of individual messages but an actual continues stream of data
> where some data may depend on previous or future data to be fully
> deciphered).
>
> I need to be able to pass the data through several processing stages (that
> manipulate the data but still produce the same order of magnitude output at
> each stage) and I need processing to be done with low-latency.
>
> The data itself CAN be segmented but the segments will be some HUGE
> (~100MB – 250MB) and I would like to be able to stream data in and out of
> the processors ASAP instead of waiting for full segments to be complete at
> each stage (so bytes will flow in/out as soon as they are available).
>
>
>
> The obvious solution would be to split the data into very small buffers,
> but since each segment would have to be sent completely to the same
> processor node (and not split between several nodes), doing such
> micro-batching would be a bad idea as it would spread a single segment’s
> buffers between multiple nodes.
>
>
>
> Is there any way to accomplish this with Flink? Or is Flink the wrong
> platform for that type of processing?
>
>
>
> Any help would be greatly appreciated!
>
>
>
> Thanks,
>
>
>
> Tal
>

Mime
View raw message