flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eleanore Jin <eleanore....@gmail.com>
Subject Re: run flink on edge vs hub
Date Tue, 19 May 2020 04:39:26 GMT
Hi Arvid,

Thanks for the suggestion! I will tryout to see how it works.

Best,
Eleanore

On Mon, May 18, 2020 at 8:04 AM Arvid Heise <arvid@ververica.com> wrote:

> Hi Eleanore,
>
> The question in general is what you understand under edge data centers as
> the term is pretty fuzzy. Since Flink is running on Java, it's not suitable
> for embedded clusters as of now. There is plenty of work done already to
> tests that Flink runs on ARM clusters [1].
>
> If you just mean in general moving away from a monolithic hub cluster to
> smaller clusters, then this is easily done with Flink on the compute side.
> The question is rather how data storage should look in such an edge setting
> and how the interfaces look.
>
> From your example, it seems as if you want to use Flink as a reactive
> server, possibly easily scalable. If so, then yes it is possible with
> Flink, even though I'd say it's not the primary use case for Flink. In any
> case, synchronous requests will be a bit difficult/unnatural. I'd probably
> go for an async job pattern. So Flink listens to some port for requests (
> socketTextStream [2]) with a job id, processes data and keeps the data in
> state keyed by job id. The client then uses the job id to fetch the job
> state through queryable state [2]. The responses eventually time out
> through TTL [4].
>
> Of course, you'd put a small proxy in front of that composited job
> (separate input/query port) that translates the queries from the client to
> the Flink job. The proxy would most likely also generate the job id and
> return it to the client. Ultimately, that proxy could offer a synchronous
> interface and pull for the result itself, but that makes the proxy suddenly
> quite heavy.
>
> The proxy setup can be reused for different edge clusters making it a one
> time investment. Note that there are other software stacks for reactive
> servers that offer the functionality out of the box.
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-ARM-support-for-Flink-td30298.html
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/datastream_api.html#data-sources
> [3]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html
> [4]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl
>
> On Mon, May 18, 2020 at 4:39 AM Eleanore Jin <eleanore.jin@gmail.com>
> wrote:
>
>> Hi Community,
>>
>> Currently we are running flink in 'hub' data centers where data is
>> ingested into the platform via kafka, and flink job will read from kafka,
>> do the transformations, and publish to another kafka topic.
>>
>> I would also like to see if the same logic (read input message -> do
>> transformation -> return output message) can be applied on 'edge' data
>> centers.
>>
>> The requirement for run on 'edge' is to return the response
>> synchronously. Like the synchronous http based request/response.
>>
>> Can you please provide some guidance/thoughts on this?
>>
>> Thanks a lot!
>> Eleanore
>>
>>
>
> --
>
> Arvid Heise | Senior Java Developer
>
> <https://www.ververica.com/>
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Toni) Cheng
>

Mime
View raw message