flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasanna kumar <prasannakumarram...@gmail.com>
Subject Re: Need Help on Flink suitability to our usecase
Date Fri, 29 May 2020 07:13:40 GMT
Thanks Robert for the reply.

On Fri 29 May, 2020, 12:31 Robert Metzger, <rmetzger@apache.org> wrote:

> Hey Prasanna,
> (Side note: there is not need to send this email to multiple mailing
> lists. The user@ list is the right one)
> Let me quickly go through your questions:
> Is this usecase suited for flink ?
> Based on the details you've provided: Yes
> What you also need to consider are the hardware requirements you'll have
> for processing such amounts of data. I can strongly recommend setting up a
> small demo environment to measure the throughput of a smaller Flink cluster
> (say 10 machines).
> 1) If you do not have any consistency guarantees (data loss is
> acceptable), and you have good infrastructure in place to deploy and
> monitor such microservices then a microservice might also be an option.
> Flink is pretty well suited for heavy IO use-cases. Afaik Netflix has
> talked at several Flink Forward conferences about similar cases (check
> Youtube for recorded talks)
> 2) It should not be too difficult to build a small, generic framework on
> top of the Flink APIs
> 3) If you are deploying Flink on a resource manager like Kubernetes or
> YARN, they will take care of recovering your cluster if it goes down. Your
> recovery time will mostly depend on the state size that you are
> checkpointing (and the ability of your resource manager to bring up new
> resources). I don't think you'll be able to recover in < 500 milliseconds,
> but within a few seconds.
> I don't think that the other frameworks you are looking at are going to be
> much better at this.
> Best,
> Robert
> On Tue, May 19, 2020 at 1:28 PM Prasanna kumar <
> prasannakumarramani@gmail.com> wrote:
>> Hi,
>> I have the following usecase to implement in my organization.
>> Say there is huge relational database(1000 tables for each of our 30k
>> customers) in our monolith setup
>> We want to reduce the load on the DB and prevent the applications from
>> hitting it for latest events. So an extract is done from redo logs on to
>> kafka.
>> We need to set up a streaming platform based on the table updates that
>> happen(read from kafka) , we need to form events and send it consumer.
>> Each consumer may be interested in same table but different
>> updates/columns respective of their business needs and then deliver it to
>> their endpoint/kinesis/SQS/a kafka topic.
>> So the case here is *1* table update : *m* events : *n* sink.
>> Peak Load expected is easily a  100k-million table updates per second(all
>> customers put together)
>> Latency expected by most customers is less than a second. Mostly in
>> 100-500ms.
>> Is this usecase suited for flink ?
>> I went through the Flink book and documentation. These are the following
>> questions i have
>> 1). If we have situation like this *1* table update : *m* events : *n*
>> sink , is it better to write our micro service on our own or it it better
>> to implement through flink.
>>       1 a)  How does checkpointing happens if we have *1* input: *n*
>> output situations.
>>       1 b)  There are no heavy transformations maximum we might do is to
>> check the required columns are present in the db updates and decide whether
>> to create an event. So there is an alternative thought process to write a
>> service in node since it more IO and less process.
>> 2)  I see that we are writing a Job and it is deployed and flink takes
>> care of the rest in handling parallelism, latency and throughput.
>>      But what i need is to write a generic framework so that we should be
>> able to handle any table structure. we should not end up writing one job
>> driver for each case.
>>     There are at least 200 type of events in the existing monolith system
>> which might move to this new system once built.
>> 3)  How do we maintain flink cluster HA . From the book , i get that
>> internal task level failures are handled gracefully in flink.  But what if
>> the flink cluster goes down, how do we make sure its HA ?
>>     I had earlier worked with spark and we had issues managing it. (Not
>> much problem was there since there the latency requirement is 15 min and we
>> could make sure to ramp another one up within that time).
>>     These are absolute realtime cases and we cannot miss even one
>> message/event.
>> There are also thoughts whether to use kafka streams/apache storm for the
>> same. [They are investigated by different set of folks]
>> Thanks,
>> Prasanna.

View raw message