flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harikrishnan S <hihari...@gmail.com>
Subject Re: JDBC sink in flink
Date Tue, 05 Jul 2016 11:27:58 GMT
Awesome ! Thanks a lot ! I should probably write a blog post somewhere
explaining this.

On Tue, Jul 5, 2016 at 4:28 PM, Chesnay Schepler <chesnay@apache.org> wrote:

> They serve a similar purpose.
>
> OutputFormats originate from the Batch API, whereas SinkFunctions are a
> Streaming API concept.
>
> You can however use OutputFormats in the Streaming API using the
> DataStrea#writeUsingOutputFormat.
>
> Regards,
> Chesnay
>
>
> On 05.07.2016 12:51, Harikrishnan S wrote:
>
> Ah that makes send. Also what's the difference between a RichOutputFormat
> and a RichSinkFunction ? Can I use JDBCOutputFormat as a sink in a stream ?
>
> On Tue, Jul 5, 2016 at 3:53 PM, Chesnay Schepler <chesnay@apache.org>
> wrote:
>
>> Hello,
>>
>> an instance of the JDBCOutputFormat will use a single connection to send
>> all values.
>>
>> Essentially
>> - open(...) is called at the very start to create the connection
>> - then all invoke/writeRecord calls are executed (using the same
>> connection)
>> - then close() is called to clean up.
>>
>> The total number of connections made to the database depends on the
>> parallelism of the Sink, as every parallel instance creates it's own
>> connection.
>>
>> Regards,
>> Chesnay
>>
>>
>> On 05.07.2016 12:04, Harikrishnan S wrote:
>>
>> The basic idea was that I would create a pool of connections in the
>> open() method in a custom sink and each invoke() method gets one connection
>> from the pool and does the upserts needed. I might have misunderstood how
>> sinks work in flink though.
>>
>> On Tue, Jul 5, 2016 at 2:22 PM, Flavio Pompermaier <
>> <pompermaier@okkam.it>pompermaier@okkam.it> wrote:
>>
>>> why do you need a connection pool?
>>> On 5 Jul 2016 11:41, "Harikrishnan S" < <hihari777@gmail.com>
>>> hihari777@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Are there any examples of implementing a jdbc sink in flink using a
>>>> connection pool ?
>>>>
>>>> Thanks
>>>>
>>>> On Tue, Jul 5, 2016 at 2:00 PM, Harikrishnan S < <hihari777@gmail.com>
>>>> hihari777@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Are there any examples of implementing a jdbc sink in flink using a
>>>>> connection pool ?
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Tue, Jul 5, 2016 at 1:57 PM, Harikrishnan S < <hihari777@gmail.com>
>>>>> hihari777@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Are there any examples of implementing a jdbc sink in flink using
a
>>>>>> connection pool ?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>
>>>>>
>>>>
>>
>>
>
>

Mime
View raw message