flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <c.schep...@web.de>
Subject Re: JDBC Streaming Connector
Date Fri, 26 Feb 2016 09:55:41 GMT
I'm actually working on a JDBC streaming connector with transactions.

On 26.02.2016 10:33, Stephan Ewen wrote:
> Hi Timothy!
>
> I would first just write a stream data sink based on the JDBCOutputFormat,
> and then let's look into making it exactly-once.
>
> For JDBC with support for transactions, the basic idea would be to start a
> transaction, insert elements, commit when a checkpoint is confirmed, and
> start a new transaction.
>
> On Fri, Feb 26, 2016 at 10:28 AM, Till Rohrmann <trohrmann@apache.org>
> wrote:
>
>> Hi Timothy,
>>
>> you can use the `JDBCOutputFormat` in combination with the
>> `DataStream.writeUsingOutputFormat` method. However, this won't give you
>> exactly once guarantees since the output formats don't take part in the
>> checkpointing mechanism.
>>
>> Currently, Chesnay is working on a generic framework for almost exactly
>> once sinks. Maybe you can sync with him. You'll find the corresponding JIRA
>> ticket here [1].
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-3332
>>
>> Cheers,
>> Till
>>
>>
>> On Fri, Feb 26, 2016 at 1:23 AM, Timothy Farkas <
>> timothytiborfarkas@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> I'm new to Flink and was looking for a JDBC stream sink connector, and
>>> didn't see one in flink-streaming-connectors. Is there one somewhere
>> else,
>>> or is there one currently in development. If not could I pick up a ticket
>>> to add it?
>>>
>>> Thanks,
>>> Tim
>>>


Mime
View raw message