flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: JDBC table source
Date Wed, 27 Sep 2017 03:28:03 GMT
Yes, there's no built-in TableSource for that.
However, it is certainly possible to implement a custom TableSource for
your use case. The code of the JdbcInputFormat should be a good starting
point. So you could run a query every n seconds (assuming you can consume
the data of the last n seconds in n seconds). If you want to run the
TableSource in parallel, you would need to partition the query (as for the
JdbcInputFormat).

2017-09-26 19:19 GMT-04:00 Mohit Anchlia <mohitanchlia@gmail.com>:

> Thanks. Idea was to query for 'x' records in last 'n' seconds using an
> indexed column. Looks like that is not possible?
>
> On Tue, Sep 26, 2017 at 3:24 PM, Fabian Hueske <fhueske@gmail.com> wrote:
>
>> Hi Mohit,
>>
>> no, a JdbcTableSource does not exist yet. However, since there is a
>> JdbcInputFormat it should not be hard to wrap that in a TableSource.
>> However, this would rather be a batch TableSource in the sense that it
>> would just return the data that the query returns. Once all data is read it
>> would terminate. You can of course wrap the JdbcInputFormat in a
>> StreamingTableSource, but as I said it would terminate when all data was
>> read.
>>
>> If you are thinking of streaming a changelog stream from a database to
>> the Table API / SQL, this would not be possible at the moment due to
>> limitation in the Table API / SQL (these will be removed in the future).
>> Moreover, not many DBMS expose their changelog (such as PostgreSQL) and
>> there is no common interface for that such as JDBC. Instead they use custom
>> formats. There is a tool called Bottled Water that ingests PostgreSQL
>> streams into Kafka.
>>
>> So, to make a long story short: implementing a JDBC TableSource for batch
>> query should be fairly easy. A true streaming solution that hooks into the
>> changelog stream of a table is not possible at the moment.
>>
>> Cheers, Fabian
>>
>> 2017-09-26 15:04 GMT-04:00 Mohit Anchlia <mohitanchlia@gmail.com>:
>>
>>> We are looking to stream data from the database. Is there already a jdbc
>>> table source available for streaming?
>>>
>>
>>
>

Mime
View raw message