flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sathi Chowdhury <Sathi.Chowdh...@elliemae.com>
Subject Data stream to write to multiple rds instances
Date Thu, 02 Mar 2017 06:21:40 GMT
Hi All,
Is there any preferred way to manage multiple jdbc connections from flink..? I am new to flink
and looking for some guidance around the right pattern and apis to do this. The usecase needs
to route a stream to a particular jdbc connection depending on a field value.So the records
are written to multiple destination dbs.
Thanks
Sathi
On 02/07/2017 04:12 PM, Robert Metzger wrote:
Currently, there is no streaming JDBC connector.
Check out this thread from last year: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/JDBC-Streaming-Connector-td10508.html<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-mailing-list-archive.1008284.n3.nabble.com%2FJDBC-Streaming-Connector-td10508.html&data=01%7C01%7C%7C38def12a718e41d76a0808d45007bf5c%7C0d009d13c2cd47d891dd2ae838b00d4b%7C0&sdata=ncxXmugcAakxfZgRbTqT%2FVU3KqILr1zXB4UCeH%2B9910%3D&reserved=0>

Sent from my iPhone

On Feb 8, 2017, at 1:49 AM, Punit Tandel <punit.tandel@ericsson.com<mailto:punit.tandel@ericsson.com>>
wrote:


Hi Chesnay

Currently that is what i have done, reading the schema from database in order to create a
new table in jdbc database and writing the rows coming from jdbcinputformat.

Overall i am trying to implement the solution which reads the streaming data from one source
which either could be coming from kafka, Jdbc, Hive, Hdfs and writing those streaming data
to output source which is again could be any of those.

For a simple use case i have just taken one scenario using jdbc in and jdbc out, Since the
jdbc input source returns the datastream of Row and to write them into jdbc database we have
to create a table which requires schema.

Thanks
Punit


On 02/08/2017 08:22 AM, Chesnay Schepler wrote:
Hello,

I don't understand why you explicitly need the schema since the batch JDBCInput-/Outputformats
don't require it.
That's kind of the nice thing about Rows.

Would be cool if you could tell us what you're planning to do with the schema :)

In any case, to get the schema within the plan then you will have to query the DB and build
it yourself. Note that this
is executed on the client.

Regards,
Chesnay

On 08.02.2017 00:39, Punit Tandel wrote:

Hi Robert

Thanks for the response, So in near future release of the flink version , is this functionality
going to be implemented ?

Thanks

On 02/07/2017 04:12 PM, Robert Metzger wrote:
Currently, there is no streaming JDBC connector.
Check out this thread from last year: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/JDBC-Streaming-Connector-td10508.html<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-mailing-list-archive.1008284.n3.nabble.com%2FJDBC-Streaming-Connector-td10508.html&data=01%7C01%7C%7C38def12a718e41d76a0808d45007bf5c%7C0d009d13c2cd47d891dd2ae838b00d4b%7C0&sdata=ncxXmugcAakxfZgRbTqT%2FVU3KqILr1zXB4UCeH%2B9910%3D&reserved=0>



On Mon, Feb 6, 2017 at 5:00 PM, Ufuk Celebi <uce@apache.org<mailto:uce@apache.org>>
wrote:
I'm not sure how well this works for the streaming API. Looping in
Chesnay, who worked on this.

On Mon, Feb 6, 2017 at 11:09 AM, Punit Tandel <punit.tandel@ericsson.com<mailto:punit.tandel@ericsson.com>>
wrote:
> Hi ,
>
> I was looking into flink streaming api and trying to implement the solution
> for reading the data from jdbc database and writing them to jdbc databse
> again.
>
> At the moment i can see the datastream is returning Row from the database.
> dataStream.getType().getGenericParameters() retuning an empty list of
> collection.
>
> I am right now manually creating a database connection and getting the
> schema from ResultMetadata and constructing the schema for the table which
> is a bit heavy operation.
>
> So is there any other way to get the schema for the table in order to create
> a new table and write those records in the database ?
>
> Please let me know
>
> Thanks
> Punit




=============Notice to Recipient: This e-mail transmission, and any documents, files or previous
e-mail messages attached to it may contain information that is confidential or legally privileged,
and intended for the use of the individual or entity named above. If you are not the intended
recipient, or a person responsible for delivering it to the intended recipient, you are hereby
notified that you must not read this transmission and that any disclosure, copying, printing,
distribution or use of any of the information contained in or attached to this transmission
is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately
notify the sender by telephone or return e-mail and delete the original transmission and its
attachments without reading or saving in any manner. Thank you. =============
Mime
View raw message