flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lucas Kinne <lucas.ki...@stud-mail.uni-wuerzburg.de>
Subject How to use OpenTSDB as Source?
Date Tue, 21 Apr 2020 16:55:53 GMT
Hey guys,

in a university project we are storing our collected sensor data in an
OpenTSDB <http://opentsdb.net/>database.
I am now trying to use this database as a source in Apache Flink, but I
can't seem to figure out how to do it.

I have seen that there is no existing connector for this Database, but I
read in the docs
<https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sourceSinks.html>
that is is possible to implement a custom (Batch/Streaming)TableSource.
There is a Java client for OpenTSDB
<http://opentsdb.net/docs/javadoc/net/opentsdb/core/TSDB.html>, which
could be used for that.

So I created a new Java Class "OpenTSDBTableSource" that implements
"StreamTableSource", "DefinedProctimeAttribute",
"DefinedRowtimeAttribute" and "LookupableTableSource", as suggested in
the docs.
However, I have no idea how to register this TableSource. The
"StreamExecutionEnvironment.addSource" requires a "SourceFunction"
parameter instead of my "TableSource" and the
"StreamTableEnvironment.registerTableSource"-Method is deprecated. There
is a link to the topic "register a TableSource" on linked docs page, but
the link seems to be dead, hence I found no other method on how to
register a TableSource.

I could also write a "SourceFunction" myself, pull the OpenTSDB database
in there and return the DataStream from the fetched Collection, but I am
not sure whether this is an efficient way.
And if I did it this "manual" way, how do I avoid pulling the whole
database everytime?

Any help is much appreciated, even if it is just a small pointer to the
right direction.

Thanks in advance!

Sincerely,
Lucas


Mime
View raw message