flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: can flink do streaming from data sources other than Kafka?
Date Thu, 07 Sep 2017 08:34:11 GMT
Yes I can indeed create them but I wonder if that is even possible? I
haven't see any framework doing this as of today. Flink has something
called AsyncDataStream? and I wonder if this can be leveraged to create a
Stream out of Cassandra source?

Thanks!

On Thu, Sep 7, 2017 at 1:16 AM, Tzu-Li (Gordon) Tai <tzulitai@apache.org>
wrote:

> Ah, I see. I’m not aware of any existing work / JIRAs on streaming sources
> for Cassandra or HBase, only sinks.
> If you are interested in one, could you open JIRAs for them?
>
>
> On 7 September 2017 at 4:11:05 PM, kant kodali (kanth909@gmail.com) wrote:
>
> Hi Gordon,
>
> Thanks for the response, I did go over the links for sources and sinks
> prior to posting my question. Maybe, I didn't get my question across
> correctly so let me rephrase it. Can I get data out of data stores like
> Cassandra, Hbase in a streaming manner? coz, currently more or less all the
> sources are of message queue family.
>
> Thanks,
> Kant
>
> On Thu, Sep 7, 2017 at 1:04 AM, Tzu-Li (Gordon) Tai <tzulitai@apache.org>
> wrote:
>
>> Hi!
>>
>>
>> I am wondering if Flink can do streaming from data sources other than
>> Kafka. For example can Flink do streaming from a database like Cassandra,
>> HBase, MongoDb to sinks like says Elastic search or Kafka.
>>
>>
>> Yes, Flink currently supports various connectors for different sources
>> and sinks. For an overview you can check out this documentation [1]
>> Apache Bahir [2] also maintains some Flink connectors and is released
>> separately.
>>
>> Also for out of core stateful streaming. Is RocksDB the only option?
>>
>> Currently, RocksDB is the only option for out-of-core state. There was
>> some previous discussion for a Cassandra state backend, though [3].
>>
>> - Gordon
>> [1] https://ci.apache.org/projects/flink/flink-docs-release-
>> 1.3/dev/connectors/index.html
>> [2] http://bahir.apache.org/
>> [3] https://issues.apache.org/jira/browse/FLINK-4266
>>
>> On 7 September 2017 at 2:58:38 PM, kant kodali (kanth909@gmail.com)
>> wrote:
>>
>> Hi All,
>>
>> I am wondering if Flink can do streaming from data sources other than
>> Kafka. For example can Flink do streaming from a database like Cassandra,
>> HBase, MongoDb to sinks like says Elastic search or Kafka.
>>
>> Also for out of core stateful streaming. Is RocksDB the only option? Can
>> I use some other key value store that has SQL interface (since RocksDB
>> doesn't)?
>>
>> Thanks,
>> kant
>>
>>
>

Mime
View raw message