flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timo Walther <twal...@apache.org>
Subject Re: Using HiveBolt from storm-hive with Flink-Storm compatibility wrapper
Date Mon, 25 Sep 2017 14:43:51 GMT
Hi Federico,

I think going through a Storm compatibility layer could work, but did 
you thought about using the flink-jdbc connector? That should be the 
easiest solution.

Otherwise I think it would be easier to quickly implement your our 
SinkFunction. It is just one method that you have to implement, you 
could call some Hive commands there.

Regards,
Timo


Am 9/25/17 um 4:16 PM schrieb Nico Kruber:
> Hi Federico,
> I also did not find any implementation of a hive sink, nor much details on this
> topic in general. Let me forward this to Timo and Fabian (cc'd) who may know
> more.
>
> Nico
>
> On Friday, 22 September 2017 12:14:32 CEST Federico D'Ambrosio wrote:
>> Hello everyone,
>>
>> I'd like to use the HiveBolt from storm-hive inside a flink job using the
>> Flink-Storm compatibility layer but I'm not sure how to integrate it. Let
>> me explain, I would have the following:
>>
>> val mapper = ...
>>
>> val hiveOptions = ...
>>
>> streamByID
>>    .transform[OUT]("hive-sink", new BoltWrapper[IN, OUT](new
>> HiveBolt(hiveOptions)))
>>
>> where streamByID is a DataStream[Event].
>>
>> What would be the IN and OUT types? HiveBolt executes on a storm Tuple, so,
>> I'd think that In should be an Event "tuple-d" ( event => (field1, field2,
>> field3 ...) ), while OUT, since I don't want the stream to keep flowing
>> would be null or None?
>>
>> Alternatively, do you know any implementation of an hive sink in Flink?
>> Other than the adaptation of the said HiveBolt in a RichSinkFunction?
>>
>> Thanks for your attention,
>>   Federico



Mime
View raw message