Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <eb036f09-59ab-4605-8f1c-99800f2b583f@apache.org>
References: <CANNGFpi44kOEMe=-Y8tZCoi+8Fzhm0CQ8WHOParZb4v+4XNvpA@mail.gmail.com>
 <1654823.QftJd3NfD3@nico-work> <eb036f09-59ab-4605-8f1c-99800f2b583f@apache.org>
From: "Federico D'Ambrosio" <fedexist@gmail.com>
Date: Mon, 25 Sep 2017 17:15:55 +0200
Message-ID: <CANNGFpiPxz2OxN_pDKY7hZN1Y3O7ndFq4CsyHW0_FGCnu+t_6g@mail.gmail.com>
Subject: Re: Using HiveBolt from storm-hive with Flink-Storm compatibility wrapper
To: Timo Walther <twalthr@apache.org>
Cc: user <user@flink.apache.org>
Content-Type: multipart/alternative; boundary="f403045e69f8ff10e7055a0508d8"
archived-at: Mon, 25 Sep 2017 15:16:01 -0000

--f403045e69f8ff10e7055a0508d8
Content-Type: text/plain; charset="UTF-8"

Hi Timo,

I didn't think about the jdbc-connector (I actually forgot that this was a
thing) and I'll surely look into it.
So far, I was trying to implement a simple sink (only for JSON so far)
starting from the base provided by the storm HiveBolt implementation, my
goal was to make use of Hive Streaming API.
I noticed that there are some potentially blocking calls in Hive API, for
example when a TransactionBatch is being committed or the
StreamingConnection is being closed, in your opinion what would be the best
way to deal with this kind of calls in Flink? Wrapping them in an
AsyncFunction? Simply spawning a new thread?

Kind regards,
Federico

2017-09-25 16:43 GMT+02:00 Timo Walther <twalthr@apache.org>:

> Hi Federico,
>
> I think going through a Storm compatibility layer could work, but did you
> thought about using the flink-jdbc connector? That should be the easiest
> solution.
>
> Otherwise I think it would be easier to quickly implement your our
> SinkFunction. It is just one method that you have to implement, you could
> call some Hive commands there.
>
> Regards,
> Timo
>
>
> Am 9/25/17 um 4:16 PM schrieb Nico Kruber:
>
> Hi Federico,
>> I also did not find any implementation of a hive sink, nor much details
>> on this
>> topic in general. Let me forward this to Timo and Fabian (cc'd) who may
>> know
>> more.
>>
>> Nico
>>
>> On Friday, 22 September 2017 12:14:32 CEST Federico D'Ambrosio wrote:
>>
>>> Hello everyone,
>>>
>>> I'd like to use the HiveBolt from storm-hive inside a flink job using the
>>> Flink-Storm compatibility layer but I'm not sure how to integrate it. Let
>>> me explain, I would have the following:
>>>
>>> val mapper = ...
>>>
>>> val hiveOptions = ...
>>>
>>> streamByID
>>>    .transform[OUT]("hive-sink", new BoltWrapper[IN, OUT](new
>>> HiveBolt(hiveOptions)))
>>>
>>> where streamByID is a DataStream[Event].
>>>
>>> What would be the IN and OUT types? HiveBolt executes on a storm Tuple,
>>> so,
>>> I'd think that In should be an Event "tuple-d" ( event => (field1,
>>> field2,
>>> field3 ...) ), while OUT, since I don't want the stream to keep flowing
>>> would be null or None?
>>>
>>> Alternatively, do you know any implementation of an hive sink in Flink?
>>> Other than the adaptation of the said HiveBolt in a RichSinkFunction?
>>>
>>> Thanks for your attention,
>>>   Federico
>>>
>>
>
>


-- 
Federico D'Ambrosio

--f403045e69f8ff10e7055a0508d8
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Hi Timo,</div><div><br></div><div>I didn&#39;t t=
hink about the jdbc-connector (I actually forgot that this was a thing) and=
 I&#39;ll surely look into it.<br></div>So far, I was trying to implement a
 simple sink (only for JSON so far) starting from the base provided by the =
storm HiveBolt implementation, my goal was to make use of Hive Streaming AP=
I. <br></div><div>I noticed that there are some potentially blocking calls =
in Hive API, for example when a TransactionBatch is being committed or the =
StreamingConnection is being closed, in your opinion what would be the best=
 way to deal with this kind of calls in Flink? Wrapping them in an AsyncFun=
ction? Simply spawning a new thread?</div><div><br></div><div>Kind regards,=
</div><div>Federico<br></div></div><div class=3D"gmail_extra"><br><div clas=
s=3D"gmail_quote">2017-09-25 16:43 GMT+02:00 Timo Walther <span dir=3D"ltr"=
>&lt;<a href=3D"mailto:twalthr@apache.org" target=3D"_blank">twalthr@apache=
.org</a>&gt;</span>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0=
 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Federico,<br>
<br>
I think going through a Storm compatibility layer could work, but did you t=
hought about using the flink-jdbc connector? That should be the easiest sol=
ution.<br>
<br>
Otherwise I think it would be easier to quickly implement your our SinkFunc=
tion. It is just one method that you have to implement, you could call some=
 Hive commands there.<br>
<br>
Regards,<br>
Timo<br>
<br>
<br>
Am 9/25/17 um 4:16 PM schrieb Nico Kruber:<div class=3D"HOEnZb"><div class=
=3D"h5"><br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
Hi Federico,<br>
I also did not find any implementation of a hive sink, nor much details on =
this<br>
topic in general. Let me forward this to Timo and Fabian (cc&#39;d) who may=
 know<br>
more.<br>
<br>
Nico<br>
<br>
On Friday, 22 September 2017 12:14:32 CEST Federico D&#39;Ambrosio wrote:<b=
r>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
Hello everyone,<br>
<br>
I&#39;d like to use the HiveBolt from storm-hive inside a flink job using t=
he<br>
Flink-Storm compatibility layer but I&#39;m not sure how to integrate it. L=
et<br>
me explain, I would have the following:<br>
<br>
val mapper =3D ...<br>
<br>
val hiveOptions =3D ...<br>
<br>
streamByID<br>
=C2=A0 =C2=A0.transform[OUT](&quot;hive-sink&quot;, new BoltWrapper[IN, OUT=
](new<br>
HiveBolt(hiveOptions)))<br>
<br>
where streamByID is a DataStream[Event].<br>
<br>
What would be the IN and OUT types? HiveBolt executes on a storm Tuple, so,=
<br>
I&#39;d think that In should be an Event &quot;tuple-d&quot; ( event =3D&gt=
; (field1, field2,<br>
field3 ...) ), while OUT, since I don&#39;t want the stream to keep flowing=
<br>
would be null or None?<br>
<br>
Alternatively, do you know any implementation of an hive sink in Flink?<br>
Other than the adaptation of the said HiveBolt in a RichSinkFunction?<br>
<br>
Thanks for your attention,<br>
=C2=A0 Federico<br>
</blockquote></blockquote>
<br>
<br>
</div></div></blockquote></div><br><br clear=3D"all"><br>-- <br><div class=
=3D"gmail_signature" data-smartmail=3D"gmail_signature">Federico D&#39;Ambr=
osio</div>
</div>

--f403045e69f8ff10e7055a0508d8--