flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex De Castro <alex.decas...@lab49.com>
Subject Re: Http Requests from Flink
Date Thu, 02 Mar 2017 13:17:13 GMT
Hi Ulf,
I’ve had similar problem, before but from a sink perspective: I had to create a HTTP sink
for a Kafka REST API. I’ve used scalaj-http https://github.com/scalaj/scalaj-http which
is a wrapper for the corresponding Java lib.

For example,
https://github.com/scalaj/scalaj-http

For example

class HttpSink extends SinkFunction[Message]{
  private val secretkey = new GetToken().token

  def sendMessage(message: Message):String = Http("http://XXX.XXX.XX.XXX:5000/api/message")
// <-- GLOBAL var
    .header("Content-Type","application/json")
    .header("Authorization", s"Bearer $secretkey")
    .postData(message.data).asString.body

  @throws[Exception]
  override def invoke(message: Message): Unit = {
    log.info(sendMessage(message))
  }
}

I image for a http source, you could send a request to the REST API periodically and convert
the micro-batches into a stream. I’d love to know about other alternatives.

Cheers,
Alex
From: Ulf Thomas <ulf@relayr.de>
Reply-To: "user@flink.apache.org" <user@flink.apache.org>
Date: Thursday, March 2, 2017 at 12:58 PM
To: "user@flink.apache.org" <user@flink.apache.org>
Subject: Http Requests from Flink

Hello,

I've been trying to perform HTTP requests from a Flink Program but I wasn't successful :-(.

Does anybody here has done this before and can point me to an working library?

I've attached a small demo project in case someone wants to try to solve this.

Best,

--
--
Ulf Thomas
Software Developer
relayr
This email and any attachments may contain information which is confidential and/or privileged.
The information is intended exclusively for the addressee and the views expressed may not
be official policy, but the personal views of the originator. If you are not the intended
recipient, be aware that any disclosure, copying, distribution or use of the contents is prohibited.
If you have received this email and any file transmitted with it in error, please notify the
sender by telephone or return email immediately and delete the material from your computer.
Internet communications are not secure and Lab49 is not responsible for their abuse by third
parties, nor for any alteration or corruption in transmission, nor for any damage or loss
caused by any virus or other defect. Lab49 accepts no liability or responsibility arising
out of or in any way connected to this email.
Mime
View raw message