flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Georg Heiler <georg.kf.hei...@gmail.com>
Subject Re: Flink first project
Date Sun, 23 Apr 2017 18:23:51 GMT
So you would suggest flume over a custom akka-source from bahir?

Jörn Franke <jornfranke@gmail.com> schrieb am So., 23. Apr. 2017 um
18:59 Uhr:

> I would use flume to import these sources to HDFS and then use flink or
> Hadoop or whatever to process them. While it is possible to do it in flink,
> you do not want that your processing fails because the web service is not
> available etc.
> Via flume which is suitable for this kind of tasks it is more controlled
> and reliable.
> On 23. Apr 2017, at 18:02, Georg Heiler <georg.kf.heiler@gmail.com> wrote:
> New to flink I would like to do a small project to get a better feeling
> for flink. I am thinking of getting some stats from several REST api (i.e.
> Bitcoin course values from different exchanges) and comparing prices over
> different exchanges in real time.
> Are there already some REST api sources for flink as a sample to get
> started implementing a custom REST source?
> I was thinking about using https://github.com/timmolter/XChange to
> connect to several exchanges. E.g. to make a single api call by hand would
> look similar to
> val currencyPair = new CurrencyPair(Currency.XMR, Currency.BTC)
>   CertHelper.trustAllCerts()
>   val poloniex =
> ExchangeFactory.INSTANCE.createExchange(classOf[PoloniexExchange].getName)
>   val dataService = poloniex.getMarketDataService
>   generic(dataService)
>   raw(dataService.asInstanceOf[PoloniexMarketDataServiceRaw])
> System.out.println(dataService.getTicker(currencyPair))
> How would be a proper way to make this available as a flink source? I have
> seen
> https://github.com/apache/flink/blob/master/flink-contrib/flink-connector-wikiedits/src/main/java/org/apache/flink/streaming/connectors/wikiedits/WikipediaEditsSource.java
> new to flink am a bit unsure how to proceed.
> Regards,
> Georg

View raw message