flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Georg Heiler <georg.kf.hei...@gmail.com>
Subject Re: Flink first project
Date Sun, 23 Apr 2017 18:25:27 GMT
Wouldn't adding flume -> Kafka -> flink also introduce additional latency?

Georg Heiler <georg.kf.heiler@gmail.com> schrieb am So., 23. Apr. 2017 um
20:23 Uhr:

> So you would suggest flume over a custom akka-source from bahir?
>
> Jörn Franke <jornfranke@gmail.com> schrieb am So., 23. Apr. 2017 um
> 18:59 Uhr:
>
>> I would use flume to import these sources to HDFS and then use flink or
>> Hadoop or whatever to process them. While it is possible to do it in flink,
>> you do not want that your processing fails because the web service is not
>> available etc.
>> Via flume which is suitable for this kind of tasks it is more controlled
>> and reliable.
>>
>> On 23. Apr 2017, at 18:02, Georg Heiler <georg.kf.heiler@gmail.com>
>> wrote:
>>
>> New to flink I would like to do a small project to get a better feeling
>> for flink. I am thinking of getting some stats from several REST api (i.e.
>> Bitcoin course values from different exchanges) and comparing prices over
>> different exchanges in real time.
>>
>> Are there already some REST api sources for flink as a sample to get
>> started implementing a custom REST source?
>>
>> I was thinking about using https://github.com/timmolter/XChange to
>> connect to several exchanges. E.g. to make a single api call by hand would
>> look similar to
>>
>> val currencyPair = new CurrencyPair(Currency.XMR, Currency.BTC)
>>   CertHelper.trustAllCerts()
>>   val poloniex =
>> ExchangeFactory.INSTANCE.createExchange(classOf[PoloniexExchange].getName)
>>   val dataService = poloniex.getMarketDataService
>>
>>   generic(dataService)
>>   raw(dataService.asInstanceOf[PoloniexMarketDataServiceRaw])
>> System.out.println(dataService.getTicker(currencyPair))
>>
>> How would be a proper way to make this available as a flink source? I
>> have seen
>> https://github.com/apache/flink/blob/master/flink-contrib/flink-connector-wikiedits/src/main/java/org/apache/flink/streaming/connectors/wikiedits/WikipediaEditsSource.java
but
>> new to flink am a bit unsure how to proceed.
>>
>> Regards,
>> Georg
>>
>>

Mime
View raw message