flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ronald Van de Kuil <ronald.van.de.k...@nl.ibm.com>
Subject Re: Twitter Source problem
Date Thu, 12 May 2016 14:05:19 GMT
Hello,

I used the following class:

a1.sources.r1.type = com.cloudera.flume.source.TwitterSource

It is running stable since Februari 2016.

Hope that helps.



Met Vriendelijke Groet, 
Ronald van de Kuil 




From:   chandra sekar <thittaisekar@hotmail.com>
To:     "user@flume.apache.org" <user@flume.apache.org>
Date:   12-05-16 03:38
Subject:        Re: Twitter Source problem



Dear Ronald, 

I have a twitter apps account and created all the keys. here I did removed 
 the keys due to the security purpose. 

Regards 
Chandrasekar.

Sent from Outlook



From: Ronald Van de Kuil <ronald.van.de.kuil@nl.ibm.com>
Sent: Tuesday, May 10, 2016 2:10 PM
To: user@flume.apache.org
Subject: Re: Twitter Source problem 
 
You would have to create a twitter app (account) to get your key, token 
and so on.

Op 10 mei 2016 om 05:04 heeft chandra sekar <thittaisekar@hotmail.com> het 
volgende geschreven:

Hi  I am using Twitter source to collect  the data from twitter. Suddenly 
the process throw an error and there is no data from twitter feed. 

Error: 

404:The URI requested is invalid or the resource requested, such as a 
user, does not exist.
Unknown URL. See Twitter Streaming API documentation at 
http://dev.twitter.com/pages/streaming_api

Configuration:

TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS

TwitterAgent.sources.Twitter.type = 
com.cloudera.flume.source.TwitterSource
# TwitterAgent.sources.Twitter.type = 
org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = 
TwitterAgent.sources.Twitter.consumerSecret = 
TwitterAgent.sources.Twitter.accessToken = 
TwitterAgent.sources.Twitter.accessTokenSecret = 

TwitterAgent.sources.Twitter.keywords = data analytics, data science , 
hadoop , big data 

TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:8020/data_analytics
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 10000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 100000

TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 500000
TwitterAgent.channels.MemChannel.transactionCapacity = 3000


Thanks & Regards 
Chandrasekar
Tenzij hierboven anders aangegeven: / Unless stated otherwise above:
IBM Nederland B.V.
Gevestigd te Amsterdam
Inschrijving Handelsregister Amsterdam Nr. 33054214

Tenzij hierboven anders aangegeven: / Unless stated otherwise above:
IBM Nederland B.V.
Gevestigd te Amsterdam
Inschrijving Handelsregister Amsterdam Nr. 33054214
Mime
View raw message