flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saket (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLUME-3066) Twitter Streaming using flume in hdfs
Date Tue, 07 Mar 2017 02:59:32 GMT

    [ https://issues.apache.org/jira/browse/FLUME-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898648#comment-15898648
] 

Saket edited comment on FLUME-3066 at 3/7/17 2:58 AM:
------------------------------------------------------

Any one please help me to resolve this issue.

Please let me know if any further info is required.


Thanx Saket.


was (Author: peta1):
Any one please help me to resolve this issue.

> Twitter Streaming using flume in hdfs
> -------------------------------------
>
>                 Key: FLUME-3066
>                 URL: https://issues.apache.org/jira/browse/FLUME-3066
>             Project: Flume
>          Issue Type: Question
>    Affects Versions: v1.7.0
>            Reporter: Saket
>
> I was trying to stream twitter data in HDFS using flume.Succeffuly configured agent but
the data population in HDFS doesnt seems proper please help.
> flume agent config:
> TwitterAgent.sources= Twitter
> TwitterAgent.channels= MemChannel
> TwitterAgent.sinks=HDFS
> TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
> TwitterAgent.sources.Twitter.channels=MemChannel
> TwitterAgent.sources.Twitter.consumerKey=xxxxx
> TwitterAgent.sources.Twitter.consumerSecret=xxxx
> TwitterAgent.sources.Twitter.accessToken=xxxx
> TwitterAgent.sources.Twitter.accessTokenSecret=	xxxx
> TwitterAgent.sinks.HDFS.channel=MemChannel
> TwitterAgent.sinks.HDFS.type=hdfs
> TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:8020/user/xxx/flume/events
> TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream
> TwitterAgent.sinks.HDFS.hdfs.writeformat=Text
> TwitterAgent.sinks.HDFS.hdfs.batchSize=1000
> TwitterAgent.sinks.HDFS.hdfs.rollSize=0
> TwitterAgent.sinks.HDFS.hdfs.rollCount=10000
> TwitterAgent.sinks.HDFS.hdfs.rollInterval=600
> TwitterAgent.channels.MemChannel.type=memory
> TwitterAgent.channels.MemChannel.capacity=10000
> TwitterAgent.channels.MemChannel.transactionCapacity=100
> =======================Twitter file in HDFS==============
> Objavro.schema�
> {"type":"record","name":"Doc","doc":"adoc","fields":[{"name":"id","type":"string"},{"name":"user_friends_count","type":["int","null"]},{"name":"user_location","type":["string","null"]},{"name":"user_description","type":["string","null"]},{"name":"user_statuses_count","type":["int","null"]},{"name":"user_followers_count","type":["int","null"]},{"name":"user_name","type":["string","null"]},{"name":"user_screen_name","type":["string","null"]},{"name":"created_at","type":["string","null"]},{"name":"text","type":["string","null"]},{"name":"retweet_count","type":["long","null"]},{"name":"retweeted","type":["boolean","null"]},{"name":"in_reply_to_user_id","type":["long","null"]},{"name":"source","type":["string","null"]},{"name":"in_reply_to_status_id","type":["long","null"]},{"name":"media_url_https","type":["string","null"]},{"name":"expanded_url","type":["string","null"]}]}��rm���R��d�q�#�$838697771628544001Jぽやしみ��F$おやすみ世界eZ_XL1200C(2017-03-06T05:28:25Z�30ゲームから頑張ってはめ続けたのにやめて
https://t.co/SmtQDJHAdz�<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter
for iPhone</a>^https://pbs.twimg.com/media/C6OnJgRVUAExHzQ.jpg�https://twitter.com/eZ_XL1200C/status/838697771628544001/photo/1��rm���R��d�q�#
> Objavro.schema�
> {"type":"record","name":"Doc","doc":"adoc","fields":[{"name":"id","type":"string"},{"name":"user_friends_count","type":["int","null"]},{"name":"user_location","type":["string","null"]},{"name":"user_description","type":["string","null"]},{"name":"user_statuses_count","type":["int","null"]},{"name":"user_followers_count","type":["int","null"]},{"name":"user_name","type":["string","null"]},{"name":"user_screen_name","type":["string","null"]},{"name":"created_at","type":["string","null"]},{"name":"text","type":["string","null"]},{"name":"retweet_count","type":["long","null"]},{"name":"retweeted","type":["boolean","null"]},{"name":"in_reply_to_user_id","type":["long","null"]},{"name":"source","type":["string","null"]},{"name":"in_reply_to_status_id","type":["long","null"]},{"name":"media_url_https","type":["string","null"]},{"name":"expanded_url","type":["string","null"]}]}I��
> $
> ��b�u�St�XTޥ$838697775802040320��Exénekakavshi(2017-03-06T05:28:26Z�SUPER
on est en alerte orange ils nous sortent quoi??? "faut pas s'approcher des toits les tuiles
c fragile encore" mdrrrrr mrci fdp�<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter
for iPhone</a>$838697775797792768�	�full time working mom. convinced Runner.
canadian at heart and #fantasydiamondrace Team holder. عبد الله الهرري���
running falahrunningfalah(2017-03-06T05:28:26Z�Heute läufts "wir brauchen dringendst
Pasta zu Mittag" gut im Büro.�<a href="http://twitter.com" rel="nofollow">Twitter
Web Client</a>$838697775818747904�|*دولة الكويت�‏من بغإني
صرٺ #لعيونه ـ فدآ ‏                                             ومن تركنيء
جعل ربيء يحفظه 💯 ‏�m��Nضــّمـآ آلـمـشــاعـر
> 8vvvp(2017-03-06T05:28:26Z�RT @Sm_F__16: -



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message