flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Wang <chen.apache.s...@gmail.com>
Subject Re: Help on HDFS sink hangs
Date Thu, 09 Jan 2014 01:38:37 GMT
David,
I am using flume-ng 1.4.0.
Thanks,
Chen


On Wed, Jan 8, 2014 at 5:26 PM, David Sinclair <
dsinclair@chariotsolutions.com> wrote:

> What version of flume are you using?
>
>
> On Wed, Jan 8, 2014 at 8:01 PM, Chen Wang <chen.apache.solr@gmail.com>wrote:
>
>> Hey, Guys,
>> First time using flume. It goes all well, however, I found that in my
>> custom source when I am calling
>>
>> Event event = EventBuilder.withBody(mySignal.toString().getBytes(),
>> headers);
>>
>>   try {
>>
>>   logger.debug("submitting to channel");
>>
>>   channel.processEvent(event);
>>
>> }catch (ChannelException e) {
>>
>>   logger.debug("failed to submit to channel, exception {}",e);
>>
>>   throw e;
>>
>>   }
>>
>>
>> It just hangs there on "submitting to channel", and hdfs file apparently
>> is not created. Just wondering how can I trouble shoot this? I am running
>> it as
>>
>> bin/flume-ng agent -c conf -f conf/flume.conf -n SocketAgent
>> -Dflume.root.logger=DEBUG,console
>>
>> and my config file looks like:
>>
>> SocketAgent.sinks.HDFS.channel = MemChannel
>>
>> SocketAgent.sinks.HDFS.type = hdfs
>>
>> SocketAgent.sinks.HDFS.hdfs.path =
>> maprfs:///hive/myname.db/targeting_storm/%{epoch}
>>
>> SocketAgent.sinks.HDFS.hdfs.fileType = SequenceFile
>>
>> SocketAgent.sinks.HDFS.hdfs.writeFormat = Text
>>
>> SocketAgent.sinks.HDFS.hdfs.batchSize = 1000
>>
>> SocketAgent.sinks.HDFS.hdfs.rollSize = 0
>>
>> SocketAgent.sinks.HDFS.hdfs.rollCount = 10000
>>
>> SocketAgent.sinks.HDFS.hdfs.rollInterval = 600
>>
>>
>> SocketAgent.channels.MemChannel.type = memory
>>
>> SocketAgent.channels.MemChannel.capacity = 10000
>>
>> SocketAgent.channels.MemChannel.transactionCapacity = 100
>>
>>
>> Thank you very much!!
>>
>> Chen
>>
>
>

Mime
View raw message