flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jagadish Bihani <jagadish.bih...@pubmatic.com>
Subject Re: flume-ng agent startup problem
Date Sat, 11 Aug 2012 09:09:49 GMT
Hi

In my case flume is not transferring data to HDFS with my hadoop version
being 0.20.1 and it doesn't show any error even in DEBUG log mode.
It works fine for other sinks.

Is there any known compatibility problem with hadoop 0.20.1 ? OR
  can there be a problem due to an particular hadoop version?
(I know its an old version but it is on production machine and cant upgrade
as of now...)

Details of configuration and log records are in the following mail

Thanks ,
Jagadish

On 08/10/2012 03:30 PM, Jagadish Bihani wrote:
> Hi
>
> Thanks all for the inputs. After the initial problem I was able to 
> start flume except in one scenario in
> which I use HDFS as sink.
>
> I have a production machine with hadoop-0.20.1 installed. I have 
> installed latest flume 1.2.0.
> It works fine for all the configurations (at least which I tried) 
> except when HDFS sink is used.
>
> Test:
> ---------
>  I used both netcat listener as the source of the agent and HDFS is 
> sink. Then I start the agent using
> the command *"bin/flume-ng agent -n agent1 -c conf -f 
> conf/flume_hdfs.conf --classpath 
> /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar 
> -Dflume.root.logger=DEBUG,console"*
>  with DEBUG logging mode enabled. I don't get any error/exception. I 
> use *"/usr/sbin/lsof -i:<port_no>"* command to check whether the source
> is actually bound to that port and it doesn't return any port. But 
> when I use *file sink instead of HDFS sink* and run lsof it correctly 
> shows me the port on which
> it is listening.
> Thus when HDFS sink is used even source part of agent doesn't work and 
> it doesn't give any exception. And nothing is written to
> HDFS sink.
>
> P.S. I have checked the user,permission details of HDFS. They are fine.
>
> I have run flume on my other machines with different version of hadoop 
> (0.23 & 1.0). It has run HDFS sink properly there.
> Does flume support hadoop-0.20.1 or there is something I am missing???
>
> This is my Configuration:
> -----------------------------------------
> agent1.sources = sequencer
> agent1.sinks  =hdfsSink fileSink
> agent1.sinks  =fileSink
> agent1.channels =memoryChannel fileChannel
>
> agent1.sources.sequencer.type=seq
>
> agent1.sinks.hdfsSink.type=hdfs
> agent1.sinks.hdfsSink.hdfs.path=hdfs://MLNameNode2001:54310/flume
>
> agent1.sinks.fileSink.type=file_roll
> agent1.sinks.fileSink.sink.directory=/home/hadoop/flume/output
>
>
> agent1.channels.memoryChannel.type=memory
> agent1.channels.memoryChannel.capacity=10000
> agent1.channels.memoryChannel.transactionCapacity=100
>
>
> agent1.channels.fileChannel.type=file
> agent1.channels.fileChannel.checkpointDir=/home/hadoop/flume/channel/checkpointDir
> agent1.channels.fileChannel.dataDirs=/home/hadoop/flume/channel/dataDir
>
> agent1.sources.sequencer.channels = fileChannel
> agent1.sinks.hdfsSink.channel = fileChannel
>
> This is the log which I get:
> ----------------------------------------------------------
>
> bin/flume-ng agent -n agent1 -c conf -f conf/flume_hdfs.conf 
> --classpath /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop
> -0.20.1-core.jar -Dflume.root.logger=DEBUG,console
> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m 
> -Dflume.root.logger=DEBUG,console -cp 
> '/home/hadoop/flume/apache-flume-1.2.0/conf:/home/hadoop/flume/apache-flume-1.2.0/li
> b/*:/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar' 
> -Djava.library.path= org.apache.flume.node.Application -n agent1 -f 
> conf/flume_hdfs.conf
> 2012-08-10 10:56:50,604 (main) [INFO - 
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] 
> Starting lifecycle supervisor 1
> 2012-08-10 10:56:50,607 (main) [INFO - 
> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node 
> starting - agent1
> 2012-08-10 10:56:50,611 (lifecycleSupervisor-1-2) [INFO - 
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
>  Configuration provider starting
> 2012-08-10 10:56:50,611 (lifecycleSupervisor-1-0) [INFO - 
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)]

> Node ma
> nager starting
> 2012-08-10 10:56:50,613 (lifecycleSupervisor-1-0) [INFO - 
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] 
> Starting lifecycle supervis
> or 9
> 2012-08-10 10:56:50,614 (lifecycleSupervisor-1-0) [DEBUG - 
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)]

> Node manager started
> 2012-08-10 10:56:50,614 (lifecycleSupervisor-1-2) [DEBUG - 
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)
> ] Configuration provider started
> 2012-08-10 10:56:50,614 (conf-file-poller-0) [DEBUG - 
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProv
> ider.java:188)] Checking file:conf/flume_hdfs.conf for changes
> 2012-08-10 10:56:50,614 (conf-file-poller-0) [INFO - 
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvi
> der.java:195)] Reloading configuration file:conf/flume_hdfs.conf
> 2012-08-10 10:56:50,618 (conf-file-poller-0) [INFO - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]

> Processing:fi
> leSink
> 2012-08-10 10:56:50,619 (conf-file-poller-0) [DEBUG - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]

> Created cont
> ext for fileSink: sink.directory
> 2012-08-10 10:56:50,619 (conf-file-poller-0) [INFO - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]

> Processing:hd
> fsSink
> 2012-08-10 10:56:50,620 (conf-file-poller-0) [DEBUG - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]

> Created cont
> ext for hdfsSink: type
> 2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]

> Processing:hd
> fsSink
> 2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]

> Processing:hd
> fsSink
> 2012-08-10 10:56:50,614 (lifecycleSupervisor-1-0) [DEBUG - 
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)]

> Node m
> anager started
> 2012-08-10 10:56:50,614 (lifecycleSupervisor-1-2) [DEBUG - 
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)
> ] Configuration provider started
> 2012-08-10 10:56:50,614 (conf-file-poller-0) [DEBUG - 
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProv
> ider.java:188)] Checking file:conf/flume_hdfs.conf for changes
> 2012-08-10 10:56:50,614 (conf-file-poller-0) [INFO - 
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvi
> der.java:195)] Reloading configuration file:conf/flume_hdfs.conf
> 2012-08-10 10:56:50,618 (conf-file-poller-0) [INFO - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]

> Processing:fi
> leSink
> 2012-08-10 10:56:50,619 (conf-file-poller-0) [DEBUG - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]

> Created cont
> ext for fileSink: sink.directory
> 2012-08-10 10:56:50,619 (conf-file-poller-0) [INFO - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]

> Processing:hd
> fsSink
> 2012-08-10 10:56:50,620 (conf-file-poller-0) [DEBUG - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]

> Created cont
> ext for hdfsSink: type
> 2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]

> Processing:hd
> fsSink
> 2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]

> Processing:hd
> fsSink
> 2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]

> Processing:fileSink
> 2012-08-10 10:56:50,622 (conf-file-poller-0) [INFO - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)]

> Added sinks: fileSink Agent: agent1
> 2012-08-10 10:56:50,622 (conf-file-poller-0) [DEBUG - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:295)]

> Starting validation of configuration for agent: agent1, 
> initial-configuration: AgentConfiguration[agent1]
> SOURCES: {sequencer={ parameters:{channels=fileChannel, type=seq} }}
> CHANNELS: {memoryChannel={ parameters:{transactionCapacity=100, 
> capacity=10000, type=memory} }, fileChannel={ 
> parameters:{checkpointDir=/home/hadoop/flume/channel/checkpointDir, 
> dataDirs=/home/hadoop/flume/channel/dataDir, type=file} }}
> SINKS: {hdfsSink={ 
> parameters:{hdfs.path=hdfs://MLNameNode2001:54310/flume, type=hdfs, 
> channel=fileChannel} }, fileSink={ parameters:{type=file_roll, 
> sink.directory=/home/hadoop/flume/output} }}
>
> 2012-08-10 10:56:50,627 (conf-file-poller-0) [DEBUG - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)]

> Created channel memoryChannel
> 2012-08-10 10:56:50,627 (conf-file-poller-0) [DEBUG - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)]

> Created channel fileChannel
> 2012-08-10 10:56:50,639 (conf-file-poller-0) [DEBUG - 
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)]

> Creating s
>
>
>
>
>
> On 08/08/2012 10:27 PM, Hari Shreedharan wrote:
>> Could you share your configuration with us? Your class path is quite 
>> bare, and remember that Hadoop does not ship with Flume. Can you take 
>> a look at the logs and see if you notice any errors?
>>
>> Note that Flume "not printing" stuff is quite normal - flume is not 
>> exactly interactive. You should look for logs - it does not print 
>> stuff on the console. It will print to the console, only the exec 
>> command it uses to start up flume - which is what you are seeing. The 
>> logs should give you more details. Did you check the logs to make 
>> sure Flume is not working at all?
>>
>>
>> Hari
>>
>> -- 
>> Hari Shreedharan
>>
>> On Wednesday, August 8, 2012 at 9:49 AM, Patrick Wendell wrote:
>>
>>> Jagadish,
>>>
>>> If you use the new page and it still doesn't work, can you send me 
>>> the output of jstack against the flume pid? I've noticed some 
>>> similar issues in the past (silent startup failures) and I'm also 
>>> trying to debug.
>>>
>>> - Patrick
>>>
>>> On Wed, Aug 8, 2012 at 12:49 AM, alo alt <wget.null@gmail.com 
>>> <mailto:wget.null@gmail.com>> wrote:
>>>> HI,
>>>>
>>>> This page is outdated. Please follow:
>>>> http://flume.apache.org/FlumeUserGuide.html#setup
>>>>
>>>> Since you haven't attached logs or error messages it's hard to say 
>>>> what happen.
>>>>
>>>> best
>>>> - Alex
>>>>
>>>>
>>>> Jagadish Bihani wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> I have downloaded the tarball of latest flume-ng 1.2.0.
>>>>> I have JAVA_HOME properly set.
>>>>>
>>>>> To begin with I have followed the instructions in "
>>>>> https://cwiki.apache.org/FLUME/getting-started.html"
>>>>> as it is. And even for that basic example:
>>>>>
>>>>> My flume agent stucks printing the following output and then nothing
>>>>> happens:
>>>>>
>>>>> *bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
>>>>>
>>>>> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp
>>>>> '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*'

>>>>>
>>>>> -Djava.library.path= org.apache.flume.node.Application -f
>>>>> conf/flume.conf -n agent1*
>>>>>
>>>>>
>>>>> Please guide me what might be the cause.
>>>>>
>>>>> Regards,
>>>>> Jagadish
>>>>
>>>>
>>>>> Jagadish Bihani <mailto:jagadish.bihani@pubmatic.com>
>>>>> August 8, 2012 9:45 AM
>>>>> Hi
>>>>>
>>>>> I have downloaded the tarball of latest flume-ng 1.2.0.
>>>>> I have JAVA_HOME properly set.
>>>>>
>>>>> To begin with I have followed the instructions in " 
>>>>> https://cwiki.apache.org/FLUME/getting-started.html"
>>>>> as it is. And even for that basic example:
>>>>>
>>>>> My flume agent stucks printing the following output and then 
>>>>> nothing happens:
>>>>>
>>>>> * bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
>>>>> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp 
>>>>> '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*'

>>>>> -Djava.library.path= org.apache.flume.node.Application -f 
>>>>> conf/flume.conf -n agent1*
>>>>>
>>>>> Please guide me what might be the cause.
>>>>>
>>>>> Regards,
>>>>> Jagadish
>>>>
>>>> -- 
>>>> Alexander Alten-Lorenz
>>>> http://mapredit.blogspot.com
>>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>>
>>
>


Mime
View raw message