flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mardan Khan <mardan8...@gmail.com>
Subject Re: Use of Flume for the sensor network data
Date Tue, 24 Jul 2012 12:04:56 GMT
Followed the following links

1)
https://ccp.cloudera.com/display/CDH4B2/CDH4+Installation#CDH4Installation-AddingaDebianRepository

2) https://ccp.cloudera.com/display/CDH4B2/Flume+Installation




cheers



On Tue, Jul 24, 2012 at 3:55 AM, Will McQueen <will@cloudera.com> wrote:

> Hi Mardan,
>
> How did you install Flume in SLES? Are you installing Flume straight from
> Apache, or from a distro?
>
> Cheers,
> Will
>
>
> On Mon, Jul 23, 2012 at 4:46 PM, mardan Khan <mardan8310@gmail.com> wrote:
>
>> Hi,
>>
>> I am still getting missing class error.
>>
>> I have specify the full path. My flume is installed in usr/lib/flume-ng
>> and my configuration file is :/usr/lib/flume-ng/conf/flume.conf.
>>
>> you have mentioned that -c option with command, sorry i dont know exactly
>> how I can use, I tried but give me the error message.
>>
>> I am apply the following command:
>>
>> $: /usr/bin/flume-ng agent -n agent -f /usr/lib/flume-ng/conf/flume.conf
>>
>>
>> Could you please write me the exact command to override this problem.ch
>>
>> The configuration file exactly same you have posted,  just change the
>> hdfs path and location of log file.
>>
>>
>> For remind the error message as:
>>
>>
>> 12/07/24 00:33:51 ERROR channel.ChannelProcessor: Builder class not
>> found. Exception follows.
>>
>> java.lang.ClassNotFoundException:
>> org.apache.flume.interceptor.HostInterceptor$Builder
>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>     at java.lang.Class.forName0(Native Method)
>>
>> I am really straggling to run one command successful
>>
>> Thanks
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Jul 23, 2012 at 9:09 AM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>
>>> Hi mardan,
>>>
>>>      You need to use the -c option with tour command to specify the
>>> directory where your configuration file is kept. Just look at the the
>>> other thread of yours.
>>>
>>> Regards,
>>>     Mohammad Tariq
>>>
>>>
>>> On Mon, Jul 23, 2012 at 10:19 AM, mardan Khan <mardan8310@gmail.com>
>>> wrote:
>>> > Dear Mohammad Tariq,
>>> >
>>> > Many thanks for your valuable information.
>>> >
>>> > For the testing purpose , i have installed flume on SuSE Linux system.
>>> when
>>> > i have type the command $/etc/init.d/flume-ng-agent start. Then i have
>>> > receive message that Starting Flume NG agent daemon (flume-ng-agent):
>>> >
>>> > I think this mean my flume agent is working properly. I have the
>>> following
>>> > changes in configuration file according to your example. The
>>> configuration
>>> > file as:
>>> >
>>> >
>>> >
>>> > agent.sources = seqGenSrc
>>> > agent.channels = memoryChannel
>>> > agent.sinks = loggerSink
>>> >
>>> > # For each one of the sources, the type is defined
>>> > agent.sources.seqGenSrc.type = seq
>>> >
>>> > # The channel can be defined as follows.
>>> > agent.sources.seqGenSrc.channels = memoryChannel
>>> >
>>> > # Each sink's type must be defined
>>> > agent.sinks.loggerSink.type = logger
>>> >
>>> > #Specify the channel the sink should use
>>> > agent.sinks.loggerSink.channel = memoryChannel
>>> >
>>> > # Each channel's type is defined.
>>> > agent.channels.memoryChannel.type = memory
>>> >
>>> > # Other config values specific to each type of channel(sink or source)
>>> > # can be defined as well
>>> > # In this case, it specifies the capacity of the memory channel
>>> > agent.channels.memoryChannel.capacity = 100
>>> >
>>> >
>>> > agent1.sources = tail
>>> > agent1.channels = MemoryChannel-2
>>> > agent1.sinks = HDFS
>>> >
>>> > agent1.sources.tail.type = exec
>>> > agent1.sources.tail.command = tail -F /var/log/flume-ng/flume-init.log
>>> >
>>> > agent1.sources.tail.channels = MemoryChannel-2
>>> >
>>> > agent1.sources.tail.interceptors = hostint
>>> > agent1.sources.tail.interceptors.hostint.type =
>>> > org.apache.flume.interceptor.HostInterceptor$Builder
>>> > agent1.sources.tail.interceptors.hostint.preserverExisting = true
>>> >
>>> > agent1.sources.tail.interceptors.hostint.useIP = true
>>> >
>>> > agent1.sinks.HDFS.channel = MemoryChannel-2
>>> > agent1.channels.MemoryChannel-2.type = memory
>>> > agent1.sinks.HDFS.type =hdfs
>>> > agent1.sinks.HDFS.hdfs.path = hdfs://134.83.35.24/user/mardan/
>>> >
>>> > agent1.sinks.HDFS.hdfs.file.Type = DataStream
>>> > agent1.sinks.HDFS.hdfs.writeFormat = Text
>>> >
>>> >
>>> >
>>> > when i have type the following command
>>> >
>>> > $ /usr/bin/flume-ng agent -n agent1 -f /etc/flume-ng/conf/flume.conf
>>> >
>>> >
>>> >
>>> > I got the following warring / error  messages
>>> >
>>> > Warning: No configuration directory set! Use --conf <dir> to override.
>>> > Info: Including Hadoop libraries found via (/usr/bin/hadoop) for HDFS
>>> access
>>> > Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.6.1.jar from classpath
>>> > Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar from
>>> classpath
>>> > Info: Excluding /usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar from
>>> classpath
>>> > Info: Excluding /usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar
>>> from
>>> > classpath
>>> > Info: Including HBASE libraries found via (/usr/bin/hbase) for HBASE
>>> access
>>> > Info: Excluding /usr/lib/hbase/bin/../lib/slf4j-api-1.6.1.jar from
>>> classpath
>>> > Info: Excluding /usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar from
>>> classpath
>>> > Info: Excluding /usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar from
>>> > classpath
>>> > Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.6.1.jar from classpath
>>> > Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar from
>>> classpath
>>> > Info: Excluding /usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar from
>>> classpath
>>> > + exec /usr/java/jdk1.6.0_31/bin/java -Xmx20m -cp
>>> >
>>> '/usr/lib/flume-ng/lib/*:/etc/hadoop/conf:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/avro-1.5.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/commons-logging-api-1.1.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/hue-plugins-2.0.0-cdh4.0.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.
>>> >
>>> >
>>> ................................................................................
>>> >
>>> >
>>> > 12/07/23 05:41:29 INFO lifecycle.LifecycleSupervisor: Starting
>>> lifecycle
>>> > supervisor 1
>>> > 12/07/23 05:41:29 INFO node.FlumeNode: Flume node starting - agent1
>>> > 12/07/23 05:41:29 INFO nodemanager.DefaultLogicalNodeManager: Node
>>> manager
>>> > starting
>>> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
>>> > Configuration provider starting
>>> > 12/07/23 05:41:29 INFO lifecycle.LifecycleSupervisor: Starting
>>> lifecycle
>>> > supervisor 10
>>> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
>>> > Reloading configuration file:/etc/flume-ng/conf/flume.conf
>>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
>>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
>>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
>>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Added sinks: loggerSink
>>> > Agent: agent
>>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:loggerSink
>>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
>>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Added sinks: HDFS
>>> Agent:
>>> > agent1
>>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
>>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:loggerSink
>>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Post-validation flume
>>> > configuration contains configuration  for agents: [agent, agent1]
>>> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
>>> > Creating channels
>>> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
>>> > created channel MemoryChannel-2
>>> > 12/07/23 05:41:29 ERROR channel.ChannelProcessor: Builder class not
>>> found.
>>> > Exception follows.
>>> > java.lang.ClassNotFoundException:
>>> > org.apache.flume.interceptor.HostInterceptor$Builder
>>> >     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> >     at java.security.AccessController.doPrivileged(Native Method)
>>> >     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> >     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> >     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> >     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>> >     at java.lang.Class.forName0(Native Method)
>>> >     at java.lang.Class.forName(Class.java:169)
>>> >     at
>>> >
>>> org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:103)
>>> >     at
>>> >
>>> org.apache.flume.channel.ChannelProcessor.configure(ChannelProcessor.java:79)
>>> >     at
>>> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>>> >     at
>>> >
>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSources(PropertiesFileConfigurationProvider.java:337)
>>> >     at
>>> >
>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:222)
>>> >     at
>>> >
>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
>>> >     at
>>> >
>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
>>> >     at
>>> >
>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
>>> >     at
>>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>> >     at
>>> >
>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>> >     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>> >     at
>>> >
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>> >     at
>>> >
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>> >     at
>>> >
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>> >     at
>>> >
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> >     at
>>> >
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> >     at java.lang.Thread.run(Thread.java:662)
>>> >
>>> >
>>> >
>>> > Could you please let me know why give me this message / class missing.
>>> >
>>> > Many thanks
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Sun, Jul 22, 2012 at 10:12 PM, Mohammad Tariq <dontariq@gmail.com>
>>> wrote:
>>> >>
>>> >> Hello Mardan,
>>> >>
>>> >>         In order to aggregate data into your Hadoop cluster you need
>>> >> to set up a Flume agent first. In order to do that you have to write
a
>>> >> config file having desired properties. An example file would be
>>> >> somewhat like this :
>>> >>
>>> >> agent1.sources = tail
>>> >> agent1.channels = MemoryChannel-2
>>> >> agent1.sinks = HDFS
>>> >>
>>> >> agent1.sources.tail.type = exec
>>> >> agent1.sources.tail.command = tail -F /var/log/apache2/access.log
>>> >> agent1.sources.tail.channels = MemoryChannel-2
>>> >>
>>> >> agent1.sources.tail.interceptors = hostint
>>> >> agent1.sources.tail.interceptors.hostint.type =
>>> >> org.apache.flume.interceptor.HostInterceptor$Builder
>>> >> agent1.sources.tail.interceptors.hostint.preserveExisting = true
>>> >> agent1.sources.tail.interceptors.hostint.useIP = true
>>> >>
>>> >> agent1.sinks.HDFS.channel = MemoryChannel-2
>>> >> agent1.sinks.HDFS.type = hdfs
>>> >> agent1.sinks.HDFS.hdfs.path = hdfs://localhost:9000/flume/%{host}
>>> >> agent1.sinks.HDFS.hdfs.file.Type = DataStream
>>> >> agent1.sinks.HDFS.hdfs.writeFormat = Text
>>> >>
>>> >> agent1.channels.MemoryChannel-2.type = memory
>>> >>
>>> >> You can visit this link as the starting point, if you want -
>>> >>
>>> http://cloudfront.blogspot.in/2012/06/how-to-build-and-use-flume-ng.html
>>> >>
>>> >> And, it is quite possible to run Flume-1.x o windows. Here is a great
>>> >> post by Alex on how to do that -
>>> >> http://mapredit.blogspot.in/2012/07/run-flume-13x-on-windows.html
>>> >>
>>> >> Hope it helps.
>>> >>
>>> >> Regards,
>>> >>     Mohammad Tariq
>>> >>
>>> >>
>>> >> On Mon, Jul 23, 2012 at 2:17 AM, mardan Khan <mardan8310@gmail.com>
>>> wrote:
>>> >> > Yeah, my cluster is always running. But i dont know how to setup
the
>>> >> > flume
>>> >> > that directly stream the data to hadoop. I have must install the
>>> flume
>>> >> > agent
>>> >> > on window machine. As per my study the flume version-0.9.4 agent
can
>>> >> > install
>>> >> > on window machine. Can we install flume version 1.x on window
>>> machine?
>>> >> > If any one have done, please let me guide.
>>> >> >
>>> >> >
>>> >> >
>>> >> > Many thanks
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Sun, Jul 22, 2012 at 7:26 PM, Mohammad Tariq <dontariq@gmail.com
>>> >
>>> >> > wrote:
>>> >> >>
>>> >> >> NameNode and DataNode must be running if we need to write anything
>>> to
>>> >> >> the
>>> >> >> Hdfs.
>>> >> >>
>>> >> >> Regards,
>>> >> >>     Mohammad Tariq
>>> >> >>
>>> >> >>
>>> >> >> On Sun, Jul 22, 2012 at 11:41 PM, Henry Larson <
>>> newtoflume@gmail.com>
>>> >> >> wrote:
>>> >> >> > You can have flume write to HDFS: however, do you have
your
>>> hadoop
>>> >> >> > cluster running all the time?
>>> >> >
>>> >> >
>>> >
>>> >
>>>
>>
>>
>

Mime
View raw message