flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Tariq <donta...@gmail.com>
Subject Re: Use of Flume for the sensor network data
Date Mon, 23 Jul 2012 08:09:35 GMT
Hi mardan,

     You need to use the -c option with tour command to specify the
directory where your configuration file is kept. Just look at the the
other thread of yours.

Regards,
    Mohammad Tariq


On Mon, Jul 23, 2012 at 10:19 AM, mardan Khan <mardan8310@gmail.com> wrote:
> Dear Mohammad Tariq,
>
> Many thanks for your valuable information.
>
> For the testing purpose , i have installed flume on SuSE Linux system. when
> i have type the command $/etc/init.d/flume-ng-agent start. Then i have
> receive message that Starting Flume NG agent daemon (flume-ng-agent):
>
> I think this mean my flume agent is working properly. I have the following
> changes in configuration file according to your example. The configuration
> file as:
>
>
>
> agent.sources = seqGenSrc
> agent.channels = memoryChannel
> agent.sinks = loggerSink
>
> # For each one of the sources, the type is defined
> agent.sources.seqGenSrc.type = seq
>
> # The channel can be defined as follows.
> agent.sources.seqGenSrc.channels = memoryChannel
>
> # Each sink's type must be defined
> agent.sinks.loggerSink.type = logger
>
> #Specify the channel the sink should use
> agent.sinks.loggerSink.channel = memoryChannel
>
> # Each channel's type is defined.
> agent.channels.memoryChannel.type = memory
>
> # Other config values specific to each type of channel(sink or source)
> # can be defined as well
> # In this case, it specifies the capacity of the memory channel
> agent.channels.memoryChannel.capacity = 100
>
>
> agent1.sources = tail
> agent1.channels = MemoryChannel-2
> agent1.sinks = HDFS
>
> agent1.sources.tail.type = exec
> agent1.sources.tail.command = tail -F /var/log/flume-ng/flume-init.log
>
> agent1.sources.tail.channels = MemoryChannel-2
>
> agent1.sources.tail.interceptors = hostint
> agent1.sources.tail.interceptors.hostint.type =
> org.apache.flume.interceptor.HostInterceptor$Builder
> agent1.sources.tail.interceptors.hostint.preserverExisting = true
>
> agent1.sources.tail.interceptors.hostint.useIP = true
>
> agent1.sinks.HDFS.channel = MemoryChannel-2
> agent1.channels.MemoryChannel-2.type = memory
> agent1.sinks.HDFS.type =hdfs
> agent1.sinks.HDFS.hdfs.path = hdfs://134.83.35.24/user/mardan/
>
> agent1.sinks.HDFS.hdfs.file.Type = DataStream
> agent1.sinks.HDFS.hdfs.writeFormat = Text
>
>
>
> when i have type the following command
>
> $ /usr/bin/flume-ng agent -n agent1 -f /etc/flume-ng/conf/flume.conf
>
>
>
> I got the following warring / error  messages
>
> Warning: No configuration directory set! Use --conf <dir> to override.
> Info: Including Hadoop libraries found via (/usr/bin/hadoop) for HDFS access
> Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.6.1.jar from classpath
> Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar from classpath
> Info: Excluding /usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar from classpath
> Info: Excluding /usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar from
> classpath
> Info: Including HBASE libraries found via (/usr/bin/hbase) for HBASE access
> Info: Excluding /usr/lib/hbase/bin/../lib/slf4j-api-1.6.1.jar from classpath
> Info: Excluding /usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar from classpath
> Info: Excluding /usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar from
> classpath
> Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.6.1.jar from classpath
> Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar from classpath
> Info: Excluding /usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar from classpath
> + exec /usr/java/jdk1.6.0_31/bin/java -Xmx20m -cp
> '/usr/lib/flume-ng/lib/*:/etc/hadoop/conf:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/avro-1.5.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/commons-logging-api-1.1.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/hue-plugins-2.0.0-cdh4.0.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.
>
> ................................................................................
>
>
> 12/07/23 05:41:29 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
> supervisor 1
> 12/07/23 05:41:29 INFO node.FlumeNode: Flume node starting - agent1
> 12/07/23 05:41:29 INFO nodemanager.DefaultLogicalNodeManager: Node manager
> starting
> 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
> Configuration provider starting
> 12/07/23 05:41:29 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
> supervisor 10
> 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
> Reloading configuration file:/etc/flume-ng/conf/flume.conf
> 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Added sinks: loggerSink
> Agent: agent
> 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:loggerSink
> 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent:
> agent1
> 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:loggerSink
> 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Post-validation flume
> configuration contains configuration  for agents: [agent, agent1]
> 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
> Creating channels
> 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
> created channel MemoryChannel-2
> 12/07/23 05:41:29 ERROR channel.ChannelProcessor: Builder class not found.
> Exception follows.
> java.lang.ClassNotFoundException:
> org.apache.flume.interceptor.HostInterceptor$Builder
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>     at java.lang.Class.forName0(Native Method)
>     at java.lang.Class.forName(Class.java:169)
>     at
> org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:103)
>     at
> org.apache.flume.channel.ChannelProcessor.configure(ChannelProcessor.java:79)
>     at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>     at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSources(PropertiesFileConfigurationProvider.java:337)
>     at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:222)
>     at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
>     at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
>     at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
>     at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>     at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>     at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>     at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>     at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:662)
>
>
>
> Could you please let me know why give me this message / class missing.
>
> Many thanks
>
>
>
>
>
>
>
>
>
>
>
> On Sun, Jul 22, 2012 at 10:12 PM, Mohammad Tariq <dontariq@gmail.com> wrote:
>>
>> Hello Mardan,
>>
>>         In order to aggregate data into your Hadoop cluster you need
>> to set up a Flume agent first. In order to do that you have to write a
>> config file having desired properties. An example file would be
>> somewhat like this :
>>
>> agent1.sources = tail
>> agent1.channels = MemoryChannel-2
>> agent1.sinks = HDFS
>>
>> agent1.sources.tail.type = exec
>> agent1.sources.tail.command = tail -F /var/log/apache2/access.log
>> agent1.sources.tail.channels = MemoryChannel-2
>>
>> agent1.sources.tail.interceptors = hostint
>> agent1.sources.tail.interceptors.hostint.type =
>> org.apache.flume.interceptor.HostInterceptor$Builder
>> agent1.sources.tail.interceptors.hostint.preserveExisting = true
>> agent1.sources.tail.interceptors.hostint.useIP = true
>>
>> agent1.sinks.HDFS.channel = MemoryChannel-2
>> agent1.sinks.HDFS.type = hdfs
>> agent1.sinks.HDFS.hdfs.path = hdfs://localhost:9000/flume/%{host}
>> agent1.sinks.HDFS.hdfs.file.Type = DataStream
>> agent1.sinks.HDFS.hdfs.writeFormat = Text
>>
>> agent1.channels.MemoryChannel-2.type = memory
>>
>> You can visit this link as the starting point, if you want -
>> http://cloudfront.blogspot.in/2012/06/how-to-build-and-use-flume-ng.html
>>
>> And, it is quite possible to run Flume-1.x o windows. Here is a great
>> post by Alex on how to do that -
>> http://mapredit.blogspot.in/2012/07/run-flume-13x-on-windows.html
>>
>> Hope it helps.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Mon, Jul 23, 2012 at 2:17 AM, mardan Khan <mardan8310@gmail.com> wrote:
>> > Yeah, my cluster is always running. But i dont know how to setup the
>> > flume
>> > that directly stream the data to hadoop. I have must install the flume
>> > agent
>> > on window machine. As per my study the flume version-0.9.4 agent can
>> > install
>> > on window machine. Can we install flume version 1.x on window machine?
>> > If any one have done, please let me guide.
>> >
>> >
>> >
>> > Many thanks
>> >
>> >
>> >
>> > On Sun, Jul 22, 2012 at 7:26 PM, Mohammad Tariq <dontariq@gmail.com>
>> > wrote:
>> >>
>> >> NameNode and DataNode must be running if we need to write anything to
>> >> the
>> >> Hdfs.
>> >>
>> >> Regards,
>> >>     Mohammad Tariq
>> >>
>> >>
>> >> On Sun, Jul 22, 2012 at 11:41 PM, Henry Larson <newtoflume@gmail.com>
>> >> wrote:
>> >> > You can have flume write to HDFS: however, do you have your hadoop
>> >> > cluster running all the time?
>> >
>> >
>
>

Mime
View raw message