flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mardan Khan <mardan8...@gmail.com>
Subject Re: Use of Flume for the sensor network data
Date Mon, 23 Jul 2012 04:49:22 GMT
Dear Mohammad Tariq,

Many thanks for your valuable information.

For the testing purpose , i have installed flume on SuSE Linux system. when
i have type the command $/etc/init.d/flume-ng-agent start. Then i have
receive message that Starting Flume NG agent daemon (flume-ng-agent):

I think this mean my flume agent is working properly. I have the following
changes in configuration file according to your example. The configuration
file as:



agent.sources = seqGenSrc
agent.channels = memoryChannel
agent.sinks = loggerSink

# For each one of the sources, the type is defined
agent.sources.seqGenSrc.type = seq

# The channel can be defined as follows.
agent.sources.seqGenSrc.channels = memoryChannel

# Each sink's type must be defined
agent.sinks.loggerSink.type = logger

#Specify the channel the sink should use
agent.sinks.loggerSink.channel = memoryChannel

# Each channel's type is defined.
agent.channels.memoryChannel.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 100

agent1.sources = tail
agent1.channels = MemoryChannel-2
agent1.sinks = HDFS

agent1.sources.tail.type = exec
agent1.sources.tail.command = tail -F /var/log/flume-ng/flume-init.log
agent1.sources.tail.channels = MemoryChannel-2

agent1.sources.tail.interceptors = hostint
agent1.sources.tail.interceptors.hostint.type =
org.apache.flume.interceptor.HostInterceptor$Builder
agent1.sources.tail.interceptors.hostint.preserverExisting = true
agent1.sources.tail.interceptors.hostint.useIP = true

agent1.sinks.HDFS.channel = MemoryChannel-2
agent1.channels.MemoryChannel-2.type = memory
agent1.sinks.HDFS.type =hdfs
agent1.sinks.HDFS.hdfs.path = hdfs://134.83.35.24/user/mardan/
agent1.sinks.HDFS.hdfs.file.Type = DataStream
agent1.sinks.HDFS.hdfs.writeFormat = Text



when i have type the following command

$ /usr/bin/flume-ng agent -n agent1 -f /etc/flume-ng/conf/flume.conf



I got the following warring / error  messages

Warning: No configuration directory set! Use --conf <dir> to override.
Info: Including Hadoop libraries found via (/usr/bin/hadoop) for HDFS access
Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.6.1.jar from classpath
Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar from classpath
Info: Excluding /usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar from classpath
Info: Excluding /usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar from
classpath
Info: Including HBASE libraries found via (/usr/bin/hbase) for HBASE access
Info: Excluding /usr/lib/hbase/bin/../lib/slf4j-api-1.6.1.jar from classpath
Info: Excluding /usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar from classpath
Info: Excluding /usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar from
classpath
Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.6.1.jar from classpath
Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar from classpath
Info: Excluding /usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar from classpath
+ exec /usr/java/jdk1.6.0_31/bin/java -Xmx20m -cp
'/usr/lib/flume-ng/lib/*:/etc/hadoop/conf:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/avro-1.5.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/commons-logging-api-1.1.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/hue-plugins-2.0.0-cdh4.0.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.

................................................................................


12/07/23 05:41:29 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 1
12/07/23 05:41:29 INFO node.FlumeNode: Flume node starting - agent1
12/07/23 05:41:29 INFO nodemanager.DefaultLogicalNodeManager: Node manager
starting
12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
Configuration provider starting
12/07/23 05:41:29 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 10
12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
Reloading configuration file:/etc/flume-ng/conf/flume.conf
12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
12/07/23 05:41:29 INFO conf.FlumeConfiguration: Added sinks: loggerSink
Agent: agent
12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:loggerSink
12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
12/07/23 05:41:29 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent:
agent1
12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:loggerSink
12/07/23 05:41:29 INFO conf.FlumeConfiguration: Post-validation flume
configuration contains configuration  for agents: [agent, agent1]
12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
Creating channels
12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
created channel MemoryChannel-2
12/07/23 05:41:29 ERROR channel.ChannelProcessor: Builder class not found.
Exception follows.
java.lang.ClassNotFoundException:
org.apache.flume.interceptor.HostInterceptor$Builder
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:169)
    at
org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:103)
    at
org.apache.flume.channel.ChannelProcessor.configure(ChannelProcessor.java:79)
    at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
    at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSources(PropertiesFileConfigurationProvider.java:337)
    at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:222)
    at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
    at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
    at
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)



Could you please let me know why give me this message / class missing.

Many thanks










On Sun, Jul 22, 2012 at 10:12 PM, Mohammad Tariq <dontariq@gmail.com> wrote:

> Hello Mardan,
>
>         In order to aggregate data into your Hadoop cluster you need
> to set up a Flume agent first. In order to do that you have to write a
> config file having desired properties. An example file would be
> somewhat like this :
>
> agent1.sources = tail
> agent1.channels = MemoryChannel-2
> agent1.sinks = HDFS
>
> agent1.sources.tail.type = exec
> agent1.sources.tail.command = tail -F /var/log/apache2/access.log
> agent1.sources.tail.channels = MemoryChannel-2
>
> agent1.sources.tail.interceptors = hostint
> agent1.sources.tail.interceptors.hostint.type =
> org.apache.flume.interceptor.HostInterceptor$Builder
> agent1.sources.tail.interceptors.hostint.preserveExisting = true
> agent1.sources.tail.interceptors.hostint.useIP = true
>
> agent1.sinks.HDFS.channel = MemoryChannel-2
> agent1.sinks.HDFS.type = hdfs
> agent1.sinks.HDFS.hdfs.path = hdfs://localhost:9000/flume/%{host}
> agent1.sinks.HDFS.hdfs.file.Type = DataStream
> agent1.sinks.HDFS.hdfs.writeFormat = Text
>
> agent1.channels.MemoryChannel-2.type = memory
>
> You can visit this link as the starting point, if you want -
> http://cloudfront.blogspot.in/2012/06/how-to-build-and-use-flume-ng.html
>
> And, it is quite possible to run Flume-1.x o windows. Here is a great
> post by Alex on how to do that -
> http://mapredit.blogspot.in/2012/07/run-flume-13x-on-windows.html
>
> Hope it helps.
>
> Regards,
>     Mohammad Tariq
>
>
> On Mon, Jul 23, 2012 at 2:17 AM, mardan Khan <mardan8310@gmail.com> wrote:
> > Yeah, my cluster is always running. But i dont know how to setup the
> flume
> > that directly stream the data to hadoop. I have must install the flume
> agent
> > on window machine. As per my study the flume version-0.9.4 agent can
> install
> > on window machine. Can we install flume version 1.x on window machine?
> > If any one have done, please let me guide.
> >
> >
> >
> > Many thanks
> >
> >
> >
> > On Sun, Jul 22, 2012 at 7:26 PM, Mohammad Tariq <dontariq@gmail.com>
> wrote:
> >>
> >> NameNode and DataNode must be running if we need to write anything to
> the
> >> Hdfs.
> >>
> >> Regards,
> >>     Mohammad Tariq
> >>
> >>
> >> On Sun, Jul 22, 2012 at 11:41 PM, Henry Larson <newtoflume@gmail.com>
> >> wrote:
> >> > You can have flume write to HDFS: however, do you have your hadoop
> >> > cluster running all the time?
> >
> >
>

Mime
View raw message