flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mardan Khan <mardan8...@gmail.com>
Subject Re: Use of Flume for the sensor network data
Date Mon, 23 Jul 2012 23:46:59 GMT
Hi,

I am still getting missing class error.

I have specify the full path. My flume is installed in usr/lib/flume-ng and
my configuration file is :/usr/lib/flume-ng/conf/flume.conf.

you have mentioned that -c option with command, sorry i dont know exactly
how I can use, I tried but give me the error message.

I am apply the following command:

$: /usr/bin/flume-ng agent -n agent -f /usr/lib/flume-ng/conf/flume.conf


Could you please write me the exact command to override this problem.ch

The configuration file exactly same you have posted,  just change the hdfs
path and location of log file.


For remind the error message as:


12/07/24 00:33:51 ERROR channel.ChannelProcessor: Builder class not found.
Exception follows.
java.lang.ClassNotFoundException:
org.apache.flume.interceptor.HostInterceptor$Builder
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)

I am really straggling to run one command successful

Thanks






On Mon, Jul 23, 2012 at 9:09 AM, Mohammad Tariq <dontariq@gmail.com> wrote:

> Hi mardan,
>
>      You need to use the -c option with tour command to specify the
> directory where your configuration file is kept. Just look at the the
> other thread of yours.
>
> Regards,
>     Mohammad Tariq
>
>
> On Mon, Jul 23, 2012 at 10:19 AM, mardan Khan <mardan8310@gmail.com>
> wrote:
> > Dear Mohammad Tariq,
> >
> > Many thanks for your valuable information.
> >
> > For the testing purpose , i have installed flume on SuSE Linux system.
> when
> > i have type the command $/etc/init.d/flume-ng-agent start. Then i have
> > receive message that Starting Flume NG agent daemon (flume-ng-agent):
> >
> > I think this mean my flume agent is working properly. I have the
> following
> > changes in configuration file according to your example. The
> configuration
> > file as:
> >
> >
> >
> > agent.sources = seqGenSrc
> > agent.channels = memoryChannel
> > agent.sinks = loggerSink
> >
> > # For each one of the sources, the type is defined
> > agent.sources.seqGenSrc.type = seq
> >
> > # The channel can be defined as follows.
> > agent.sources.seqGenSrc.channels = memoryChannel
> >
> > # Each sink's type must be defined
> > agent.sinks.loggerSink.type = logger
> >
> > #Specify the channel the sink should use
> > agent.sinks.loggerSink.channel = memoryChannel
> >
> > # Each channel's type is defined.
> > agent.channels.memoryChannel.type = memory
> >
> > # Other config values specific to each type of channel(sink or source)
> > # can be defined as well
> > # In this case, it specifies the capacity of the memory channel
> > agent.channels.memoryChannel.capacity = 100
> >
> >
> > agent1.sources = tail
> > agent1.channels = MemoryChannel-2
> > agent1.sinks = HDFS
> >
> > agent1.sources.tail.type = exec
> > agent1.sources.tail.command = tail -F /var/log/flume-ng/flume-init.log
> >
> > agent1.sources.tail.channels = MemoryChannel-2
> >
> > agent1.sources.tail.interceptors = hostint
> > agent1.sources.tail.interceptors.hostint.type =
> > org.apache.flume.interceptor.HostInterceptor$Builder
> > agent1.sources.tail.interceptors.hostint.preserverExisting = true
> >
> > agent1.sources.tail.interceptors.hostint.useIP = true
> >
> > agent1.sinks.HDFS.channel = MemoryChannel-2
> > agent1.channels.MemoryChannel-2.type = memory
> > agent1.sinks.HDFS.type =hdfs
> > agent1.sinks.HDFS.hdfs.path = hdfs://134.83.35.24/user/mardan/
> >
> > agent1.sinks.HDFS.hdfs.file.Type = DataStream
> > agent1.sinks.HDFS.hdfs.writeFormat = Text
> >
> >
> >
> > when i have type the following command
> >
> > $ /usr/bin/flume-ng agent -n agent1 -f /etc/flume-ng/conf/flume.conf
> >
> >
> >
> > I got the following warring / error  messages
> >
> > Warning: No configuration directory set! Use --conf <dir> to override.
> > Info: Including Hadoop libraries found via (/usr/bin/hadoop) for HDFS
> access
> > Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.6.1.jar from classpath
> > Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar from
> classpath
> > Info: Excluding /usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar from
> classpath
> > Info: Excluding /usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar
> from
> > classpath
> > Info: Including HBASE libraries found via (/usr/bin/hbase) for HBASE
> access
> > Info: Excluding /usr/lib/hbase/bin/../lib/slf4j-api-1.6.1.jar from
> classpath
> > Info: Excluding /usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar from classpath
> > Info: Excluding /usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar from
> > classpath
> > Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.6.1.jar from classpath
> > Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar from
> classpath
> > Info: Excluding /usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar from
> classpath
> > + exec /usr/java/jdk1.6.0_31/bin/java -Xmx20m -cp
> >
> '/usr/lib/flume-ng/lib/*:/etc/hadoop/conf:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/avro-1.5.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/commons-logging-api-1.1.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/hue-plugins-2.0.0-cdh4.0.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.
> >
> >
> ................................................................................
> >
> >
> > 12/07/23 05:41:29 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
> > supervisor 1
> > 12/07/23 05:41:29 INFO node.FlumeNode: Flume node starting - agent1
> > 12/07/23 05:41:29 INFO nodemanager.DefaultLogicalNodeManager: Node
> manager
> > starting
> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
> > Configuration provider starting
> > 12/07/23 05:41:29 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
> > supervisor 10
> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
> > Reloading configuration file:/etc/flume-ng/conf/flume.conf
> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Added sinks: loggerSink
> > Agent: agent
> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:loggerSink
> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent:
> > agent1
> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS
> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:loggerSink
> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Post-validation flume
> > configuration contains configuration  for agents: [agent, agent1]
> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
> > Creating channels
> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider:
> > created channel MemoryChannel-2
> > 12/07/23 05:41:29 ERROR channel.ChannelProcessor: Builder class not
> found.
> > Exception follows.
> > java.lang.ClassNotFoundException:
> > org.apache.flume.interceptor.HostInterceptor$Builder
> >     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >     at java.security.AccessController.doPrivileged(Native Method)
> >     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >     at java.lang.Class.forName0(Native Method)
> >     at java.lang.Class.forName(Class.java:169)
> >     at
> >
> org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:103)
> >     at
> >
> org.apache.flume.channel.ChannelProcessor.configure(ChannelProcessor.java:79)
> >     at
> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
> >     at
> >
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSources(PropertiesFileConfigurationProvider.java:337)
> >     at
> >
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:222)
> >     at
> >
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
> >     at
> >
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
> >     at
> >
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
> >     at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >     at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> >     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> >     at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> >     at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> >     at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >     at java.lang.Thread.run(Thread.java:662)
> >
> >
> >
> > Could you please let me know why give me this message / class missing.
> >
> > Many thanks
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Sun, Jul 22, 2012 at 10:12 PM, Mohammad Tariq <dontariq@gmail.com>
> wrote:
> >>
> >> Hello Mardan,
> >>
> >>         In order to aggregate data into your Hadoop cluster you need
> >> to set up a Flume agent first. In order to do that you have to write a
> >> config file having desired properties. An example file would be
> >> somewhat like this :
> >>
> >> agent1.sources = tail
> >> agent1.channels = MemoryChannel-2
> >> agent1.sinks = HDFS
> >>
> >> agent1.sources.tail.type = exec
> >> agent1.sources.tail.command = tail -F /var/log/apache2/access.log
> >> agent1.sources.tail.channels = MemoryChannel-2
> >>
> >> agent1.sources.tail.interceptors = hostint
> >> agent1.sources.tail.interceptors.hostint.type =
> >> org.apache.flume.interceptor.HostInterceptor$Builder
> >> agent1.sources.tail.interceptors.hostint.preserveExisting = true
> >> agent1.sources.tail.interceptors.hostint.useIP = true
> >>
> >> agent1.sinks.HDFS.channel = MemoryChannel-2
> >> agent1.sinks.HDFS.type = hdfs
> >> agent1.sinks.HDFS.hdfs.path = hdfs://localhost:9000/flume/%{host}
> >> agent1.sinks.HDFS.hdfs.file.Type = DataStream
> >> agent1.sinks.HDFS.hdfs.writeFormat = Text
> >>
> >> agent1.channels.MemoryChannel-2.type = memory
> >>
> >> You can visit this link as the starting point, if you want -
> >>
> http://cloudfront.blogspot.in/2012/06/how-to-build-and-use-flume-ng.html
> >>
> >> And, it is quite possible to run Flume-1.x o windows. Here is a great
> >> post by Alex on how to do that -
> >> http://mapredit.blogspot.in/2012/07/run-flume-13x-on-windows.html
> >>
> >> Hope it helps.
> >>
> >> Regards,
> >>     Mohammad Tariq
> >>
> >>
> >> On Mon, Jul 23, 2012 at 2:17 AM, mardan Khan <mardan8310@gmail.com>
> wrote:
> >> > Yeah, my cluster is always running. But i dont know how to setup the
> >> > flume
> >> > that directly stream the data to hadoop. I have must install the flume
> >> > agent
> >> > on window machine. As per my study the flume version-0.9.4 agent can
> >> > install
> >> > on window machine. Can we install flume version 1.x on window machine?
> >> > If any one have done, please let me guide.
> >> >
> >> >
> >> >
> >> > Many thanks
> >> >
> >> >
> >> >
> >> > On Sun, Jul 22, 2012 at 7:26 PM, Mohammad Tariq <dontariq@gmail.com>
> >> > wrote:
> >> >>
> >> >> NameNode and DataNode must be running if we need to write anything
to
> >> >> the
> >> >> Hdfs.
> >> >>
> >> >> Regards,
> >> >>     Mohammad Tariq
> >> >>
> >> >>
> >> >> On Sun, Jul 22, 2012 at 11:41 PM, Henry Larson <newtoflume@gmail.com
> >
> >> >> wrote:
> >> >> > You can have flume write to HDFS: however, do you have your hadoop
> >> >> > cluster running all the time?
> >> >
> >> >
> >
> >
>

Mime
View raw message