flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Johnson <st...@webninja.com>
Subject Unable to start FileChannel
Date Wed, 29 Aug 2012 15:38:30 GMT
Good morning.  I'm trying to test Flume for use for our companies log
collection.  These are application transaction logs (not syslogs) so I need
the agents channel data to be fully recoverable.  I have done some initial
setup testing using the memory channel for my initial tests and it's been
going well.  However, I'm unable to get the FileChannel operable.  Below is
the exception I'm getting.  I'm currently running this on OSX 10.7.4
(Darwin Kernel Version 11.4.0).  I've installed using the flume-ng 1.2.0
binary.

I searched the flume mail lists and didn't find anything, but I was able to
find something on google here, but I didn't see any more updates on the
issues since late June:
http://www.mail-archive.com/flume-dev@incubator.apache.org/msg07419.html

I've pasted the exception below I'm getting at startup.  Also, here is my
confiugration, thanks in advance for any help:
testagent.sources = netcatSource
testagent.channels = fileChannel
testagent.sinks = fileSink

# For each one of the sources, the type is defined
testagent.sources.netcatSource.type = netcat

# The channel can be defined as follows.
testagent.sources.netcatSource.channels = fileChannel

# Each sink's type must be defined
testagent.sinks.fileSink.type = FILE_ROLL

#Specify the channel the sink should use
testagent.sinks.fileSink.channel = fileChannel

# Each channel's type is defined.
testagent.channels.fileChannel.type = file

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
testagent.sources.netcatSource.bind = 0.0.0.0
testagent.sources.netcatSource.port = 6166
testagent.sources.netcatSource.max-line-length = 32768
testagent.channels.fileChannel.transactionCapacity = 1024000
testagent.channels.fileChannel.maxFileSize = 10240000000
testagent.channels.fileChannel.checkpointDir =
/opt/dotomi/flume/file_channel/checkpoint
testagent.channels.fileChannel.dataDirs =
/opt/dotomi/flume/file_channel/data
testagent.channels.fileChannel.capacity = 512000000
testagent.sinks.fileSink.sink.directory = /opt/dotomi/flume/sinkdir
testagent.sinks.fileSink.sink.rollInterval = 0


2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager:
Starting new configuration:{
sourceRunners:{netcatSource=EventDrivenSourceRunner: { source:o
rg.apache.flume.source.NetcatSource@3c50507 }}
sinkRunners:{fileSink=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@7bcd107f counterGroup:{
nam
e:null counters:{} } }} channels:{fileChannel=FileChannel fileChannel {
dataDirs: [/opt/dotomi/flume/file_channel/data] }} }
2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager:
Starting Channel fileChannel
2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager: Waiting
for channel: fileChannel to start. Sleeping for 500 ms
2012-08-28 16:32:39,162 INFO file.FileChannel: Starting FileChannel
fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }...
2012-08-28 16:32:39,165 ERROR lifecycle.LifecycleSupervisor: Unable to
start FileChannel fileChannel { dataDirs:
[/opt/dotomi/flume/file_channel/data] } - Excep
tion follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
        at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at org.apache.flume.channel.file.Log$Builder.build(Log.java:139)
        at
org.apache.flume.channel.file.FileChannel.start(FileChannel.java:226)
        at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:237)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        ... 24 more
2012-08-28 16:32:39,167 INFO file.FileChannel: Stopping FileChannel
fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }...

-- 
Steve Johnson
steve@webninja.com

Mime
View raw message