flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: FileChannel on Windows
Date Wed, 10 Apr 2013 20:41:48 GMT
Yes. We will need to remove the format upgrade code and the old format code from the file channel
as well. I think it should be possible to do it, but I suspect it is just that no one had
the time to look at this as yet. 

-- 
Hari Shreedharan


On Wednesday, April 10, 2013 at 1:38 PM, Roshan Naik wrote:

> Hari,
>   i recall some discussion around the desire to get rid of this dependency. i guess it
means removing that piece of code that supports the format upgrade ? are there any plans to
do that ?
> -roshan
> 
> 
> 
> On Wed, Apr 10, 2013 at 1:25 PM, Paul Chavez <pchavez@verticalsearchworks.com (mailto:pchavez@verticalsearchworks.com)>
wrote:
> > I did not realize I could use pre-built binaries on Windows. Thank you, I used this
to grab the hadoop-common.jar from our existing Hadoop cluster and there are no startup errors
now. 
> >  
> > Thanks to you, Brock and Hari.
> >  
> > -Paul
> > 
> > From: Israel Ekpo [mailto:israel@aicer.org] 
> > Sent: Wednesday, April 10, 2013 12:07 PM
> > To: user@flume.apache.org (mailto:user@flume.apache.org)
> > Subject: Re: FileChannel on Windows
> > 
> > Paul,
> > 
> > If Flume 1.3.1 is what you are looking for, you don't have to build it from source.
> > 
> > You can just download it directly from the site. Its already released.
> > 
> > Paul Chavez <pchavez@verticalsearchworks.com (mailto:pchavez@verticalsearchworks.com)>
wrote:
> > 
> > Hello, 
> >  
> > I've been using flume on Windows to send logs to an avro source for awhile now,
using the AvroClient class. I'm now trying to get a full agent running to hopefully leverage
the spoolDir and httpSource functionality. Part of my requirements is a persistent channel
so I'm attempting to get a config running the utilizes the fileChannel.
> >  
> > I followed the instructions at http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html
to build flume, same as I did for the AvroClient. However it seems I need some HDFS libraries.
Can anyone point me to instructions for building the JAR(s) I need to get this working?
> >  
> > Currently I'm running into the following exception:
> > 10 Apr 2013 11:50:29,317 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.start:285)
 - Starting FileChannel fc_Default { dataDirs: [c:\flume_data\log] }...
> > 10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.start:314)
 - Failed to start the file channel [channel=fc_Default]
> > java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
> >  at java.lang.ClassLoader.defineClass1(Native Method)
> >  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
> >  at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
> >  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> >  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> >  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> >  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> >  at java.security.AccessController.doPrivileged(Native Method)
> >  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >  at org.apache.flume.channel.file.Log$Builder.build(Log.java:212)
> >  at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
> >  at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
> >  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> >  at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> >  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> >  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> >  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> >  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> >  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >  at java.lang.Thread.run(Thread.java:662)
> > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable
> >  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >  at java.security.AccessController.doPrivileged(Native Method)
> >  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >  ... 24 more
> > 10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:238)
 - Unable to start FileChannel fc_Default { dataDirs: [c:\flume_data\log] } - Exception follows.
> > java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
> >  at java.lang.ClassLoader.defineClass1(Native Method)
> >  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
> >  at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
> >  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> >  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> >  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> >  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> >  at java.security.AccessController.doPrivileged(Native Method)
> >  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >  at org.apache.flume.channel.file.Log$Builder.build(Log.java:212)
> >  at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
> >  at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
> >  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> >  at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> >  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> >  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> >  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> >  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> >  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >  at java.lang.Thread.run(Thread.java:662)
> > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable
> >  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >  at java.security.AccessController.doPrivileged(Native Method)
> >  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >  ... 24 more
> > 10 Apr 2013 11:50:29,317 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.stop:329)
 - Stopping FileChannel fc_Default { dataDirs: [c:\flume_data\log] }... 
> >  
> >  
> >  
> > Thank you,
> > Paul Chavez
> > 
> > 
> > 
> > 
> 
> 
> 


Mime
View raw message