flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Israel Ekpo <isr...@aicer.org>
Subject Re: FileChannel on Windows
Date Wed, 10 Apr 2013 21:20:07 GMT
Brock, do you have the JIRA issue where the patches were uploaded?

If I get some spare time this weekend I can take a shot at it.



On 10 April 2013 16:44, Brock Noland <brock@cloudera.com> wrote:

> I think we just need to copy 4-5 classes over from hadoop. Someone
> uploaded a patch I just haven't had time to work with it.
>
>
> On Wed, Apr 10, 2013 at 3:41 PM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
>>  Yes. We will need to remove the format upgrade code and the old format
>> code from the file channel as well. I think it should be possible to do it,
>> but I suspect it is just that no one had the time to look at this as yet.
>>
>> --
>> Hari Shreedharan
>>
>> On Wednesday, April 10, 2013 at 1:38 PM, Roshan Naik wrote:
>>
>> Hari,
>>   i recall some discussion around the desire to get rid of this
>> dependency. i guess it means removing that piece of code that supports the
>> format upgrade ? are there any plans to do that ?
>>  -roshan
>>
>>
>> On Wed, Apr 10, 2013 at 1:25 PM, Paul Chavez <
>> pchavez@verticalsearchworks.com> wrote:
>>
>> **
>> I did not realize I could use pre-built binaries on Windows. Thank you, I
>> used this to grab the hadoop-common.jar from our existing Hadoop cluster
>> and there are no startup errors now.
>>
>> Thanks to you, Brock and Hari.
>>
>> -Paul
>>
>>  ------------------------------
>> *From:* Israel Ekpo [mailto:israel@aicer.org]
>> *Sent:* Wednesday, April 10, 2013 12:07 PM
>> *To:* user@flume.apache.org
>> *Subject:* Re: FileChannel on Windows
>>
>> Paul,
>>
>> If Flume 1.3.1 is what you are looking for, you don't have to build it
>> from source.
>>
>> You can just download it directly from the site. Its already released.
>>
>> Paul Chavez <pchavez@verticalsearchworks.com> wrote:
>>
>> Hello,
>>
>> I've been using flume on Windows to send logs to an avro source for
>> awhile now, using the AvroClient class. I'm now trying to get a full agent
>> running to hopefully leverage the spoolDir and httpSource functionality.
>> Part of my requirements is a persistent channel so I'm attempting to get a
>> config running the utilizes the fileChannel.
>>
>> I followed the instructions at
>> http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html to
>> build flume, same as I did for the AvroClient. However it seems I need some
>> HDFS libraries. Can anyone point me to instructions for building the JAR(s)
>> I need to get this working?
>>
>> Currently I'm running into the following exception:
>> 10 Apr 2013 11:50:29,317 INFO  [lifecycleSupervisor-1-0]
>> (org.apache.flume.channel.file.FileChannel.start:285)  - Starting
>> FileChannel fc_Default { dataDirs: [c:\flume_data\log] }...
>> 10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0]
>> (org.apache.flume.channel.file.FileChannel.start:314)  - Failed to start
>> the file channel [channel=fc_Default]
>> java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
>>  at java.lang.ClassLoader.defineClass1(Native Method)
>>  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>>  at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>>  at
>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>>  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>>  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>  at org.apache.flume.channel.file.Log$Builder.build(Log.java:212)
>>  at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
>>  at
>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
>>  at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>>  at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable
>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>  ... 24 more
>> 10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0]
>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:238)  -
>> Unable to start FileChannel fc_Default { dataDirs: [c:\flume_data\log] } -
>> Exception follows.
>> java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
>>  at java.lang.ClassLoader.defineClass1(Native Method)
>>  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>>  at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>>  at
>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>>  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>>  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>  at org.apache.flume.channel.file.Log$Builder.build(Log.java:212)
>>  at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
>>  at
>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
>>  at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>>  at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable
>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>  ... 24 more
>> 10 Apr 2013 11:50:29,317 INFO  [lifecycleSupervisor-1-0]
>> (org.apache.flume.channel.file.FileChannel.stop:329)  - Stopping
>> FileChannel fc_Default { dataDirs: [c:\flume_data\log] }...
>>
>>
>>
>> Thank you,
>> Paul Chavez
>>
>>
>>
>>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>

Mime
View raw message