flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: FileChannel on Windows
Date Wed, 10 Apr 2013 21:32:08 GMT
http://issues.apache.org/jira/browse/FLUME-1285


On Wed, Apr 10, 2013 at 4:20 PM, Israel Ekpo <israel@aicer.org> wrote:

> Brock, do you have the JIRA issue where the patches were uploaded?
>
> If I get some spare time this weekend I can take a shot at it.
>
>
>
> On 10 April 2013 16:44, Brock Noland <brock@cloudera.com> wrote:
>
>> I think we just need to copy 4-5 classes over from hadoop. Someone
>> uploaded a patch I just haven't had time to work with it.
>>
>>
>> On Wed, Apr 10, 2013 at 3:41 PM, Hari Shreedharan <
>> hshreedharan@cloudera.com> wrote:
>>
>>>  Yes. We will need to remove the format upgrade code and the old format
>>> code from the file channel as well. I think it should be possible to do it,
>>> but I suspect it is just that no one had the time to look at this as yet.
>>>
>>> --
>>> Hari Shreedharan
>>>
>>> On Wednesday, April 10, 2013 at 1:38 PM, Roshan Naik wrote:
>>>
>>> Hari,
>>>   i recall some discussion around the desire to get rid of this
>>> dependency. i guess it means removing that piece of code that supports the
>>> format upgrade ? are there any plans to do that ?
>>>  -roshan
>>>
>>>
>>> On Wed, Apr 10, 2013 at 1:25 PM, Paul Chavez <
>>> pchavez@verticalsearchworks.com> wrote:
>>>
>>> **
>>> I did not realize I could use pre-built binaries on Windows. Thank you,
>>> I used this to grab the hadoop-common.jar from our existing Hadoop cluster
>>> and there are no startup errors now.
>>>
>>> Thanks to you, Brock and Hari.
>>>
>>> -Paul
>>>
>>>  ------------------------------
>>> *From:* Israel Ekpo [mailto:israel@aicer.org]
>>> *Sent:* Wednesday, April 10, 2013 12:07 PM
>>> *To:* user@flume.apache.org
>>> *Subject:* Re: FileChannel on Windows
>>>
>>> Paul,
>>>
>>> If Flume 1.3.1 is what you are looking for, you don't have to build it
>>> from source.
>>>
>>> You can just download it directly from the site. Its already released.
>>>
>>> Paul Chavez <pchavez@verticalsearchworks.com> wrote:
>>>
>>> Hello,
>>>
>>> I've been using flume on Windows to send logs to an avro source for
>>> awhile now, using the AvroClient class. I'm now trying to get a full agent
>>> running to hopefully leverage the spoolDir and httpSource functionality.
>>> Part of my requirements is a persistent channel so I'm attempting to get a
>>> config running the utilizes the fileChannel.
>>>
>>> I followed the instructions at
>>> http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html to
>>> build flume, same as I did for the AvroClient. However it seems I need some
>>> HDFS libraries. Can anyone point me to instructions for building the JAR(s)
>>> I need to get this working?
>>>
>>> Currently I'm running into the following exception:
>>> 10 Apr 2013 11:50:29,317 INFO  [lifecycleSupervisor-1-0]
>>> (org.apache.flume.channel.file.FileChannel.start:285)  - Starting
>>> FileChannel fc_Default { dataDirs: [c:\flume_data\log] }...
>>> 10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0]
>>> (org.apache.flume.channel.file.FileChannel.start:314)  - Failed to start
>>> the file channel [channel=fc_Default]
>>> java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
>>>  at java.lang.ClassLoader.defineClass1(Native Method)
>>>  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>>>  at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>>>  at
>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>>>  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>>>  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>>>  at java.security.AccessController.doPrivileged(Native Method)
>>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>  at org.apache.flume.channel.file.Log$Builder.build(Log.java:212)
>>>  at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
>>>  at
>>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
>>>  at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>>>  at
>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.hadoop.io.Writable
>>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>  at java.security.AccessController.doPrivileged(Native Method)
>>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>  ... 24 more
>>> 10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:238)  -
>>> Unable to start FileChannel fc_Default { dataDirs: [c:\flume_data\log] } -
>>> Exception follows.
>>> java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
>>>  at java.lang.ClassLoader.defineClass1(Native Method)
>>>  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>>>  at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>>>  at
>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>>>  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>>>  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>>>  at java.security.AccessController.doPrivileged(Native Method)
>>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>  at org.apache.flume.channel.file.Log$Builder.build(Log.java:212)
>>>  at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
>>>  at
>>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
>>>  at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>>>  at
>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.hadoop.io.Writable
>>>  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>  at java.security.AccessController.doPrivileged(Native Method)
>>>  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>  ... 24 more
>>> 10 Apr 2013 11:50:29,317 INFO  [lifecycleSupervisor-1-0]
>>> (org.apache.flume.channel.file.FileChannel.stop:329)  - Stopping
>>> FileChannel fc_Default { dataDirs: [c:\flume_data\log] }...
>>>
>>>
>>>
>>> Thank you,
>>> Paul Chavez
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>>
>
>


-- 
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org

Mime
View raw message