Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A46CCF213 for ; Wed, 10 Apr 2013 20:42:18 +0000 (UTC) Received: (qmail 72111 invoked by uid 500); 10 Apr 2013 20:42:18 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 72053 invoked by uid 500); 10 Apr 2013 20:42:18 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 72044 invoked by uid 99); 10 Apr 2013 20:42:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Apr 2013 20:42:18 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hshreedharan@cloudera.com designates 209.85.210.44 as permitted sender) Received: from [209.85.210.44] (HELO mail-da0-f44.google.com) (209.85.210.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Apr 2013 20:42:13 +0000 Received: by mail-da0-f44.google.com with SMTP id z20so371460dae.3 for ; Wed, 10 Apr 2013 13:41:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:date:from:to:message-id:in-reply-to:references:subject :x-mailer:mime-version:content-type:x-gm-message-state; bh=EYBjkDwmc0783/ZNtUi9TEKlCdq17H3nzUeI2xUBcFc=; b=ABwomZctX7LuMGYcyKrZNQ4yhJUiKHyeY6DZoqcWW5rZWxdPHPaEQvWYXDKNco+HYM XPFTZpweZ+BWjfSBPL1+wjBy8npZBJcMdH8esM6ZVBgv1uIAdTjAZRWDR8do6DWfy6n9 kWotjd357JxOewOhQuQ6SnENiaHuMN9zNaiEoJgE1exp5lkLJi5BXAG9X09kn7B1aYyA b0db/dUyaY0Jl2EUkGtpr4lRVrIZbpWS22whqI57T7HMwkBHG397pdHFgfru8GOURHOL urEOUVJV5hGhwWLJZbB2Hpt/m6cKq7xJ9CKwd659sK36Jkpklw4ZxJpz4qhZzpuX3rbW zC2A== X-Received: by 10.68.244.5 with SMTP id xc5mr5154873pbc.66.1365626512976; Wed, 10 Apr 2013 13:41:52 -0700 (PDT) Received: from [10.0.1.8] (c-67-161-24-53.hsd1.ca.comcast.net. [67.161.24.53]) by mx.google.com with ESMTPS id dr6sm1777439pac.11.2013.04.10.13.41.51 (version=TLSv1 cipher=RC4-SHA bits=128/128); Wed, 10 Apr 2013 13:41:52 -0700 (PDT) Date: Wed, 10 Apr 2013 13:41:48 -0700 From: Hari Shreedharan To: user@flume.apache.org Message-ID: In-Reply-To: References: <6q31wbdqcd7t16dle8npou1m.1365620803041@email.android.com> Subject: Re: FileChannel on Windows X-Mailer: sparrow 1.6.4 (build 1178) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="5165ce8c_6b057295_246" X-Gm-Message-State: ALoCoQnWuogrBAYMTujOws1aHU3oqGXmynWsOELqWfwRDhg3vRX0G8WlA4IrnsRm8SrZk3plaZLO X-Virus-Checked: Checked by ClamAV on apache.org --5165ce8c_6b057295_246 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Yes. We will need to remove the format upgrade code and the old format code from the file channel as well. I think it should be possible to do it, but I suspect it is just that no one had the time to look at this as yet. -- Hari Shreedharan On Wednesday, April 10, 2013 at 1:38 PM, Roshan Naik wrote: > Hari, > i recall some discussion around the desire to get rid of this dependency. i guess it means removing that piece of code that supports the format upgrade ? are there any plans to do that ? > -roshan > > > > On Wed, Apr 10, 2013 at 1:25 PM, Paul Chavez wrote: > > I did not realize I could use pre-built binaries on Windows. Thank you, I used this to grab the hadoop-common.jar from our existing Hadoop cluster and there are no startup errors now. > > > > Thanks to you, Brock and Hari. > > > > -Paul > > > > From: Israel Ekpo [mailto:israel@aicer.org] > > Sent: Wednesday, April 10, 2013 12:07 PM > > To: user@flume.apache.org (mailto:user@flume.apache.org) > > Subject: Re: FileChannel on Windows > > > > Paul, > > > > If Flume 1.3.1 is what you are looking for, you don't have to build it from source. > > > > You can just download it directly from the site. Its already released. > > > > Paul Chavez wrote: > > > > Hello, > > > > I've been using flume on Windows to send logs to an avro source for awhile now, using the AvroClient class. I'm now trying to get a full agent running to hopefully leverage the spoolDir and httpSource functionality. Part of my requirements is a persistent channel so I'm attempting to get a config running the utilizes the fileChannel. > > > > I followed the instructions at http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html to build flume, same as I did for the AvroClient. However it seems I need some HDFS libraries. Can anyone point me to instructions for building the JAR(s) I need to get this working? > > > > Currently I'm running into the following exception: > > 10 Apr 2013 11:50:29,317 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.start:285) - Starting FileChannel fc_Default { dataDirs: [c:\flume_data\log] }... > > 10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.start:314) - Failed to start the file channel [channel=fc_Default] > > java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable > > at java.lang.ClassLoader.defineClass1(Native Method) > > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > > at org.apache.flume.channel.file.Log$Builder.build(Log.java:212) > > at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302) > > at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236) > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > > at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable > > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > > ... 24 more > > 10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:238) - Unable to start FileChannel fc_Default { dataDirs: [c:\flume_data\log] } - Exception follows. > > java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable > > at java.lang.ClassLoader.defineClass1(Native Method) > > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > > at org.apache.flume.channel.file.Log$Builder.build(Log.java:212) > > at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302) > > at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236) > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > > at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable > > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > > ... 24 more > > 10 Apr 2013 11:50:29,317 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.stop:329) - Stopping FileChannel fc_Default { dataDirs: [c:\flume_data\log] }... > > > > > > > > Thank you, > > Paul Chavez > > > > > > > > > > > --5165ce8c_6b057295_246 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
Yes. We will need to remove the format upgrade code a= nd the old format code from the file channel as well. I think it should b= e possible to do it, but I suspect it is just that no one had the time to= look at this as yet.

-- 
Hari Shreedha= ran

=20

On Wednesday, April 10= , 2013 at 1:38 PM, Roshan Naik wrote:

Hari,
 = i recall some discussion around the desire to get rid of this dependency= . i guess it means removing that piece of code that supports the format u= pgrade =3F are there any plans to do that =3F
-roshan


On Wed, Apr 10, 2013 at 1:25 PM, Pau= l Chavez <pchavez=40verticalsearchworks.= com> wrote:
I did not realize I could use pre-built binaries on W= indows. =20 Thank you, I used this to grab the hadoop-common.jar from our existing Ha= doop =20 cluster and there are no startup errors now.
 
Thanks to you, Brock and Hari.
 
-Paul

=46rom: Israel Ekpo =5Bmailto:israel=40aicer.or= g=5D =20
Sent: Wednesday, April 10, 2013 12:07 PM
To: =20 us= er=40flume.apache.org
Subject: Re: =46ileChannel on =20 Windows

Paul,

If =46lume 1.3.1 is what you are looking for, you don't =20 have to build it from source.

You can just download it directly fr= om the =20 site. Its already released.

Paul Chavez =20 <pchavez=40verticalsearchworks.com> wrote:

Hello,
 
I've been using =20 flume on Windows to send logs to an avro source for awhile now, using the= =20 AvroClient class. I'm now trying to get a full agent running to hopefully= =20 leverage the spoolDir and httpSource functionality. Part of my requiremen= ts is a =20 persistent channel so I'm attempting to get a config running the utilizes= the =20 fileChannel.
 
I followed the =20 instructions at http://mapredit.blogspot.com/2012/07/run-flume= -13x-on-windows.html to =20 build flume, same as I did for the AvroClient. However it seems I need so= me HD=46S =20 libraries. Can anyone point me to instructions for building the JAR(s) I = need to =20 get this working=3F
 
Currently I'm =20 running into the following exception:
10 Apr 2013 =20 11:50:29,317 IN=46O  =5BlifecycleSupervisor-1-0=5D =20 (org.apache.flume.channel.file.=46ileChannel.start:285)  - Starting = =20 =46ileChannel fc=5FDefault =7B dataDirs: =5Bc:=5Cflume=5Fdata=5Clog=5D =7D= ...
10 Apr 2013 =20 11:50:29,317 ERROR =5BlifecycleSupervisor-1-0=5D =20 (org.apache.flume.channel.file.=46ileChannel.start:314)  - =46ailed = to start =20 the file channel =5Bchannel=3Dfc=5FDefault=5D
java.lang.NoClassDef=46o= undError: =20 org/apache/hadoop/io/Writable
 at =20 java.lang.ClassLoader.defineClass1(Native Method)
 at =20 java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
 at =20 java.lang.ClassLoader.defineClass(ClassLoader.java:615)
 at =20 java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at =20 java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
 at = =20 java.net.URLClassLoader.access=24000(URLClassLoader.java:58)
 at = =20 java.net.URLClassLoader=241.run(URLClassLoader.java:197)
 at =20 java.security.AccessController.doPrivileged(Native Method)
 at =20 java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at =20 java.lang.ClassLoader.loadClass(ClassLoader.java:306)
 at =20 sun.misc.Launcher=24AppClassLoader.loadClass(Launcher.java:301)
 = at =20 java.lang.ClassLoader.loadClass(ClassLoader.java:247)
 at =20 org.apache.flume.channel.file.Log=24Builder.build(Log.java:212)
 = at =20 org.apache.flume.channel.file.=46ileChannel.start(=46ileChannel.java:302)=
 at =20 org.apache.flume.lifecycle.LifecycleSupervisor=24MonitorRunnable.run(Life= cycleSupervisor.java:236)
 at =20 java.util.concurrent.Executors=24RunnableAdapter.call(Executors.java:439)=
 at =20 java.util.concurrent.=46utureTask=24Sync.innerRunAndReset(=46utureTask.ja= va:317)
 at =20 java.util.concurrent.=46utureTask.runAndReset(=46utureTask.java:150)
&= nbsp;at =20 java.util.concurrent.ScheduledThreadPoolExecutor=24Scheduled=46utureTask.= access=24101(ScheduledThreadPoolExecutor.java:98)
 at =20 java.util.concurrent.ScheduledThreadPoolExecutor=24Scheduled=46utureTask.= runPeriodic(ScheduledThreadPoolExecutor.java:180)
 at =20 java.util.concurrent.ScheduledThreadPoolExecutor=24Scheduled=46utureTask.= run(ScheduledThreadPoolExecutor.java:204)
 at =20 java.util.concurrent.ThreadPoolExecutor=24Worker.runTask(ThreadPoolExecut= or.java:886)
 at =20 java.util.concurrent.ThreadPoolExecutor=24Worker.run(ThreadPoolExecutor.j= ava:908)
 at =20 java.lang.Thread.run(Thread.java:662)
Caused by: =20 java.lang.ClassNot=46oundException: org.apache.hadoop.io.Writable
&nbs= p;at =20 java.net.URLClassLoader=241.run(URLClassLoader.java:202)
 at =20 java.security.AccessController.doPrivileged(Native Method)
 at =20 java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at =20 java.lang.ClassLoader.loadClass(ClassLoader.java:306)
 at =20 sun.misc.Launcher=24AppClassLoader.loadClass(Launcher.java:301)
 = at =20 java.lang.ClassLoader.loadClass(ClassLoader.java:247)
 ... 24 mor= e
10 =20 Apr 2013 11:50:29,317 ERROR =5BlifecycleSupervisor-1-0=5D =20 (org.apache.flume.lifecycle.LifecycleSupervisor=24MonitorRunnable.run:238= )  - =20 Unable to start =46ileChannel fc=5FDefault =7B dataDirs: =5Bc:=5Cflume=5F= data=5Clog=5D =7D - =20 Exception follows.
java.lang.NoClassDef=46oundError: =20 org/apache/hadoop/io/Writable
 at =20 java.lang.ClassLoader.defineClass1(Native Method)
 at =20 java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
 at =20 java.lang.ClassLoader.defineClass(ClassLoader.java:615)
 at =20 java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at =20 java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
 at = =20 java.net.URLClassLoader.access=24000(URLClassLoader.java:58)
 at = =20 java.net.URLClassLoader=241.run(URLClassLoader.java:197)
 at =20 java.security.AccessController.doPrivileged(Native Method)
 at =20 java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at =20 java.lang.ClassLoader.loadClass(ClassLoader.java:306)
 at =20 sun.misc.Launcher=24AppClassLoader.loadClass(Launcher.java:301)
 = at =20 java.lang.ClassLoader.loadClass(ClassLoader.java:247)
 at =20 org.apache.flume.channel.file.Log=24Builder.build(Log.java:212)
 = at =20 org.apache.flume.channel.file.=46ileChannel.start(=46ileChannel.java:302)=
 at =20 org.apache.flume.lifecycle.LifecycleSupervisor=24MonitorRunnable.run(Life= cycleSupervisor.java:236)
 at =20 java.util.concurrent.Executors=24RunnableAdapter.call(Executors.java:439)=
 at =20 java.util.concurrent.=46utureTask=24Sync.innerRunAndReset(=46utureTask.ja= va:317)
 at =20 java.util.concurrent.=46utureTask.runAndReset(=46utureTask.java:150)
&= nbsp;at =20 java.util.concurrent.ScheduledThreadPoolExecutor=24Scheduled=46utureTask.= access=24101(ScheduledThreadPoolExecutor.java:98)
 at =20 java.util.concurrent.ScheduledThreadPoolExecutor=24Scheduled=46utureTask.= runPeriodic(ScheduledThreadPoolExecutor.java:180)
 at =20 java.util.concurrent.ScheduledThreadPoolExecutor=24Scheduled=46utureTask.= run(ScheduledThreadPoolExecutor.java:204)
 at =20 java.util.concurrent.ThreadPoolExecutor=24Worker.runTask(ThreadPoolExecut= or.java:886)
 at =20 java.util.concurrent.ThreadPoolExecutor=24Worker.run(ThreadPoolExecutor.j= ava:908)
 at =20 java.lang.Thread.run(Thread.java:662)
Caused by: =20 java.lang.ClassNot=46oundException: org.apache.hadoop.io.Writable
&nbs= p;at =20 java.net.URLClassLoader=241.run(URLClassLoader.java:202)
 at =20 java.security.AccessController.doPrivileged(Native Method)
 at =20 java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at =20 java.lang.ClassLoader.loadClass(ClassLoader.java:306)
 at =20 sun.misc.Launcher=24AppClassLoader.loadClass(Launcher.java:301)
 = at =20 java.lang.ClassLoader.loadClass(ClassLoader.java:247)
 ... 24 mor= e
10 =20 Apr 2013 11:50:29,317 IN=46O  =5BlifecycleSupervisor-1-0=5D =20 (org.apache.flume.channel.file.=46ileChannel.stop:329)  - Stopping =20 =46ileChannel fc=5FDefault =7B dataDirs: =5Bc:=5Cflume=5Fdata=5Clog=5D =7D= ...
 
 
 
Thank =20 you,
Paul =20 Chavez

 =20 =20 =20 =20
=20

--5165ce8c_6b057295_246--