Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (nike.apache.org: domain of hshreedharan@cloudera.com
 designates 209.85.161.179 as permitted sender)
Date: Thu, 2 Aug 2012 16:21:45 -0700
From: Hari Shreedharan <hshreedharan@cloudera.com>
To: user@flume.apache.org
Message-ID: <57C87BF73C504954AAF17A9E6560E14B@cloudera.com>
In-Reply-To: 
 <CAB=riaBSp0+BkjsqELqUcKmASdWdFc=L=iSrg4eZ11cp4Urr6g@mail.gmail.com>
References: 
 <CAB=riaBSp0+BkjsqELqUcKmASdWdFc=L=iSrg4eZ11cp4Urr6g@mail.gmail.com>
Subject: Re: unable to write to HDFS with flume 1.2.0
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="501b0b89_2a9c34e2_16f28"

--501b0b89_2a9c34e2_16f28
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

agent.sinks.hdfsSink.channels = memoryChannel   <-needs to be "agent.sinks.hdfsSink.channel = memoryChannel" 

Hari
-- 
Hari Shreedharan


On Thursday, August 2, 2012 at 4:19 PM, David Capwell wrote:

> I downloaded the latest release (1.2.0) of flume and tried to get it sending data to HDFS; for some reason it isn't adding hdfs as a sink.  Here is my properties file:
> 
> 
> agent.sources = seqGenSrc 
> agent.channels = memoryChannel
> agent.sinks = hdfsSink
> 
> # For each one of the sources, the type is defined
> agent.sources.seqGenSrc.type = seq
> agent.sources.seqGenSrc.channels = memoryChannel
> 
> # Each sink's type must be defined
> agent.sinks.loggerSink.type = logger
> agent.sinks.loggerSink.channel = memoryChannel
> 
> # Each channel's type is defined. 
> agent.channels.memoryChannel.type = memory
> agent.channels.memoryChannel.capacity = 100
> 
> agent.sinks.hdfsSink.type = hdfs
> 
> agent.sinks.hdfsSink.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
> agent.sinks.hdfsSink.hdfs.filePrefix = events-#agent.sinks.hdfsSink.hdfs.round = true
> agent.sinks.hdfsSink.hdfs.roundValue = 10#agent.sinks.hdfsSink.hdfs.roundUnit = second
> 
> 
> 
> Here is what I see in the logs with debugging enabled:
> 
> 2012-08-02 16:11:31,961 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)] 649 Creating sink: hdfsSink using HDFS 
> 2012-08-02 16:11:31,963 (conf-file-poller-0) [WARN - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:668)] 668 Configuration empty for: hdfsSink.Removed.
> 
> 
> Going over the code I wasn't sure where the ConfigurationException was thrown so I did a test and commented out agent.sinks.hdfsSink.channels (to get linke 655 to fail [sinkConf.configure(sinkContext);] ) but still get the same error. 
> 
> 
> am I missing something?  
> 
> Thanks for your time reading this email. 


--501b0b89_2a9c34e2_16f28
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline


                <div>
                    agent.sinks.hdfsSink.channels =3D memoryChannel &nbsp=
; &lt;-needs to be =22agent.sinks.hdfsSink.channel =3D memoryChannel=22
                </div>
                <div><div><br></div><div>Hari</div><div>--&nbsp;</div><di=
v>Hari Shreedharan</div><div><br></div></div>
                =20
                <p style=3D=22color: =23A0A0A8;=22>On Thursday, August 2,=
 2012 at 4:19 PM, David Capwell wrote:</p>
                <blockquote type=3D=22cite=22 style=3D=22border-left-styl=
e:solid;border-width:1px;margin-left:0px;padding-left:10px;=22>
                    <span>I downloaded the latest release (1.2.0) of flum=
e and tried to get it sending data to HD=46S; for some reason it isn't ad=
ding hdfs as a sink. &nbsp;Here is my properties file:<div><br></div><div=
><br></div><div><div>agent.sources =3D seqGenSrc</div>
<div>agent.channels =3D memoryChannel</div><div>agent.sinks =3D hdfsSink<=
/div><div><br></div><div>=23 =46or each one of the sources, the type is d=
efined</div><div>agent.sources.seqGenSrc.type =3D seq</div><div>agent.sou=
rces.seqGenSrc.channels =3D memoryChannel</div>
<div><br></div><div>=23 Each sink's type must be defined</div><div>agent.=
sinks.loggerSink.type =3D logger</div><div>agent.sinks.loggerSink.channel=
 =3D memoryChannel</div><div><br></div><div>=23 Each channel's type is de=
fined.</div>
<div>agent.channels.memoryChannel.type =3D memory</div><div>agent.channel=
s.memoryChannel.capacity =3D 100</div><div><br></div><div>agent.sinks.hdf=
sSink.type =3D hdfs</div><div><br></div>
<div>agent.sinks.hdfsSink.hdfs.path =3D /flume/events/%y-%m-%d/%H%M/%S</d=
iv><div>agent.sinks.hdfsSink.hdfs.filePrefix =3D events-=23agent.sinks.hd=
fsSink.hdfs.round =3D true</div><div>agent.sinks.hdfsSink.hdfs.roundValue=
 =3D 10=23agent.sinks.hdfsSink.hdfs.roundUnit =3D second</div>
</div><div><br></div><div><br></div><div>Here is what I see in the logs w=
ith debugging enabled:</div><div><br></div><div><div>2012-08-02 16:11:31,=
961 (conf-file-poller-0) =5BDEBUG - org.apache.flume.conf.=46lumeConfigur=
ation=24AgentConfiguration.validateSinks(=46lumeConfiguration.java:649)=5D=
 649 Creating sink: hdfsSink using HD=46S</div>
<div>2012-08-02 16:11:31,963 (conf-file-poller-0) =5BWARN - org.apache.fl=
ume.conf.=46lumeConfiguration=24AgentConfiguration.validateSinks(=46lumeC=
onfiguration.java:668)=5D 668 Configuration empty for: hdfsSink.Removed.<=
/div></div><div>
<br></div><div>Going over the code I wasn't sure where the&nbsp;Configura=
tionException was thrown so I did a test and commented out agent.sinks.hd=
fsSink.channels (to get linke 655 to fail =5BsinkConf.configure(sinkConte=
xt);=5D ) but still get the same error.</div>
<div><br></div><div><br></div><div>am I missing something=3F &nbsp;</div>=
<div><br></div><div>Thanks for your time reading this email.</div>
</span>
                =20
                =20
                =20
                =20
                </blockquote>
                =20
                <div>
                    <br>
                </div>
            
--501b0b89_2a9c34e2_16f28--