Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4976B18F07 for ; Mon, 16 Nov 2015 08:34:27 +0000 (UTC) Received: (qmail 92378 invoked by uid 500); 16 Nov 2015 08:34:22 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 92322 invoked by uid 500); 16 Nov 2015 08:34:22 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 92306 invoked by uid 99); 16 Nov 2015 08:34:21 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2015 08:34:21 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 8ACE41A0C0A for ; Mon, 16 Nov 2015 08:34:21 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.88 X-Spam-Level: ** X-Spam-Status: No, score=2.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 1xBDvgZ3TaMc for ; Mon, 16 Nov 2015 08:34:14 +0000 (UTC) Received: from mail-wm0-f46.google.com (mail-wm0-f46.google.com [74.125.82.46]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id A86222114D for ; Mon, 16 Nov 2015 08:34:13 +0000 (UTC) Received: by wmec201 with SMTP id c201so163783681wme.0 for ; Mon, 16 Nov 2015 00:34:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=3KAwGd3CS8uoyn+YuZX1O9agid0DgyqsIKB16xr5L1I=; b=k5Z7LFsjCjOwXJaEaXzcOHLrL3HpBGTmQoBDDkHnBCvBhYUSDICbR4cHCI6rqEJCbE t1cctpZinO2NG08CyCF9SWVhUTnGGCxP/1iveOkfPl5MERAERe0XQnDC1Ccai0jx7FUe oZLNGxtUYxxa5+nMcEVSftVKwHgrL4h+LcWYO9r/Gc1C4SqqkfSRbCw0mmaodz7uHbwh 9j9mbr/FjO7nnvPYtf3+s8OJHpKUo7qLmp/5cJmrTR7FV7KTvfBrAK+k9eNCpW1Jm12V R79DQz5F3rJ9g0k6rQ0We9ZyVsIkGJfsqnIXiVcSEsOaE4szqxFRwz2yf7D6WUBmaTa9 wbVw== MIME-Version: 1.0 X-Received: by 10.194.90.169 with SMTP id bx9mr34691046wjb.1.1447662846354; Mon, 16 Nov 2015 00:34:06 -0800 (PST) Received: by 10.194.25.234 with HTTP; Mon, 16 Nov 2015 00:34:06 -0800 (PST) In-Reply-To: References: Date: Mon, 16 Nov 2015 08:34:06 +0000 Message-ID: Subject: Re: Flume-agent conf on ambari From: Gonzalo Herreros To: user Content-Type: multipart/alternative; boundary=047d7bfd0df8b62a6c0524a445d0 --047d7bfd0df8b62a6c0524a445d0 Content-Type: text/plain; charset=UTF-8 I see your problem now. You are using "memory-channel" and "memoryChannel" to refer to the same thing. Change the 3rd line to: hdfs-agent.channels= memoryChannel On 16 November 2015 at 08:31, zaenal rifai wrote: > 1.Yes i get error, the error is " Agent configuration for hdfs-agent does > not contain any valid channels. Marking it as invalid " > here is for more about the error > > http://tempel.blankon.in/1527288 > > 2.No, i don't have enable the kerberos > > 3.i'm not sure about the problem, i tried another method sink and it still > failed. > > can you give me some simple example for agent configuration to collect log > and write on hdfs ? > > > > > > > On 16 November 2015 at 14:47, Gonzalo Herreros > wrote: > >> I looks fine to me. Do you get any errors? >> Do you have kerberos enabled? maybe is a security issue >> Are you sure the problem is hdfs and not some netcat error? Try with >> another sink to confirm that. >> >> Regards, >> Gonzalo >> >> On 16 November 2015 at 07:35, zaenal rifai >> wrote: >> >>> Hello guys, i'm newbie on flume, i want to ask something about >>> configuration on flume-agent which can write event on HDFS. >>> >>> for example which i can find on internet is like this. >>> >>> hdfs-agent.sources= netcat-collect >>> hdfs-agent.sinks = hdfs-write >>> hdfs-agent.channels= memory-channel >>> hdfs-agent.sources.netcat-collect.type = netcat >>> hdfs-agent.sources.netcat-collect.bind = 127.0.0.1 >>> hdfs-agent.sources.netcat-collect.port = 11111 >>> hdfs-agent.sinks.hdfs-write.type = hdfs >>> hdfs-agent.sinks.hdfs-write.hdfs.path =/user/user/togatta/ >>> hdfs-agent.sinks.hdfs-write.rollInterval = 10 >>> hdfs-agent.sinks.hdfs-write.hdfs.writeFormat=Text >>> hdfs-agent.sinks.hdfs-write.hdfs.fileType=DataStream >>> hdfs-agent.channels.memoryChannel.type = memory >>> hdfs-agent.channels.memoryChannel.capacity=10000 >>> hdfs-agent.sources.netcat-collect.channels=memoryChannel >>> hdfs-agent.sinks.hdfs-write.channel=memoryChannel >>> >>> this configuration will collect about log netcat event, but when i run >>> it, it cant write on path hdfs which i had to config. So any suggest from >>> all of you guys? >>> >> >> > --047d7bfd0df8b62a6c0524a445d0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I see your problem now. You are using "memory-channel" and "memoryChannel" to refer to the same thing.
<= span style=3D"font-size:12.8px">Change the 3rd line to:

hdfs-agent.channels=3D=C2=A0memoryChannel
<= /div>

On 16 Novemb= er 2015 at 08:31, zaenal rifai <togatta.fudo@gmail.com>= wrote:
1.Yes = i get error, the error is " Agent configuration for hdfs-agent does no= t contain any valid channels. Marking it as invalid "
=C2=A0= =C2=A0 here is for more about the error


2.No, i don't = have enable the kerberos

3.i'm not sure about = the problem, i tried another method sink and it still failed.
can you give me some simple example for agent configuration to = collect log and write on hdfs ?

=





On 16 November 2015 at 14:= 47, Gonzalo Herreros <gherreros@gmail.com> wrote:
I looks fine to me. Do you get any errors?Do you have kerberos enabled? maybe is a security issue
Are you = sure the problem is hdfs and not some netcat error? Try with another sink t= o confirm that.

Regards,
Gonzalo

On = 16 November 2015 at 07:35, zaenal rifai <togatta.fudo@gmail.com&g= t; wrote:
Hello guys, i'm newbie on flume, i want to ask something ab= out configuration on flume-agent which can write event on HDFS.

for ex= ample which i can find on internet is like this.

hdfs-agent.source= s=3D netcat-collect
hdfs-agent.sinks =3D hdfs-write
hdf= s-agent.channels=3D memory-channel
hdfs-agent.sources.netcat-coll= ect.type =3D netcat
hdfs-agent.sources.netcat-collect.bind =3D 12= 7.0.0.1
hdfs-agent.sources.netcat-collect.port =3D 11111
hdfs-agent.sinks.hdfs-write.type =3D hdfs
hdfs-agent.sinks.hdfs= -write.hdfs.path =3D/user/user/togatta/
hdfs-agent.sinks.hdfs-wri= te.rollInterval =3D 10
hdfs-agent.sinks.hdfs-write.hdfs.writeForm= at=3DText
hdfs-agent.sinks.hdfs-write.hdfs.fileType=3DDataStream<= /div>
hdfs-agent.channels.memoryChannel.type =3D memory
hdfs-= agent.channels.memoryChannel.capacity=3D10000
hdfs-agent.sources.= netcat-collect.channels=3DmemoryChannel
hdfs-agent.sinks.hdfs-wri= te.channel=3DmemoryChannel

this configuration will= collect about log netcat event, but when i run it, it cant write on path h= dfs which i had to config. So any suggest from all of you guys?
=



--047d7bfd0df8b62a6c0524a445d0--