Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1645218F5A for ; Mon, 16 Nov 2015 08:48:06 +0000 (UTC) Received: (qmail 10239 invoked by uid 500); 16 Nov 2015 08:48:05 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 10182 invoked by uid 500); 16 Nov 2015 08:48:05 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 10172 invoked by uid 99); 16 Nov 2015 08:48:05 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2015 08:48:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 2F40518099A for ; Mon, 16 Nov 2015 08:48:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.88 X-Spam-Level: ** X-Spam-Status: No, score=2.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id FvjoPhV5hi5o for ; Mon, 16 Nov 2015 08:48:00 +0000 (UTC) Received: from mail-lb0-f174.google.com (mail-lb0-f174.google.com [209.85.217.174]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 8CA9543CD6 for ; Mon, 16 Nov 2015 08:47:59 +0000 (UTC) Received: by lbbkw15 with SMTP id kw15so84120919lbb.0 for ; Mon, 16 Nov 2015 00:47:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=P1MXhWHjKnqO9zRqD2eX5hUwPMl7JA0Fx05KHqsO8as=; b=A5XFpjFDCVnT3JrgkA4ltmYXW0/eIV9FlNsa4IiyTm/QRc00ESWfs53WFtp6g01Jjv da4PBJ5x5ptFvElfinQ4p2XOpsrra3k3BmKlwfO0lrLJqoNwxkOv+9ncXmi0VQccFK7B IgZjNpmFFwxMEwOgLHFMQnvxRm+g+BUSxZ93YxrmGKg8ApDKOycakIavxS0MpEJKz3xx AuliRNEKa5jwCfnRq8P2LNwRuBBlcWtHKVN9pDHcAr3NKg+CKzGgdFO1pRu1yZYOCzxm 2F4d9ew0DtXFbpueArGvsHuXFBlsU4pXQ2WxbjzmWsxQQJb0jZut0r4EIdXZGOJOPHPa ypUQ== MIME-Version: 1.0 X-Received: by 10.112.200.40 with SMTP id jp8mr16482288lbc.104.1447663678341; Mon, 16 Nov 2015 00:47:58 -0800 (PST) Received: by 10.25.24.24 with HTTP; Mon, 16 Nov 2015 00:47:58 -0800 (PST) In-Reply-To: References: Date: Mon, 16 Nov 2015 15:47:58 +0700 Message-ID: Subject: Re: Flume-agent conf on ambari From: zaenal rifai To: user@flume.apache.org Content-Type: multipart/alternative; boundary=001a11c36b2a4d528d0524a47735 --001a11c36b2a4d528d0524a47735 Content-Type: text/plain; charset=UTF-8 thanks gonzalo, it's running but still cant write on hdfs. the error is http://tempel.blankon.in/1527360 when Starting connection handler it always Checking file:/home/hdfs/adaw.conf for changes and i check on hdfs, there is no file On 16 November 2015 at 15:34, Gonzalo Herreros wrote: > I see your problem now. You are using "memory-channel" and "memoryChannel" > to refer to the same thing. > Change the 3rd line to: > > hdfs-agent.channels= memoryChannel > > On 16 November 2015 at 08:31, zaenal rifai wrote: > >> 1.Yes i get error, the error is " Agent configuration for hdfs-agent does >> not contain any valid channels. Marking it as invalid " >> here is for more about the error >> >> http://tempel.blankon.in/1527288 >> >> 2.No, i don't have enable the kerberos >> >> 3.i'm not sure about the problem, i tried another method sink and it >> still failed. >> >> can you give me some simple example for agent configuration to collect >> log and write on hdfs ? >> >> >> >> >> >> >> On 16 November 2015 at 14:47, Gonzalo Herreros >> wrote: >> >>> I looks fine to me. Do you get any errors? >>> Do you have kerberos enabled? maybe is a security issue >>> Are you sure the problem is hdfs and not some netcat error? Try with >>> another sink to confirm that. >>> >>> Regards, >>> Gonzalo >>> >>> On 16 November 2015 at 07:35, zaenal rifai >>> wrote: >>> >>>> Hello guys, i'm newbie on flume, i want to ask something about >>>> configuration on flume-agent which can write event on HDFS. >>>> >>>> for example which i can find on internet is like this. >>>> >>>> hdfs-agent.sources= netcat-collect >>>> hdfs-agent.sinks = hdfs-write >>>> hdfs-agent.channels= memory-channel >>>> hdfs-agent.sources.netcat-collect.type = netcat >>>> hdfs-agent.sources.netcat-collect.bind = 127.0.0.1 >>>> hdfs-agent.sources.netcat-collect.port = 11111 >>>> hdfs-agent.sinks.hdfs-write.type = hdfs >>>> hdfs-agent.sinks.hdfs-write.hdfs.path =/user/user/togatta/ >>>> hdfs-agent.sinks.hdfs-write.rollInterval = 10 >>>> hdfs-agent.sinks.hdfs-write.hdfs.writeFormat=Text >>>> hdfs-agent.sinks.hdfs-write.hdfs.fileType=DataStream >>>> hdfs-agent.channels.memoryChannel.type = memory >>>> hdfs-agent.channels.memoryChannel.capacity=10000 >>>> hdfs-agent.sources.netcat-collect.channels=memoryChannel >>>> hdfs-agent.sinks.hdfs-write.channel=memoryChannel >>>> >>>> this configuration will collect about log netcat event, but when i run >>>> it, it cant write on path hdfs which i had to config. So any suggest from >>>> all of you guys? >>>> >>> >>> >> > --001a11c36b2a4d528d0524a47735 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
thanks gonzalo, it's running but still cant write on h= dfs.

the error is=C2=A0
<= br>
when=C2=A0Starting connection handler=

it always=C2= =A0Checking file:/h= ome/hdfs/adaw.conf for changes

and i check on hdfs, there is no file=C2=A0

On 16 November 2015= at 15:34, Gonzalo Herreros <gherreros@gmail.com> wrote:
I see your problem now. Y= ou are using "memory-channel" an= d "memoryChannel" to refe= r to the same thing.
Change the= 3rd line to:

=
hdfs-agent.channels=3D=C2=A0memoryChannel

On 16 November= 2015 at 08:31, zaenal rifai <togatta.fudo@gmail.com> w= rote:
1.Yes i = get error, the error is " Agent configuration for hdfs-agent does not = contain any valid channels. Marking it as invalid "
=C2=A0 = =C2=A0 here is for more about the error


2.No, i don't hav= e enable the kerberos

3.i'm not sure about the= problem, i tried another method sink and it still failed.

can you give me some simple example for agent configuration to col= lect log and write on hdfs ?


<= div>



On 16 November 2015 at 14:47, Gonzalo Herr= eros <gherreros@gmail.com> wrote:
I looks fine to me. Do you get any errors?
Do you have ker= beros enabled? maybe is a security issue
Are you sure the problem= is hdfs and not some netcat error? Try with another sink to confirm that.<= /div>

Regards,
Gonzalo

On 16 November 2015= at 07:35, zaenal rifai <togatta.fudo@gmail.com> wrote:=
Hell= o guys, i'm newbie on flume, i want to ask something about configuratio= n on flume-agent which can write event on HDFS.

for example which i ca= n find on internet is like this.

<= /div>
hdfs-agent.sources=3D netcat-coll= ect
hdfs-agent.sinks =3D hdfs-write
hdfs-agent.channels= =3D memory-channel
hdfs-agent.sources.netcat-collect.type =3D net= cat
hdfs-agent.sources.netcat-collect.bind =3D 127.0.0.1
hdfs-agent.sources.netcat-collect.port =3D 11111
hdfs-agent.sin= ks.hdfs-write.type =3D hdfs
hdfs-agent.sinks.hdfs-write.hdfs.path= =3D/user/user/togatta/
hdfs-agent.sinks.hdfs-write.rollInterval = =3D 10
hdfs-agent.sinks.hdfs-write.hdfs.writeFormat=3DText
<= div>hdfs-agent.sinks.hdfs-write.hdfs.fileType=3DDataStream
hdfs-a= gent.channels.memoryChannel.type =3D memory
hdfs-agent.channels.m= emoryChannel.capacity=3D10000
hdfs-agent.sources.netcat-collect.c= hannels=3DmemoryChannel
hdfs-agent.sinks.hdfs-write.channel=3Dmem= oryChannel

this configuration will collect about l= og netcat event, but when i run it, it cant write on path hdfs which i had = to config. So any suggest from all of you guys?




--001a11c36b2a4d528d0524a47735--