Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E0D1A10E66 for ; Thu, 9 Jan 2014 01:39:03 +0000 (UTC) Received: (qmail 34749 invoked by uid 500); 9 Jan 2014 01:39:03 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 34689 invoked by uid 500); 9 Jan 2014 01:39:03 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 34681 invoked by uid 99); 9 Jan 2014 01:39:03 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jan 2014 01:39:03 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of chen.apache.solr@gmail.com designates 209.85.214.41 as permitted sender) Received: from [209.85.214.41] (HELO mail-bk0-f41.google.com) (209.85.214.41) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jan 2014 01:38:59 +0000 Received: by mail-bk0-f41.google.com with SMTP id v15so925510bkz.0 for ; Wed, 08 Jan 2014 17:38:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=85VhE2m4+sSXYkOxoJomExqAwf2G/LI0D6qg44q9XFg=; b=ZB+6JmL8fCekSxkjSSujQsmlvy5vdlErfyvZleI4tlPRohdofzIl0G1JahspaCIAoH NfQ1lBKQkPIDfmMRmj6bg5WhsGfPhtn4kFB/LGuMtsE9JsP7Bn5ZuBXS5FIBQN1x8BSk lIICPMXG6HKXudq8ZHgQhMphD3PQEvvQa/JHsgHxr8rt0pyaQRC542C0I8lHiE2Eb8uF WNu9+RArHwc1AQf58a7uHmJQVzpgdmtprvgs5NqYEUvstJ62T/tT1sru2LfQWnr8AJnD pdi1WZBZxyVKR63ZHFR4Q8T7nsQWjQYflfuN3ViD3r4e4+ObgYvCZBAeLQNuazrjuVeK +PeA== MIME-Version: 1.0 X-Received: by 10.205.8.199 with SMTP id ot7mr250101bkb.82.1389231517637; Wed, 08 Jan 2014 17:38:37 -0800 (PST) Received: by 10.204.172.144 with HTTP; Wed, 8 Jan 2014 17:38:37 -0800 (PST) In-Reply-To: References: Date: Wed, 8 Jan 2014 17:38:37 -0800 Message-ID: Subject: Re: Help on HDFS sink hangs From: Chen Wang To: user@flume.apache.org Content-Type: multipart/alternative; boundary=20cf302234cb1ec80504ef7fabb1 X-Virus-Checked: Checked by ClamAV on apache.org --20cf302234cb1ec80504ef7fabb1 Content-Type: text/plain; charset=ISO-8859-1 David, I am using flume-ng 1.4.0. Thanks, Chen On Wed, Jan 8, 2014 at 5:26 PM, David Sinclair < dsinclair@chariotsolutions.com> wrote: > What version of flume are you using? > > > On Wed, Jan 8, 2014 at 8:01 PM, Chen Wang wrote: > >> Hey, Guys, >> First time using flume. It goes all well, however, I found that in my >> custom source when I am calling >> >> Event event = EventBuilder.withBody(mySignal.toString().getBytes(), >> headers); >> >> try { >> >> logger.debug("submitting to channel"); >> >> channel.processEvent(event); >> >> }catch (ChannelException e) { >> >> logger.debug("failed to submit to channel, exception {}",e); >> >> throw e; >> >> } >> >> >> It just hangs there on "submitting to channel", and hdfs file apparently >> is not created. Just wondering how can I trouble shoot this? I am running >> it as >> >> bin/flume-ng agent -c conf -f conf/flume.conf -n SocketAgent >> -Dflume.root.logger=DEBUG,console >> >> and my config file looks like: >> >> SocketAgent.sinks.HDFS.channel = MemChannel >> >> SocketAgent.sinks.HDFS.type = hdfs >> >> SocketAgent.sinks.HDFS.hdfs.path = >> maprfs:///hive/myname.db/targeting_storm/%{epoch} >> >> SocketAgent.sinks.HDFS.hdfs.fileType = SequenceFile >> >> SocketAgent.sinks.HDFS.hdfs.writeFormat = Text >> >> SocketAgent.sinks.HDFS.hdfs.batchSize = 1000 >> >> SocketAgent.sinks.HDFS.hdfs.rollSize = 0 >> >> SocketAgent.sinks.HDFS.hdfs.rollCount = 10000 >> >> SocketAgent.sinks.HDFS.hdfs.rollInterval = 600 >> >> >> SocketAgent.channels.MemChannel.type = memory >> >> SocketAgent.channels.MemChannel.capacity = 10000 >> >> SocketAgent.channels.MemChannel.transactionCapacity = 100 >> >> >> Thank you very much!! >> >> Chen >> > > --20cf302234cb1ec80504ef7fabb1 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
David,=A0
I am using flume-ng=A01.4.0.
Thank= s,
Chen


On Wed, Jan 8, 2014 at 5:26 PM, David Sinclair <dsinclair@chariotsolutions.com> wrote:
What version of flume are y= ou using?


On Wed, Jan 8, 2014 at 8:01 PM, Chen Wang <chen.apache.solr@gmail.com> wrote:
Hey, Guys,
First time u= sing flume. It goes all well, however, I found that in my custom source whe= n I am calling=A0

Event event =3D EventBuilder.withBody(mySignal.toString().= getBytes(), headers);

try= {

logger.debug("submitting to chann= el");

channel.processEvent(event);

}catch (ChannelE= xception e) {

logger.debug("failed to submit to= channel, exception {}",e);

throw e;

}


=

It just hangs there on "submitting to channel", and hdfs file = apparently is not created. Just wondering how can I trouble shoot this? I a= m running it as=A0

bin/flume-ng agent -c conf -f conf/flume.conf -n SocketAgent -Dflume.roo= t.logger=3DDEBUG,console

and my config file looks like:

Soc= ketAgent.sinks.HDFS.channel =3D MemChannel

SocketAgent.sinks.HDFS.type =3D hdfs

SocketAgent.sinks.HDFS.hdfs.path= =3D maprfs:///hive/myname.db/targeting_storm/%{epoch}

SocketAgent.si= nks.HDFS.hdfs.fileType =3D SequenceFile

SocketAgent.sinks.HDFS.hdfs.w= riteFormat =3D Text

SocketAgent.sinks.HDFS.hdfs.batchSize =3D 1000

SocketAgent.sinks.H= DFS.hdfs.rollSize =3D 0

SocketAgent.sinks.HDFS.hdfs.rollCount =3D 100= 00

SocketAgent.sinks.HDFS.hdfs.rollInterval =3D 600


SocketAgent.channels.MemChannel.type =3D memory

SocketA= gent.channels.MemChannel.capacity =3D 10000

SocketAgent.channels.MemC= hannel.transactionCapacity =3D 100


Thank you very much!!

Chen



--20cf302234cb1ec80504ef7fabb1--