Return-Path: X-Original-To: apmail-incubator-flume-user-archive@minotaur.apache.org Delivered-To: apmail-incubator-flume-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E9BB77919 for ; Wed, 28 Sep 2011 15:47:50 +0000 (UTC) Received: (qmail 52005 invoked by uid 500); 28 Sep 2011 15:47:50 -0000 Delivered-To: apmail-incubator-flume-user-archive@incubator.apache.org Received: (qmail 51941 invoked by uid 500); 28 Sep 2011 15:47:50 -0000 Mailing-List: contact flume-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: flume-user@incubator.apache.org Delivered-To: mailing list flume-user@incubator.apache.org Received: (qmail 51932 invoked by uid 99); 28 Sep 2011 15:47:50 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Sep 2011 15:47:50 +0000 X-ASF-Spam-Status: No, hits=4.1 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jonny2112@gmail.com designates 209.85.210.41 as permitted sender) Received: from [209.85.210.41] (HELO mail-pz0-f41.google.com) (209.85.210.41) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Sep 2011 15:47:46 +0000 Received: by pzk5 with SMTP id 5so21565342pzk.0 for ; Wed, 28 Sep 2011 08:47:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=NclrhQq8bWFOC00SdWzb2usWF/Q+4TiZkWVk4h32Zdg=; b=oRuiES/jcmY+EKJ8I6GHHqvfgScmboyhyviwEOXVwsm5KRM7VKvfC2WuxOMUcvsGAE 70Ny9sLSF6wR2Wt/S3UT+PiewOlQDO3XSVyhNStJjxNwBc5SJgSGw3HjWrF7mOQ/Utt2 YGYUmahkINo045rwC2QvYAPRc272IOZXdq1LU= Received: by 10.68.10.70 with SMTP id g6mr44833868pbb.65.1317224845184; Wed, 28 Sep 2011 08:47:25 -0700 (PDT) MIME-Version: 1.0 Received: by 10.142.207.16 with HTTP; Wed, 28 Sep 2011 08:47:05 -0700 (PDT) In-Reply-To: References: From: Jonathan Date: Wed, 28 Sep 2011 11:47:05 -0400 Message-ID: Subject: Re: Having trouble writing to HDFS To: flume-user@incubator.apache.org Content-Type: multipart/alternative; boundary=bcaec53af23afc90a104ae024e16 --bcaec53af23afc90a104ae024e16 Content-Type: text/plain; charset=ISO-8859-1 If It helps, here is my log file, 2011-09-28 11:31:38,923 INFO com.cloudera.flume.handlers.rolling.RollSink: Created RollSink: trigger=[TimeTrigger: maxAge=30000 tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@162e07d] checkPeriodM$ 2011-09-28 11:31:43,933 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 1 heartbeats 2011-09-28 11:31:48,943 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 2 heartbeats 2011-09-28 11:31:48,943 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 3 heartbeats 2011-09-28 11:31:53,953 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 4 heartbeats 2011-09-28 11:31:53,953 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 5 heartbeats 2011-09-28 11:31:58,963 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 6 heartbeats 2011-09-28 11:31:58,963 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 7 heartbeats 2011-09-28 11:31:59,473 WARN com.cloudera.flume.conf.FlumeBuilder: Deprecated syntax: Expected a format spec but instead had a (String) avrojson 2011-09-28 11:32:03,973 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 8 heartbeats 2011-09-28 11:32:03,973 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 9 heartbeats 2011-09-28 11:32:08,932 ERROR com.cloudera.flume.agent.LogicalNode: Forcing driver to exit uncleanly 2011-09-28 11:32:08,932 ERROR com.cloudera.flume.core.connector.DirectDriver: Closing down due to exception during append calls java.io.IOException: Waiting for queue element was interrupted! null at com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:222) at com.cloudera.flume.collector.CollectorSource.next(CollectorSource.java:72) at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:105) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1961) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2038) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:209) ... 2 more 2011-09-28 11:32:08,933 INFO com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode collector1-2137 exited with error: Waiting for queue element was interrupted! null java.io.IOException: Waiting for queue element was interrupted! null at com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:222) at com.cloudera.flume.collector.CollectorSource.next(CollectorSource.java:72) at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:105) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1961) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2038) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:209) ... 2 more 2011-09-28 11:32:08,933 INFO com.cloudera.flume.collector.CollectorSource: closed 2011-09-28 11:32:08,933 INFO com.cloudera.flume.agent.LogicalNode: Node config successfully set to com.cloudera.flume.conf.FlumeConfigData@1348e2f 2011-09-28 11:32:08,936 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Closed server on port 35853... 2011-09-28 11:32:08,936 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Queue still has 0 elements ... 2011-09-28 11:32:08,936 INFO com.cloudera.flume.handlers.rolling.RollSink: closing RollSink 'escapedCustomDfs("hdfs:// ec2-174-129-89-0.compute-1.amazonaws.com:54310/user/flume/","syslog%{rolltag}" )' 2011-09-28 11:32:08,937 ERROR com.cloudera.flume.core.connector.DirectDriver: Exiting driver logicalNode collector1-2137 in error state CollectorSource | Collector because Waiting for queue element was inte$ 2011-09-28 11:32:08,937 INFO com.cloudera.flume.collector.CollectorSource: opened 2011-09-28 11:32:08,937 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Starting blocking thread pool server on port 35853... 2011-09-28 11:32:08,937 INFO com.cloudera.flume.handlers.rolling.RollSink: opening RollSink 'escapedCustomDfs("hdfs://10.116.98.79:54310/user/flume/","syslog%{rolltag}" )' 2011-09-28 11:32:08,940 WARN com.cloudera.flume.conf.FlumeBuilder: Deprecated syntax: Expected a format spec but instead had a (String) avrojson 2011-09-28 11:32:08,940 INFO com.cloudera.flume.handlers.debug.InsistentOpenDecorator: Opened MaskDecorator on try 0 Jonathan On Wed, Sep 28, 2011 at 11:16 AM, Jonathan wrote: > To the best of my knowledge they are both running the same hdfs version. I > installed both from Cloudera's CDH > > Jonathan > > > > On Wed, Sep 28, 2011 at 11:12 AM, Justin Workman > wrote: > >> I have also seem this happen if the hadoop version on the collector nodes >> is a different version than the hdfs version you are writhing to. >> >> Sent from my iPhone >> >> On Sep 28, 2011, at 8:40 AM, Jonathan wrote: >> >> Hi, >> >> Yeah, that doesn't seem to help. Thanks for trying though. >> >> Jonathan >> >> >> On Wed, Sep 28, 2011 at 10:35 AM, steve layland < >> stevieplayland@gmail.com> wrote: >> >>> 54310 >> >> >> > --bcaec53af23afc90a104ae024e16 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable If It helps, here is my log file,
2011-09-28 11:31:38,923 INFO com= .cloudera.flume.handlers.rolling.RollSink: Created RollSink: trigger=3D[Tim= eTrigger: maxAge=3D30000 tagger=3Dcom.cloudera.flume.handlers.rolling.Proce= ssTagger@162e07d] checkPeriodM$
2011-09-28 11:31:43,933 WARN com.cloudera.flume.agent.LivenessManager:= Heartbeats are backing up, currently behind by 1 heartbeats
2011= -09-28 11:31:48,943 WARN com.cloudera.flume.agent.LivenessManager: Heartbea= ts are backing up, currently behind by 2 heartbeats
2011-09-28 11:31:48,943 WARN com.cloudera.flume.agent.LivenessManager:= Heartbeats are backing up, currently behind by 3 heartbeats
2011= -09-28 11:31:53,953 WARN com.cloudera.flume.agent.LivenessManager: Heartbea= ts are backing up, currently behind by 4 heartbeats
2011-09-28 11:31:53,953 WARN com.cloudera.flume.agent.LivenessManager:= Heartbeats are backing up, currently behind by 5 heartbeats
2011= -09-28 11:31:58,963 WARN com.cloudera.flume.agent.LivenessManager: Heartbea= ts are backing up, currently behind by 6 heartbeats
2011-09-28 11:31:58,963 WARN com.cloudera.flume.agent.LivenessManager:= Heartbeats are backing up, currently behind by 7 heartbeats
2011= -09-28 11:31:59,473 WARN com.cloudera.flume.conf.FlumeBuilder: Deprecated s= yntax: Expected a format spec but instead had a (String) avrojson
2011-09-28 11:32:03,973 WARN com.cloudera.flume.agent.LivenessManager:= Heartbeats are backing up, currently behind by 8 heartbeats
2011= -09-28 11:32:03,973 WARN com.cloudera.flume.agent.LivenessManager: Heartbea= ts are backing up, currently behind by 9 heartbeats
2011-09-28 11:32:08,932 ERROR com.cloudera.flume.agent.LogicalNode: Fo= rcing driver to exit uncleanly
2011-09-28 11:32:08,932 ERROR com.= cloudera.flume.core.connector.DirectDriver: Closing down due to exception d= uring append calls
java.io.IOException: Waiting for queue element was interrupted! null
=A0 =A0 =A0 =A0 at com.cloudera.flume.handlers.thrift.ThriftEventS= ource.next(ThriftEventSource.java:222)
=A0 =A0 =A0 =A0 at com.clo= udera.flume.collector.CollectorSource.next(CollectorSource.java:72)
=A0 =A0 =A0 =A0 at com.cloudera.flume.core.connector.DirectDriver$Pump= erThread.run(DirectDriver.java:105)
Caused by: java.lang.Interrup= tedException
=A0 =A0 =A0 =A0 at java.util.concurrent.locks.Abstra= ctQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueue= dSynchronizer.java:1961)
=A0 =A0 =A0 =A0 at java.util.concurrent.locks.AbstractQueuedSynchroniz= er$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2038)
=A0 =A0 =A0 =A0 at java.util.concurrent.LinkedBlockingQueue.poll(LinkedB= lockingQueue.java:424)
=A0 =A0 =A0 =A0 at com.cloudera.flume.handlers.thrift.ThriftEventSourc= e.next(ThriftEventSource.java:209)
=A0 =A0 =A0 =A0 ... 2 more
2011-09-28 11:32:08,933 INFO com.cloudera.flume.core.connector.Direc= tDriver: Connector logicalNode collector1-2137 exited with error: Waiting f= or queue element was interrupted! null
java.io.IOException: Waiting for queue element was interrupted! null
=A0 =A0 =A0 =A0 at com.cloudera.flume.handlers.thrift.ThriftEventS= ource.next(ThriftEventSource.java:222)
=A0 =A0 =A0 =A0 at com.clo= udera.flume.collector.CollectorSource.next(CollectorSource.java:72)
=A0 =A0 =A0 =A0 at com.cloudera.flume.core.connector.DirectDriver$Pump= erThread.run(DirectDriver.java:105)
Caused by: java.lang.Interrup= tedException
=A0 =A0 =A0 =A0 at java.util.concurrent.locks.Abstra= ctQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueue= dSynchronizer.java:1961)
=A0 =A0 =A0 =A0 at java.util.concurrent.locks.AbstractQueuedSynchroniz= er$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2038)
=A0 =A0 =A0 =A0 at java.util.concurrent.LinkedBlockingQueue.poll(LinkedB= lockingQueue.java:424)
=A0 =A0 =A0 =A0 at com.cloudera.flume.handlers.thrift.ThriftEventSourc= e.next(ThriftEventSource.java:209)
=A0 =A0 =A0 =A0 ... 2 more
2011-09-28 11:32:08,933 INFO com.cloudera.flume.collector.CollectorS= ource: closed
2011-09-28 11:32:08,933 INFO com.cloudera.flume.agent.LogicalNode: Nod= e config successfully set to com.cloudera.flume.conf.FlumeConfigData@1348e2= f
2011-09-28 11:32:08,936 INFO com.cloudera.flume.handlers.thrift= .ThriftEventSource: Closed server on port 35853...
2011-09-28 11:32:08,936 INFO com.cloudera.flume.handlers.thrift.Thrift= EventSource: Queue still has 0 elements ...
2011-09-28 11:32:08,9= 36 INFO com.cloudera.flume.handlers.rolling.RollSink: closing RollSink '= ;escapedCustomDfs("hdfs://ec2-174-129-89-0.compute-1.amazonaws.com:= 54310/user/flume/ ","syslog%{rolltag}" )'
2011-09-28 11:32:08,937 ERROR com.cloudera.flume.core.connector.Direct= Driver: Exiting driver logicalNode collector1-2137 in error state Collector= Source | Collector because Waiting for queue element was inte$
2011-09-28 11:32:08,937 INFO com.cloudera.flume.collector.CollectorSource: = opened
2011-09-28 11:32:08,937 INFO com.cloudera.flume.handlers.t= hrift.ThriftEventSource: Starting blocking thread pool server on port 35853= ...
2011-09-28 11:32:08,937 INFO com.cloudera.flume.handlers.rolling.RollS= ink: opening RollSink =A0'escapedCustomDfs("hdfs://10.116.98.79:54310/user/flume/ "= ;,"syslog%{rolltag}" )'
2011-09-28 11:32:08,940 WARN com.cloudera.flume.conf.FlumeBuilder: Dep= recated syntax: Expected a format spec but instead had a (String) avrojson<= /div>
2011-09-28 11:32:08,940 INFO com.cloudera.flume.handlers.debug.In= sistentOpenDecorator: Opened MaskDecorator on try 0

Jonathan


On Wed, Sep 28, 2011 at 11:16 AM, Jonath= an <jonny2112@g= mail.com> wrote:
To the best of my=A0knowledge they are both running the same hdfs version. = I installed both from Cloudera's CDH
=A0Jonathan



On Wed, Sep 28, 2011 at 11:12 AM, Justin= Workman <justinjworkman@gmail.com> wrote:
I have also seem this happen if the hadoop ve= rsion on the collector nodes is a different version than the hdfs version y= ou are writhing to.=A0

Sent from my iPhone

On Sep 28, 2011, at 8:40 AM, Jonathan <jonny2112@gmail.com> wrote:

=
Hi,

Yeah= , that doesn't seem to help. Thanks for trying though.

Jonathan


On Wed, Sep 28, 2011 at 10:35 AM, steve = layland <stevieplayland@gmail.com> wrote:
54310



--bcaec53af23afc90a104ae024e16--