Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DD8FFDBFC for ; Fri, 30 Nov 2012 07:34:39 +0000 (UTC) Received: (qmail 21516 invoked by uid 500); 30 Nov 2012 07:34:39 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 21481 invoked by uid 500); 30 Nov 2012 07:34:39 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 21452 invoked by uid 99); 30 Nov 2012 07:34:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Nov 2012 07:34:38 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of abhishekit@gmail.com designates 209.85.210.51 as permitted sender) Received: from [209.85.210.51] (HELO mail-da0-f51.google.com) (209.85.210.51) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Nov 2012 07:34:30 +0000 Received: by mail-da0-f51.google.com with SMTP id i30so93872dad.38 for ; Thu, 29 Nov 2012 23:34:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:message-id:mime-version:subject:date:references :to:in-reply-to:x-mailer; bh=5O/BCpv1h/lK4yCY7R6CBMxuCs7LOHttJ/Z00VpP6Ak=; b=YtzdaScwuXJumvuERcZba+Sm6aSSBvw2hdWnaHVsOnZ82gJgW5b8Jda7Xkd7iv2fKv vAlurgsF7F0/A29PgFid3l5oze2DbDhCbICC1ZeiASQL54FRw7rK+RK4dXEIQVLqCNMg XV5RzXoE0Ro81M89f0sCDk79BWZKh7H6qn02E60eWNitieXBxCW70XkyZg+AlexnVtWu JluucjNXkbC9Dtio0VcVOInV/1vdXd4MHKCLa+whvusuzEwXgFUWzNTtg0SLcNTJOvrt f5qSOEanlT1V2kjcK2Gi60+J5tM0HsVhXSRfZmK74RZDE+55WEdLyUjh1Kvv2NR87P8x Ptjg== Received: by 10.66.77.199 with SMTP id u7mr1124805paw.7.1354260849236; Thu, 29 Nov 2012 23:34:09 -0800 (PST) Received: from new-host-2.home (pool-96-247-75-113.lsanca.fios.verizon.net. [96.247.75.113]) by mx.google.com with ESMTPS id vx2sm961238pbc.33.2012.11.29.23.34.08 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 29 Nov 2012 23:34:08 -0800 (PST) From: Abhi Dubey Content-Type: multipart/alternative; boundary="Apple-Mail=_CBD172DC-55F2-40AA-A2EA-C9360AA9197F" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: error writing to collectorSink on mac os x Date: Thu, 29 Nov 2012 23:34:07 -0800 References: <4179D3AE-2789-4D0F-9E3C-5AF5B5CF5140@gmail.com> <0DBB48FE-FEF6-45D0-AE99-898E64CC3D01@gmail.com> To: user@flume.apache.org In-Reply-To: <0DBB48FE-FEF6-45D0-AE99-898E64CC3D01@gmail.com> X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_CBD172DC-55F2-40AA-A2EA-C9360AA9197F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Alex, Thanks for the quick reply. I am afraid I don't understand which config = file are you asking for. Do you mean the config file I used to setup = flume source and sink? If that is so, I just used the web interface = http://localhost:35871/flumemaster.jsp to set the config.=20 I used source: text("/etc/services") I used Sink: = collectorSink("hdfs://localhost:8020/user/abhi/flume/","test file") This throws the error that I wrote in the previous message. I also tried = a different datacollectod sink - writing to file system. source: text("/etc/services") Sink: collectorSink("file:///Users/abhi/","testfile") It throws the same error.=20 But if I use source: text("/etc/services") sink: cosole It works. It also works if I use a text sink. source: text("/etc/services") sink: text("services.copy") Can you tell me once again which config file do you need to look at? As per the homebrew formulas, hadoop is version 1.0.4. And flume is = version 0.9.4-cdh3u2. After installing flume, I replaced the jar file = with that from hadoop install. Thanks, Abhi On Nov 29, 2012, at 11:17 PM, Alexander Alten-Lorenz = wrote: > Hi, >=20 > Can you connect to your HDFS instance? Attach the config file for = further debug. > Btw, per homebrew installed HDFS / Flume / whatever depends on the = recipes homebrew uses. Check this first please. >=20 > Best > - Alex >=20 >=20 > On Nov 30, 2012, at 6:50 AM, Abhi Dubey wrote: >=20 >>> Hi, >>>=20 >>> I am running hadoop in pseudo-distributed mode on mac os x 10.8.2. I = installed flume (via homebrew) and I can write to console and local file = system using text. Hadoop was also installed via homebrew. >>>=20 >>> But if I use collectorSink and either a file target file:/// or = hdfs://, I get an error. The error is same for both types of targets. >>>=20 >>>=20 >>> 2012-11-29 20:22:47,182 [logicalNode new-host-2.home-21] INFO = debug.InsistentAppendDecorator: append attempt 6 failed, backoff = (60000ms): failure to login >>> 2012-11-29 20:23:47,181 [pool-8-thread-1] INFO = hdfs.EscapedCustomDfsSink: Opening = file:///Users/abhi/flume/testfile20121129-202317290-0800.13542493972903100= 00.00000037 >>> 2012-11-29 20:23:47,183 [logicalNode new-host-2.home-21] INFO = debug.StubbornAppendSink: append failed on event 'new-host-2.home [INFO = Thu Nov 29 20:21:44 PST 2012] #' with error: failure to login >>> 2012-11-29 20:23:47,183 [logicalNode new-host-2.home-21] INFO = rolling.RollSink: closing RollSink = 'escapedCustomDfs("file:///Users/abhi/flume","testfile%{rolltag}" )' >>> 2012-11-29 20:23:47,183 [logicalNode new-host-2.home-21] INFO = rolling.RollSink: opening RollSink = 'escapedCustomDfs("file:///Users/abhi/flume","testfile%{rolltag}" )' >>> 2012-11-29 20:23:47,184 [logicalNode new-host-2.home-21] INFO = debug.InsistentOpenDecorator: Opened MaskDecorator on try 0 >>> 2012-11-29 20:23:47,185 [pool-9-thread-1] INFO = hdfs.EscapedCustomDfsSink: Opening = file:///Users/abhi/flume/testfile20121129-202347184-0800.13542494271841780= 00.00000021 >>> 2012-11-29 20:23:47,192 [logicalNode new-host-2.home-21] INFO = debug.InsistentAppendDecorator: append attempt 7 failed, backoff = (60000ms): failure to login >>>=20 >>> I am not sure what is going on. The permissions to the directory is = 777. Can anyone help with this error? >>>=20 >>>=20 >>> Thanks, >>>=20 >>> Abhi >>=20 >=20 > -- > Alexander Alten-Lorenz > http://mapredit.blogspot.com > German Hadoop LinkedIn Group: http://goo.gl/N8pCF >=20 --Apple-Mail=_CBD172DC-55F2-40AA-A2EA-C9360AA9197F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii http://localhost:35871/flu= memaster.jsp to set the config. 
I used source: = text("/etc/services")
I used Sink: collectorSink("hdfs://localhost:8020/user= /abhi/flume/","test file")

This throws the = error that I wrote in the previous message. I also tried a different = datacollectod sink - writing to file system.
source: = text("/etc/services")
Sink: collectorSink("file:///Users/abhi/","testfile")
It throws the same error. 

But if I = use
source: text("/etc/services")
sink: = cosole

It works. It also works if I use a text = sink.
source: = text("/etc/services")
sink: text("services.copy")


Can you tell me once again which config = file do you need to look at?

As per the = homebrew formulas, hadoop is version 1.0.4. And flume is = version 0.9.4-cdh3u2. After installing flume, I replaced the jar = file with that from hadoop = install.


Thanks,

Abhi


On = Nov 29, 2012, at 11:17 PM, Alexander Alten-Lorenz <wget.null@gmail.com> = wrote:

Hi,

Can you connect to your HDFS instance? Attach = the config file for further debug.
Btw, per homebrew installed HDFS / = Flume / whatever depends on the recipes homebrew uses. Check this first = please.

Best
- Alex


On Nov 30, 2012, at 6:50 AM, = Abhi Dubey <abhishekit@gmail.com> = wrote:

Hi,

I am running hadoop in pseudo-distributed mode = on mac os x 10.8.2. I installed flume (via homebrew) and I can write to = console and local file system using text. Hadoop was also installed via = homebrew.

But if I use collectorSink and either a file target file:/// or hdfs://, I get an error. The error is = same for both types of targets.


2012-11-29 20:22:47,182 = [logicalNode new-host-2.home-21] INFO debug.InsistentAppendDecorator: = append attempt 6 failed, backoff (60000ms): failure to = login
2012-11-29 20:23:47,181 [pool-8-thread-1] INFO = hdfs.EscapedCustomDfsSink: Opening file:///Users/abhi/flume/testfile20121129-202317290-0= 800.1354249397290310000.00000037
2012-11-29 20:23:47,183 = [logicalNode new-host-2.home-21] INFO debug.StubbornAppendSink: append = failed on event 'new-host-2.home [INFO Thu Nov 29 20:21:44 PST 2012] #' = with error: failure to login
2012-11-29 20:23:47,183 [logicalNode = new-host-2.home-21] INFO rolling.RollSink: closing RollSink = 'escapedCustomDfs("file:///Users/abhi/flume","testfile%= {rolltag}" )'
2012-11-29 20:23:47,183 [logicalNode = new-host-2.home-21] INFO rolling.RollSink: opening RollSink =  'escapedCustomDfs("file:///Users/abhi/flume","testfile%= {rolltag}" )'
2012-11-29 20:23:47,184 [logicalNode = new-host-2.home-21] INFO debug.InsistentOpenDecorator: Opened = MaskDecorator on try 0
2012-11-29 20:23:47,185 [pool-9-thread-1] INFO = hdfs.EscapedCustomDfsSink: Opening file:///Users/abhi/flume/testfile20121129-202347184-0= 800.1354249427184178000.00000021
2012-11-29 20:23:47,192 = [logicalNode new-host-2.home-21] INFO debug.InsistentAppendDecorator: = append attempt 7 failed, backoff (60000ms): failure to login

I am = not sure what is going on. The permissions to the directory is 777. Can = anyone help with this = error?


Thanks,

Abhi

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
= German Hadoop LinkedIn Group: = http://goo.gl/N8pCF


= --Apple-Mail=_CBD172DC-55F2-40AA-A2EA-C9360AA9197F--