Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5CE01D2CD for ; Sun, 2 Dec 2012 02:02:33 +0000 (UTC) Received: (qmail 12068 invoked by uid 500); 2 Dec 2012 02:02:33 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 11995 invoked by uid 500); 2 Dec 2012 02:02:32 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 11982 invoked by uid 99); 2 Dec 2012 02:02:32 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 Dec 2012 02:02:32 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of abhishekit@gmail.com designates 209.85.220.51 as permitted sender) Received: from [209.85.220.51] (HELO mail-pa0-f51.google.com) (209.85.220.51) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 Dec 2012 02:02:26 +0000 Received: by mail-pa0-f51.google.com with SMTP id fb11so983522pad.38 for ; Sat, 01 Dec 2012 18:02:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:message-id:mime-version:subject:date:references :to:in-reply-to:x-mailer; bh=zDdA1Nc0v5opSR4fXOgqBye6AkeqedLolO4j0HppgXQ=; b=Lvm6gs4YoLklPCtR74QFnn9Y5e1vVVCKy1SIIYvsa99jVCRd/Hlf2OOw1Iov5V0RZ2 cYcI9L8z2aaY+4fSwXw9mc6kIDJ6rQwrE1kuJRDPtm2yT7StE91izJTZN7Nhi4QTwWWY ZGtaM6I/x4ao4w/YZ0BTQaNkD2Lg2zd1zb7mnZaM5YJGkluUG0oLOCyetmLJlEYcJt31 oseHAe7zlggnp3ActCfQPpRVTcovns64MlP13PmbvQGd7/sMuYWWy22itypWq6/I8/aR LxAJKri1TfOGmlCY7TepIo8+H17CPRtHkVXKITStvtleFhZMdJx5XVnfIq1TjSid6mjs MLHg== Received: by 10.66.9.2 with SMTP id v2mr15066173paa.18.1354413725903; Sat, 01 Dec 2012 18:02:05 -0800 (PST) Received: from new-host-2.home (pool-96-247-75-113.lsanca.fios.verizon.net. [96.247.75.113]) by mx.google.com with ESMTPS id c1sm5497602pav.23.2012.12.01.18.01.59 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 01 Dec 2012 18:02:01 -0800 (PST) From: Abhi Dubey Content-Type: multipart/alternative; boundary="Apple-Mail=_40AD1430-11F2-45AB-87DA-19D05303A0B6" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: error writing to collectorSink on mac os x Date: Sat, 1 Dec 2012 18:01:57 -0800 References: <4179D3AE-2789-4D0F-9E3C-5AF5B5CF5140@gmail.com> <0DBB48FE-FEF6-45D0-AE99-898E64CC3D01@gmail.com> <2E99CD4A-6A58-433D-A8F5-EC2FA7B21C6C@gmail.com> To: user@flume.apache.org In-Reply-To: <2E99CD4A-6A58-433D-A8F5-EC2FA7B21C6C@gmail.com> X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_40AD1430-11F2-45AB-87DA-19D05303A0B6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Alex, I just wanted to let you know that your tip to install flume 1.x worked. = I am able to set an hdfs sink and write correctly. Thanks! Abhi On Nov 29, 2012, at 11:33 PM, Abhi Dubey wrote: > Alex, >=20 > Thanks for the quick reply. I am afraid I don't understand which = config file are you asking for. Do you mean the config file I used to = setup flume source and sink? If that is so, I just used the web = interface http://localhost:35871/flumemaster.jsp to set the config.=20 > I used source: text("/etc/services") > I used Sink: = collectorSink("hdfs://localhost:8020/user/abhi/flume/","test file") >=20 > This throws the error that I wrote in the previous message. I also = tried a different datacollectod sink - writing to file system. > source: text("/etc/services") > Sink: collectorSink("file:///Users/abhi/","testfile") > It throws the same error.=20 >=20 > But if I use > source: text("/etc/services") > sink: cosole >=20 > It works. It also works if I use a text sink. > source: text("/etc/services") > sink: text("services.copy") >=20 >=20 > Can you tell me once again which config file do you need to look at? >=20 > As per the homebrew formulas, hadoop is version 1.0.4. And flume is = version 0.9.4-cdh3u2. After installing flume, I re >=20 >=20 > On Nov 29, 2012, at 11:17 PM, Alexander Alten-Lorenz = wrote: >=20 >> Hi, >>=20 >> Can you connect to your HDFS instance? Attach the config file for = further debug. >> Btw, per homebrew installed HDFS / Flume / whatever depends on the = recipes homebrew uses. Check this first please. >>=20 >> Best >> - Alex >>=20 >>=20 >> On Nov 30, 2012, at 6:50 AM, Abhi Dubey wrote: >>=20 >>>> Hi, >>>>=20 >>>> I am running hadoop in pseudo-distributed mode on mac os x 10.8.2. = I installed flume (via homebrew) and I can write to console and local = file system using text. Hadoop was also installed via homebrew. >>>>=20 >>>> But if I use collectorSink and either a file target file:/// or = hdfs://, I get an error. The error is same for both types of targets. >>>>=20 >>>>=20 >>>> 2012-11-29 20:22:47,182 [logicalNode new-host-2.home-21] INFO = debug.InsistentAppendDecorator: append attempt 6 failed, backoff = (60000ms): failure to login >>>> 2012-11-29 20:23:47,181 [pool-8-thread-1] INFO = hdfs.EscapedCustomDfsSink: Opening = file:///Users/abhi/flume/testfile20121129-202317290-0800.13542493972903100= 00.00000037 >>>> 2012-11-29 20:23:47,183 [logicalNode new-host-2.home-21] INFO = debug.StubbornAppendSink: append failed on event 'new-host-2.home [INFO = Thu Nov 29 20:21:44 PST 2012] #' with error: failure to login >>>> 2012-11-29 20:23:47,183 [logicalNode new-host-2.home-21] INFO = rolling.RollSink: closing RollSink = 'escapedCustomDfs("file:///Users/abhi/flume","testfile%{rolltag}" )' >>>> 2012-11-29 20:23:47,183 [logicalNode new-host-2.home-21] INFO = rolling.RollSink: opening RollSink = 'escapedCustomDfs("file:///Users/abhi/flume","testfile%{rolltag}" )' >>>> 2012-11-29 20:23:47,184 [logicalNode new-host-2.home-21] INFO = debug.InsistentOpenDecorator: Opened MaskDecorator on try 0 >>>> 2012-11-29 20:23:47,185 [pool-9-thread-1] INFO = hdfs.EscapedCustomDfsSink: Opening = file:///Users/abhi/flume/testfile20121129-202347184-0800.13542494271841780= 00.00000021 >>>> 2012-11-29 20:23:47,192 [logicalNode new-host-2.home-21] INFO = debug.InsistentAppendDecorator: append attempt 7 failed, backoff = (60000ms): failure to login >>>>=20 >>>> I am not sure what is going on. The permissions to the directory is = 777. Can anyone help with this error? >>>>=20 >>>>=20 >>>> Thanks, >>>>=20 >>>> Abhi >>>=20 >>=20 >> -- >> Alexander Alten-Lorenz >> http://mapredit.blogspot.com >> German Hadoop LinkedIn Group: http://goo.gl/N8pCF >>=20 >=20 --Apple-Mail=_40AD1430-11F2-45AB-87DA-19D05303A0B6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii abhishekit@gmail.com> = wrote:
http://localhost:35871/flu= memaster.jsp to set the config. 
I used source: = text("/etc/services")
I used Sink: collectorSink("hdfs://localhost:8020/user= /abhi/flume/","test file")

This throws the = error that I wrote in the previous message. I also tried a different = datacollectod sink - writing to file system.
source: = text("/etc/services")
Sink: collectorSink("file:///Users/abhi/","testfile")
It throws the same error. 

But if I = use
source: text("/etc/services")
sink: = cosole

It works. It also works if I use a text = sink.
source: = text("/etc/services")
sink: text("services.copy")


Can you tell me once again which config = file do you need to look at?

As per the = homebrew formulas, hadoop is version 1.0.4. And flume is = version 0.9.4-cdh3u2. After installing flume, I = re


On Nov 29, 2012, at = 11:17 PM, Alexander Alten-Lorenz <wget.null@gmail.com> = wrote:

Hi,

Can you connect to your HDFS instance? Attach = the config file for further debug.
Btw, per homebrew installed HDFS / = Flume / whatever depends on the recipes homebrew uses. Check this first = please.

Best
- Alex


On Nov 30, 2012, at 6:50 AM, = Abhi Dubey <abhishekit@gmail.com> = wrote:

Hi,

I am running hadoop in pseudo-distributed mode = on mac os x 10.8.2. I installed flume (via homebrew) and I can write to = console and local file system using text. Hadoop was also installed via = homebrew.

But if I use collectorSink and either a file target file:/// or hdfs://, I get an error. The error is = same for both types of targets.


2012-11-29 20:22:47,182 = [logicalNode new-host-2.home-21] INFO debug.InsistentAppendDecorator: = append attempt 6 failed, backoff (60000ms): failure to = login
2012-11-29 20:23:47,181 [pool-8-thread-1] INFO = hdfs.EscapedCustomDfsSink: Opening file:///Users/abhi/flume/testfile20121129-202317290-0= 800.1354249397290310000.00000037
2012-11-29 20:23:47,183 = [logicalNode new-host-2.home-21] INFO debug.StubbornAppendSink: append = failed on event 'new-host-2.home [INFO Thu Nov 29 20:21:44 PST 2012] #' = with error: failure to login
2012-11-29 20:23:47,183 [logicalNode = new-host-2.home-21] INFO rolling.RollSink: closing RollSink = 'escapedCustomDfs("file:///Users/abhi/flume","testfile%= {rolltag}" )'
2012-11-29 20:23:47,183 [logicalNode = new-host-2.home-21] INFO rolling.RollSink: opening RollSink =  'escapedCustomDfs("file:///Users/abhi/flume","testfile%= {rolltag}" )'
2012-11-29 20:23:47,184 [logicalNode = new-host-2.home-21] INFO debug.InsistentOpenDecorator: Opened = MaskDecorator on try 0
2012-11-29 20:23:47,185 [pool-9-thread-1] INFO = hdfs.EscapedCustomDfsSink: Opening file:///Users/abhi/flume/testfile20121129-202347184-0= 800.1354249427184178000.00000021
2012-11-29 20:23:47,192 = [logicalNode new-host-2.home-21] INFO debug.InsistentAppendDecorator: = append attempt 7 failed, backoff (60000ms): failure to login

I am = not sure what is going on. The permissions to the directory is 777. Can = anyone help with this = error?


Thanks,

Abhi

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
= German Hadoop LinkedIn Group: = http://goo.gl/N8pCF



= --Apple-Mail=_40AD1430-11F2-45AB-87DA-19D05303A0B6--