Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 37889172EA for ; Tue, 3 Feb 2015 23:12:43 +0000 (UTC) Received: (qmail 33286 invoked by uid 500); 3 Feb 2015 23:12:31 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 33230 invoked by uid 500); 3 Feb 2015 23:12:31 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 33220 invoked by uid 99); 3 Feb 2015 23:12:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Feb 2015 23:12:31 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of stefan@nextdoor.com designates 209.85.212.169 as permitted sender) Received: from [209.85.212.169] (HELO mail-wi0-f169.google.com) (209.85.212.169) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Feb 2015 23:12:27 +0000 Received: by mail-wi0-f169.google.com with SMTP id h11so25857616wiw.0 for ; Tue, 03 Feb 2015 15:09:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=B3JA5C+6pRIN4INuDv70/tWCEH21emS1xDatLq0EtGU=; b=XvtP9HWEG7Qu1bqZOBQfVtj54z2WXQlhJwGyCMXf930ZGFNJPbbRkWm3llQhQtPoNJ rP9Uoe2D8K74dYi2FUa9eLtR1HZQI4d/KzrEfmh66q/4oWKnyBGgkQJ6Xa3XMipaCbyi Cq72VKKVc6jgYyz+LbtNoF/PquCNvtDZxTmUOkt/ndg/nYHgtAXB85qmiE5TWvVDawyS aYdvyovjle4jC7V6V8KoLOyEMG5ZNxHQ3w9+vpCuPKDiCqxLonL+f/C2/7g28SD8h6OL NC0S1EWtGmDKE52SCMxhTZc/Hk5l03HrtDNvdnp1qQJ6H5iWkeIz2vsimO2tBSrdN+LH 5o8w== X-Gm-Message-State: ALoCoQmb4bEMcF5Jn4KfNUZx3WsxNV6b5hFMxIzNddJtxFyy1trTY7Fgp/Y32m7W2wgwmrnhlp7C X-Received: by 10.180.108.202 with SMTP id hm10mr38872189wib.68.1423004991308; Tue, 03 Feb 2015 15:09:51 -0800 (PST) MIME-Version: 1.0 Received: by 10.180.89.38 with HTTP; Tue, 3 Feb 2015 15:09:31 -0800 (PST) In-Reply-To: <1423003794755.69c4013b@Nodemailer> References: <1423003794755.69c4013b@Nodemailer> From: Stefan Krawczyk Date: Tue, 3 Feb 2015 15:09:31 -0800 Message-ID: Subject: Re: HDFS Sink fails to rename file in Flume 1.4 To: user Content-Type: multipart/alternative; boundary=e89a8f3bae77055dd9050e372bac X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f3bae77055dd9050e372bac Content-Type: text/plain; charset=UTF-8 Yes I have some logs, but they don't show any errors as far as I can tell; in comparison to other files, the log statement that the file is being renamed is missing. Do you have something in particular I should look for? What's the timeline for 1.6 at this stage? Cheers, Stefan On Tue, Feb 3, 2015 at 2:49 PM, Hari Shreedharan wrote: > Do you have some logs around the time the rename does not happen? We > recently pushed a workaround for this - when HDFS is unable to close the > file, we were not renaming it. But since the data is safe (it is hflushed > anyway), we decided to change that and rename the file anyway: > https://issues.apache.org/jira/browse/FLUME-2586 > > > Thanks, > Hari > > > On Tue, Feb 3, 2015 at 2:48 PM, Stefan Krawczyk > wrote: > >> Hi Fellow Flume Users, >> >> I'm noticing that we get some "stranded" files in S3 that never get >> renamed from .tmp to .txt. >> >> Questions: >> 1) has this problem been fixed in 1.5, I see some jiras that might >> address the problem, but none are quite that descriptive for me to figure >> that out? >> 2) if this hasn't been addressed, is there a bug for it that I can track? >> >> Otherwise any tips or advice appreciated. >> >> Cheers, >> >> Stefan >> > > --e89a8f3bae77055dd9050e372bac Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Yes I have some logs, but they don't show any errors a= s far as I can tell; in comparison to other files, the log statement that t= he file is being renamed is missing.
Do you have something in particula= r I should look for?

What's the timeline for 1.6= at this stage?

Cheers,

Stefan
=

On Tu= e, Feb 3, 2015 at 2:49 PM, Hari Shreedharan <hshreedharan@cloude= ra.com> wrote:
Do you have some logs around the time the rename does not happen= ? We recently pushed a workaround for this - when HDFS is unable to close t= he file, we were not renaming it. But since the data is safe (it is hflushe= d anyway), we decided to change that and rename the file anyway: https:= //issues.apache.org/jira/browse/FLUME-2586


Thanks,=C2=A0
Hari


On Tue, Feb 3, 2015 at 2:48 PM, Stefa= n Krawczyk <stefan@nextdoor.com> wrote:

Hi Fellow Flume Users,

I'm noticing that we get some "stranded" files in S3 tha= t never get renamed from .tmp to .txt.

Questions:
1) has this problem been fixed in 1.5, I see some jiras that might add= ress the problem, but none are quite that descriptive for me to figure that= out?
2) if this hasn't been addressed, is there a bug for it that I can= track?

Otherwise any tips or advice appreciated.

Cheers,

Stefan


--e89a8f3bae77055dd9050e372bac--