Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 4FF29200B66 for ; Thu, 18 Aug 2016 14:56:17 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 4E78A160AAE; Thu, 18 Aug 2016 12:56:17 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 94A0F160AAB for ; Thu, 18 Aug 2016 14:56:16 +0200 (CEST) Received: (qmail 10337 invoked by uid 500); 18 Aug 2016 12:56:15 -0000 Mailing-List: contact dev-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flume.apache.org Delivered-To: mailing list dev@flume.apache.org Received: (qmail 10313 invoked by uid 99); 18 Aug 2016 12:56:15 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Aug 2016 12:56:15 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 320AE2CC71A; Thu, 18 Aug 2016 12:56:14 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============2563999021310686507==" MIME-Version: 1.0 Subject: Review Request 51204: FLUME-2973: Deadlock in hdfs sink From: Denes Arvay To: Mike Percy , Attila Simon , =?utf-8?q?Bal=C3=A1zs_Don=C3=A1t_Bessenyei?= Cc: Flume , Denes Arvay Date: Thu, 18 Aug 2016 12:56:14 -0000 Message-ID: <20160818125614.26606.58162@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: Denes Arvay X-ReviewGroup: Flume X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/51204/ X-Sender: Denes Arvay Reply-To: Denes Arvay X-ReviewRequest-Repository: flume-git archived-at: Thu, 18 Aug 2016 12:56:17 -0000 --===============2563999021310686507== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51204/ ----------------------------------------------------------- Review request for Flume, Balázs Donát Bessenyei, Mike Percy, and Attila Simon. Bugs: FLUME-2973 https://issues.apache.org/jira/browse/FLUME-2973 Repository: flume-git Description ------- Automatic close of BucketWriters (when open file count reached `hdfs.maxOpenFiles`) and the file rolling thread can end up in deadlock. Diffs ----- flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java b096410 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 741f01e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java 742deb0 Diff: https://reviews.apache.org/r/51204/diff/ Testing ------- `flume-hdfs-sink` tests pass. No new tests were added. I created this script to validate the solution: https://gist.github.com/adenes/96503a6e737f9604ab3ee9397a5809ff Thanks, Denes Arvay --===============2563999021310686507==--