Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Date: Wed, 29 Jul 2015 09:17:04 +0000 (UTC)
From: "Jan Filipiak (JIRA)" <jira@apache.org>
To: hdfs-dev@hadoop.apache.org
Message-ID: <JIRA.12849876.1438161407000.316948.1438161424763@Atlassian.JIRA>
In-Reply-To: <JIRA.12849876.1438161407000@Atlassian.JIRA>
References: <JIRA.12849876.1438161407000@Atlassian.JIRA>
 <JIRA.12849876.1438161407646@arcas>
Subject: [jira] [Created] (HDFS-8836) Skip newline on empty files with
 getMerge -nl
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Jan Filipiak created HDFS-8836:
----------------------------------

             Summary: Skip newline on empty files with getMerge -nl
                 Key: HDFS-8836
                 URL: https://issues.apache.org/jira/browse/HDFS-8836
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: hdfs-client
    Affects Versions: 2.7.1, 2.6.0
            Reporter: Jan Filipiak
            Priority: Trivial


Hello everyone,

I recently was in the need of using the new line option -nl with getMerge because the files I needed to merge simply didn't had one. I was merging all the files from one directory and unfortunately this directory also included empty files, which effectively led to multiple newlines append after some files. I needed to remove them manually afterwards.

In this situation it is maybe good to have another argument that allows
skipping empty files.

Thing one could try to implement this feature:

The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't
return the number of bytes copied which would be convenient as one could
skip append the new line when 0 bytes where copied or one would check the file size before.

I posted this Idea on the mailing list http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E but I didn't really get many responses, so I thought I my try this way.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)