Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Fri, 19 Jul 2013 15:02:50 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12647529.1368545774244.81710.1374246170368@arcas>
In-Reply-To: <JIRA.12647529.1368545774244@arcas>
References: <JIRA.12647529.1368545774244@arcas>
Subject: [jira] [Commented] (MAPREDUCE-5247) FileInputFormat should filter
 files with '._COPYING_' sufix
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-5247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13713743#comment-13713743 ] 

Robert Joseph Evans commented on MAPREDUCE-5247:
------------------------------------------------

Why are you running a Map/Reduce job with input from a directory that has not finished being copied?  MR was not designed to run on data that is changing underneath it.  When the job is done how do you know which of the input files were actually used to produce the output?  This issue existed prior to 2.0 but was even worse without the ._COPYING_ suffix. In those cases the files were opened in place and data started to be copied into them.  You may have only even gotten a part of the file in your MR job, not all of it.  The file could have disappeared out from under the MR job if an error occurred.

This is not behavior that I want to make a common park of Map/Reduce.  If you want to do this and you know the risks then you can filter ._COPYING_ files out of your list of input files to the MR job.  But I don't want the framework to do it automatically for everyone.
                
> FileInputFormat should filter files with '._COPYING_' sufix
> -----------------------------------------------------------
>
>                 Key: MAPREDUCE-5247
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5247
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Stan Rosenberg
>
> FsShell copy/put creates staging files with '._COPYING_' suffix.  These files should be considered hidden by FileInputFormat.  (A simple fix is to add the following conjunct to the existing hiddenFilter: 
> {code}
> !name.endsWith("._COPYING_")
> {code}
> After upgrading to CDH 4.2.0 we encountered this bug. We have a legacy data loader which uses 'hadoop fs -put' to load data into hourly partitions.  We also have intra-hourly jobs which are scheduled to execute several times per hour using the same hourly partition as input.  Thus, as the new data is continuously loaded, these staging files (i.e., ._COPYING_) are breaking our jobs (since when copy/put completes staging files are moved).
> As a workaround, we've defined a custom input path filter and loaded it with "mapred.input.pathFilter.class".

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira