Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <1319399.1193300871081.JavaMail.jira@brutus>
Date: Thu, 25 Oct 2007 01:27:51 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Issue Comment Edited: (HADOOP-2098) File handles for log
 files are still open in case of jobs with 0 maps
In-Reply-To: <28553113.1193290010782.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537539 ] 

acmurthy edited comment on HADOOP-2098 at 10/25/07 1:26 AM:
-----------------------------------------------------------------

We should also check for zero input files in {{FileInputFormat.validateInput(JobConf)}} and throw an {{InvalidInputException}} in the first place. 

Something along the lines of:
{noformat}
Index: /home/arun/dev/java/hadoop/HADOOP-1881/src/java/org/apache/hadoop/mapred/FileInputFormat.java
===================================================================
---src/java/org/apache/hadoop/mapred/FileInputFormat.java	(revision 588140)
+++ src/java/org/apache/hadoop/mapred/FileInputFormat.java	(working copy)
@@ -150,6 +150,9 @@
         }
       }
     }
+    if (totalFiles == 0) {
+      result.add(new IOException("Found zero input files"));
+    }
     if (!result.isEmpty()) {
       throw new InvalidInputException(result);
     }
{noformat}


      was (Author: acmurthy):
    We should also check for zero input files in {{FileInputFormat.validateInput(JobConf)}} and throw an {{InvalidInputException}} in the first place. Prevention is better than cure...
  
> File handles for log files are still open in case of jobs with 0 maps
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-2098
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2098
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.15.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>             Fix For: 0.16.0
>
>         Attachments: HADOOP-2098.patch
>
>
> When a job with zero maps is submitted the handle for the log file for that job is still open and can be seen using {{lsof}}. This over time could lead to {{Too many open files Exception}}.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.