Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 84BFDF2DC for ; Tue, 28 May 2013 09:16:49 +0000 (UTC) Received: (qmail 35878 invoked by uid 500); 28 May 2013 09:16:44 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 35794 invoked by uid 500); 28 May 2013 09:16:44 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 35770 invoked by uid 99); 28 May 2013 09:16:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 May 2013 09:16:43 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [212.122.43.157] (HELO datameer.com) (212.122.43.157) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 May 2013 09:16:38 +0000 Received: from ssanyal-lan.datameer.lan (unknown [192.168.178.41]) by datameer.com (Postfix) with ESMTPSA id D53353182926 for ; Tue, 28 May 2013 11:16:15 +0200 (CEST) From: Subroto Content-Type: multipart/signed; boundary="Apple-Mail=_900B809B-C235-4BE2-B1B1-AFA747F62839"; protocol="application/pgp-signature"; micalg=pgp-sha1 Subject: LocalJobRunner is not using the correct JobConf to setup the OutputCommitter Date: Tue, 28 May 2013 11:16:14 +0200 Message-Id: To: user@hadoop.apache.org Mime-Version: 1.0 (Apple Message framework v1283) X-Mailer: Apple Mail (2.1283) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_900B809B-C235-4BE2-B1B1-AFA747F62839 Content-Type: multipart/alternative; boundary="Apple-Mail=_D7283F37-4F4E-4964-93DC-A4B5A8180E11" --Apple-Mail=_D7283F37-4F4E-4964-93DC-A4B5A8180E11 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, I am reusing JobClient object which internally holds a LocalJobRunner = instance. When I submit the Job via the JobClient; LocalJobRunner is not using the = correct JobConf to set the OutputCommitter.setupJob(). Following is the code snippet from = LocalJobRunner#org.apache.hadoop.mapred.LocalJobRunner.Job.run(): public void run() { JobID jobId =3D profile.getJobID(); JobContext jContext =3D new JobContext(conf, jobId); OutputCommitter outputCommitter =3D job.getOutputCommitter(); try { TaskSplitMetaInfo[] taskSplitMetaInfos =3D SplitMetaInfoReader.readSplitMetaInfo(jobId, localFs, conf, = systemJobDir); =20 int numReduceTasks =3D job.getNumReduceTasks(); if (numReduceTasks > 1 || numReduceTasks < 0) { // we only allow 0 or 1 reducer in local mode numReduceTasks =3D 1; job.setNumReduceTasks(1); } outputCommitter.setupJob(jContext); status.setSetupProgress(1.0f); // Some more code to start map and reduce } The JobContext created in the second line of snippet is being created = with the JobConf with which LocalJobRunner is instantiated; instead the = JobContext should be created with JobConf with which the Job is = instantiated. Same context is being used to call outputcommitter.setupJob. Please let me know if this is a bug or there is some specific intention = behind this ?? Cheers, Subroto Sanyal= --Apple-Mail=_D7283F37-4F4E-4964-93DC-A4B5A8180E11 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii public = void run() {
profile.getJobID();
      JobContext = jContext =3D new JobContext(conf, jobId);
job.getOutputCommitter();
try = {
          = SplitMetaInfoReader.readSplitMetaInfo(jobId, localFs, conf, = systemJobDir);     =    
        int numReduceTasks =3D job.getNumReduceTasks();
        = if (numReduceTasks > 1 || = numReduceTasks < 0) {
          // we = only allow 0 or 1 reducer in local mode
        =   numReduceTasks =3D 1;
          = job.setNumReduceTasks(1);
        = }
        status.setSetupProgress(1.0f);
// = Some more code to start map and reduce
}
The JobContext = created in the second line of snippet is being created with the JobConf = with which LocalJobRunner is instantiated; instead the JobContext should = be created with JobConf with which the Job is = instantiated.
Same context is = being used to call outputcommitter.setupJob.

Please let me know = if this is a bug or there is some specific intention behind this = ??