Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DA4A9D68D for ; Tue, 7 Aug 2012 23:20:45 +0000 (UTC) Received: (qmail 90938 invoked by uid 500); 7 Aug 2012 23:20:42 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 90856 invoked by uid 500); 7 Aug 2012 23:20:42 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 90844 invoked by uid 99); 7 Aug 2012 23:20:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Aug 2012 23:20:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mohitanchlia@gmail.com designates 209.85.160.176 as permitted sender) Received: from [209.85.160.176] (HELO mail-gh0-f176.google.com) (209.85.160.176) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Aug 2012 23:20:34 +0000 Received: by ghbz10 with SMTP id z10so227368ghb.35 for ; Tue, 07 Aug 2012 16:20:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=wKX1tochWIaEaHRHAeeOTnFvTDTw3EcFHDwT/ktYmNk=; b=yJJEFWrdP3IOUkIfJPkkg+RC2th5TrLCaRBx1RvJZgD9JBCLCnarKLNXkRDzuLrWHC lt/5orpGgnN44TU1ZwyO7jgs9PFZfGKzh1/jW5ZE3xM+m7/MyHIFM36QFUvffpOwCIqt LhpnpzjPKknIVZ1M34rcopMW2jIOllcWDjrTFqx1B/mWlBAA/qV2dj9/biAaR8Q0k7au hSr57LdFDFNpGQdABJInjh56i+94TGdGReerpnMyHnDTo0N9wVd5ViIodbZZjcGPHhZU BRC5UJ31DOpHYT5wJGT/dInjZP7cc6FeySqILgTl/wTZOcbLCN19udGofyl5czgUr/22 jofg== MIME-Version: 1.0 Received: by 10.50.203.71 with SMTP id ko7mr111005igc.72.1344381613488; Tue, 07 Aug 2012 16:20:13 -0700 (PDT) Received: by 10.64.65.1 with HTTP; Tue, 7 Aug 2012 16:20:13 -0700 (PDT) In-Reply-To: References: Date: Tue, 7 Aug 2012 16:20:13 -0700 Message-ID: Subject: Re: Local jobtracker in test env? From: Mohit Anchlia To: common-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=14dae934078383bafe04c6b53c7e --14dae934078383bafe04c6b53c7e Content-Type: text/plain; charset=ISO-8859-1 On Tue, Aug 7, 2012 at 2:08 PM, Harsh J wrote: > It used the local mode of operation: > org.apache.hadoop.mapred.LocalJobRunner > > In localmode everything is done inside the same JVM i.e. tasktracker,jobtracker etc. all run in the same JVM. Or does it mean that none of those processes run everything is pipelined in the same process on the local file system. > A JobTracker (via MiniMRCluster) is only required for simulating > distributed tests. > > On Wed, Aug 8, 2012 at 2:27 AM, Mohit Anchlia > wrote: > > I just wrote a test where fs.default.name is file:/// and > > mapred.job.tracker is set to local. The test ran fine, I also see mapper > > and reducer were invoked but what I am trying to understand is that how > did > > this run without specifying the job tracker port and which port task > > tracker connected with job tracker. It's not clear from the output: > > > > Also what's the difference between this and bringing up miniDFS cluster? > > > > INFO org.apache.hadoop.mapred.FileInputFormat [main]: Total input paths > to > > proc > > ess : 1 > > INFO org.apache.hadoop.mapred.JobClient [main]: Running job: > job_local_0001 > > INFO org.apache.hadoop.mapred.Task [Thread-11]: Using > > ResourceCalculatorPlugin > > : null > > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: numReduceTasks: 1 > > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: io.sort.mb = 100 > > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: data buffer = > > 79691776/99614 > > 720 > > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: record buffer = > > 262144/32768 > > 0 > > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup > [Thread-11]: z > > ip 92127 > > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup > [Thread-11]: z > > ip 1 > > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup > [Thread-11]: z > > ip 92127 > > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup > [Thread-11]: z > > ip 1 > > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: Starting flush of map > > output > > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: Finished spill 0 > > INFO org.apache.hadoop.mapred.Task [Thread-11]: > > Task:attempt_local_0001_m_00000 > > 0_0 is done. And is in the process of commiting > > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: > > file:/c:/upb/dp/manch > > lia-dp/depot/services/data-platform/trunk/analytics/geoinput/geo.dat:0+18 > > INFO org.apache.hadoop.mapred.Task [Thread-11]: Task > > 'attempt_local_0001_m_0000 > > 00_0' done. > > INFO org.apache.hadoop.mapred.Task [Thread-11]: Using > > ResourceCalculatorPlugin > > : null > > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: > > INFO org.apache.hadoop.mapred.Merger [Thread-11]: Merging 1 sorted > segments > > INFO org.apache.hadoop.mapred.Merger [Thread-11]: Down to the last > > merge-pass, > > with 1 segments left of total size: 26 bytes > > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: > > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup > [Thread-11]: I > > nside reduce > > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup > [Thread-11]: O > > utside reduce > > INFO org.apache.hadoop.mapred.Task [Thread-11]: > > Task:attempt_local_0001_r_00000 > > 0_0 is done. And is in the process of commiting > > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: > > INFO org.apache.hadoop.mapred.Task [Thread-11]: Task > > attempt_local_0001_r_00000 > > 0_0 is allowed to commit now > > INFO org.apache.hadoop.mapred.FileOutputCommitter [Thread-11]: Saved > > output of > > task 'attempt_local_0001_r_000000_0' to > > file:/c:/upb/dp/manchlia-dp/depot/servic > > es/data-platform/trunk/analytics/geooutput > > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: reduce > > reduce > > INFO org.apache.hadoop.mapred.Task [Thread-11]: Task > > 'attempt_local_0001_r_0000 > > 00_0' done. > > INFO org.apache.hadoop.mapred.JobClient [main]: map 100% reduce 100% > > INFO org.apache.hadoop.mapred.JobClient [main]: Job complete: > > job_local_0001 > > INFO org.apache.hadoop.mapred.JobClient [main]: Counters: 15 > > INFO org.apache.hadoop.mapred.JobClient [main]: FileSystemCounters > > INFO org.apache.hadoop.mapred.JobClient [main]: FILE_BYTES_READ=458 > > INFO org.apache.hadoop.mapred.JobClient [main]: > > FILE_BYTES_WRITTEN=96110 > > INFO org.apache.hadoop.mapred.JobClient [main]: Map-Reduce Framework > > INFO org.apache.hadoop.mapred.JobClient [main]: Map input records=2 > > INFO org.apache.hadoop.mapred.JobClient [main]: Reduce shuffle > bytes=0 > > INFO org.apache.hadoop.mapred.JobClient [main]: Spilled Records=4 > > INFO org.apache.hadoop.mapred.JobClient [main]: Map output bytes=20 > > INFO org.apache.hadoop.mapred.JobClient [main]: Total committed heap > > usage > > (bytes)=321527808 > > INFO org.apache.hadoop.mapred.JobClient [main]: Map input bytes=18 > > INFO org.apache.hadoop.mapred.JobClient [main]: SPLIT_RAW_BYTES=142 > > INFO org.apache.hadoop.mapred.JobClient [main]: Combine input > records=0 > > INFO org.apache.hadoop.mapred.JobClient [main]: Reduce input > records=2 > > INFO org.apache.hadoop.mapred.JobClient [main]: Reduce input > groups=1 > > INFO org.apache.hadoop.mapred.JobClient [main]: Combine output > > records=0 > > INFO org.apache.hadoop.mapred.JobClient [main]: Reduce output > records=1 > > INFO org.apache.hadoop.mapred.JobClient [main]: Map output records=2 > > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [main]: > Inside > > reduce > > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [main]: > Outsid > > e reduce > > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.547 sec > > Results : > > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 > > > > -- > Harsh J > --14dae934078383bafe04c6b53c7e--