Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3A5759521 for ; Sat, 24 Mar 2012 17:40:23 +0000 (UTC) Received: (qmail 82885 invoked by uid 500); 24 Mar 2012 17:40:21 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 82831 invoked by uid 500); 24 Mar 2012 17:40:21 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 82823 invoked by uid 99); 24 Mar 2012 17:40:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Mar 2012 17:40:21 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of bejoy.hadoop@gmail.com designates 209.85.160.48 as permitted sender) Received: from [209.85.160.48] (HELO mail-pb0-f48.google.com) (209.85.160.48) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Mar 2012 17:40:15 +0000 Received: by pbbjt11 with SMTP id jt11so4666138pbb.35 for ; Sat, 24 Mar 2012 10:39:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=TxxeAJ07Rx3xqokE71a+mN8qhG785QH8VxfmqMKLFSc=; b=YsytWeC540XZ/W8DoxappSIUbGzq6iwxWGbuzAYUIi0kKd5jL4DVxGWhzOhmV1JNkM ROpw5AQJIksMqo3+lICxG/kMddapdrvEFpwg15E9edtJ/aYytcgMUHhh28QsYy132On9 tE1dz/RcO/+DwS/0Tfv9DoIB29+kDpkGVjtwuzDBD/wGTinYtlf/SccE5+R9w3l/em5Q p4FjEsFkaXI5oPWv4FvFaVFizkFaeSWvSnDyE4FHsnUSVm/EDMLvwFsC2M37j7NXFeUn 2JsShUsA4zd78s5aqR40gj6vYgdZlj0L5FUoVk1hiBP2NmCBeOb2ePIzRG2il+qDX/ec 0QNw== MIME-Version: 1.0 Received: by 10.68.236.72 with SMTP id us8mr39767377pbc.41.1332610793810; Sat, 24 Mar 2012 10:39:53 -0700 (PDT) Received: by 10.143.16.14 with HTTP; Sat, 24 Mar 2012 10:39:53 -0700 (PDT) In-Reply-To: <4F6E04C0.50204@filez.com> References: <4F6E04C0.50204@filez.com> Date: Sat, 24 Mar 2012 23:09:53 +0530 Message-ID: Subject: Re: Mappers only job, output sorted? From: Bejoy Ks To: mapreduce-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b33d9aefd29b904bc00a09b X-Virus-Checked: Checked by ClamAV on apache.org --047d7b33d9aefd29b904bc00a09b Content-Type: text/plain; charset=ISO-8859-1 Hi Radim You are correct. If there is no reduce process then there won't be any sort and shuffle phase. The output from mapper is copied directly to hdfs itself. Regards Bejoy KS 2012/3/24 Radim Kolar > i have mappers only job - number of reducers set to 0. Its hadoop 0.22 and > output from job is this: > > 2012-03-24 18:24:22,117 WARN org.apache.hadoop.util.**NativeCodeLoader: > Unable to load native-hadoop library for your platform... using > builtin-java classes where applicable > 2012-03-24 18:24:22,734 INFO org.apache.hadoop.metrics.jvm.**JvmMetrics: > Initializing JVM Metrics with processName=MAP, sessionId= > 2012-03-24 18:24:37,748 WARN org.apache.hadoop.conf.**Configuration: > user.name is deprecated. Instead, use mapreduce.job.user.name > 2012-03-24 18:24:38,718 INFO org.apache.hadoop.mapred.Task: Using > ResourceCalculatorPlugin : null > 2012-03-24 18:24:41,447 INFO org.apache.hadoop.mapred.Task: > Task:attempt_201201250916_**0074_m_000036_0 is done. And is in the > process of commiting > 2012-03-24 18:24:42,472 INFO org.apache.hadoop.mapred.Task: Task > attempt_201201250916_0074_m_**000036_0 is allowed to commit now > 2012-03-24 18:24:42,478 INFO org.apache.hadoop.mapreduce.**lib.output.**FileOutputCommitter: > Saved output of task 'attempt_201201250916_0074_m_**000036_0' to > /cznewgen/db-fixed > 2012-03-24 18:24:42,479 INFO org.apache.hadoop.mapred.Task: Task > 'attempt_201201250916_0074_m_**000036_0' done. > > is this job still sorting keys in mapper output? Because task status for > mapper says "map > sort". I do not need output sorted. > --047d7b33d9aefd29b904bc00a09b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Radim
=A0 =A0 =A0 =A0 You are correct. If there is no reduce process= then there won't be any sort and shuffle phase. The output from mapper= is copied directly to hdfs itself.

Regards
<= div>Bejoy KS

2012/3/24 Radim Kolar <= hsn@filez.com>
i have mappers only job - number of reducers set to 0. Its hadoop 0.22 and = output from job is this:

2012-03-24 18:24:22,117 WARN org.apache.hadoop.util.NativeCodeLoader= : Unable to load native-hadoop library for your platform... using builtin-j= ava classes where applicable
2012-03-24 18:24:22,734 INFO org.apache.hadoop.metrics.jvm.JvmMetric= s: Initializing JVM Metrics with processName=3DMAP, sessionId=3D
2012-03-24 18:24:37,748 WARN org.apache.hadoop.conf.Configuration: <= a href=3D"http://user.name" target=3D"_blank">user.name is deprecated. = Instead, use m= apreduce.job.user.name
2012-03-24 18:24:38,718 INFO org.apache.hadoop.mapred.Task: =A0Using Resour= ceCalculatorPlugin : null
2012-03-24 18:24:41,447 INFO org.apache.hadoop.mapred.Task: Task:attempt_20= 1201250916_0074_m_000036_0 is done. And is in the process of commiti= ng
2012-03-24 18:24:42,472 INFO org.apache.hadoop.mapred.Task: Task attempt_20= 1201250916_0074_m_000036_0 is allowed to commit now
2012-03-24 18:24:42,478 INFO org.apache.hadoop.mapreduce.lib.output.= FileOutputCommitter: Saved output of task 'attempt_201201250916_= 0074_m_000036_0' to /cznewgen/db-fixed
2012-03-24 18:24:42,479 INFO org.apache.hadoop.mapred.Task: Task 'attem= pt_201201250916_0074_m_000036_0' done.

is this job still sorting keys in mapper output? Because task status for ma= pper says "map > sort". I do not need output sorted.

--047d7b33d9aefd29b904bc00a09b--