Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BD31E18F80 for ; Mon, 14 Sep 2015 23:42:49 +0000 (UTC) Received: (qmail 37973 invoked by uid 500); 14 Sep 2015 23:42:44 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 37855 invoked by uid 500); 14 Sep 2015 23:42:44 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 37845 invoked by uid 99); 14 Sep 2015 23:42:43 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2015 23:42:43 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 722001A19A6 for ; Mon, 14 Sep 2015 23:42:43 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.9 X-Spam-Level: ** X-Spam-Status: No, score=2.9 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id iKpRY-04CBLE for ; Mon, 14 Sep 2015 23:42:38 +0000 (UTC) Received: from mail-io0-f196.google.com (mail-io0-f196.google.com [209.85.223.196]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 352D7206EF for ; Mon, 14 Sep 2015 23:42:37 +0000 (UTC) Received: by ioii196 with SMTP id i196so17958948ioi.2 for ; Mon, 14 Sep 2015 16:42:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-type; bh=O7ehWLV68cWGJUUjOYBf2J4uBcCNgE63V+aoJNbBjZ0=; b=GWooMoWZEhOHOs4Dj20XrK135R7ZLs9+Hw+gDE8hQGP4XK7U0iLMCFk2gorqjNhOva HKzEd4wrOtMtsvjO9/1ZxHBK3hnwl3q0yAcfVQdzqzX61nqPzEXqtEY3n6GKzA8yos1T qTZ9vQ5EVuVn99X2SUphvNB12SRReBhE1wGokRfdnfP+1jobxkitHfgWkvVAqSVn6Vze WrXWk+kePZLqlfaCjfDDiJEEmsNpH3Qoxiwxnr4fmFHyGlXAGNdzVgtMW5hkJt22QlqV 1heL6u0h7vkmBwnU11OaMOr2Px9vSB7/p053OQStURiBOzWZF5AIGJ1TQRYBshneSU4D +REw== X-Received: by 10.107.129.65 with SMTP id c62mr32070662iod.4.1442274149892; Mon, 14 Sep 2015 16:42:29 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: mehdi benchoufi Date: Mon, 14 Sep 2015 23:42:20 +0000 Message-ID: Subject: hadoop output in local not in remote To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a113f89de5f8ab8051fbd9e75 --001a113f89de5f8ab8051fbd9e75 Content-Type: text/plain; charset=UTF-8 Hi I have some strange issue. I have some `MapR` code running fine on my local, producing the desired output. With the same `input`, I get no output when I use this code on some `google-cloud` remote cluster. Here's the log : 15/09/14 09:23:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/09/14 09:23:35 INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.4.0-hadoop2 15/09/14 09:23:36 INFO Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 15/09/14 09:23:36 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. Deleted ../output mehdibenchoufi@metamap-m:~$ hadoop jar jar/full_metamap.jar avro_input output 15/09/14 09:23:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/09/14 09:23:49 INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.4.0-hadoop2 15/09/14 09:23:50 INFO client.RMProxy: Connecting to ResourceManager at metamap-m/10.240.231.143:8032 15/09/14 09:23:51 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 15/09/14 09:23:55 INFO input.FileInputFormat: Total input paths to process : 2 15/09/14 09:23:56 INFO mapreduce.JobSubmitter: number of splits:2 15/09/14 09:23:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1442217060046_0001 15/09/14 09:23:59 INFO impl.YarnClientImpl: Submitted application application_1442217060046_0001 15/09/14 09:23:59 INFO mapreduce.Job: The url to track the job: http://metamap-m:8088/proxy/application_1442217060046_0001/ 15/09/14 09:23:59 INFO mapreduce.Job: Running job: job_1442217060046_0001 15/09/14 09:24:22 INFO mapreduce.Job: Job job_1442217060046_0001 running in uber mode : false 15/09/14 09:24:22 INFO mapreduce.Job: map 0% reduce 0% 15/09/14 09:24:41 INFO mapreduce.Job: map 16% reduce 0% 15/09/14 09:24:44 INFO mapreduce.Job: map 32% reduce 0% 15/09/14 09:24:45 INFO mapreduce.Job: map 51% reduce 0% 15/09/14 09:24:47 INFO mapreduce.Job: map 52% reduce 0% 15/09/14 09:24:53 INFO mapreduce.Job: map 53% reduce 0% 15/09/14 09:24:56 INFO mapreduce.Job: map 68% reduce 0% 15/09/14 09:24:57 INFO mapreduce.Job: map 68% reduce 1% 15/09/14 09:24:58 INFO mapreduce.Job: map 68% reduce 3% 15/09/14 09:24:59 INFO mapreduce.Job: map 100% reduce 3% 15/09/14 09:25:01 INFO mapreduce.Job: map 100% reduce 8% 15/09/14 09:25:03 INFO mapreduce.Job: map 100% reduce 32% 15/09/14 09:25:04 INFO mapreduce.Job: map 100% reduce 35% 15/09/14 09:25:08 INFO mapreduce.Job: map 100% reduce 40% 15/09/14 09:25:11 INFO mapreduce.Job: map 100% reduce 45% 15/09/14 09:25:12 INFO mapreduce.Job: map 100% reduce 55% 15/09/14 09:25:13 INFO mapreduce.Job: map 100% reduce 65% 15/09/14 09:25:14 INFO mapreduce.Job: map 100% reduce 70% 15/09/14 09:25:15 INFO mapreduce.Job: map 100% reduce 80% 15/09/14 09:25:19 INFO mapreduce.Job: map 100% reduce 85% 15/09/14 09:25:20 INFO mapreduce.Job: map 100% reduce 100% 15/09/14 09:26:08 INFO mapreduce.Job: Job job_1442217060046_0001 completed successfully 15/09/14 09:26:08 INFO mapreduce.Job: Counters: 51 File System Counters FILE: Number of bytes read=128 FILE: Number of bytes written=6484874 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 GS: Number of bytes read=72553378 GS: Number of bytes written=2 GS: Number of read operations=0 GS: Number of large read operations=0 GS: Number of write operations=0 Job Counters Killed map tasks=1 Killed reduce tasks=1 Launched map tasks=3 Launched reduce tasks=21 Rack-local map tasks=3 Total time spent by all maps in occupied slots (ms)=127110 Total time spent by all reduces in occupied slots (ms)=998415 Total time spent by all map tasks (ms)=63555 Total time spent by all reduce tasks (ms)=199683 Total vcore-seconds taken by all map tasks=63555 Total vcore-seconds taken by all reduce tasks=399366 Total megabyte-seconds taken by all map tasks=130160640 Total megabyte-seconds taken by all reduce tasks=1022376960 Map-Reduce Framework Map input records=5978 Map output records=5978 Map output bytes=11956 Map output materialized bytes=248 Input split bytes=208 Combine input records=5978 Combine output records=2 Reduce input groups=1 Reduce shuffle bytes=248 Reduce input records=2 Reduce output records=1 Spilled Records=4 Shuffled Maps =40 Failed Shuffles=0 Merged Map outputs=40 GC time elapsed (ms)=916 CPU time spent (ms)=60290 Physical memory (bytes) snapshot=6689841152 Virtual memory (bytes) snapshot=104437014528 Total committed heap usage (bytes)=6258425856 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=72553170 File Output Format Counters Bytes Written=2 Could someone help ? Best regards, Mehdi --001a113f89de5f8ab8051fbd9e75 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi

I have some strange issue= . I have some `MapR` code running fine on my local, producing the desired o= utput. With the same `input`, I get no output when I use this code on some = `google-cloud` remote cluster.=C2=A0

Here's th= e log :=C2=A0

=C2=A0 =C2=A0 15/09/14 09:23:35 WARN= util.NativeCodeLoader: Unable to load native-hadoop library for your platf= orm... using builtin-java classes where applicable
=C2=A0 =C2=A0 = 15/09/14 09:23:35 INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.4.0-= hadoop2
=C2=A0 =C2=A0 15/09/14 09:23:36 INFO Configuration.deprec= ation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-chec= ksum
=C2=A0 =C2=A0 15/09/14 09:23:36 INFO fs.TrashPolicyDefault: = Namenode trash configuration: Deletion interval =3D 0 minutes, Emptier inte= rval =3D 0 minutes.
=C2=A0 =C2=A0 Deleted ../output
=C2= =A0 =C2=A0 mehdibenchoufi@metamap-m:~$ hadoop jar jar/full_metamap.jar avro= _input output
=C2=A0 =C2=A0 15/09/14 09:23:48 WARN util.NativeCod= eLoader: Unable to load native-hadoop library for your platform... using bu= iltin-java classes where applicable
=C2=A0 =C2=A0 15/09/14 09:23:= 49 INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.4.0-hadoop2
=C2=A0 =C2=A0 15/09/14 09:23:50 INFO client.RMProxy: Connecting to Resou= rceManager at metamap-m/10.240.231.1= 43:8032
=C2=A0 =C2=A0 15/09/14 09:23:51 WARN mapreduce.JobSub= mitter: Hadoop command-line option parsing not performed. Implement the Too= l interface and execute your application with ToolRunner to remedy this.
=C2=A0 =C2=A0 15/09/14 09:23:55 INFO input.FileInputFormat: Total i= nput paths to process : 2
=C2=A0 =C2=A0 15/09/14 09:23:56 INFO ma= preduce.JobSubmitter: number of splits:2
=C2=A0 =C2=A0 15/09/14 0= 9:23:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1442217= 060046_0001
=C2=A0 =C2=A0 15/09/14 09:23:59 INFO impl.YarnClientI= mpl: Submitted application application_1442217060046_0001
=C2=A0 = =C2=A0 15/09/14 09:23:59 INFO mapreduce.Job: The url to track the job: http://= metamap-m:8088/proxy/application_1442217060046_0001/
=C2=A0 = =C2=A0 15/09/14 09:23:59 INFO mapreduce.Job: Running job: job_1442217060046= _0001
=C2=A0 =C2=A0 15/09/14 09:24:22 INFO mapreduce.Job: Job job= _1442217060046_0001 running in uber mode : false
=C2=A0 =C2=A0 15= /09/14 09:24:22 INFO mapreduce.Job: =C2=A0map 0% reduce 0%
=C2=A0= =C2=A0 15/09/14 09:24:41 INFO mapreduce.Job: =C2=A0map 16% reduce 0%
=
=C2=A0 =C2=A0 15/09/14 09:24:44 INFO mapreduce.Job: =C2=A0map 32% redu= ce 0%
=C2=A0 =C2=A0 15/09/14 09:24:45 INFO mapreduce.Job: =C2=A0m= ap 51% reduce 0%
=C2=A0 =C2=A0 15/09/14 09:24:47 INFO mapreduce.J= ob: =C2=A0map 52% reduce 0%
=C2=A0 =C2=A0 15/09/14 09:24:53 INFO = mapreduce.Job: =C2=A0map 53% reduce 0%
=C2=A0 =C2=A0 15/09/14 09:= 24:56 INFO mapreduce.Job: =C2=A0map 68% reduce 0%
=C2=A0 =C2=A0 1= 5/09/14 09:24:57 INFO mapreduce.Job: =C2=A0map 68% reduce 1%
=C2= =A0 =C2=A0 15/09/14 09:24:58 INFO mapreduce.Job: =C2=A0map 68% reduce 3%
=C2=A0 =C2=A0 15/09/14 09:24:59 INFO mapreduce.Job: =C2=A0map 100% = reduce 3%
=C2=A0 =C2=A0 15/09/14 09:25:01 INFO mapreduce.Job: =C2= =A0map 100% reduce 8%
=C2=A0 =C2=A0 15/09/14 09:25:03 INFO mapred= uce.Job: =C2=A0map 100% reduce 32%
=C2=A0 =C2=A0 15/09/14 09:25:0= 4 INFO mapreduce.Job: =C2=A0map 100% reduce 35%
=C2=A0 =C2=A0 15/= 09/14 09:25:08 INFO mapreduce.Job: =C2=A0map 100% reduce 40%
=C2= =A0 =C2=A0 15/09/14 09:25:11 INFO mapreduce.Job: =C2=A0map 100% reduce 45%<= /div>
=C2=A0 =C2=A0 15/09/14 09:25:12 INFO mapreduce.Job: =C2=A0map 100= % reduce 55%
=C2=A0 =C2=A0 15/09/14 09:25:13 INFO mapreduce.Job: = =C2=A0map 100% reduce 65%
=C2=A0 =C2=A0 15/09/14 09:25:14 INFO ma= preduce.Job: =C2=A0map 100% reduce 70%
=C2=A0 =C2=A0 15/09/14 09:= 25:15 INFO mapreduce.Job: =C2=A0map 100% reduce 80%
=C2=A0 =C2=A0= 15/09/14 09:25:19 INFO mapreduce.Job: =C2=A0map 100% reduce 85%
= =C2=A0 =C2=A0 15/09/14 09:25:20 INFO mapreduce.Job: =C2=A0map 100% reduce 1= 00%
=C2=A0 =C2=A0 15/09/14 09:26:08 INFO mapreduce.Job: Job job_1= 442217060046_0001 completed successfully
=C2=A0 =C2=A0 15/09/14 0= 9:26:08 INFO mapreduce.Job: Counters: 51
=C2=A0 =C2=A0 File System Counters=
=C2=A0 =C2=A0 FILE: Number of bytes read=3D128
=C2=A0 =C2=A0 FILE: Numbe= r of bytes written=3D6484874
=C2=A0 =C2=A0 FILE: Number of read operations= =3D0
=C2=A0 =C2=A0 FILE: Number of large read operations=3D0
=C2= =A0 =C2=A0 FILE: Number of write operations=3D0
=C2=A0 =C2=A0 GS: Number of bytes = read=3D72553378
=C2=A0 =C2=A0 GS: Number of bytes written=3D2
= =C2=A0 =C2=A0 GS: Number of read operations=3D0
=C2=A0 =C2=A0 GS: Number of large = read operations=3D0
=C2=A0 =C2=A0 GS: Number of write operations=3D0
=
=C2=A0 =C2=A0 Job Counters=C2=A0
=C2=A0 =C2=A0 Killed map tasks=3D1
= =C2=A0 =C2=A0 Killed reduce tasks=3D1
=C2=A0 =C2=A0 Launched map tasks=3D3
=C2=A0 =C2=A0 = Launched reduce tasks=3D21
=C2=A0 =C2=A0 Rack-local map tasks=3D3
=C2=A0 =C2=A0 Total time spent by all maps in occupied slots (ms)=3D127110<= /div>
=C2=A0 =C2=A0 Total time spent by all reduces in occupied slots (ms)=3D998= 415
=C2=A0 =C2=A0 Total time spent by all map tasks (ms)=3D63555
=C2=A0 =C2=A0 <= /span>Total time spent by all reduce tasks (ms)=3D199683
=C2=A0 = =C2=A0 To= tal vcore-seconds taken by all map tasks=3D63555
=C2=A0 =C2=A0 Total vcore= -seconds taken by all reduce tasks=3D399366
=C2=A0 =C2=A0 Total megabyte-s= econds taken by all map tasks=3D130160640
=C2=A0 =C2=A0 Total megabyte-sec= onds taken by all reduce tasks=3D1022376960
=C2=A0 =C2=A0 Map-Reduce Framew= ork
=C2=A0 =C2=A0 Map input records=3D5978
=C2=A0 =C2=A0 Map output recor= ds=3D5978
=C2=A0 =C2=A0 Map output bytes=3D11956
=C2=A0 =C2=A0 <= span class=3D"Apple-tab-span" style=3D"white-space:pre"> Map output= materialized bytes=3D248
=C2=A0 =C2=A0 Input split bytes=3D208
= =C2=A0 =C2=A0 Combine input records=3D5978
=C2=A0 =C2=A0 Combine output records=3D2=
=C2=A0 =C2=A0 Reduce input groups=3D1
=C2=A0 =C2=A0 Reduce shuffle bytes= =3D248
=C2=A0 =C2=A0 Reduce input records=3D2
=C2=A0 =C2=A0 Reduce output= records=3D1
=C2=A0 =C2=A0 Spilled Records=3D4
=C2=A0 =C2=A0 <= span class=3D"Apple-tab-span" style=3D"white-space:pre"> Shuffled M= aps =3D40
=C2=A0 =C2=A0 Failed Shuffles=3D0
=C2=A0 =C2=A0 Merged Map outp= uts=3D40
=C2=A0 =C2=A0 GC time elapsed (ms)=3D916
=C2=A0 =C2=A0 = CPU time = spent (ms)=3D60290
=C2=A0 =C2=A0 Physical memory (bytes) snapshot=3D668984= 1152
=C2=A0 =C2=A0 Virtual memory (bytes) snapshot=3D104437014528
=C2=A0 =C2=A0 = Total committed heap usage (bytes)=3D6258425856
=C2=A0 =C2= =A0 Shuffl= e Errors
=C2=A0 =C2=A0 BAD_ID=3D0
=C2=A0 =C2=A0 CONNECTION=3D0
= =C2=A0 =C2=A0 IO_ERROR=3D0
=C2=A0 =C2=A0 WRONG_LENGTH=3D0
=C2=A0 =C2=A0 <= span class=3D"Apple-tab-span" style=3D"white-space:pre"> WRONG_MAP= =3D0
=C2=A0 =C2=A0 WRONG_REDUCE=3D0
=C2=A0 =C2=A0 File Input Format Counte= rs=C2=A0
=C2=A0 =C2=A0 Bytes Read=3D72553170
=C2=A0 =C2=A0 File Output For= mat Counters=C2=A0
=C2=A0 =C2=A0 Bytes Written=3D2=C2=A0

Could someone help ?=C2=A0

Best regards,= =C2=A0
Mehdi
--001a113f89de5f8ab8051fbd9e75--