Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5646F173F1 for ; Fri, 6 Feb 2015 05:18:43 +0000 (UTC) Received: (qmail 32973 invoked by uid 500); 6 Feb 2015 05:18:36 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 32880 invoked by uid 500); 6 Feb 2015 05:18:36 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 32870 invoked by uid 99); 6 Feb 2015 05:18:35 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Feb 2015 05:18:35 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of daemeonr@gmail.com designates 209.85.213.178 as permitted sender) Received: from [209.85.213.178] (HELO mail-ig0-f178.google.com) (209.85.213.178) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Feb 2015 05:18:30 +0000 Received: by mail-ig0-f178.google.com with SMTP id hl2so5601567igb.5 for ; Thu, 05 Feb 2015 21:17:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=FbdrCT9qzxqiL66NWHS+AwIH+XAT68ynGiEgwssdUbM=; b=y21hYp2HNuA302lqAxzZ7UedPqXHF9imSGjzSmtm5IbwGDJEYeq1vkVNSgse+BNrmJ fvguBD4Z2U3YhQxxVokmA2pp5AOb/mb8J9gLEYsn/PwgmFmY2HA0+VznoJvVz5Qohx/U acQonvVSMJcMfDuIfnu2exScAJeGJvclyVphuxKMgBPwgEQ0I5bdF9RLmczxxZuEVYTW lPAGUXInD3CU2XSZXFFDAT+TxtSosrt+aPuogMlpZXvTdRVh7rTgz9tabsfSPTCQ97x0 cUUB0hcRyDajslFi5/tQr8+78y5GFxK9bL1ryTR2yYcqedZ7QLyGgLJksYFu3UrHCAkI YZ0Q== X-Received: by 10.43.140.196 with SMTP id jb4mr812074icc.90.1423199844552; Thu, 05 Feb 2015 21:17:24 -0800 (PST) MIME-Version: 1.0 Received: by 10.50.245.44 with HTTP; Thu, 5 Feb 2015 21:16:53 -0800 (PST) In-Reply-To: References: From: daemeon reiydelle Date: Thu, 5 Feb 2015 21:16:53 -0800 Message-ID: Subject: Re: Localizer failed but it works after I restart yarn To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c1e6082dd476050e64892a X-Virus-Checked: Checked by ClamAV on apache.org --001a11c1e6082dd476050e64892a Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable My first concern is that temp is a virtual file system and cannot exceed real memory plus swap space. You may not see file system full errors flow up as the root cause of the stack trace, but you will see null pointers. So move your temp directory onto your file system (~/Documents/HDFSTMP or something) and retry. *.......* *=E2=80=9CLife should not be a journey to the grave with the intention of a= rriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of smoke,thoroughly used up, totally worn out, and loudly proclaiming =E2=80=9CWow! What a Ride!=E2=80=9D - Hunter ThompsonDaemeon C.= M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Thu, Feb 5, 2015 at 7:36 PM, Jeff Zhang wrote: > BTW, I am using hadoop 2.6 and it's a single node cluster on mac. > > > > On Fri, Feb 6, 2015 at 11:35 AM, Jeff Zhang wrote: > >> I intermittently meet this issue of Localizer failed, but after I restar= t >> the yarn, then the problem is gone. Is this a known issue ? >> >> Here's log in node manager >> >> >> 2015-02-06 11:31:24,898 INFO >> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: >> Localizer CWD set to >> /tmp/hadoop-jzhang/nm-local-dir/usercache/jzhang/appcache/application_14= 23126607079_0007 >> =3D >> file:/tmp/hadoop-jzhang/nm-local-dir/usercache/jzhang/appcache/applicati= on_1423126607079_0007 >> 2015-02-06 11:31:24,906 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.Res= ourceLocalizationService: >> Localizer failed >> java.lang.NullPointerException >> at java.util.zip.ZipFile.getZipEntry(ZipFile.java:531) >> at java.util.zip.ZipFile.access$900(ZipFile.java:56) >> at java.util.zip.ZipFile$1.nextElement(ZipFile.java:513) >> at java.util.zip.ZipFile$1.nextElement(ZipFile.java:483) >> at java.util.jar.JarFile$1.nextElement(JarFile.java:244) >> at java.util.jar.JarFile$1.nextElement(JarFile.java:239) >> at sun.misc.URLClassPath$JarLoader.validIndex(URLClassPath.java:803) >> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:937= ) >> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:848= ) >> at sun.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:81= 8) >> at sun.misc.URLClassPath.findResource(URLClassPath.java:176) >> at java.net.URLClassLoader$2.run(URLClassLoader.java:551) >> at java.net.URLClassLoader$2.run(URLClassLoader.java:549) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findResource(URLClassLoader.java:548) >> at java.lang.ClassLoader.getResource(ClassLoader.java:1147) >> at >> java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:227) >> at org.apache.xerces.parsers.SecuritySupport$6.run(Unknown Source) >> at java.security.AccessController.doPrivileged(Native Method) >> at >> org.apache.xerces.parsers.SecuritySupport.getResourceAsStream(Unknown >> Source) >> at >> org.apache.xerces.parsers.ObjectFactory.findJarServiceProvider(Unknown >> Source) >> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown >> Source) >> at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown >> Source) >> at org.apache.xerces.parsers.DOMParser.(Unknown Source) >> at org.apache.xerces.parsers.DOMParser.(Unknown Source) >> at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source) >> at >> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unk= nown >> Source) >> at >> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:240= 2) >> at >> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:23= 64) >> at >> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2281) >> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1019) >> at org.apache.hadoop.conf.Configuration.set(Configuration.java:991) >> at >> org.apache.hadoop.conf.Configuration.setStrings(Configuration.java:1791) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.Con= tainerLocalizer.initDirs(ContainerLocalizer.java:407) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.Con= tainerLocalizer.runLocalization(ContainerLocalizer.java:129) >> at >> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.start= Localizer(DefaultContainerExecutor.java:129) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.Res= ourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.ja= va:1088) >> >> -- >> Best Regards >> >> Jeff Zhang >> > > > > -- > Best Regards > > Jeff Zhang > --001a11c1e6082dd476050e64892a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
My first concern is that temp is a vir= tual file system and cannot exceed real memory plus swap space. You may not= see file system full errors flow up as the root cause of the stack trace, = but you will see null pointers. So move your temp directory onto your file = system (~/Documents/HDFSTMP or something) and retry.

<= div dir=3D"ltr">
=
.......
= =E2=80=9CLife should not be a journey to the grave with the = intention of arriving safely in a
pretty and well preserved body, but rather to skid in broadside in a cloud of smoke,
thoroughly used up, totally worn out, and loudly proclaiming =E2=80=9CWow! What a Ride!=E2=80=9D
- = Hunter Thompson

Daemeon C.M. Reiydelle
USA (+1) 415.501.0198
L= ondon (+44) (0) 20 8144 9872

On Thu, Feb 5, 2015 at 7:36 PM, Jeff Zhang <= span dir=3D"ltr"><= zjffdu@gmail.com> wrote:
BTW, I am using hadoop 2.6 and it's a single node cluste= r on mac.



On Fri,= Feb 6, 2015 at 11:35 AM, Jeff Zhang <zjffdu@gmail.com> wrote= :
I intermittently meet = this issue of Localizer failed, but after I restart the yarn, then the prob= lem is gone. Is this a known issue ?

Here's log in n= ode manager


2015-02-06 11:31:2= 4,898 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecut= or: Localizer CWD set to /tmp/hadoop-jzhang/nm-local-dir/usercache/jzhang/a= ppcache/application_1423126607079_0007 =3D file:/tmp/hadoop-jzhang/nm-local= -dir/usercache/jzhang/appcache/application_1423126607079_0007
201= 5-02-06 11:31:24,906 INFO org.apache.hadoop.yarn.server.nodemanager.contain= ermanager.localizer.ResourceLocalizationService: Localizer failed
java.lang.NullPointerException
=C2=A0 =C2=A0 at java.util.zip.Zi= pFile.getZipEntry(ZipFile.java:531)
=C2=A0 =C2=A0 at java.util.zi= p.ZipFile.access$900(ZipFile.java:56)
=C2=A0 =C2=A0 at java.util.= zip.ZipFile$1.nextElement(ZipFile.java:513)
=C2=A0 =C2=A0 at java= .util.zip.ZipFile$1.nextElement(ZipFile.java:483)
=C2=A0 =C2=A0 a= t java.util.jar.JarFile$1.nextElement(JarFile.java:244)
=C2=A0 = =C2=A0 at java.util.jar.JarFile$1.nextElement(JarFile.java:239)
= =C2=A0 =C2=A0 at sun.misc.URLClassPath$JarLoader.validIndex(URLClassPath.ja= va:803)
=C2=A0 =C2=A0 at sun.misc.URLClassPath$JarLoader.getResou= rce(URLClassPath.java:937)
=C2=A0 =C2=A0 at sun.misc.URLClassPath= $JarLoader.getResource(URLClassPath.java:848)
=C2=A0 =C2=A0 at su= n.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:818)
=C2=A0 =C2=A0 at sun.misc.URLClassPath.findResource(URLClassPath.java:176)=
=C2=A0 =C2=A0 at java.net.URLClassLoader$2.run(URLClassLoader.ja= va:551)
=C2=A0 =C2=A0 at java.net.URLClassLoader$2.run(URLClassLo= ader.java:549)
=C2=A0 =C2=A0 at java.security.AccessController.do= Privileged(Native Method)
=C2=A0 =C2=A0 at java.net.URLClassLoade= r.findResource(URLClassLoader.java:548)
=C2=A0 =C2=A0 at java.lan= g.ClassLoader.getResource(ClassLoader.java:1147)
=C2=A0 =C2=A0 at= java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:227)
=
=C2=A0 =C2=A0 at org.apache.xerces.parsers.SecuritySupport$6.run(Unkno= wn Source)
=C2=A0 =C2=A0 at java.security.AccessController.doPriv= ileged(Native Method)
=C2=A0 =C2=A0 at org.apache.xerces.parsers.= SecuritySupport.getResourceAsStream(Unknown Source)
=C2=A0 =C2=A0= at org.apache.xerces.parsers.ObjectFactory.findJarServiceProvider(Unknown = Source)
=C2=A0 =C2=A0 at org.apache.xerces.parsers.ObjectFactory.= createObject(Unknown Source)
=C2=A0 =C2=A0 at org.apache.xerces.p= arsers.ObjectFactory.createObject(Unknown Source)
=C2=A0 =C2=A0 a= t org.apache.xerces.parsers.DOMParser.<init>(Unknown Source)
=C2=A0 =C2=A0 at org.apache.xerces.parsers.DOMParser.<init>(Unknown= Source)
=C2=A0 =C2=A0 at org.apache.xerces.jaxp.DocumentBuilderI= mpl.<init>(Unknown Source)
=C2=A0 =C2=A0 at org.apache.xerc= es.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)
=
=C2=A0 =C2=A0 at org.apache.hadoop.conf.Configuration.loadResource(Con= figuration.java:2402)
=C2=A0 =C2=A0 at org.apache.hadoop.conf.Con= figuration.loadResources(Configuration.java:2364)
=C2=A0 =C2=A0 a= t org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2281)
=C2=A0 =C2=A0 at org.apache.hadoop.conf.Configuration.set(Configura= tion.java:1019)
=C2=A0 =C2=A0 at org.apache.hadoop.conf.Configura= tion.set(Configuration.java:991)
=C2=A0 =C2=A0 at org.apache.hado= op.conf.Configuration.setStrings(Configuration.java:1791)
=C2=A0 = =C2=A0 at org.apache.hadoop.yarn.server.nodemanager.containermanager.locali= zer.ContainerLocalizer.initDirs(ContainerLocalizer.java:407)
=C2= =A0 =C2=A0 at org.apache.hadoop.yarn.server.nodemanager.containermanager.lo= calizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:129)
=C2=A0 =C2=A0 at org.apache.hadoop.yarn.server.nodemanager.DefaultCo= ntainerExecutor.startLocalizer(DefaultContainerExecutor.java:129)
=C2=A0 =C2=A0 at org.apache.hadoop.yarn.server.nodemanager.containermanage= r.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocaliz= ationService.java:1088)

-= -
Best Regards

Jeff Zhang



--
Best Re= gards

Jeff Zhang

--001a11c1e6082dd476050e64892a--