Mailing-List: contact yarn-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-dev@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.173 is neither permitted
 nor denied by domain of jlowe@yahoo-inc.com)
Message-ID: <527258FB.8090506@yahoo-inc.com>
Date: Thu, 31 Oct 2013 08:19:55 -0500
From: Jason Lowe <jlowe@yahoo-inc.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:24.0) Gecko/20100101 Thunderbird/24.0
MIME-Version: 1.0
To: <common-dev@hadoop.apache.org>
CC: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>,
        "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
        "yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>
Subject: Re: test-patch failing with OOM errors in javah
References: 
 <CAJs-t7Oqx++2KRvYG06pQHO8nqMO+tJ038meR2JBWtf4D_KUsA@mail.gmail.com>
 <CA+ULb+u4EHRgkacby__FOov0U_Js7UcK4SGbsC11SFp3T47yHg@mail.gmail.com>
In-Reply-To: 
 <CA+ULb+u4EHRgkacby__FOov0U_Js7UcK4SGbsC11SFp3T47yHg@mail.gmail.com>
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: quoted-printable

I don't think that OOM error below indicates it needs more heap space,=20
as it's complaining about the ability to create a new native thread. =20
That usually is caused by lack of available virtual address space or=20
hitting process ulimits.

What's most likely going on is the jenkins user is hitting a process=20
ulimit.  This can occur if processes have "leaked" from previous=20
build/test runs and are using a large number of threads, or a large=20
number of processes have leaked overall.  Could someone with access to=20
the build machines check if that is indeed the case?  If it has, bonus=20
points for indentifying the source of the leak. ;-)

Thanks!

Jason

On 10/30/2013 05:39 PM, Roman Shaposhnik wrote:
> I can take a look sometime later today. Meantime I can only
> say that I've been running into 1Gb limit in a few builds as
> of late. These days -- I just go with 2G by default.
>
> Thanks,
> Roman.
>
> On Wed, Oct 30, 2013 at 3:33 PM, Alejandro Abdelnur <tucu@cloudera.com>=
 wrote:
>> The following is happening in builds for MAPREDUCE and YARN patches.
>> I've seen the failures in hadoop5 and hadoop7 machines. I've increased=

>> Maven memory to 1GB (export MAVEN_OPTS=3D"-Xmx1024m" in the jenkins
>> jobs) but still some failures persist:
>> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4159/
>>
>> Does anybody has an idea of what may be going on?
>>
>>
>>
>> thx
>>
>>
>> [INFO] --- native-maven-plugin:1.0-alpha-7:javah (default) @ hadoop-co=
mmon ---
>> [INFO] /bin/sh -c cd
>> /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/=
hadoop-common-project/hadoop-common
>> && /home/jenkins/tools/java/latest/bin/javah -d
>> /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/=
hadoop-common-project/hadoop-common/target/native/javah
>> -classpath /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-B=
uild/trunk/hadoop-common-project/hadoop-common/target/classes:/home/jenki=
ns/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-common-=
project/hadoop-annotations/target/classes:/home/jenkins/tools/java/jdk1.6=
=2E0_26/jre/../lib/tools.jar:/home/jenkins/.m2/repository/com/google/guav=
a/guava/11.0.2/guava-11.0.2.jar:/home/jenkins/.m2/repository/com/google/c=
ode/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/home/jenkins/.m2/repository/c=
ommons-cli/commons-cli/1.2/commons-cli-1.2.jar:/home/jenkins/.m2/reposito=
ry/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:/home/jenkins=
/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/home/jenkins/.m2/repo=
sitory/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.j=
ar:/home/jenkins/.m2/repository/commons-codec/commons-codec/1.4/commons-c=
odec-1.4.jar:/home/jenkins/.m2/repository/commons-io/commons-io/2.1/commo=
ns-io-2.1.jar:/home/jenkins/.m2/repository/commons-net/commons-net/3.1/co=
mmons-net-3.1.jar:/home/jenkins/.m2/repository/javax/servlet/servlet-api/=
2.5/servlet-api-2.5.jar:/home/jenkins/.m2/repository/org/mortbay/jetty/je=
tty/6.1.26/jetty-6.1.26.jar:/home/jenkins/.m2/repository/org/mortbay/jett=
y/jetty-util/6.1.26/jetty-util-6.1.26.jar:/home/jenkins/.m2/repository/co=
m/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar:/home/jenkins/.m2/reposi=
tory/com/sun/jersey/jersey-json/1.9/jersey-json-1.9.jar:/home/jenkins/.m2=
/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/home/jen=
kins/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/home/jenkins/=
=2Em2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar=
:/home/jenkins/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.=
2.jar:/home/jenkins/.m2/repository/javax/activation/activation/1.1/activa=
tion-1.1.jar:/home/jenkins/.m2/repository/org/codehaus/jackson/jackson-ja=
xrs/1.8.8/jackson-jaxrs-1.8.8.jar:/home/jenkins/.m2/repository/org/codeha=
us/jackson/jackson-xc/1.8.8/jackson-xc-1.8.8.jar:/home/jenkins/.m2/reposi=
tory/com/sun/jersey/jersey-server/1.9/jersey-server-1.9.jar:/home/jenkins=
/.m2/repository/asm/asm/3.2/asm-3.2.jar:/home/jenkins/.m2/repository/comm=
ons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/home/jenkins=
/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/home/jenkins/.m2/rep=
ository/net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar:/home/jenkins/.=
m2/repository/commons-lang/commons-lang/2.5/commons-lang-2.5.jar:/home/je=
nkins/.m2/repository/commons-configuration/commons-configuration/1.6/comm=
ons-configuration-1.6.jar:/home/jenkins/.m2/repository/commons-collection=
s/commons-collections/3.2.1/commons-collections-3.2.1.jar:/home/jenkins/.=
m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.=
jar:/home/jenkins/.m2/repository/commons-beanutils/commons-beanutils/1.7.=
0/commons-beanutils-1.7.0.jar:/home/jenkins/.m2/repository/commons-beanut=
ils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/home/j=
enkins/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar:/home=
/jenkins/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jacks=
on-core-asl-1.8.8.jar:/home/jenkins/.m2/repository/org/codehaus/jackson/j=
ackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar:/home/jenkins/.m2/re=
pository/org/apache/avro/avro/1.7.4/avro-1.7.4.jar:/home/jenkins/.m2/repo=
sitory/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar:/home/j=
enkins/.m2/repository/org/xerial/snappy/snappy-java/1.0.4.1/snappy-java-1=
=2E0.4.1.jar:/home/jenkins/.m2/repository/com/google/protobuf/protobuf-ja=
va/2.5.0/protobuf-java-2.5.0.jar:/home/jenkins/jenkins-slave/workspace/Pr=
eCommit-MAPREDUCE-Build/trunk/hadoop-common-project/hadoop-auth/target/cl=
asses:/home/jenkins/.m2/repository/com/jcraft/jsch/0.1.42/jsch-0.1.42.jar=
:/home/jenkins/.m2/repository/org/apache/zookeeper/zookeeper/3.4.5/zookee=
per-3.4.5.jar:/home/jenkins/.m2/repository/org/apache/commons/commons-com=
press/1.4.1/commons-compress-1.4.1.jar:/home/jenkins/.m2/repository/org/t=
ukaani/xz/1.0/xz-1.0.jar
>> org.apache.hadoop.io.compress.zlib.ZlibCompressor
>> org.apache.hadoop.io.compress.zlib.ZlibDecompressor
>> org.apache.hadoop.io.compress.bzip2.Bzip2Compressor
>> org.apache.hadoop.io.compress.bzip2.Bzip2Decompressor
>> org.apache.hadoop.security.JniBasedUnixGroupsMapping
>> org.apache.hadoop.io.nativeio.NativeIO
>> org.apache.hadoop.security.JniBasedUnixGroupsNetgroupMapping
>> org.apache.hadoop.io.compress.snappy.SnappyCompressor
>> org.apache.hadoop.io.compress.snappy.SnappyDecompressor
>> org.apache.hadoop.io.compress.lz4.Lz4Compressor
>> org.apache.hadoop.io.compress.lz4.Lz4Decompressor
>> org.apache.hadoop.util.NativeCrc32
>> org.apache.hadoop.net.unix.DomainSocket
>> Error occurred during initialization of VM
>> java.lang.OutOfMemoryError: unable to create new native thread
>>
>>
>>
>> --
>> Alejandro