Return-Path: X-Original-To: apmail-ambari-user-archive@www.apache.org Delivered-To: apmail-ambari-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8685518F15 for ; Fri, 19 Feb 2016 15:00:45 +0000 (UTC) Received: (qmail 83703 invoked by uid 500); 19 Feb 2016 15:00:45 -0000 Delivered-To: apmail-ambari-user-archive@ambari.apache.org Received: (qmail 83667 invoked by uid 500); 19 Feb 2016 15:00:45 -0000 Mailing-List: contact user-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ambari.apache.org Delivered-To: mailing list user@ambari.apache.org Received: (qmail 83655 invoked by uid 99); 19 Feb 2016 15:00:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2016 15:00:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D8E60C17F4 for ; Fri, 19 Feb 2016 15:00:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.993 X-Spam-Level: * X-Spam-Status: No, score=1.993 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RP_MATCHES_RCVD=-0.006, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id ZcVzk5Hs3yZ1 for ; Fri, 19 Feb 2016 15:00:42 +0000 (UTC) Received: from secmailbal101.susq.com (mail2.sig.com [141.162.101.28]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 4326A5FAE0 for ; Fri, 19 Feb 2016 15:00:41 +0000 (UTC) X-SIG-Rating: None X-SIG-Detect: None X-IronPort-AV: E=Sophos;i="5.22,470,1449550800"; d="scan'208,217";a="68821433" Received: from secdlpbal101.ds.susq.com ([192.168.17.73]) by secmailbal101.susq.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 19 Feb 2016 10:00:34 -0500 Received: from msgbal834.susq.com (msgbal834.balmail.susq.com [10.11.247.15]) by secdlpbal101.ds.susq.com (8.13.8/8.13.8) with ESMTP id u1JF0XL3006194 for ; Fri, 19 Feb 2016 10:00:34 -0500 Received: from xchbal502.ds.susq.com (xchbal502.ds.susq.com [10.10.151.72]) by msgbal834.susq.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id u1JF0X0g014530 for ; Fri, 19 Feb 2016 10:00:33 -0500 Received: from xchbal501.ds.susq.com (10.10.151.71) by xchbal502.ds.susq.com (10.10.151.72) with Microsoft SMTP Server (TLS) id 15.0.1076.9; Fri, 19 Feb 2016 10:00:33 -0500 Received: from xchbal501.ds.susq.com ([10.10.151.71]) by xchbal501.ds.susq.com ([10.10.151.71]) with mapi id 15.00.1076.000; Fri, 19 Feb 2016 10:00:33 -0500 From: "LaStrange, Adam" To: "user@ambari.apache.org" Subject: RE: Yarn nodemanager java.lang.OutOfMemory Thread-Topic: Yarn nodemanager java.lang.OutOfMemory Thread-Index: AdFqfLzRZrFN7tOLRkam+VduO+FSZwAGeLx8ACPgrbA= Date: Fri, 19 Feb 2016 15:00:32 +0000 Message-ID: References: <8dc945d4688c488990282f01ae4a8b2f@xchbal501.ds.susq.com> <1455832462946.20586@hortonworks.com> In-Reply-To: <1455832462946.20586@hortonworks.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.10.25.96] Content-Type: multipart/alternative; boundary="_000_ef479679171a497b97755079ebafd05axchbal501dssusqcom_" MIME-Version: 1.0 --_000_ef479679171a497b97755079ebafd05axchbal501dssusqcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable The Ambari version is 2.1.0. Thanks From: Siddharth Wagle [mailto:swagle@hortonworks.com] Sent: Thursday, February 18, 2016 4:54 PM To: user@ambari.apache.org Subject: Re: Yarn nodemanager java.lang.OutOfMemory Thanks Adam for reporting this. Could you please tell us which version of Ambari are you running? This bit = is very important to be known. We do not have previous report of this issue. I have filed an Apache Jira for this: https://issues.apache.org/jira/browse/AMBARI-15100 BR, Sid ________________________________ From: LaStrange, Adam > Sent: Thursday, February 18, 2016 10:52 AM To: user@ambari.apache.org Subject: Yarn nodemanager java.lang.OutOfMemory I have seen several instances of the Yarn nodemanager shutting down due to = this exception: 2016-02-16 18:04:24,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.jav= a:run(373)) - java.util.ConcurrentModificationException 2016-02-16 18:04:34,509 WARN impl.MetricsSystemImpl (MetricsSystemImpl.jav= a:run(373)) - java.util.ConcurrentModificationException 2016-02-16 18:04:44,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.jav= a:run(373)) - java.util.ConcurrentModificationException 2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaug= htExceptionHandler.java:uncaughtException(51)) - Thread Thread[timeline,5,m= ain] threw an Error. Shutting down now... java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.resize(HashMap.java:703) at java.util.HashMap.putVal(HashMap.java:662) at java.util.HashMap.put(HashMap.java:611) at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCac= he$TimelineMetricHolder.put(TimelineMetricsCache.java:123) at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCac= he.putTimelineMetric(TimelineMetricsCache.java:154) at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCac= he.putTimelineMetric(TimelineMetricsCache.java:177) at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSin= k.putMetrics(HadoopTimelineMetricsSink.java:195) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(Metric= sSinkAdapter.java:186) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(Metric= sSinkAdapter.java:43) at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.ja= va:87) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetrics= FromQueue(MetricsSinkAdapter.java:134) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsS= inkAdapter.java:88) 2016-02-16 18:04:59,433 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Hal= t with status -1 Message: HaltException This code is located in the ambari-metrics project. Has anyone seen this b= efore or have advice on how to proceed? Running with HDP-2.3.0.0-2557, Amb= ari Metrics 0.1.0, YARN 2.7.1.2.3. Thanks -Adam L ________________________________ IMPORTANT: The information contained in this email and/or its attachments i= s confidential. If you are not the intended recipient, please notify the se= nder immediately by reply and immediately delete this message and all its a= ttachments. Any review, use, reproduction, disclosure or dissemination of t= his message or any attachment by an unintended recipient is strictly prohib= ited. Neither this message nor any attachment is intended as or should be c= onstrued as an offer, solicitation or recommendation to buy or sell any sec= urity or other financial instrument. Neither the sender, his or her employe= r nor any of their respective affiliates makes any warranties as to the com= pleteness or accuracy of any of the information contained herein or that th= is message or any of its attachments is free of viruses. ________________________________ IMPORTANT: The information contained in this email and/or its attachments i= s confidential. If you are not the intended recipient, please notify the se= nder immediately by reply and immediately delete this message and all its a= ttachments. Any review, use, reproduction, disclosure or dissemination of t= his message or any attachment by an unintended recipient is strictly prohib= ited. Neither this message nor any attachment is intended as or should be c= onstrued as an offer, solicitation or recommendation to buy or sell any sec= urity or other financial instrument. Neither the sender, his or her employe= r nor any of their respective affiliates makes any warranties as to the com= pleteness or accuracy of any of the information contained herein or that th= is message or any of its attachments is free of viruses. --_000_ef479679171a497b97755079ebafd05axchbal501dssusqcom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

The Ambari version is = 2.1.0.  Thanks

 

From: Siddharth Wagle [mailto:swagle@hortonwo= rks.com]
Sent: Thursday, February 18, 2016 4:54 PM
To: user@ambari.apache.org
Subject: Re: Yarn nodemanager java.lang.OutOfMemory

 

T= hanks Adam for reporting this.

<= o:p> 

C= ould you please tell us which version of Ambari are you running? This bit i= s very important to be known.

<= o:p> 

W= e do not have previous report of this issue.

I= have filed an Apache Jira for this:

<= a href=3D"https://issues.apache.org/jira/browse/AMBARI-15100">https://issue= s.apache.org/jira/browse/AMBARI-15100

<= o:p> 

B= R,

S= id

<= o:p> 

From: LaStrange, Adam <Adam.LaStrange@sig.com>
Sent: Thursday, February 18, 2016 10:52 AM
To: user@ambari.apache.org=
Subject: Yarn nodemanager java.lang.OutOfMemory

 = ;

I have seen several in= stances of the Yarn nodemanager shutting down due to this exception:

 

2016-02-16 18:04:24,510 W= ARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.u= til.ConcurrentModificationException

2016-02-16 18:04:34,509 W= ARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.u= til.ConcurrentModificationException

2016-02-16 18:04:44,510 W= ARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.u= til.ConcurrentModificationException

2016-02-16 18:04:59,424 F= ATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:u= ncaughtException(51)) - Thread Thread[timeline,5,main] threw an Error.  Shutting down now...

java.lang.OutOfMemoryErro= r: Java heap space

    &= nbsp;  at java.util.HashMap.resize(HashMap.java:703)

    &= nbsp;  at java.util.HashMap.putVal(HashMap.java:662)

    &= nbsp;  at java.util.HashMap.put(HashMap.java:611)

    &= nbsp;  at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetri= csCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)

    &= nbsp;  at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetri= csCache.putTimelineMetric(TimelineMetricsCache.java:154)

    &= nbsp;  at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetri= csCache.putTimelineMetric(TimelineMetricsCache.java:177)

    &= nbsp;  at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetri= csSink.putMetrics(HadoopTimelineMetricsSink.java:195)

    &= nbsp;  at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(M= etricsSinkAdapter.java:186)=

    &= nbsp;  at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(M= etricsSinkAdapter.java:43)<= /span>

    &= nbsp;  at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQue= ue.java:87)

    &= nbsp;  at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMe= tricsFromQueue(MetricsSinkAdapter.java:134)

    &= nbsp;  at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(Met= ricsSinkAdapter.java:88)

2016-02-16 18:04:59,433 I= NFO  util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Mes= sage: HaltException<= /p>

 

 

This code is located i= n the ambari-metrics project.  Has anyone seen this before or have adv= ice on how to proceed?  Running with HDP-2.3.0.0-2557, Ambari Metrics = 0.1.0, YARN 2.7.1.2.3.  Thanks

 

-Adam L

=  


IMPORTANT: The information contained in this email and/or its attachments i= s confidential. If you are not the intended recipient, please notify the se= nder immediately by reply and immediately delete this message and all its a= ttachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an uninte= nded recipient is strictly prohibited. Neither this message nor any attachm= ent is intended as or should be construed as an offer, solicitation or reco= mmendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor= any of their respective affiliates makes any warranties as to the complete= ness or accuracy of any of the information contained herein or that this me= ssage or any of its attachments is free of viruses.<= o:p>



IMPORTANT: The information contained in this email and/or its attachments i= s confidential. If you are not the intended recipient, please notify the se= nder immediately by reply and immediately delete this message and all its a= ttachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an uninte= nded recipient is strictly prohibited. Neither this message nor any attachm= ent is intended as or should be construed as an offer, solicitation or reco= mmendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor= any of their respective affiliates makes any warranties as to the complete= ness or accuracy of any of the information contained herein or that this me= ssage or any of its attachments is free of viruses.
 --_000_ef479679171a497b97755079ebafd05axchbal501dssusqcom_--