Return-Path: X-Original-To: apmail-ambari-dev-archive@www.apache.org Delivered-To: apmail-ambari-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EBC01184B1 for ; Mon, 26 Oct 2015 10:46:13 +0000 (UTC) Received: (qmail 60384 invoked by uid 500); 26 Oct 2015 10:46:13 -0000 Delivered-To: apmail-ambari-dev-archive@ambari.apache.org Received: (qmail 60347 invoked by uid 500); 26 Oct 2015 10:46:13 -0000 Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ambari.apache.org Delivered-To: mailing list dev@ambari.apache.org Received: (qmail 60335 invoked by uid 99); 26 Oct 2015 10:46:13 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Oct 2015 10:46:13 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 0650B1D9C32; Mon, 26 Oct 2015 10:46:13 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============7661196513738098993==" MIME-Version: 1.0 Subject: Review Request 39648: App Timeline Server unexpectedly turn down [rarely reproduced] From: "Dmitro Lisnichenko" To: "Vitalyi Brodetskyi" Cc: "Ambari" Date: Mon, 26 Oct 2015 10:46:13 -0000 Message-ID: <20151026104613.22461.38775@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Dmitro Lisnichenko" X-ReviewGroup: Ambari X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/39648/ X-Sender: "Dmitro Lisnichenko" Reply-To: "Dmitro Lisnichenko" X-ReviewRequest-Repository: ambari --===============7661196513738098993== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/39648/ ----------------------------------------------------------- Review request for Ambari and Vitalyi Brodetskyi. Bugs: AMBARI-13558 https://issues.apache.org/jira/browse/AMBARI-13558 Repository: ambari Description ------- App Timeline Server unexpectedly turn down reproduced only for one test run {code}2015-09-29 15:16:53,742 INFO impl.MetricsConfig (MetricsConfig.java:loadFirst(112)) - loaded properties from hadoop-metrics2.properties 2015-09-29 15:16:53,931 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(377)) - Scheduled snapshot period at 60 second(s). 2015-09-29 15:16:53,931 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:start(192)) - ApplicationHistoryServer metrics system started 2015-09-29 15:16:54,185 INFO timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:serviceInit(228)) - Using leveldb path /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb 2015-09-29 15:16:54,196 INFO service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore failed in state INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource temporarily unavailable org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource temporarily unavailable at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218) at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168) at org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.serviceInit(LeveldbTimelineStore.java:229) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:103) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:161) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:171) 2015-09-29 15:16:54,199 INFO service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource temporarily unavailable org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource temporarily unavailable at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:103) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:161) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:171) Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource temporarily unavailable at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218) at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168) at org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.serviceInit(LeveldbTimelineStore.java:229) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ... 5 more 2015-09-29 15:16:54,200 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(211)) - Stopping ApplicationHistoryServer metrics system... 2015-09-29 15:16:54,201 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - ApplicationHistoryServer metrics system stopped. 2015-09-29 15:16:54,201 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(606)) - ApplicationHistoryServer metrics system shutdown complete. 2015-09-29 15:16:54,201 FATAL applicationhistoryservice.ApplicationHistoryServer (ApplicationHistoryServer.java:launchAppHistoryServer(164)) - Error starting ApplicationHistoryServer org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource temporarily unavailable at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:103) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:161) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:171) Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /grid/0/hadoop/yarn/timeline/leveldb-timeline-store.ldb/LOCK: Resource temporarily unavailable at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218) at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168) at org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.serviceInit(LeveldbTimelineStore.java:229) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ... 5 more 2015-09-29 15:16:54,203 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status -1 2015-09-29 15:16:54,206 INFO applicationhistoryservice.ApplicationHistoryServer (LogAdapter.java:info(45)) - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down ApplicationHistoryServer at ambari-us-roll-maint-sle113-4235-split3-3/172.22.122.12 ************************************************************/ 2015-09-29 15:18:30,769 INFO timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1532)) - Discarded 0 entities for timestamp 1440861510769 and earlier in 0.0 seconds 2015-09-29 15:23:30,771 INFO timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1532)) - Discarded 0 entities for timestamp 1440861810770 and earlier in 0.0 seconds {code} Diffs ----- ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 929269d ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service.py f368bd4 ambari-server/src/test/python/stacks/2.1/YARN/test_apptimelineserver.py 0e467d8 Diff: https://reviews.apache.org/r/39648/diff/ Testing ------- mvn clean test Thanks, Dmitro Lisnichenko --===============7661196513738098993==--