Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5723618EC4 for ; Tue, 23 Jun 2015 17:07:46 +0000 (UTC) Received: (qmail 55891 invoked by uid 500); 23 Jun 2015 17:07:40 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 55793 invoked by uid 500); 23 Jun 2015 17:07:40 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 55783 invoked by uid 99); 23 Jun 2015 17:07:40 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Jun 2015 17:07:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id E9D59CFBA0 for ; Tue, 23 Jun 2015 17:07:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.15 X-Spam-Level: *** X-Spam-Status: No, score=3.15 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, KAM_LOTSOFHASH=0.25, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 0Kyj2meqvDqn for ; Tue, 23 Jun 2015 17:07:26 +0000 (UTC) Received: from mail-pa0-f43.google.com (mail-pa0-f43.google.com [209.85.220.43]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 8DF5227DD0 for ; Tue, 23 Jun 2015 17:07:25 +0000 (UTC) Received: by pactm7 with SMTP id tm7so10710612pac.2 for ; Tue, 23 Jun 2015 10:07:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type; bh=jAD/kWG7WR+wCUO7lwHRQEzaes0BXwGefQNyib7O3nQ=; b=DDPYI3T8tE2RtsVDNpR2OTJh8YSTT0B0DL4mRRLfw+1dYPYx/qjEv992Dm5kNa6kvz ylVBJWdJ8pQ22VXE0HigDxuciLSxFa5G8syuKOlz0KYA5TvMs8najiGXzYonhZKagJ37 Y5H67sbgz6s4mZJm0bikUGqzo3+O/QShMEepKIxRhettXdreta/MV+m43g/VsLi2FX4h tVnhNhJ7POmz4HZ6wcQWEKAYBuNo/B3raEf0ayPglD3Nzt0i4LpHJy6g8QnkGgaKzjSM QInoZEesmlD1eBzJV57NrXkVEObXDefIb8jSsembqD3TPBjJ5xXd8W9e5OveuD7oXFXO UMyQ== X-Received: by 10.68.136.169 with SMTP id qb9mr70950220pbb.47.1435079244255; Tue, 23 Jun 2015 10:07:24 -0700 (PDT) Received: from [192.168.125.221] (aswan.sscsinc.com. [199.96.38.41]) by mx.google.com with ESMTPSA id us3sm23794991pbc.94.2015.06.23.10.07.21 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Jun 2015 10:07:22 -0700 (PDT) Message-ID: <55899249.3020103@gmail.com> Date: Tue, 23 Jun 2015 10:07:21 -0700 From: Motty Cruz User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: cdh-user@cloudera.org CC: user@hadoop.apache.org Subject: Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue References: <556F2965.5070604@gmail.com> <556F399C.6070601@gmail.com> <93dabfc4-4055-4050-8c3e-7b13df475530@cloudera.org> <55705F89.40801@gmail.com> <55896C7C.3090205@gmail.com> <690181c0-6936-4618-be77-dfe5429a7648@cloudera.org> In-Reply-To: Content-Type: multipart/alternative; boundary="------------060704090203060406080700" This is a multi-part message in MIME format. --------------060704090203060406080700 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit yes, I did upgrade to 5.4.1: *Version*: Cloudera Express 5.4.1 (#197 built by jenkins on 20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736) I did not enable trace level as suggest! Please let me know if you find a solution, I had tried increasing memory to that service but no successful results yet. Thanks, -Motty On 06/23/2015 09:31 AM, Terry Siu wrote: > Motty, > > Did you upgrade your Cloudera Manager to 5.4.1? I was comparing your > INFO message for the EventCatcherService and noticed that your Version > number at the end says 5.3.1. > > > On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote: > > Bit skeptical that enabling trace would "fix" this issue, but I > did and no change. Event Server keeps restarting and no errors. > Posted a message on the Cloudera Manager forum to see if anybody > has any other ideas. > > On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote: > > Hi Motty, > > No, I haven't. The Event Server has been restarting constantly > since I left it last night and the logs do not provide any > errors on what might be wrong. Did you try enabling the trace > level as described by Atul? I'll see if I can figure out where > the setting is. This is getting annoying and I'm going to > engage Cloudera Support for this. > > -Terry > > On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz wrote: > > Hello Terry, > did you find a solution to this problem? We're > experiencing this issue, no solution thus far. > > Thanks, > -Motty > > On 06/22/2015 03:12 PM, Terry Siu wrote: >> Hi guys, >> >> I just updated CDH from 5.3.3 to 5.4.2 and am seeing the >> same issue with the Clouder Manager EventServer >> (unexpected exits). I tailed the event server log and saw >> not ERROR logs and it looked like the EventServer keeps >> restarting itself over and over again. The only notable >> thing I've seen in the log is WARN level message: >> >> 2015-06-22 15:04:10,885 WARN >> com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: >> F >> ailed to publish event: >> SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER], >> CATEGORY=[LOG_MESS >> AGE], >> ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b], >> SEVERITY=[IMPORTANT], SERVI >> CE=[mgmt], >> HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec], >> SERVICE_TYPE=[MGMT], LOG_LEVEL= >> [WARN], HOSTS=[8K04.corp.pivotlink.com >> ], >> EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC >> E is not ROLE for input role type KMS-KMS., >> timestamp=1435010650675} >> >> Anyone else got this resolved/found a workaround? >> >> Thanks, >> -Terry >> >> On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz >> wrote: >> >> Thanks for your reply Atul, >> I believe it has to do with version of Java. We Also >> updated Java to version 8 previously we were using >> java version 1.7-67. >> >> I have not resolve the issue yet. do you mind sharing >> the steps you taken to enable "trace". >> >> Thanks, >> Motty >> >> On 06/03/2015 06:18 PM, Atul Kulkarni wrote: >>> We had the same issue (upgrade from 5.3.3 to 5.4.0) >>> - while trying to debug I enabled trace and it >>> started working correctly - it was very odd, in the >>> mean time we did not care about the previous events >>> data, hence we also whacked directory (after backing >>> it up, of course!) where it stores some intermediate >>> data. >>> >>> As this is not a perfect or even sensible solution >>> this may not work for you, we are still >>> investigating the real reason but since we have not >>> been able reproduce it - we are not sure how to >>> debug further. >>> >>> Just thought of sharing this random act that we >>> still can't believe turned out to be the solution. >>> Let us know if this works for you too. >>> >>> Regards, >>> Atul. >>> >>> On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, >>> motty cruz wrote: >>> >>> Hello, >>> Just finished upgrading CDH from 5.3.1 to 5.4.1 >>> using >>> >>> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html >>> >>> >>> >>> however Event health issues warnings " >>> eventserver (cloudera1) >>> >>> Process Status, Unexpected Exits" >>> >>> 2015-02-24 14:18:06,012 INFO >>> com.cloudera.cmf.eventcatcher.server.EventCatcherService: >>> Starting EventCatcherService. JVM Args: >>> [-XX:+UseConcMarkSweepGC, >>> -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC, >>> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out, >>> -Djava.awt.headless=true, >>> -Djava.net.preferIPv4Stack=true, -Xms52428800, >>> -Xmx52428800, >>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh], >>> Args: [], Version: 5.3.1 (#191 built by jenkins >>> on 20150123-2020 git: >>> b0377087cf605a686591e659eb14078923bc3c83) >>> >>> Any ideas? we're using Java 8 >>> >>> Thanks, >>> Motty >>> >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed >>> to the Google Groups "CDH Users" group. >>> To unsubscribe from this group and stop receiving >>> emails from it, send an email to >>> cdh-user+u...@cloudera.org. >>> For more options, visit >>> https://groups.google.com/a/cloudera.org/d/optout >>> . >> >> -- >> >> --- >> You received this message because you are subscribed to >> the Google Groups "CDH Users" group. >> To unsubscribe from this group and stop receiving emails >> from it, send an email to cdh-user+u...@cloudera.org. >> For more options, visit >> https://groups.google.com/a/cloudera.org/d/optout >> . > > -- > > --- > You received this message because you are subscribed to the Google > Groups "CDH Users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to cdh-user+unsubscribe@cloudera.org > . > For more options, visit https://groups.google.com/a/cloudera.org/d/optout. --------------060704090203060406080700 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit yes, I did upgrade to 5.4.1:
Version: Cloudera Express 5.4.1 (#197 built by jenkins on 20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)

I did not enable trace level as suggest!

Please let me know if you find a solution, I had tried increasing memory to that service but no successful results yet.

Thanks,
-Motty
On 06/23/2015 09:31 AM, Terry Siu wrote:
Motty,

Did you upgrade your Cloudera Manager to 5.4.1? I was comparing your INFO message for the EventCatcherService and noticed that your Version number at the end says 5.3.1.


On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote:
Bit skeptical that enabling trace would "fix" this issue, but I did and no change. Event Server keeps restarting and no errors. Posted a message on the Cloudera Manager forum to see if anybody has any other ideas.

On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote:
Hi Motty,

No, I haven't. The Event Server has been restarting constantly since I left it last night and the logs do not provide any errors on what might be wrong. Did you try enabling the trace level as described by Atul? I'll see if I can figure out where the setting is. This is getting annoying and I'm going to engage Cloudera Support for this.

-Terry

On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz wrote:
Hello Terry,
did you find a solution to this problem? We're experiencing this issue, no solution thus far.

Thanks,
-Motty

On 06/22/2015 03:12 PM, Terry Siu wrote:
Hi guys,

I just updated CDH from 5.3.3 to 5.4.2 and am seeing the same issue with the Clouder Manager EventServer (unexpected exits). I tailed the event server log and saw not ERROR logs and it looked like the EventServer keeps restarting itself over and over again. The only notable thing I've seen in the log is WARN level message:

2015-06-22 15:04:10,885 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: F
ailed to publish event: SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER], CATEGORY=[LOG_MESS
AGE], ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b], SEVERITY=[IMPORTANT], SERVI
CE=[mgmt], HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec], SERVICE_TYPE=[MGMT], LOG_LEVEL=
[WARN], HOSTS=[8K04.corp.pivotlink.com], EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC
E is not ROLE for input role type KMS-KMS., timestamp=1435010650675}

Anyone else got this resolved/found a workaround?

Thanks,
-Terry

On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz wrote:
Thanks for your reply Atul,
I believe it has to do with version of Java. We Also updated Java to version 8 previously we were using java version 1.7-67.

I have not resolve the issue yet. do you mind sharing the steps you taken to enable "trace".

Thanks,
Motty

On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
We had the same issue (upgrade from 5.3.3 to 5.4.0) - while trying to debug I enabled trace and it started working correctly - it was very odd, in the mean time we did not care about the previous events data, hence we also whacked directory (after backing it up, of course!) where it stores some intermediate data. 

As this is not a perfect or even sensible solution this may not work for you, we are still investigating the real reason but since we have not been able reproduce it - we are not sure how to debug further.

Just thought of sharing this random act that we still can't believe turned out to be the solution. Let us know if this works for you too.

Regards,
Atul.

On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote:
Hello,
Just finished upgrading CDH from 5.3.1 to 5.4.1 using

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html

 
however Event health issues warnings " eventserver (cloudera1) Process Status, Unexpected Exits"

2015-02-24 14:18:06,012 INFO com.cloudera.cmf.eventcatcher.server.EventCatcherService: Starting EventCatcherService. JVM Args: [-XX:+UseConcMarkSweepGC, -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC, -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out, -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true, -Xms52428800, -Xmx52428800, -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh], Args: [], Version: 5.3.1 (#191 built by jenkins on 20150123-2020 git: b0377087cf605a686591e659eb14078923bc3c83)

Any ideas? we're using Java 8

Thanks,
Motty


--

---
You received this message because you are subscribed to the Google Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+u...@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+u...@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

--------------060704090203060406080700--