hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Motty Cruz <motty.c...@gmail.com>
Subject Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Date Tue, 23 Jun 2015 14:26:04 GMT
Hello Terry,
did you find a solution to this problem? We're experiencing this issue, 
no solution thus far.

Thanks,
-Motty

On 06/22/2015 03:12 PM, Terry Siu wrote:
> Hi guys,
>
> I just updated CDH from 5.3.3 to 5.4.2 and am seeing the same issue 
> with the Clouder Manager EventServer (unexpected exits). I tailed the 
> event server log and saw not ERROR logs and it looked like the 
> EventServer keeps restarting itself over and over again. The only 
> notable thing I've seen in the log is WARN level message:
>
> 2015-06-22 15:04:10,885 WARN 
> com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: F
> ailed to publish event: 
> SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER], CATEGORY=[LOG_MESS
> AGE], ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b], 
> SEVERITY=[IMPORTANT], SERVI
> CE=[mgmt], HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec], 
> SERVICE_TYPE=[MGMT], LOG_LEVEL=
> [WARN], HOSTS=[8K04.corp.pivotlink.com], EVENTCODE=[EV_LOG_EVENT]}, 
> content=Category SERVIC
> E is not ROLE for input role type KMS-KMS., timestamp=1435010650675}
>
> Anyone else got this resolved/found a workaround?
>
> Thanks,
> -Terry
>
> On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz wrote:
>
>     Thanks for your reply Atul,
>     I believe it has to do with version of Java. We Also updated Java
>     to version 8 previously we were using java version 1.7-67.
>
>     I have not resolve the issue yet. do you mind sharing the steps
>     you taken to enable "trace".
>
>     Thanks,
>     Motty
>
>     On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>>     We had the same issue (upgrade from 5.3.3 to 5.4.0) - while
>>     trying to debug I enabled trace and it started working correctly
>>     - it was very odd, in the mean time we did not care about the
>>     previous events data, hence we also whacked directory (after
>>     backing it up, of course!) where it stores some intermediate data.
>>
>>     As this is not a perfect or even sensible solution this may not
>>     work for you, we are still investigating the real reason but
>>     since we have not been able reproduce it - we are not sure how to
>>     debug further.
>>
>>     Just thought of sharing this random act that we still can't
>>     believe turned out to be the solution. Let us know if this works
>>     for you too.
>>
>>     Regards,
>>     Atul.
>>
>>     On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote:
>>
>>         Hello,
>>         Just finished upgrading CDH from 5.3.1 to 5.4.1 using
>>
>>         http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>>         <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>
>>
>>         however Event health issues warnings " eventserver
>>         (cloudera1)
>>         <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>>         Process Status, Unexpected Exits"
>>
>>         2015-02-24 14:18:06,012 INFO
>>         com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>>         Starting EventCatcherService. JVM Args:
>>         [-XX:+UseConcMarkSweepGC, -XX:-CMSConcurrentMTEnabled,
>>         -XX:+UseParNewGC,
>>         -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>>         -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true,
>>         -Xms52428800, -Xmx52428800,
>>         -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>>         Args: [], Version: 5.3.1 (#191 built by jenkins on
>>         20150123-2020 git: b0377087cf605a686591e659eb14078923bc3c83)
>>
>>         Any ideas? we're using Java 8
>>
>>         Thanks,
>>         Motty
>>
>>
>>     -- 
>>
>>     ---
>>     You received this message because you are subscribed to the
>>     Google Groups "CDH Users" group.
>>     To unsubscribe from this group and stop receiving emails from it,
>>     send an email to cdh-user+u...@cloudera.org <javascript:>.
>>     For more options, visit
>>     https://groups.google.com/a/cloudera.org/d/optout
>>     <https://groups.google.com/a/cloudera.org/d/optout>.
>
> -- 
>
> ---
> You received this message because you are subscribed to the Google 
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to cdh-user+unsubscribe@cloudera.org 
> <mailto:cdh-user+unsubscribe@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.


Mime
View raw message