hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Motty Cruz <motty.c...@gmail.com>
Subject Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Date Tue, 23 Jun 2015 17:07:21 GMT
yes, I did upgrade to 5.4.1:
*Version*: Cloudera Express 5.4.1 (#197 built by jenkins on 
20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)

I did not enable trace level as suggest!

Please let me know if you find a solution, I had tried increasing memory 
to that service but no successful results yet.

Thanks,
-Motty
On 06/23/2015 09:31 AM, Terry Siu wrote:
> Motty,
>
> Did you upgrade your Cloudera Manager to 5.4.1? I was comparing your 
> INFO message for the EventCatcherService and noticed that your Version 
> number at the end says 5.3.1.
>
>
> On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote:
>
>     Bit skeptical that enabling trace would "fix" this issue, but I
>     did and no change. Event Server keeps restarting and no errors.
>     Posted a message on the Cloudera Manager forum to see if anybody
>     has any other ideas.
>
>     On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote:
>
>         Hi Motty,
>
>         No, I haven't. The Event Server has been restarting constantly
>         since I left it last night and the logs do not provide any
>         errors on what might be wrong. Did you try enabling the trace
>         level as described by Atul? I'll see if I can figure out where
>         the setting is. This is getting annoying and I'm going to
>         engage Cloudera Support for this.
>
>         -Terry
>
>         On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz wrote:
>
>             Hello Terry,
>             did you find a solution to this problem? We're
>             experiencing this issue, no solution thus far.
>
>             Thanks,
>             -Motty
>
>             On 06/22/2015 03:12 PM, Terry Siu wrote:
>>             Hi guys,
>>
>>             I just updated CDH from 5.3.3 to 5.4.2 and am seeing the
>>             same issue with the Clouder Manager EventServer
>>             (unexpected exits). I tailed the event server log and saw
>>             not ERROR logs and it looked like the EventServer keeps
>>             restarting itself over and over again. The only notable
>>             thing I've seen in the log is WARN level message:
>>
>>             2015-06-22 15:04:10,885 WARN
>>             com.cloudera.cmf.event.publish.EventStorePublisherWithRetry:
>>             F
>>             ailed to publish event:
>>             SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER],
>>             CATEGORY=[LOG_MESS
>>             AGE],
>>             ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
>>             SEVERITY=[IMPORTANT], SERVI
>>             CE=[mgmt],
>>             HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
>>             SERVICE_TYPE=[MGMT], LOG_LEVEL=
>>             [WARN], HOSTS=[8K04.corp.pivotlink.com
>>             <http://8K04.corp.pivotlink.com>],
>>             EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC
>>             E is not ROLE for input role type KMS-KMS.,
>>             timestamp=1435010650675}
>>
>>             Anyone else got this resolved/found a workaround?
>>
>>             Thanks,
>>             -Terry
>>
>>             On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz
>>             wrote:
>>
>>                 Thanks for your reply Atul,
>>                 I believe it has to do with version of Java. We Also
>>                 updated Java to version 8 previously we were using
>>                 java version 1.7-67.
>>
>>                 I have not resolve the issue yet. do you mind sharing
>>                 the steps you taken to enable "trace".
>>
>>                 Thanks,
>>                 Motty
>>
>>                 On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>>>                 We had the same issue (upgrade from 5.3.3 to 5.4.0)
>>>                 - while trying to debug I enabled trace and it
>>>                 started working correctly - it was very odd, in the
>>>                 mean time we did not care about the previous events
>>>                 data, hence we also whacked directory (after backing
>>>                 it up, of course!) where it stores some intermediate
>>>                 data.
>>>
>>>                 As this is not a perfect or even sensible solution
>>>                 this may not work for you, we are still
>>>                 investigating the real reason but since we have not
>>>                 been able reproduce it - we are not sure how to
>>>                 debug further.
>>>
>>>                 Just thought of sharing this random act that we
>>>                 still can't believe turned out to be the solution.
>>>                 Let us know if this works for you too.
>>>
>>>                 Regards,
>>>                 Atul.
>>>
>>>                 On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7,
>>>                 motty cruz wrote:
>>>
>>>                     Hello,
>>>                     Just finished upgrading CDH from 5.3.1 to 5.4.1
>>>                     using
>>>
>>>                     http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>>>                     <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>>
>>>
>>>                     however Event health issues warnings "
>>>                     eventserver (cloudera1)
>>>                     <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>>>                     Process Status, Unexpected Exits"
>>>
>>>                     2015-02-24 14:18:06,012 INFO
>>>                     com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>>>                     Starting EventCatcherService. JVM Args:
>>>                     [-XX:+UseConcMarkSweepGC,
>>>                     -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC,
>>>                     -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>>>                     -Djava.awt.headless=true,
>>>                     -Djava.net.preferIPv4Stack=true, -Xms52428800,
>>>                     -Xmx52428800,
>>>                     -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>>>                     Args: [], Version: 5.3.1 (#191 built by jenkins
>>>                     on 20150123-2020 git:
>>>                     b0377087cf605a686591e659eb14078923bc3c83)
>>>
>>>                     Any ideas? we're using Java 8
>>>
>>>                     Thanks,
>>>                     Motty
>>>
>>>
>>>                 -- 
>>>
>>>                 ---
>>>                 You received this message because you are subscribed
>>>                 to the Google Groups "CDH Users" group.
>>>                 To unsubscribe from this group and stop receiving
>>>                 emails from it, send an email to
>>>                 cdh-user+u...@cloudera.org.
>>>                 For more options, visit
>>>                 https://groups.google.com/a/cloudera.org/d/optout
>>>                 <https://groups.google.com/a/cloudera.org/d/optout>.
>>
>>             -- 
>>
>>             ---
>>             You received this message because you are subscribed to
>>             the Google Groups "CDH Users" group.
>>             To unsubscribe from this group and stop receiving emails
>>             from it, send an email to cdh-user+u...@cloudera.org.
>>             For more options, visit
>>             https://groups.google.com/a/cloudera.org/d/optout
>>             <https://groups.google.com/a/cloudera.org/d/optout>.
>
> -- 
>
> ---
> You received this message because you are subscribed to the Google 
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to cdh-user+unsubscribe@cloudera.org 
> <mailto:cdh-user+unsubscribe@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.


Mime
View raw message