hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Motty Cruz <motty.c...@gmail.com>
Subject Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Date Wed, 24 Jun 2015 16:19:29 GMT
Thank you very much! Terry; issues is fixed. I bumped that up Java heap 
size to 512MB as well.

Thanks,
-Motty

On 06/24/2015 09:08 AM, Terry Siu wrote:
> Hi Motty,
>
> On your EventServer config, check what the java heap size setting is. 
> Ours was set to 128MB, which was much less than the default of 1GB. 
> After bumping up our value to 512MB, the EventServer started up fine 
> and no longer restarted. Did you have see any stderr logs for your 
> EventServer? The only ones I saw were from stdout and they did not 
> record any OOM. One of the Cloudera support folks said that I should 
> see OOM errors in the stderr logs, which is one cause of the 
> EventServer restart. Hope this helps.
>
> -Terry
>
> On Tuesday, June 23, 2015 at 10:07:29 AM UTC-7, motty cruz wrote:
>
>     yes, I did upgrade to 5.4.1:
>     *Version*: Cloudera Express 5.4.1 (#197 built by jenkins on
>     20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)
>
>     I did not enable trace level as suggest!
>
>     Please let me know if you find a solution, I had tried increasing
>     memory to that service but no successful results yet.
>
>     Thanks,
>     -Motty
>     On 06/23/2015 09:31 AM, Terry Siu wrote:
>>     Motty,
>>
>>     Did you upgrade your Cloudera Manager to 5.4.1? I was comparing
>>     your INFO message for the EventCatcherService and noticed that
>>     your Version number at the end says 5.3.1.
>>
>>
>>     On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote:
>>
>>         Bit skeptical that enabling trace would "fix" this issue, but
>>         I did and no change. Event Server keeps restarting and no
>>         errors. Posted a message on the Cloudera Manager forum to see
>>         if anybody has any other ideas.
>>
>>         On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote:
>>
>>             Hi Motty,
>>
>>             No, I haven't. The Event Server has been restarting
>>             constantly since I left it last night and the logs do not
>>             provide any errors on what might be wrong. Did you try
>>             enabling the trace level as described by Atul? I'll see
>>             if I can figure out where the setting is. This is getting
>>             annoying and I'm going to engage Cloudera Support for this.
>>
>>             -Terry
>>
>>             On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz
>>             wrote:
>>
>>                 Hello Terry,
>>                 did you find a solution to this problem? We're
>>                 experiencing this issue, no solution thus far.
>>
>>                 Thanks,
>>                 -Motty
>>
>>                 On 06/22/2015 03:12 PM, Terry Siu wrote:
>>>                 Hi guys,
>>>
>>>                 I just updated CDH from 5.3.3 to 5.4.2 and am seeing
>>>                 the same issue with the Clouder Manager EventServer
>>>                 (unexpected exits). I tailed the event server log
>>>                 and saw not ERROR logs and it looked like the
>>>                 EventServer keeps restarting itself over and over
>>>                 again. The only notable thing I've seen in the log
>>>                 is WARN level message:
>>>
>>>                 2015-06-22 15:04:10,885 WARN
>>>                 com.cloudera.cmf.event.publish.EventStorePublisherWithRetry:
>>>                 F
>>>                 ailed to publish event:
>>>                 SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER],
>>>                 CATEGORY=[LOG_MESS
>>>                 AGE],
>>>                 ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
>>>                 SEVERITY=[IMPORTANT], SERVI
>>>                 CE=[mgmt],
>>>                 HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
>>>                 SERVICE_TYPE=[MGMT], LOG_LEVEL=
>>>                 [WARN], HOSTS=[8K04.corp.pivotlink.com
>>>                 <http://8K04.corp.pivotlink.com>],
>>>                 EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC
>>>                 E is not ROLE for input role type KMS-KMS.,
>>>                 timestamp=1435010650675}
>>>
>>>                 Anyone else got this resolved/found a workaround?
>>>
>>>                 Thanks,
>>>                 -Terry
>>>
>>>                 On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty
>>>                 cruz wrote:
>>>
>>>                     Thanks for your reply Atul,
>>>                     I believe it has to do with version of Java. We
>>>                     Also updated Java to version 8 previously we
>>>                     were using java version 1.7-67.
>>>
>>>                     I have not resolve the issue yet. do you mind
>>>                     sharing the steps you taken to enable "trace".
>>>
>>>                     Thanks,
>>>                     Motty
>>>
>>>                     On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>>>>                     We had the same issue (upgrade from 5.3.3 to
>>>>                     5.4.0) - while trying to debug I enabled trace
>>>>                     and it started working correctly - it was very
>>>>                     odd, in the mean time we did not care about the
>>>>                     previous events data, hence we also whacked
>>>>                     directory (after backing it up, of course!)
>>>>                     where it stores some intermediate data.
>>>>
>>>>                     As this is not a perfect or even sensible
>>>>                     solution this may not work for you, we are
>>>>                     still investigating the real reason but since
>>>>                     we have not been able reproduce it - we are not
>>>>                     sure how to debug further.
>>>>
>>>>                     Just thought of sharing this random act that we
>>>>                     still can't believe turned out to be the
>>>>                     solution. Let us know if this works for you too.
>>>>
>>>>                     Regards,
>>>>                     Atul.
>>>>
>>>>                     On Wednesday, June 3, 2015 at 10:30:13 AM
>>>>                     UTC-7, motty cruz wrote:
>>>>
>>>>                         Hello,
>>>>                         Just finished upgrading CDH from 5.3.1 to
>>>>                         5.4.1 using
>>>>
>>>>                         http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>>>>                         <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>>>
>>>>
>>>>                         however Event health issues warnings "
>>>>                         eventserver (cloudera1)
>>>>                         <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>>>>                         Process Status, Unexpected Exits"
>>>>
>>>>                         2015-02-24 14:18:06,012 INFO
>>>>                         com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>>>>                         Starting EventCatcherService. JVM Args:
>>>>                         [-XX:+UseConcMarkSweepGC,
>>>>                         -XX:-CMSConcurrentMTEnabled,
>>>>                         -XX:+UseParNewGC,
>>>>                         -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>>>>                         -Djava.awt.headless=true,
>>>>                         -Djava.net.preferIPv4Stack=true,
>>>>                         -Xms52428800, -Xmx52428800,
>>>>                         -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>>>>                         Args: [], Version: 5.3.1 (#191 built by
>>>>                         jenkins on 20150123-2020 git:
>>>>                         b0377087cf605a686591e659eb14078923bc3c83)
>>>>
>>>>                         Any ideas? we're using Java 8
>>>>
>>>>                         Thanks,
>>>>                         Motty
>>>>
>>>>
>>>>                     -- 
>>>>
>>>>                     ---
>>>>                     You received this message because you are
>>>>                     subscribed to the Google Groups "CDH Users" group.
>>>>                     To unsubscribe from this group and stop
>>>>                     receiving emails from it, send an email to
>>>>                     cdh-user+u...@cloudera.org.
>>>>                     For more options, visit
>>>>                     https://groups.google.com/a/cloudera.org/d/optout
>>>>                     <https://groups.google.com/a/cloudera.org/d/optout>.
>>>
>>>                 -- 
>>>
>>>                 ---
>>>                 You received this message because you are subscribed
>>>                 to the Google Groups "CDH Users" group.
>>>                 To unsubscribe from this group and stop receiving
>>>                 emails from it, send an email to
>>>                 cdh-user+u...@cloudera.org.
>>>                 For more options, visit
>>>                 https://groups.google.com/a/cloudera.org/d/optout
>>>                 <https://groups.google.com/a/cloudera.org/d/optout>.
>>
>>     -- 
>>
>>     ---
>>     You received this message because you are subscribed to the
>>     Google Groups "CDH Users" group.
>>     To unsubscribe from this group and stop receiving emails from it,
>>     send an email to cdh-user+u...@cloudera.org <javascript:>.
>>     For more options, visit
>>     https://groups.google.com/a/cloudera.org/d/optout
>>     <https://groups.google.com/a/cloudera.org/d/optout>.
>
> -- 
>
> ---
> You received this message because you are subscribed to the Google 
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to cdh-user+unsubscribe@cloudera.org 
> <mailto:cdh-user+unsubscribe@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.


Mime
View raw message