fluo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Walch <mwa...@apache.org>
Subject Re: debugging fluo
Date Tue, 01 Nov 2016 20:27:14 GMT
Yes.  Keith, you are right. Logging config changes need to be made to
/path/to/fluo/apps/$APPNAME/conf/logback.xml

On Tue, Nov 1, 2016 at 4:21 PM Keith Turner <keith@deenlo.com> wrote:

> On Tue, Nov 1, 2016 at 4:02 PM, Mike Walch <mwalch@apache.org> wrote:
> > Were you able to find a worker_*.log file for each of your workers?
> >
> > Below are some tips for debugging:
> >
> > - Each YARN container should have a 'stdout' and 'stderr' file.  These
> > files may have helpful error messages.  Especially if a worker failed to
> > start.  Also, any calls to System.out and System.err in your observer
> will
> > be printed to these files.
> > - When running Fluo in YARN, Fluo must use Logback for logging (due to a
> > hard requirment by Twill). Logback is configured using
> > /path/to/fluo/conf/logback.xml.  You should review this configuration but
> > the root logger is configured by default to print any message that is the
> > debug level or higher.
>
> Each application has its own config.  That config gets copied from
> /path/to/fluo/conf/ to /path/to/fluo/apps/$APPNAME/conf when the Fluo
> app is created.  So wouldn't logging config changes need to be made to
> /path/to/fluo/apps/$APPNAME/conf/logback.xml?
>
>
> > - If you configured multiple workers, each worker will run in a different
> > container and have a different worker_*.log file.
> > - When a worker starts up, it prints its configuration to worker_*.log.
> > Make sure that you configured your observers using the property
> > 'fluo.observer.*'
> >
> > -Mike
> >
> > On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb <Caleb.Meier@parsons.com>
> wrote:
> >
> >> Do you have any tips for how to make Observers log to the log files
> found
> >> in the directory specified by 'yarn.nodemanager.log-dirs'?
> >>
> >> -----Original Message-----
> >> From: Mike Walch [mailto:mwalch@apache.org]
> >> Sent: Tuesday, November 01, 2016 2:36 PM
> >> To: dev@fluo.incubator.apache.org
> >> Subject: Re: debugging fluo
> >>
> >> Hi Caleb,
> >>
> >> The logs for a Fluo application can be found in YARN but they are tricky
> >> to find. Fluo should have better documentation on this which I will add
> now.
> >>
> >> The easiest way to view the logs for a Fluo application is to use the
> web
> >> interface for the YARN resource manager (
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_cluster&d=CwIBaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5Aewd2s&s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY&e=
> >> ).
> >> First, click on the application ID (i.e application_*) of your Fluo
> >> application and then click on the latest attempt ID (appattempt_*). You
> >> should see a list of containers.  There should be a container for the
> >> application master (typically container 1), a Fluo oracle (typically
> >> container 2), and Fluo workers (containers 3+).  You can view the log
> files
> >> produced by a container by clicking on its 'logs' link.  Logs from Fluo
> >> observers will be in the worker_*.log file for each of your worker
> >> containers.
> >>
> >> If you don't want to use the YARN resource manager web interface, you
> can
> >> also view these logs in the directory specified by
> >> 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config.  This method
> >> works well on one machine but on cluster your containers will probably
> be
> >> on different machines. See the YARN documentation below for more info
> about
> >> this property:
> >>
> >>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache.org_docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xml&d=CwIBaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5Aewd2s&s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI&e=
> >>
> >> Best,
> >> Mike
> >>
> >> On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb <Caleb.Meier@parsons.com>
> >> wrote:
> >>
> >> Hello,
> >>
> >> I'm attempting to debug a Fluo application and am having difficulty
> >> locating the logs for my observers.  I've looked within the logs for
> >> hadoop-yarn, but am not seeing any logging statements for my observers.
> >> Where do observers log out of the box in a normal cloudera distribution?
> >> Do I need to do something else in addition to logging to get my
> observers
> >> to generate logs?
> >>
> >> Thanks,
> >> Caleb
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message