fluo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: debugging fluo
Date Tue, 01 Nov 2016 20:32:40 GMT

I just ran Fluo locally with Uno.  Below is some info I am seeing.
Do you see anything about RollingFileAppender in the stdout file?

$ pwd
$ ls
stderr  stdout  worker_1_host1.log
$ grep Rolling stdout
16:17:31,557 |-INFO in ch.qos.logback.core.joran.action.AppenderAction
- About to instantiate appender of type
16:17:31,574 |-INFO in
ch.qos.logback.core.rolling.FixedWindowRollingPolicy@bc1b008 - No
compression will be used
16:17:31,583 |-INFO in
ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log
file name: /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_000004/worker_1_host1
16:17:31,583 |-INFO in
ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property
is set to [/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_000004/worker_1_host1]


On Tue, Nov 1, 2016 at 4:22 PM, Meier, Caleb <Caleb.Meier@parsons.com> wrote:
> Hey Mike,
> So I’m not seeing any worker logs on my machines.  The search
> find / | grep worker
> yielded nothing.  Any ideas as to why these don't exist?  I'm not sure
> that my fluo.log.dir system property is successfully being set by the LogbackUtil class.
> Maybe this has something to do with it?
> -----Original Message-----
> From: Mike Walch [mailto:mwalch@apache.org]
> Sent: Tuesday, November 01, 2016 4:03 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: debugging fluo
> Were you able to find a worker_*.log file for each of your workers?
> Below are some tips for debugging:
> - Each YARN container should have a 'stdout' and 'stderr' file.  These files may have
helpful error messages.  Especially if a worker failed to start.  Also, any calls to System.out
and System.err in your observer will be printed to these files.
> - When running Fluo in YARN, Fluo must use Logback for logging (due to a hard requirment
by Twill). Logback is configured using /path/to/fluo/conf/logback.xml.  You should review
this configuration but the root logger is configured by default to print any message that
is the debug level or higher.
> - If you configured multiple workers, each worker will run in a different container and
have a different worker_*.log file.
> - When a worker starts up, it prints its configuration to worker_*.log.
> Make sure that you configured your observers using the property 'fluo.observer.*'
> -Mike
> On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb <Caleb.Meier@parsons.com> wrote:
>> Do you have any tips for how to make Observers log to the log files
>> found in the directory specified by 'yarn.nodemanager.log-dirs'?
>> -----Original Message-----
>> From: Mike Walch [mailto:mwalch@apache.org]
>> Sent: Tuesday, November 01, 2016 2:36 PM
>> To: dev@fluo.incubator.apache.org
>> Subject: Re: debugging fluo
>> Hi Caleb,
>> The logs for a Fluo application can be found in YARN but they are
>> tricky to find. Fluo should have better documentation on this which I will add now.
>> The easiest way to view the logs for a Fluo application is to use the
>> web interface for the YARN resource manager (
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_c
>> luster&d=CwIBaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzY
>> C2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6
>> mq5Tns5Aewd2s&s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY&e=
>> ).
>> First, click on the application ID (i.e application_*) of your Fluo
>> application and then click on the latest attempt ID (appattempt_*).
>> You should see a list of containers.  There should be a container for
>> the application master (typically container 1), a Fluo oracle
>> (typically container 2), and Fluo workers (containers 3+).  You can
>> view the log files produced by a container by clicking on its 'logs'
>> link.  Logs from Fluo observers will be in the worker_*.log file for
>> each of your worker containers.
>> If you don't want to use the YARN resource manager web interface, you
>> can also view these logs in the directory specified by
>> 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config.  This
>> method works well on one machine but on cluster your containers will
>> probably be on different machines. See the YARN documentation below
>> for more info about this property:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache.org
>> _docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xml&d
>> =CwIBaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR
>> 5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5A
>> ewd2s&s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI&e=
>> Best,
>> Mike
>> On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb <Caleb.Meier@parsons.com>
>> wrote:
>> Hello,
>> I'm attempting to debug a Fluo application and am having difficulty
>> locating the logs for my observers.  I've looked within the logs for
>> hadoop-yarn, but am not seeing any logging statements for my observers.
>> Where do observers log out of the box in a normal cloudera distribution?
>> Do I need to do something else in addition to logging to get my
>> observers to generate logs?
>> Thanks,
>> Caleb

View raw message