ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hurley" <jhur...@hortonworks.com>
Subject Re: Review Request 27396: Pushing component logs to HDFS from ambari-agent
Date Sat, 01 Nov 2014 02:34:44 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27396/#review59451
-----------------------------------------------------------



ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py
<https://reviews.apache.org/r/27396/#comment100733>

    I believe that this will create your own scheduler instance (along with its own locks,
pools, and jobstore). However, we might want to confirm this. At the very least, maybe the
alert and log schedulers should define a jobstore aside from using the default name?



ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py
<https://reviews.apache.org/r/27396/#comment100732>

    make_cachedir seems to always be True; is it supposed to be configurable?



ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py
<https://reviews.apache.org/r/27396/#comment100735>

    Why instantiate a new Scheduler here? start(self) also creates a new Scheduler. Maybe
this should set Scheduler to None and then have start(self) check for None to instantiate
one?



ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py
<https://reviews.apache.org/r/27396/#comment100736>

    I know the documentation says that you can give the job a name on creation; I wasn't able
to get it to work; I'm guessing you weren't either.



ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py
<https://reviews.apache.org/r/27396/#comment100734>

    This will produce data in ambari-agent.out - I think we'd want to keep this ambari-agent.log
by using logger.exception()
    
    Same goes for other areas where you have traceback



ambari-agent/src/main/python/ambari_agent/loghandler/config.py
<https://reviews.apache.org/r/27396/#comment100737>

    Perhaps log what component name you were looking for? Same for other areas where an Exception
is constructed due to lack of a required parameter.



ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py
<https://reviews.apache.org/r/27396/#comment100738>

    I'm not sure this command will work for all deployments. In some cases, the "hdfs" user
is the only one that can run these commands; and in security-enabled environments, that means
using keytabs.
    
    Can you verify that this command will work in the above scenarios? 
    
    Same goes for other hdfs commands.


- Jonathan Hurley


On Oct. 31, 2014, 7:50 p.m., Cabir Zounaidou wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27396/
> -----------------------------------------------------------
> 
> (Updated Oct. 31, 2014, 7:50 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, John Speidel, Mahadev
Konar, Nate Cole, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-1522
>     https://issues.apache.org/jira/browse/AMBARI-1522
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> The idea is to run a scheduler which computes diffs in a component log and pushes the
diff (only if available) to HDFS at fixed interval.
> 
> 1. Each component will have its own monitor
> 2. The apscheduler threadpool controls the monitor execution.
> 3. If the log directory is not available if the component (or service) is not running,
it will skip the processing.
> 4. It saves the last read line index and last modified time for next iteration
> 5. It uses HDFS shell utility to push log patch to HDFS.
> 6. Right now, the component log directory is configured in a json file.  In the next
iteration, it will try to automatically detected from the stack config.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/Controller.py dc3a1cf 
>   ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/__init__.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/config.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/util.py PRE-CREATION 
>   ambari-agent/src/test/python/ambari_agent/TestLogHandler.py PRE-CREATION 
>   ambari-agent/src/test/python/ambari_agent/dummy_files/log_handler_config.json PRE-CREATION

> 
> Diff: https://reviews.apache.org/r/27396/diff/
> 
> 
> Testing
> -------
> 
> - Ran the ambari-agent with the patch and verified
> 1. Started ambari-agent with the patch 
> 2. The scheduler started successfully
> 3. The logs files are getting pushed to HDFS successfully. Verified using HDFS shell
utility.
> - Ran the tests successfully
> [INFO]
> [INFO] ------------------------------------------------------------------------
> [INFO] Building Ambari Agent 1.3.0-SNAPSHOT
> [INFO] ------------------------------------------------------------------------
> [INFO]
> :
> ----------------------------------------------------------------------
> Ran 324 tests in 8.282s
> 
> OK
> :
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 14.759s
> [INFO] Finished at: Thu Oct 30 14:58:21 PDT 2014
> [INFO] Final Memory: 10M/4079M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Cabir Zounaidou
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message