ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nate Cole" <nc...@hortonworks.com>
Subject Re: Review Request 27396: Pushing component logs to HDFS from ambari-agent
Date Mon, 03 Nov 2014 20:45:21 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27396/#review59598
-----------------------------------------------------------



ambari-agent/src/main/python/ambari_agent/loghandler/config.py
<https://reviews.apache.org/r/27396/#comment100891>

    You should be using an AmbariConfig instance and create a section in that file to handle
setting all these.  AmbariConfig is the "external property file" for things like this.



ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py
<https://reviews.apache.org/r/27396/#comment100896>

    We may have an HdfsExecute common library for things like this



ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py
<https://reviews.apache.org/r/27396/#comment100889>

    This style is generally deprecated for the string.format() method.



ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py
<https://reviews.apache.org/r/27396/#comment100899>

    Use the "with" syntax.



ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py
<https://reviews.apache.org/r/27396/#comment100890>

    No error handling here can leave an open file.  Use this style and the file won't be left
hanging.
    
    with open(self.__metadata_path, 'w') as file:
      file.write(...)



ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py
<https://reviews.apache.org/r/27396/#comment100893>

    See comment about AmbariConfig



ambari-agent/src/main/python/ambari_agent/loghandler/util.py
<https://reviews.apache.org/r/27396/#comment100894>

    All you have is statics.  This doesn't need to be in a class, can just be loghandler/util
with methods in it.


- Nate Cole


On Nov. 3, 2014, 3:30 p.m., Cabir Zounaidou wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27396/
> -----------------------------------------------------------
> 
> (Updated Nov. 3, 2014, 3:30 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, John Speidel, Mahadev
Konar, Nate Cole, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-1522
>     https://issues.apache.org/jira/browse/AMBARI-1522
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> The idea is to run a scheduler which computes diffs in a component log and pushes the
diff (only if available) to HDFS at fixed interval.
> 
> 1. Each component will have its own monitor
> 2. The apscheduler threadpool controls the monitor execution.
> 3. If the log directory is not available if the component (or service) is not running,
it will skip the processing.
> 4. It saves the last read line index and last modified time for next iteration
> 5. It uses HDFS shell utility to push log patch to HDFS.
> 6. Right now, the component log directory is configured in a json file.  In the next
iteration, it will try to automatically detected from the stack config.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/Controller.py dc3a1cf 
>   ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/__init__.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/config.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/util.py PRE-CREATION 
>   ambari-agent/src/test/python/ambari_agent/TestLogHandler.py PRE-CREATION 
>   ambari-agent/src/test/python/ambari_agent/dummy_files/log_handler_config.json PRE-CREATION

> 
> Diff: https://reviews.apache.org/r/27396/diff/
> 
> 
> Testing
> -------
> 
> - Ran the ambari-agent with the patch and verified
> 1. Started ambari-agent with the patch 
> 2. The scheduler started successfully
> 3. The logs files are getting pushed to HDFS successfully. Verified using HDFS shell
utility.
> - Ran the tests successfully
> [INFO]
> [INFO] ------------------------------------------------------------------------
> [INFO] Building Ambari Agent 1.3.0-SNAPSHOT
> [INFO] ------------------------------------------------------------------------
> [INFO]
> :
> ----------------------------------------------------------------------
> Ran 324 tests in 8.282s
> 
> OK
> :
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 14.759s
> [INFO] Finished at: Thu Oct 30 14:58:21 PDT 2014
> [INFO] Final Memory: 10M/4079M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Cabir Zounaidou
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message