samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Navina Ramesh" <nram...@linkedin.com>
Subject Re: Review Request 36006: SAMZA-724: Writing a tool to read from the coordinator stream and react to config changes accordingly.
Date Fri, 14 Aug 2015 00:19:16 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36006/#review95359
-----------------------------------------------------------

Ship it!


Tried it out. Works as expected. Thanks!

- Navina Ramesh


On Aug. 13, 2015, 11:54 p.m., Shadi A. Noghabi wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36006/
> -----------------------------------------------------------
> 
> (Updated Aug. 13, 2015, 11:54 p.m.)
> 
> 
> Review request for samza, Yi Pan (Data Infrastructure), Navina Ramesh, and Naveen Somasundaram.
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> This RBis for SAMZA-724.
> 
> After a job is submitted, it might need some configuration change, specifically it might
need more containers. In SAMZA-704 a tool is being added to write to the coordinator stream
(CoordinatorStreamWriter).  This tool can be used to write new configurations to the coordinator
stream. However, another tool (ConfigManager) is needed to read the config changes and react
to them, which is the goal of this task. This tool should be brought up after the job is submitted
and read any config changes added to the coordinator stream, and react to each accordingly.

> 
> This tool, called the Config Manager, is focusing on handling container changs by reacting
to set-config massages with key "yarn.container.count". 
> 
> The config manager is a separate standa alone module, that should be brought up separately
after the submission of a job. Therefore, you have to add two configurations to the input
config file:
> 1. yarn.rm.address= <ip of resource manager in yarn. ex: localhost >
> 2. yarn.rm.port= <the port of the resource manager http server. ex: 8088 >
> 
> The config manger will periodically poll the coordinator stream to see if there are any
new messages. This period is set to 100 ms by deafualt. However, it can be configured by adding
configManager.polling.interval=<polling interval> to the input config file. Thus, overal
the command to run the config manager along with the job would be:
> 
> 
> <path to samza deployment>/bin/run-config-manager.sh --config-factory=<config
factory> --config-path=<path to config file of a job>
> 
> 
> Diffs
> -----
> 
>   build.gradle a935088eccb3aee4fbde21275fa8e701c215a69e 
>   checkstyle/import-control.xml 24ed680785175f3cdf955602b7a813684edd813e 
>   gradle/dependency-versions.gradle fb06e8ed393d1a38abfa1a48fe5244fc7f6c7339 
>   samza-autoscaling/src/main/java/org/apache/samza/autoscaling/deployer/ConfigManager.java
PRE-CREATION 
>   samza-autoscaling/src/main/java/org/apache/samza/autoscaling/utils/YarnUtil.java PRE-CREATION

>   samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java
2277a732b9ab763edf19a0fbec288ff72b27583b 
>   samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala d7c928c7401e539a370d4e82276e7dabbce1b638

>   samza-shell/src/main/bash/run-config-manager.sh PRE-CREATION 
>   samza-test/src/test/scala/org/apache/samza/test/integration/StreamTaskTestUtil.scala
c6e994ff707af802ded57c3bc1762971892014da 
>   samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterService.scala ce88698c12c4bf6f4cf128f92d60b0b9496997d7

>   settings.gradle a8d2c885254ca3994327fda18e09c49bc9c5e830 
> 
> Diff: https://reviews.apache.org/r/36006/diff/
> 
> 
> Testing
> -------
> 
> Tested with hello samza and works properly.
> 
> 
> Thanks,
> 
> Shadi A. Noghabi
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message