Return-Path: X-Original-To: apmail-samza-dev-archive@minotaur.apache.org Delivered-To: apmail-samza-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1BC4418168 for ; Fri, 14 Aug 2015 18:21:07 +0000 (UTC) Received: (qmail 44315 invoked by uid 500); 14 Aug 2015 18:21:06 -0000 Delivered-To: apmail-samza-dev-archive@samza.apache.org Received: (qmail 44264 invoked by uid 500); 14 Aug 2015 18:21:06 -0000 Mailing-List: contact dev-help@samza.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@samza.apache.org Delivered-To: mailing list dev@samza.apache.org Received: (qmail 44249 invoked by uid 99); 14 Aug 2015 18:21:06 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2015 18:21:06 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id E5436DD859; Fri, 14 Aug 2015 18:21:03 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============6051560344550040181==" MIME-Version: 1.0 Subject: Re: Review Request 36471: Autoscaling for samza (work in progress) From: "Shadi A. Noghabi" To: "Navina Ramesh" Cc: "samza" , "Shadi A. Noghabi" Date: Fri, 14 Aug 2015 18:21:03 -0000 Message-ID: <20150814182103.30461.54832@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Shadi A. Noghabi" X-ReviewGroup: Samza X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/36471/ X-Sender: "Shadi A. Noghabi" References: <20150731042417.1539.69213@reviews.apache.org> In-Reply-To: <20150731042417.1539.69213@reviews.apache.org> Reply-To: "Shadi A. Noghabi" X-ReviewRequest-Repository: samza --===============6051560344550040181== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36471/ ----------------------------------------------------------- (Updated Aug. 14, 2015, 6:20 p.m.) Review request for samza and Navina Ramesh. Changes ------- Separated config for auto-scaling, and added stat collection into csv files. Repository: samza Description ------- This work is for SAMZA-719. Currently, a fixed number of containers is assigned to a job as an input configuration parameter. However, with this design jobs can fail due to lack of enough resources (such as memory), or they can become a bottleneck in a workflow containing many jobs. While auto-scaling is much broader term, the goal of this project will be to enable a Samza job to automatically scale its containers such that there is improved job performance. Based on the design, we need a profiler, analyzer, optimizer and deployer module. -currently, all components added based on memory usage and the loop seems to work in inital test -tests not added for all components, and further testing is needed. Diffs (updated) ----- build.gradle 0852adc4e8e0c2816afd1ebf433f1af6b44852f7 checkstyle/import-control.xml 6654319392929857bb861d77763afd8a5ea7674c docs/learn/documentation/versioned/jobs/configuration-table.html ea73b40a054002a8f4b0e1fca9e271aae59148ab gradle/dependency-versions.gradle fb06e8ed393d1a38abfa1a48fe5244fc7f6c7339 samza-core/src/main/java/org/apache/samza/autoScaling/AutoScalingConfig.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/AutoScalingMode.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/AutoScalingSystem.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/analyzer/Analyzer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/analyzer/MemoryAnalyzer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/deployer/ConfigManager.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/deployer/Deployer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/deployer/RestartingJobDeployer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/metrics/MemoryMetrics.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/metrics/MetricsHistory.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/metrics/SlidingWindowMetric.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/metrics/StatCollector.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/CapacityBasedJobModel.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/Optimizer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/PerContainerMemoryOptimizer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/PerContainerOptimizer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/profiler/Profiler.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/profiler/SnapshotReporterProfiler.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/stream/AutoScalingMetricsSystemConsumer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java b1078bdf7bddd16c9ccc6559b9efd40ca5ae67bc samza-core/src/main/scala/org/apache/samza/autoScaling/AutoScalingCommandLine.scala PRE-CREATION samza-core/src/main/scala/org/apache/samza/autoScaling/stream/AutoScalingMetricsSystemFactory.scala PRE-CREATION samza-core/src/main/scala/org/apache/samza/config/AutoScalingConfigRewriter.scala PRE-CREATION samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala e4b14f4da6649eb78753ba3b3f529373b6f2dbe4 samza-core/src/main/scala/org/apache/samza/config/MetricsConfig.scala f7a11c575807c9d47e441bb8c6b0616142700d37 samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 27b2517048ad5730762506426ee7578c66181db8 samza-core/src/main/scala/org/apache/samza/container/grouper/task/GroupByContainerCountFactory.scala eca6215a1b84c6dc0a4aa1e9e0f8d9f8bff1b467 samza-core/src/main/scala/org/apache/samza/container/grouper/task/GroupByJobModel.scala PRE-CREATION samza-core/src/main/scala/org/apache/samza/container/grouper/task/GroupByjobModelFactory.scala PRE-CREATION samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala 0dbf14b8bbb8236a8cdbe037a99dec91c5e63965 samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2 samza-core/src/main/scala/org/apache/samza/metrics/JvmMetrics.scala 64daa0f0bb7c6350a57e06705aef41b781779416 samza-core/src/main/scala/org/apache/samza/metrics/reporter/MetricsSnapshotReporter.scala b6696f823e11a0c2134fc830178e47c3ee857378 samza-core/src/main/scala/org/apache/samza/metrics/reporter/MetricsSnapshotReporterFactory.scala dcf0435351cb5deffc7dc4a0cd438013939d54c0 samza-core/src/main/scala/org/apache/samza/util/Util.scala 419452c0d48d8faa84279fcf02a03e87309821d8 samza-core/src/test/java/org/apache/samza/autoScaling/AutoScalingConfigTest.java PRE-CREATION samza-core/src/test/java/org/apache/samza/autoScaling/SlidingWindowMetricTest.java PRE-CREATION samza-core/src/test/java/org/apache/samza/autoScaling/analyzer/MockAnalyzer.java PRE-CREATION samza-core/src/test/java/org/apache/samza/autoScaling/optimizer/CapacityBasedJobModelTest.java PRE-CREATION samza-core/src/test/java/org/apache/samza/autoScaling/optimizer/MemoryOptimizerTest.java PRE-CREATION samza-core/src/test/java/org/apache/samza/autoScaling/optimizer/PerContainerMemoryOptimizerTest.java PRE-CREATION samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByJobModel.scala PRE-CREATION samza-shell/src/main/bash/run-auto-scaling.sh PRE-CREATION samza-shell/src/main/bash/run-config-manager.sh PRE-CREATION samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala ea702a919348305ff95ce0b4ca1996a13aff04ec samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala af42c6a6636953a95f79837fe372e0dbd735df70 samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMaster.scala 7b7d86a43c69e72c47eaa91f68be24e0f4022891 Diff: https://reviews.apache.org/r/36471/diff/ Testing ------- tested with hello samza Thanks, Shadi A. Noghabi --===============6051560344550040181==--