From dev-return-8731-archive-asf-public=cust-asf.ponee.io@samza.apache.org Mon Jan 22 20:13:12 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 667C5180676 for ; Mon, 22 Jan 2018 20:13:12 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 54BC3160C32; Mon, 22 Jan 2018 19:13:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C228E160C4B for ; Mon, 22 Jan 2018 20:13:11 +0100 (CET) Received: (qmail 76345 invoked by uid 500); 22 Jan 2018 19:13:10 -0000 Mailing-List: contact dev-help@samza.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@samza.apache.org Delivered-To: mailing list dev@samza.apache.org Received: (qmail 76165 invoked by uid 99); 22 Jan 2018 19:13:10 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jan 2018 19:13:10 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id BC370E96DA; Mon, 22 Jan 2018 19:13:08 +0000 (UTC) From: dnishimura To: dev@samza.apache.org Reply-To: dev@samza.apache.org Message-ID: Subject: [GitHub] samza pull request #411: Move the changelog and checkpoint stream creation t... Content-Type: text/plain Date: Mon, 22 Jan 2018 19:13:08 +0000 (UTC) GitHub user dnishimura opened a pull request: https://github.com/apache/samza/pull/411 Move the changelog and checkpoint stream creation to the job coordinators. **Overview** The purpose of this PR is to consolidate the creation of the changelog and checkpoint streams into the JobCoordinators. In the current state, the changelog stream is created from the JobModelManager and the checkpoint stream is created within the OffsetManager. The issue with creating the checkpoint in the OffsetManager is that the first call happens from the first SamzaContainer that runs and each subsequent SamzaContainer run will attempt to create the checkpoint stream. **Motivations** There are three driving forces for this refactoring. The first motivation is to assign the creation of the changelog and checkpoint streams to the JobCoordinators where it is most appropriate. This was discussed in more detail with @nickpan47 . The second motivation is to have any potential failure to stream creation happen no later than during job coordination. The third motivation is to accommodate future security work to provide a robust way to set ACLs on streams. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dnishimura/samza samza-1555-move-changelog-checkpoint-creation Alternatively you can review and apply these changes as the patch at: https://github.com/apache/samza/pull/411.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #411 ---- commit 971fa91afb7f57fc8c099a7fb17e7bc96bc826c2 Author: Daniel Nishimura Date: 2018-01-16T21:20:07Z Move the responsibility of changelog and checkpoint stream creation to the job coordinators. ---- ---