From dev-return-477-archive-asf-public=cust-asf.ponee.io@heron.incubator.apache.org Sat Mar 31 03:09:38 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 86354180647 for ; Sat, 31 Mar 2018 03:09:37 +0200 (CEST) Received: (qmail 97705 invoked by uid 500); 31 Mar 2018 01:09:36 -0000 Mailing-List: contact dev-help@heron.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@heron.incubator.apache.org Delivered-To: mailing list dev@heron.incubator.apache.org Received: (qmail 97688 invoked by uid 99); 31 Mar 2018 01:09:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Mar 2018 01:09:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 81BA6C6007 for ; Sat, 31 Mar 2018 01:09:35 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id n2CGYHi4trfn for ; Sat, 31 Mar 2018 01:09:34 +0000 (UTC) Received: from mail-it0-f52.google.com (mail-it0-f52.google.com [209.85.214.52]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 9FA615F189 for ; Sat, 31 Mar 2018 01:09:33 +0000 (UTC) Received: by mail-it0-f52.google.com with SMTP id r19-v6so13350111itc.0 for ; Fri, 30 Mar 2018 18:09:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=ne2XiqectIgf6Raev6epHCG7FqEGuJTZ5cmdF1wokUs=; b=XxSvbVgrMVA9cK32ceugIjD1S5kY3YSvGbCaShH3R2HP1miUcy3oUlD1xQLhtKG0yE RiptYdNLPB3XJ7ydf8VLV29GrlbMd1JmicfxTLGkW6qa8FGxtHAVNglm19iJNVJEFXsD Qp3TyHFQhXkbTDItqK4koXY04H00Snh8cNQexXS+fJipwvjD2xhyAN/VaNAHx3KVFGnp RJo68clU78pPm1xhCttQucB+rwzrRQsXPos0w2PADN6mgdEvLGK3bqTFQVYGu+zs5LuK uTFi+qrIbh6gto9QhRJwdMM/HQBzUjyWptvHGWin1rlgraalUhf1kNhTpwcY6XBM+svc k5cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=ne2XiqectIgf6Raev6epHCG7FqEGuJTZ5cmdF1wokUs=; b=SU7Yd4m/DFyEfqSkRxBP+r5pJjkqvQc6EbgVch2+ElEo4U18DBi/aBzYivsIb0gX1Q qx77Ace9EgIRLOdj3dqftRvVAcuDuiWUEAHWePCsHEBVefF4XPkyn18OKtrw7MRHuVSx nLHYwHiqzmUjLAQfk9aNZ7FvZ8JRlcSefcWLE9ouQvU85NDpVkdnoOkmsEveVNSZyV2T AWbyUp4SUOXChbEs3r0mg9whXdg9MAy44oLI5vEAP83ptiiG6hnzW7I/Y7LPlNCj12EH C1jcqqkBvqmgeMSF7UxKZLx4aMjabRv+Vd5z/Q1oMfOE0w09CqQiIjd8tzic7VK1JT8k scag== X-Gm-Message-State: AElRT7Gegmufblo99Y/6SRJthGuRVeBK6LCWs8+HvHqiQNaPFc+D89tZ W7y5A1vHqbQmoDxtu6eiOpKFc16Op847mtBrsZIiOkCt X-Google-Smtp-Source: AIpwx4+0E1ITeeDxDmE1Ffhw6CMagFBLTkcf23e8eno8HYwP1oMPhHgWiWMbEmgz6KfDz/FcSo+xjnlnFYyQZ9E0E/E= X-Received: by 2002:a24:32c4:: with SMTP id j187-v6mr5005384ita.85.1522458572071; Fri, 30 Mar 2018 18:09:32 -0700 (PDT) MIME-Version: 1.0 Received: by 10.79.46.78 with HTTP; Fri, 30 Mar 2018 18:08:51 -0700 (PDT) From: Thomas Cooper Date: Fri, 30 Mar 2018 18:08:51 -0700 Message-ID: Subject: Stateful Updates To: dev@heron.incubator.apache.org Content-Type: multipart/alternative; boundary="0000000000005faba50568ab027c" --0000000000005faba50568ab027c Content-Type: text/plain; charset="UTF-8" Hi all, After chatting to the Heron Team at Twitter and Microsoft I have written up a proposal for giving Heron the ability to preserve state across a topology update (change in component parallelism). This is problem for components with incoming fields based connections as keys will not hash to the same task ID after an update. I have defined the problem (as I see it) and propsoed two solutions in the google doc below: https://docs.google.com/document/d/1X0pS9uwevn16nYYqrGjEvWQMWQLwti7Q0hmW6lmgyk8/edit?usp=sharing I would love to get feedback on these proposals. Feel free to blow holes in my assumptions and correct any misinterpretations I have made! Thanks, Tom Cooper W: www.tomcooper.org.uk | Twitter: @tomncooper --0000000000005faba50568ab027c--