Return-Path: X-Original-To: apmail-flink-dev-archive@www.apache.org Delivered-To: apmail-flink-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1D53D118CC for ; Fri, 11 Jul 2014 16:01:59 +0000 (UTC) Received: (qmail 90643 invoked by uid 500); 11 Jul 2014 16:01:59 -0000 Delivered-To: apmail-flink-dev-archive@flink.apache.org Received: (qmail 90586 invoked by uid 500); 11 Jul 2014 16:01:59 -0000 Mailing-List: contact dev-help@flink.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.incubator.apache.org Delivered-To: mailing list dev@flink.incubator.apache.org Received: (qmail 90575 invoked by uid 99); 11 Jul 2014 16:01:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jul 2014 16:01:58 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [130.149.7.33] (HELO mail.tu-berlin.de) (130.149.7.33) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jul 2014 16:01:52 +0000 X-tubIT-Incoming-IP: 209.85.192.44 Received: from mail-qg0-f44.google.com ([209.85.192.44]) by mail.tu-berlin.de (exim-4.72/mailfrontend-5) with esmtpa for id 1X5dGK-0007cL-9c; Fri, 11 Jul 2014 18:01:30 +0200 Received: by mail-qg0-f44.google.com with SMTP id j107so1123668qga.3 for ; Fri, 11 Jul 2014 09:01:28 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.140.88.103 with SMTP id s94mr47883919qgd.73.1405094488114; Fri, 11 Jul 2014 09:01:28 -0700 (PDT) Received: by 10.96.52.137 with HTTP; Fri, 11 Jul 2014 09:01:28 -0700 (PDT) In-Reply-To: References: Date: Fri, 11 Jul 2014 18:01:28 +0200 Message-ID: Subject: Re: Fixpoint api From: Kostas Tzoumas To: "dev@flink.incubator.apache.org" Content-Type: multipart/alternative; boundary=001a11c11e1cd70eaa04fded0d01 X-PMX-Version: 6.0.0.2142326, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2014.7.11.155420 X-PMX-Spam: Gauge=IIIIIII, Probability=0%, Report='' X-Virus-Checked: Checked by ClamAV on apache.org --001a11c11e1cd70eaa04fded0d01 Content-Type: text/plain; charset=UTF-8 As a first prototype, you can (using env.execute): - execute the bulk iteration as one job - grab the number of elapsed iterations using the static variable that Stephan mentioned - execute the delta iteration as a second job if needed On Fri, Jul 11, 2014 at 5:06 PM, Stephan Ewen wrote: > Hi! > > That is exactly the incremental rollout variant. > > You can do this with intermediate file writing, currently. Would that be a > way to start prototyping? For benchmarking, we can even hack a caching > output and input format that keeps the data in the task manager's memory. > > Stephan > --001a11c11e1cd70eaa04fded0d01--