Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 5E7BA200C18 for ; Sat, 11 Feb 2017 10:08:46 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 5D160160B5D; Sat, 11 Feb 2017 09:08:46 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A4CDE160B51 for ; Sat, 11 Feb 2017 10:08:45 +0100 (CET) Received: (qmail 93475 invoked by uid 500); 11 Feb 2017 09:08:44 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 93465 invoked by uid 99); 11 Feb 2017 09:08:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Feb 2017 09:08:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 12FA4183A61 for ; Sat, 11 Feb 2017 09:08:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1.199 X-Spam-Level: X-Spam-Status: No, score=-1.199 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id yHidOFnW-JWr for ; Sat, 11 Feb 2017 09:08:43 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id B75D95F30B for ; Sat, 11 Feb 2017 09:08:42 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id E67ACE03A2 for ; Sat, 11 Feb 2017 09:08:41 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 9E48421D64 for ; Sat, 11 Feb 2017 09:08:41 +0000 (UTC) Date: Sat, 11 Feb 2017 09:08:41 +0000 (UTC) From: "Jingsong Lee (JIRA)" To: commits@beam.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (BEAM-1393) Update Flink Runner to Flink 1.2.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Sat, 11 Feb 2017 09:08:46 -0000 [ https://issues.apache.org/jira/browse/BEAM-1393?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15862= 325#comment-15862325 ]=20 Jingsong Lee edited comment on BEAM-1393 at 2/11/17 9:08 AM: ------------------------------------------------------------- BTW, I think SPLIT_DISTRIBUTE state maybe need a new Repartitioner, not onl= y round-robin. Let each =08element in ListState have the opportunity to sel= ect a KeyGroupIndex. {{CheckpointCoordinator}} use the KeyGroupIndex to red= istribute state.=20 I understood it. {{CheckpointCoordinator}} is run in JobManager. That is to= o heavy to read every elements. Maybe Flink can abstract KeyGroup state, provide split(snapshot to several = KeyGroups) and merge(restore by several KeyGroups) methods to manage state = with KeyGroups. was (Author: lzljs3620320): BTW, I think SPLIT_DISTRIBUTE state maybe need a new Repartitioner, not onl= y round-robin. Let each =08element in ListState have the opportunity to sel= ect a KeyGroupIndex. {{CheckpointCoordinator}} use the KeyGroupIndex to red= istribute state.=20 > Update Flink Runner to Flink 1.2.0 > ---------------------------------- > > Key: BEAM-1393 > URL: https://issues.apache.org/jira/browse/BEAM-1393 > Project: Beam > Issue Type: Improvement > Components: runner-flink > Reporter: Aljoscha Krettek > Assignee: Jingsong Lee > > When we update to 1.2.0 we can use the new internal Timer API that is ava= ilable to Flink operators: {{InternalTimerService}} and also use broadcast = state to store side-input data. -- This message was sent by Atlassian JIRA (v6.3.15#6346)