From issues-return-190211-archive-asf-public=cust-asf.ponee.io@spark.apache.org Thu Apr 26 00:54:09 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 3C83F180679 for ; Thu, 26 Apr 2018 00:54:09 +0200 (CEST) Received: (qmail 31455 invoked by uid 500); 25 Apr 2018 22:54:08 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 31444 invoked by uid 99); 25 Apr 2018 22:54:08 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Apr 2018 22:54:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id CD34218030A for ; Wed, 25 Apr 2018 22:54:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.311 X-Spam-Level: X-Spam-Status: No, score=-110.311 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id KxrfzWb9Iiqw for ; Wed, 25 Apr 2018 22:54:06 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 678B25FB9C for ; Wed, 25 Apr 2018 22:54:06 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 931DBE031B for ; Wed, 25 Apr 2018 22:54:05 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 272C1241D2 for ; Wed, 25 Apr 2018 22:54:00 +0000 (UTC) Date: Wed, 25 Apr 2018 22:54:00 +0000 (UTC) From: "Jungtaek Lim (JIRA)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (SPARK-24036) Stateful operators in continuous processing MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/SPARK-24036?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D164= 53209#comment-16453209 ]=20 Jungtaek Lim commented on SPARK-24036: -------------------------------------- Maybe better to share what I've observed from continuous mode so far. * It leverages iterator hack to make logical batch (epoch) in stream. ** While iterator works different from normal,=C2=A0it=C2=A0doesn't touch = existing operators by=C2=A0putting=C2=A0assumption that all operators are c= hained and fit=C2=A0to=C2=A0single stage. ** With this assumption, only=C2=A0WriteToContinuousDataSourceExec needs t= o know how to=C2=A0deal with=C2=A0iterator hack. ** Above assumption requires no repartition, which=C2=A0most of stateful o= perators=C2=A0need to deal with. * Based on the hack, actually it=C2=A0doesn't=C2=A0put epoch marker flow t= hrough downstreams. ** To apply distributed snapshot it is mandatory, but it might require=C2= =A0non-trivial=C2=A0change of existing model, since checkpoint should be ha= ndled from each stateful operator and stored in distributed manner,=C2=A0an= d coordinator should be able to check snapshots from all tasks=C2=A0are tak= en correctly. ** This would be unnecessary change for batch, and making existing model b= eing much complicated. ** This=C2=A0would=C2=A0bring latency concerns, since=C2=A0each operator s= hould stop processing while taking a snapshot. (I guess sending or storing = snapshot still could be done asynchronously.) ** If there're more than one upstreams, it should arrange sequences betwee= n upstreams to take a snapshot with only proper data within epoch. So=C2=A0there is a huge challenge=C2=A0with existing model to extend contin= uous mode to support stateful exactly-once (not=C2=A0about end-to-end exact= ly once, since it also depends on sink), and I'd like to see the follow-up = idea/design doc around=C2=A0continuous mode to see the direction of continu= ous mode: whether relying on such assumption and try to=C2=A0explore=C2=A0(= may need to have more hacks/workarounds), or willing to discard assumption = and redesign. Most of features are supported with micro-batch manner, so also would like = to see the goal of continuous mode. Is it to cover all or most of features = being supported with micro-batch? Or is the goal of continuous mode only to= cover low latency use cases? =C2=A0 > Stateful operators in continuous processing > ------------------------------------------- > > Key: SPARK-24036 > URL: https://issues.apache.org/jira/browse/SPARK-24036 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 2.4.0 > Reporter: Jose Torres > Priority: Major > > The=C2=A0first iteration of continuous processing in Spark 2.3 does not w= ork with stateful operators. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org