From dev-return-23615-archive-asf-public=cust-asf.ponee.io@beam.apache.org Mon Jun 8 19:57:55 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 523E2180647 for ; Mon, 8 Jun 2020 21:57:55 +0200 (CEST) Received: (qmail 61352 invoked by uid 500); 8 Jun 2020 19:57:54 -0000 Mailing-List: contact dev-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list dev@beam.apache.org Received: (qmail 61338 invoked by uid 99); 8 Jun 2020 19:57:54 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Jun 2020 19:57:54 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 646AC1A438C for ; Mon, 8 Jun 2020 19:57:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id PfFn-mZMZEpc for ; Mon, 8 Jun 2020 19:57:51 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.166.52; helo=mail-io1-f52.google.com; envelope-from=chadrik@gmail.com; receiver= Received: from mail-io1-f52.google.com (mail-io1-f52.google.com [209.85.166.52]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id BFDE7BBFD6 for ; Mon, 8 Jun 2020 19:57:50 +0000 (UTC) Received: by mail-io1-f52.google.com with SMTP id y5so20118932iob.12 for ; Mon, 08 Jun 2020 12:57:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=UJPOLULOe9iORE9sRWG2r7pgE6vGBjlxiRdGqWogR84=; b=sAZ6oE5MfkEIGfbgYGqpAmqjNdOr0nzxN81BKH/uzrynKfScDMKNTpo/YrOQFDxSwU hcdXJ0/tYvT/OArjQ84BMin9YLg0nHo7TkoIlMlObsTaGqqhawl9BzekmzQsaznjvfEU 5JJP2SPF+H11fSz0n35ZFBXxr3kSzMKq1kmwGQ/7mJtI04P0tF9xCYJ7aV1b74KhTM0L /lk5yDRjzkMIJepLDYYoSO52JzkzRb+wItoxMrJsw9dWQbpwLNQ1SQP/ar/Hv7r+/dU2 GBNRnIonowdORxTZBENH3/1Zj7iSXMnuZ4xF0fwhXYPmyEjyEA3tR0P80dE7JuMEn85a GTCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=UJPOLULOe9iORE9sRWG2r7pgE6vGBjlxiRdGqWogR84=; b=eYVSZEKw95KJegfrCdQWTxN9Oni82SUnoFNHVfA/XBQC+eQlG8E7/SyTolkOk61xxs 7ZB75YGf8gnEsSmJl148qHkjwMOP/WsDsb5OMydUmmC3NOJmAvEGlMBvIOJ4UX2YMXqJ iLDvV+jxfj7VFQrNqa/48Q3TOkc25+w/pOdaY8i8w7+3qHb+M7iCd4d1ykbEEkaggVZU GXYH2Pl9UYQmPPvkIwIRbpFEqGoN0WWBC+Q+Q6IKRiH38gTbXfeQJ0EyE2m3XFhs4vxw YE4IokQxa/RHuoqu+5EdulYGtKJlKf+YqnoFk7EscigpMNpVc8IgsmhRpm/ICuE7qu7h x5Nw== X-Gm-Message-State: AOAM532grTS1gG59cUjGzqWCojI2i9OdW+2ekzUPMHRefRilH54H6+Ba I7dXZLSqaGrJFZa3S20QEdAAlRDRYoQMW3BkTztmgHtvlo8= X-Google-Smtp-Source: ABdhPJx7N9HXtyuAwQ4xJDqWJY0oBegWKO0Xi38Feg7XQGxdXk5Ri4h2ykXsLyYCjGm052NKb8LzWEjxQZHBMiRFffw= X-Received: by 2002:a5e:a705:: with SMTP id b5mr23690329iod.12.1591646270000; Mon, 08 Jun 2020 12:57:50 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Chad Dombrova Date: Mon, 8 Jun 2020 12:57:38 -0700 Message-ID: Subject: Re: DataflowRunner | Cross-language To: dev Content-Type: multipart/alternative; boundary="00000000000088113a05a7980674" --00000000000088113a05a7980674 Content-Type: text/plain; charset="UTF-8" Hi all, quick followup question: > small correction. While the new runner will be available with Beam 2.21, >> the Cross-Language support will be available in 2.22. >> There will be limitations in the initial set of connectors you can use >> with Cross-Lang. But at least you will have something to test with, >> starting in 2.22 >> > > To clarify, we're not actually prohibiting any other > cross-langauge transforms being used, but Kafka is the only one that'll be > extensively tested and supported at this time. > We're currently using the Flink runner with external Java PubSubIO transforms in our python pipelines because there is no pure python option. In its non-portable past, Dataflow has had its own native implementation of PubSubIO, that got switched out at runtime, so there was no need to use external transforms there. What's the story around PubSubIO when using Dataflow + portability? If we were to switch from Flink to Dataflow, would we continue to use external Java PubSubIO transforms, or is there still some special treatment of pubsub for Portable Dataflow? -chad --00000000000088113a05a7980674 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi all,
quic= k followup question:
=C2=A0
small correction. Whi= le the new runner will be available with Beam 2.21, the Cross-Language supp= ort will be available in 2.22.
There will be limitations in the initial= set of connectors you can use with Cross-Lang. But at least you will have = something to test with, starting in 2.22

<= /div>
To clarify, we're not actually prohibiting any other cross-la= ngauge=C2=A0transforms being used, but Kafka is the only one that'll be= extensively tested and supported at this time.=C2=A0

We're currently using the Flink runner wi= th external Java PubSubIO transforms in our python pipelines because there = is no pure python option.=C2=A0 =C2=A0In its non-portable past, Dataflow ha= s had its own native implementation of PubSubIO, that got=C2=A0switched out= at runtime, so there was no need to use external transforms there.=C2=A0 W= hat's the story around PubSubIO when using Dataflow=C2=A0+ portability?= =C2=A0 If we were to switch from Flink to Dataflow, would we continue to us= e=C2=A0external Java PubSubIO transforms, or is there still some special tr= eatment of pubsub for Portable Dataflow?=C2=A0

-ch= ad

--00000000000088113a05a7980674--