From dev-return-18942-archive-asf-public=cust-asf.ponee.io@beam.apache.org Wed Sep 18 21:17:48 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id B9E86180634 for ; Wed, 18 Sep 2019 23:17:47 +0200 (CEST) Received: (qmail 16783 invoked by uid 500); 18 Sep 2019 21:17:47 -0000 Mailing-List: contact dev-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list dev@beam.apache.org Received: (qmail 16773 invoked by uid 99); 18 Sep 2019 21:17:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Sep 2019 21:17:46 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 23F40C8BE9 for ; Wed, 18 Sep 2019 21:17:46 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -13.701 X-Spam-Level: X-Spam-Status: No, score=-13.701 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id fhzIz5IKJojx for ; Wed, 18 Sep 2019 21:17:44 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::732; helo=mail-qk1-x732.google.com; envelope-from=chamikara@google.com; receiver= Received: from mail-qk1-x732.google.com (mail-qk1-x732.google.com [IPv6:2607:f8b0:4864:20::732]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 4AB8A7D723 for ; Wed, 18 Sep 2019 21:17:43 +0000 (UTC) Received: by mail-qk1-x732.google.com with SMTP id h126so996194qke.10 for ; Wed, 18 Sep 2019 14:17:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=/U1Ol+gb7QkKR0lsO7G8jank5niEvew/+hkUycPPohg=; b=t4BVUXrKp7VrJVWBM2lUoFdGgW7DN40xmIMY5xIzzBWW1uz/qlbhmEEWlCqIsUNZ9C Ei/wLuE5tgSiCp3OzAP6IATfESgx4kUDPaoE8EJ6UQrYxcS/meP1VCvdBfSrpyWgm6/m yO5X+ccueCM97pcu8G41oPRuhaHdbFGzSNmH+qTXEN3Kr/4/UX7IMBl2DDv6eZacE0nK LV8XETj6jvzbyMRgxLXtRXH8lT9YajAy83/D0EecRVHImKb+7CS19SI6jBIZAckoKJnl 8fKuHXnTfu+0TkgGkTHIiInmvP8Ebcdu/slxJ9uI75k/IKddIGj9jrlyRQtuSOlO7x47 SnuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=/U1Ol+gb7QkKR0lsO7G8jank5niEvew/+hkUycPPohg=; b=ssRnQDA0e5mDt5gGHT1mgYUJcJJ/60kawtqLlqfpg7adYgaDkw6WTRVwF1qj3o8zOM mspx6W5vaBHRZTSSNNr8TsSxLLjPuHfTRCJCGkA4TxiWT7Bbnkp3XuI61RtH2yRKPNB0 +vgqFy698BBA4abHAPPLL7tA5hla2/QWkRYZND4Pi9d9/y4f2CsOGL/HLi4C4xJYNfHA 4OUxaMv95MhYMA1cTZUnK4gwykV6mVbU/75DsBU/NPg9FR4Tx6sEDqHiTm1r6R3Hor/d n9Bxx0ry+qbU/ZyrCyGajFxlrVRl/2yRJiMjSPyUDWX7kt9XLAOkZiLRKNtPF8EO//iN lXkA== X-Gm-Message-State: APjAAAXJt/zN/LzzGeBUZ3jofuoshVbQ25tql6atPbubY4lldE5GUwgG pYYenONKZZnyKYf95HxVmGnLxGZcWYF+YtsOJJwBUeq+0IVWZw== X-Google-Smtp-Source: APXvYqzWp5Rc/PoLefj1z/18J4XSpc2xgr/R4vMS1Hb64xEHYd1M41mSfcMB7K9aVQ8Pj+BL/Z7/6UFXpqRNcGdD0PA= X-Received: by 2002:a37:a012:: with SMTP id j18mr6553037qke.146.1568841461034; Wed, 18 Sep 2019 14:17:41 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Chamikara Jayalath Date: Wed, 18 Sep 2019 14:17:29 -0700 Message-ID: Subject: Re: Using Beam Built-in I/O Transforms with an external framework. To: dev Content-Type: multipart/alternative; boundary="000000000000fecefe0592da5d93" --000000000000fecefe0592da5d93 Content-Type: text/plain; charset="UTF-8" Hi Pulasthi, This might be possible but I don't know if anybody has done this. API of Beam sources are no different from other Beam PTransforms and we highly recommend hiding away various implementations of source framework related abstractions in a composite transform [1]. So what you are looking for is a way to use a Beam transform in a separate system. Thanks, Cham [1] https://beam.apache.org/contribute/ptransform-style-guide/#exposing-a-ptransform-vs-something-else On Wed, Sep 18, 2019 at 1:52 PM Pulasthi Supun Wickramasinghe < pulasthi911@gmail.com> wrote: > Hi Dev's > > We have a big data processing framework named Twister2, and wanted to know > if there is any way we could leverage the I/O Transforms that are built > into Apache Beam externally. That is rather than using it in a Beam > pipeline just use them as data sources in our project. Just wanted to check > with the dev's if such an approach has been done by anyone before, so we > could get some pointers on how this could be done. Any pointers in the > right direction would be highly appreciated. > > Best Regards, > Pulasthi > > -- > Pulasthi S. Wickramasinghe > PhD Candidate | Research Assistant > School of Informatics and Computing | Digital Science Center > Indiana University, Bloomington > cell: 224-386-9035 <(224)%20386-9035> > --000000000000fecefe0592da5d93 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Pulasthi,

This might be possible but= I don't know if anybody has done this. API of Beam sources are no diff= erent from other Beam PTransforms and we highly recommend hiding away vario= us implementations of source framework related abstractions in a composite = transform [1]. So what you are looking for is a way to use a Beam transform= in a separate system.

Thanks,

On Wed, Sep 18, 2019 at 1:52 PM Pulasthi Supun Wick= ramasinghe <pulasthi911@gmail.c= om> wrote:
Hi Dev's

We have a big data proce= ssing framework named Twister2, and wanted to know if there is any way we c= ould leverage the I/O Transforms that are built into Apache Beam externally= . That is rather than using it in a Beam pipeline=C2=A0just use them as dat= a sources in our project. Just wanted to check with the dev's if such a= n approach has been done by anyone before, so we could get some pointers on= how this could be done. Any pointers in the right direction would be highl= y appreciated.

Best Regards,
Pulasthi

--
Pulasthi S. Wickramasinghe
PhD=C2=A0Candidate=C2=A0 | Res= earch Assistant
School of Informatics and Computing | Digital Science Center
Indi= ana University, Bloomington
--000000000000fecefe0592da5d93--