From issues-return-146017-archive-asf-public=cust-asf.ponee.io@beam.apache.org Sun Feb 14 17:16:07 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id D8D3F1806DD for ; Sun, 14 Feb 2021 18:16:06 +0100 (CET) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id A73D364231 for ; Sun, 14 Feb 2021 17:16:05 +0000 (UTC) Received: (qmail 29283 invoked by uid 500); 14 Feb 2021 17:16:04 -0000 Mailing-List: contact issues-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list issues@beam.apache.org Received: (qmail 29146 invoked by uid 99); 14 Feb 2021 17:16:04 -0000 Received: from mailrelay1-he-de.apache.org (HELO mailrelay1-he-de.apache.org) (116.203.21.61) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Feb 2021 17:16:04 +0000 Received: from jira2-he-de.apache.org (unknown [IPv6:2a01:4f8:242:1f49::2]) by mailrelay1-he-de.apache.org (ASF Mail Server at mailrelay1-he-de.apache.org) with ESMTPS id 112B33E8B7 for ; Sun, 14 Feb 2021 17:16:03 +0000 (UTC) Received: from jira2-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira2-he-de.apache.org (ASF Mail Server at jira2-he-de.apache.org) with ESMTP id 95CB6C808EF for ; Sun, 14 Feb 2021 17:16:02 +0000 (UTC) Date: Sun, 14 Feb 2021 17:16:02 +0000 (UTC) From: "Beam JIRA Bot (Jira)" To: issues@beam.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (BEAM-11378) Cannot run Python PortableRunner on EMR cluster MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/BEAM-11378?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1728= 4447#comment-17284447 ]=20 Beam JIRA Bot commented on BEAM-11378: -------------------------------------- This issue was marked "stale-P2" and has not received a public comment in 1= 4 days. It is now automatically moved to P3. If you are still affected by i= t, you can comment and move it back to P2. > Cannot run Python PortableRunner on EMR cluster > ----------------------------------------------- > > Key: BEAM-11378 > URL: https://issues.apache.org/jira/browse/BEAM-11378 > Project: Beam > Issue Type: Bug > Components: runner-spark > Reporter: Ratul Ray > Priority: P3 > > I have been trying to run the python word-count example on an [AWS EMR|ht= tps://aws.amazon.com/emr/]=C2=A0cluster. And it does not work. > Things I have tried: > * Running with=C2=A0 > {code:bash} > python3 py_codes/word_count_beam.py --output word_count_output --runner= =3DSparkRunner > {code} > This results in implicitly running with {{--spark-master-url local[4]}} w= hich defeats the purpose of running it in a cluster > * Tried > {code:bash} > python3 py_codes/word_count_beam.py --output word_count_output --runner= =3DSparkRunner --spark-master-url=3Dyarn > {code} > Still uses local master. > * Could not use method described in [https://beam.apache.org/documentati= on/runners/spark/]=C2=A0under "Running on a pre-deployed Spark cluster" bec= ause in yarn master is not exposed with an URL like localhost:7077 > * Tried > {code:bash} > python3 py_codes/word_ount_beam.py --output word_count_output --runner=3D= SparkRunner --output_executable_path=3Djars/beam_word_count.jar > {code} > as described in https://issues.apache.org/jira/browse/BEAM-8970 > It can create a jar file, but when I submit the jar with spark-submit I = get docker permission denied exception. Possibly related to https://issues.= apache.org/jira/browse/BEAM-6020 > *So, no way to run a python beam code in a yarn spark cluster?* > This also means no way to run TFX code (which uses beam) in a yarn clust= er. -- This message was sent by Atlassian Jira (v8.3.4#803005)