Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B465A200C69 for ; Sat, 6 May 2017 14:46:09 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id AAFD0160BAD; Sat, 6 May 2017 12:46:09 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A15EF160BA2 for ; Sat, 6 May 2017 14:46:08 +0200 (CEST) Received: (qmail 11931 invoked by uid 500); 6 May 2017 12:46:07 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 11918 invoked by uid 99); 6 May 2017 12:46:07 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 May 2017 12:46:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 6D892C1FB3 for ; Sat, 6 May 2017 12:46:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id AqY1lEjH2NRH for ; Sat, 6 May 2017 12:46:06 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 3756D5F397 for ; Sat, 6 May 2017 12:46:05 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 5E8D2E06BB for ; Sat, 6 May 2017 12:46:04 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 1167A21DE0 for ; Sat, 6 May 2017 12:46:04 +0000 (UTC) Date: Sat, 6 May 2017 12:46:04 +0000 (UTC) From: "Kenneth Knowles (JIRA)" To: commits@beam.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (BEAM-2113) Apex Runner is not able to submit any job to YARN MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Sat, 06 May 2017 12:46:09 -0000 [ https://issues.apache.org/jira/browse/BEAM-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999423#comment-15999423 ] Kenneth Knowles commented on BEAM-2113: --------------------------------------- Thanks, great! The config isn't quite runner-specific, because there is also the example/user's main class that should be able to choose its binding independent of what is going on wherever their pipeline is shipped. Not sure how easy that is to set up properly. I haven't worked on our poms and logging configs as much as others ([~dhalperi@google.com] and/or [~lcwik]?) so I can't speak to the details. Perhaps filing a JIRA to track that particular discussion would be helpful. > Apex Runner is not able to submit any job to YARN > ------------------------------------------------- > > Key: BEAM-2113 > URL: https://issues.apache.org/jira/browse/BEAM-2113 > Project: Beam > Issue Type: Bug > Components: runner-apex > Affects Versions: 2.0.0 > Environment: CentOS7 + HDP-2.5.5 > OpenJDK 1.8.0_131-b11 > Reporter: Tibor Kiss > Assignee: Thomas Weise > Priority: Blocker > > Trying to execute the WordCount example using Apex Runner: > {code} > $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount -Dexec.args="--inputFile=/tmp/input/pom.xml --output=/tmp/output/ --runner=ApexRunner --embeddedExecution=false" -Papex-runner > ... > ... > [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ beam-examples-java --- > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in [jar:file:/root/.m2/repository/org/slf4j/slf4j-jdk14/1.7.14/slf4j-jdk14-1.7.14.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in [jar:file:/root/.m2/repository/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory] > Apr 28, 2017 1:51:13 PM org.apache.beam.runners.apex.ApexYarnLauncher launchApp > INFO: Executing: hadoop org.apache.beam.runners.apex.ApexYarnLauncher /tmp/beam-runner-apex1201917541012785769params with {HADOOP_USER_CLASSPATH_FIRST=1, HADOOP_CLASSPATH=/tmp/beam-runners-apex-5810119727972429445.jar:/root/.m2/repository/org/apache/beam/beam-sdks-java-core/0.7.0-SNAPSHOT/beam-sdks-java-core-0.7.0-SNAPSHOT.j > ar:/root/.m2/repository/org/apache/beam/beam-sdks-common-runner-api/0.7.0-SNAPSHOT/beam-sdks-common-runner-api-0.7.0-SNAPSHOT.jar:/root/.m2/repository/com/google/http-client/google-http-client-jackson/1.22.0/google-http-client-jackson-1.22.0.jar:/root/.m2/repository/com/google/http-client/google-http-client-protobuf/1.22.0/g > oogle-http-client-protobuf-1.22.0.jar:/root/.m2/repository/com/google/protobuf/protobuf-java/3.2.0/protobuf-java-3.2.0.jar:/root/.m2/repository/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.jar:/root/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.8.8/jackson-core-2.8.8.jar:/root/.m2/repository/com/fasterxml/ja > ckson/core/jackson-annotations/2.8.8/jackson-annotations-2.8.8.jar:/root/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.8.8/jackson-databind-2.8.8.jar:/root/.m2/repository/net/bytebuddy/byte-buddy/1.6.8/byte-buddy-1.6.8.jar:/root/.m2/repository/org/xerial/snappy/snappy-java/1.1.4-M3/snappy-java-1.1.4-M3.jar:/ro > ot/.m2/repository/org/apache/commons/commons-compress/1.9/commons-compress-1.9.jar:/root/.m2/repository/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar:/root/.m2/repository/org/apache/beam/beam-sdks-java-extensions-gcp-core/0.7.0-SNAPSHOT/beam-sdks-java-extensions-gcp-core-0.7.0-SNAPSHOT.jar:/root/.m2/repository/c > om/google/http-client/google-http-client-jackson2/1.22.0/google-http-client-jackson2-1.22.0.jar:/root/.m2/repository/com/google/cloud/bigdataoss/gcsio/1.4.5/gcsio-1.4.5.jar:/root/.m2/repository/com/google/apis/google-api-services-cloudresourcemanager/v1-rev6-1.22.0/google-api-services-cloudresourcemanager-v1-rev6-1.22.0.jar: > /root/.m2/repository/com/google/apis/google-api-services-storage/v1-rev71-1.22.0/google-api-services-storage-v1-rev71-1.22.0.jar:/root/.m2/repository/org/apache/beam/beam-sdks-java-io-google-cloud-platform/0.7.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-0.7.0-SNAPSHOT.jar:/root/.m2/repository/org/apache/beam/beam-sdks > -java-extensions-protobuf/0.7.0-SNAPSHOT/beam-sdks-java-extensions-protobuf-0.7.0-SNAPSHOT.jar:/root/.m2/repository/com/google/api/grpc/grpc-google-pubsub-v1/0.1.0/grpc-google-pubsub-v1-0.1.0.jar:/root/.m2/repository/com/google/api/grpc/grpc-google-iam-v1/0.1.0/grpc-google-iam-v1-0.1.0.jar:/root/.m2/repository/io/grpc/grpc-a > uth/1.2.0/grpc-auth-1.2.0.jar:/root/.m2/repository/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.jar:/root/.m2/repository/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.jar:/root/.m2/repository/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.jar:/root/.m2/repository/com/google/instrumentation/ins > trumentation-api/0.3.0/instrumentation-api-0.3.0.jar:/root/.m2/repository/io/grpc/grpc-netty/1.2.0/grpc-netty-1.2.0.jar:/root/.m2/repository/io/netty/netty-codec-http2/4.1.8.Final/netty-codec-http2-4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-codec-http/4.1.8.Final/netty-codec-http-4.1.8.Final.jar:/root/.m2/repository > /io/netty/netty-handler-proxy/4.1.8.Final/netty-handler-proxy-4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-codec-socks/4.1.8.Final/netty-codec-socks-4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-handler/4.1.8.Final/netty-handler-4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-buffer/4.1.8.Final/netty-buffer- > 4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-common/4.1.8.Final/netty-common-4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-transport/4.1.8.Final/netty-transport-4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-resolver/4.1.8.Final/netty-resolver-4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-codec/4.1.8. > Final/netty-codec-4.1.8.Final.jar:/root/.m2/repository/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.jar:/root/.m2/repository/io/grpc/grpc-all/1.2.0/grpc-all-1.2.0.jar:/root/.m2/repository/io/grpc/grpc-okhttp/1.2.0/grpc-okhttp-1.2.0.jar:/root/.m2/repository/com/squareup/okhttp/okhttp/2.5.0/okhttp-2.5.0.jar:/root/.m2/repository/com > /squareup/okio/okio/1.6.0/okio-1.6.0.jar:/root/.m2/repository/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.jar:/root/.m2/repository/io/grpc/grpc-protobuf-nano/1.2.0/grpc-protobuf-nano-1.2.0.jar:/root/.m2/repository/com/google/protobuf/nano/protobuf-javanano/3.0.0-alpha-5/protobuf-javanano-3.0.0-alpha-5.jar:/root > /.m2/repository/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.jar:/root/.m2/repository/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.jar:/root/.m2/repository/com/google/code/gson/gson/2.7/gson-2.7.jar:/root/.m2/repository/com/google/cloud/bigtable/bigtable-protos/0.9.6/bigtable-protos-0.9.6.jar:/roo > t/.m2/repository/com/google/cloud/bigtable/bigtable-client-core/0.9.6/bigtable-client-core-0.9.6.jar:/root/.m2/repository/com/google/auth/google-auth-library-appengine/0.6.0/google-auth-library-appengine-0.6.0.jar:/root/.m2/repository/com/google/appengine/appengine-api-1.0-sdk/1.9.34/appengine-api-1.0-sdk-1.9.34.jar:/root/.m > 2/repository/io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.jar:/root/.m2/repository/com/google/api/grpc/grpc-google-common-protos/0.1.0/grpc-google-common-protos-0.1.0.jar:/root/.m2/repository/io/netty/netty-tcnative-boringssl-static/1.1.33.Fork26/netty-tcnative-boringssl-static-1.1.33.Fork26.jar:/root/.m2/repo > sitory/com/google/api-client/google-api-client/1.22.0/google-api-client-1.22.0.jar:/root/.m2/repository/com/google/oauth-client/google-oauth-client/1.22.0/google-oauth-client-1.22.0.jar:/root/.m2/repository/com/google/apis/google-api-services-bigquery/v2-rev295-1.22.0/google-api-services-bigquery-v2-rev295-1.22.0.jar:/root/. > m2/repository/com/google/http-client/google-http-client/1.22.0/google-http-client-1.22.0.jar:/root/.m2/repository/com/google/cloud/bigdataoss/util/1.4.5/util-1.4.5.jar:/root/.m2/repository/com/google/api-client/google-api-client-java6/1.22.0/google-api-client-java6-1.22.0.jar:/root/.m2/repository/com/google/api-client/google > -api-client-jackson2/1.22.0/google-api-client-jackson2-1.22.0.jar:/root/.m2/repository/com/google/oauth-client/google-oauth-client-java6/1.22.0/google-oauth-client-java6-1.22.0.jar:/root/.m2/repository/com/google/auth/google-auth-library-oauth2-http/0.6.1/google-auth-library-oauth2-http-0.6.1.jar:/root/.m2/repository/com/goo > gle/auth/google-auth-library-credentials/0.6.1/google-auth-library-credentials-0.6.1.jar:/root/.m2/repository/org/apache/avro/avro/1.8.1/avro-1.8.1.jar:/root/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/root/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapp > er-asl-1.9.13.jar:/root/.m2/repository/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.jar:/root/.m2/repository/org/tukaani/xz/1.5/xz-1.5.jar:/root/.m2/repository/com/google/apis/google-api-services-pubsub/v1-rev10-1.22.0/google-api-services-pubsub-v1-rev10-1.22.0.jar:/root/.m2/repository/com/google/guava/guava/20.0/g > uava-20.0.jar:/root/.m2/repository/com/google/cloud/datastore/datastore-v1-proto-client/1.4.0/datastore-v1-proto-client-1.4.0.jar:/root/.m2/repository/com/google/cloud/datastore/datastore-v1-protos/1.3.0/datastore-v1-protos-1.3.0.jar:/root/.m2/repository/joda-time/joda-time/2.4/joda-time-2.4.jar:/root/.m2/repository/org/slf4 > j/slf4j-api/1.7.14/slf4j-api-1.7.14.jar:/root/.m2/repository/org/slf4j/slf4j-jdk14/1.7.14/slf4j-jdk14-1.7.14.jar:/root/.m2/repository/org/apache/beam/beam-runners-core-construction-java/0.7.0-SNAPSHOT/beam-runners-core-construction-java-0.7.0-SNAPSHOT.jar:/root/.m2/repository/org/apache/beam/beam-runners-core-java/0.7.0-SNAP > SHOT/beam-runners-core-java-0.7.0-SNAPSHOT.jar:/root/.m2/repository/org/apache/beam/beam-runners-apex/0.7.0-SNAPSHOT/beam-runners-apex-0.7.0-SNAPSHOT.jar:/root/.m2/repository/org/apache/apex/apex-common/3.5.0/apex-common-3.5.0.jar:/root/.m2/repository/org/apache/apex/apex-api/3.5.0/apex-api-3.5.0.jar:/root/.m2/repository/com > /datatorrent/netlet/1.3.0/netlet-1.3.0.jar:/root/.m2/repository/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar:/root/.m2/repository/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar:/root/.m2/repository/org/objenesis/objenesis/2.1/objenesis-2.1.jar:/root/.m2/repository/javax/validation/validation-api/1.1.0.Final/v > alidation-api-1.1.0.Final.jar:/root/.m2/repository/com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar:/root/.m2/repository/org/apache/apex/malhar-library/3.4.0/malhar-library-3.4.0.jar:/root/.m2/repository/org/eclipse/jetty/jetty-servlet/8.1.10.v20130312/jetty-servlet-8.1.10.v20130312.jar:/root/.m2/repository/org/eclipse/jet > ty/jetty-security/8.1.10.v20130312/jetty-security-8.1.10.v20130312.jar:/root/.m2/repository/org/eclipse/jetty/jetty-server/8.1.10.v20130312/jetty-server-8.1.10.v20130312.jar:/root/.m2/repository/org/eclipse/jetty/jetty-continuation/8.1.10.v20130312/jetty-continuation-8.1.10.v20130312.jar:/root/.m2/repository/com/sun/mail/jav > ax.mail/1.5.0/javax.mail-1.5.0.jar:/root/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/root/.m2/repository/com/sun/jersey/jersey-client/1.9/jersey-client-1.9.jar:/root/.m2/repository/javax/jms/jms-api/1.1-rev-1/jms-api-1.1-rev-1.jar:/root/.m2/repository/org/apache/activemq/activemq-client/5.8.0/activemq- > client-5.8.0.jar:/root/.m2/repository/org/apache/geronimo/specs/geronimo-jms_1.1_spec/1.1.1/geronimo-jms_1.1_spec-1.1.1.jar:/root/.m2/repository/org/fusesource/hawtbuf/hawtbuf/1.9/hawtbuf-1.9.jar:/root/.m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar > :/root/.m2/repository/com/github/tony19/named-regexp/0.2.3/named-regexp-0.2.3.jar:/root/.m2/repository/org/codehaus/janino/commons-compiler/2.7.8/commons-compiler-2.7.8.jar:/root/.m2/repository/org/eclipse/jetty/jetty-websocket/8.1.10.v20130312/jetty-websocket-8.1.10.v20130312.jar:/root/.m2/repository/org/eclipse/jetty/jetty > -util/8.1.10.v20130312/jetty-util-8.1.10.v20130312.jar:/root/.m2/repository/org/eclipse/jetty/jetty-io/8.1.10.v20130312/jetty-io-8.1.10.v20130312.jar:/root/.m2/repository/org/eclipse/jetty/jetty-http/8.1.10.v20130312/jetty-http-8.1.10.v20130312.jar:/root/.m2/repository/commons-beanutils/commons-beanutils/1.8.3/commons-beanut > ils-1.8.3.jar:/root/.m2/repository/it/unimi/dsi/fastutil/7.0.6/fastutil-7.0.6.jar:/root/.m2/repository/org/apache/apex/apex-shaded-ning19/1.0.0/apex-shaded-ning19-1.0.0.jar:/root/.m2/repository/org/apache/apex/apex-engine/3.5.0/apex-engine-3.5.0.jar:/root/.m2/repository/org/apache/bval/bval-jsr303/0.5/bval-jsr303-0.5.jar:/ro > ot/.m2/repository/org/apache/bval/bval-core/0.5/bval-core-0.5.jar:/root/.m2/repository/org/apache/apex/apex-bufferserver/3.5.0/apex-bufferserver-3.5.0.jar:/root/.m2/repository/com/sun/jersey/contribs/jersey-apache-client4/1.9/jersey-apache-client4-1.9.jar:/root/.m2/repository/jline/jline/2.11/jline-2.11.jar:/root/.m2/reposit > ory/org/apache/ant/ant/1.9.2/ant-1.9.2.jar:/root/.m2/repository/org/apache/ant/ant-launcher/1.9.2/ant-launcher-1.9.2.jar:/root/.m2/repository/net/engio/mbassador/1.1.9/mbassador-1.1.9.jar:/root/.m2/repository/net/lingala/zip4j/zip4j/1.3.2/zip4j-1.3.2.jar:/root/.m2/repository/commons-codec/commons-codec/1.10/commons-codec-1.1 > 0.jar:/root/.m2/repository/org/apache/xbean/xbean-asm5-shaded/4.3/xbean-asm5-shaded-4.3.jar:/root/.m2/repository/org/jctools/jctools-core/1.1/jctools-core-1.1.jar:/root/.m2/repository/org/apache/httpcomponents/httpclient/4.3.5/httpclient-4.3.5.jar:/root/.m2/repository/org/apache/httpcomponents/httpcore/4.3.2/httpcore-4.3.2.j > ar:/root/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar} > {code} > The application never gets submitted to Yarn. > If the {{org.slf4j}} dependency is replaced by {{ch.qos.logback}} then the application submission to yarn succeeds. -- This message was sent by Atlassian JIRA (v6.3.15#6346)