beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Cwik (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2113) Apex Runner is not able to submit any job to YARN
Date Sat, 06 May 2017 16:00:05 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999476#comment-15999476
] 

Luke Cwik commented on BEAM-2113:
---------------------------------

For the examples, I think we do want some kind of binding for the user when they are executing
their application so they get local execution logs and the Runner will need to deal with it.
Even if we did something else with examples, users/libraries regularly specify their own SLF4J
logging backend so only by ensuring that the workers have the correct SLF4J backend dependencies
injected automatically earlier on the classpath can you get around this issue.



> Apex Runner is not able to submit any job to YARN
> -------------------------------------------------
>
>                 Key: BEAM-2113
>                 URL: https://issues.apache.org/jira/browse/BEAM-2113
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-apex
>    Affects Versions: 2.0.0
>         Environment: CentOS7 + HDP-2.5.5
> OpenJDK 1.8.0_131-b11
>            Reporter: Tibor Kiss
>            Assignee: Thomas Weise
>            Priority: Blocker
>             Fix For: 2.0.0
>
>
> Trying to execute the WordCount example using Apex Runner:
> {code}
> $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount -Dexec.args="--inputFile=/tmp/input/pom.xml
--output=/tmp/output/ --runner=ApexRunner --embeddedExecution=false" -Papex-runner
> ...
> ...
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ beam-examples-java ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/root/.m2/repository/org/slf4j/slf4j-jdk14/1.7.14/slf4j-jdk14-1.7.14.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/root/.m2/repository/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory]
> Apr 28, 2017 1:51:13 PM org.apache.beam.runners.apex.ApexYarnLauncher launchApp
> INFO: Executing: hadoop org.apache.beam.runners.apex.ApexYarnLauncher /tmp/beam-runner-apex1201917541012785769params
with {HADOOP_USER_CLASSPATH_FIRST=1, HADOOP_CLASSPATH=/tmp/beam-runners-apex-5810119727972429445.jar:/root/.m2/repository/org/apache/beam/beam-sdks-java-core/0.7.0-SNAPSHOT/beam-sdks-java-core-0.7.0-SNAPSHOT.j
> ar:/root/.m2/repository/org/apache/beam/beam-sdks-common-runner-api/0.7.0-SNAPSHOT/beam-sdks-common-runner-api-0.7.0-SNAPSHOT.jar:/root/.m2/repository/com/google/http-client/google-http-client-jackson/1.22.0/google-http-client-jackson-1.22.0.jar:/root/.m2/repository/com/google/http-client/google-http-client-protobuf/1.22.0/g
> oogle-http-client-protobuf-1.22.0.jar:/root/.m2/repository/com/google/protobuf/protobuf-java/3.2.0/protobuf-java-3.2.0.jar:/root/.m2/repository/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.jar:/root/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.8.8/jackson-core-2.8.8.jar:/root/.m2/repository/com/fasterxml/ja
> ckson/core/jackson-annotations/2.8.8/jackson-annotations-2.8.8.jar:/root/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.8.8/jackson-databind-2.8.8.jar:/root/.m2/repository/net/bytebuddy/byte-buddy/1.6.8/byte-buddy-1.6.8.jar:/root/.m2/repository/org/xerial/snappy/snappy-java/1.1.4-M3/snappy-java-1.1.4-M3.jar:/ro
> ot/.m2/repository/org/apache/commons/commons-compress/1.9/commons-compress-1.9.jar:/root/.m2/repository/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar:/root/.m2/repository/org/apache/beam/beam-sdks-java-extensions-gcp-core/0.7.0-SNAPSHOT/beam-sdks-java-extensions-gcp-core-0.7.0-SNAPSHOT.jar:/root/.m2/repository/c
> om/google/http-client/google-http-client-jackson2/1.22.0/google-http-client-jackson2-1.22.0.jar:/root/.m2/repository/com/google/cloud/bigdataoss/gcsio/1.4.5/gcsio-1.4.5.jar:/root/.m2/repository/com/google/apis/google-api-services-cloudresourcemanager/v1-rev6-1.22.0/google-api-services-cloudresourcemanager-v1-rev6-1.22.0.jar:
> /root/.m2/repository/com/google/apis/google-api-services-storage/v1-rev71-1.22.0/google-api-services-storage-v1-rev71-1.22.0.jar:/root/.m2/repository/org/apache/beam/beam-sdks-java-io-google-cloud-platform/0.7.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-0.7.0-SNAPSHOT.jar:/root/.m2/repository/org/apache/beam/beam-sdks
> -java-extensions-protobuf/0.7.0-SNAPSHOT/beam-sdks-java-extensions-protobuf-0.7.0-SNAPSHOT.jar:/root/.m2/repository/com/google/api/grpc/grpc-google-pubsub-v1/0.1.0/grpc-google-pubsub-v1-0.1.0.jar:/root/.m2/repository/com/google/api/grpc/grpc-google-iam-v1/0.1.0/grpc-google-iam-v1-0.1.0.jar:/root/.m2/repository/io/grpc/grpc-a
> uth/1.2.0/grpc-auth-1.2.0.jar:/root/.m2/repository/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.jar:/root/.m2/repository/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.jar:/root/.m2/repository/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.jar:/root/.m2/repository/com/google/instrumentation/ins
> trumentation-api/0.3.0/instrumentation-api-0.3.0.jar:/root/.m2/repository/io/grpc/grpc-netty/1.2.0/grpc-netty-1.2.0.jar:/root/.m2/repository/io/netty/netty-codec-http2/4.1.8.Final/netty-codec-http2-4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-codec-http/4.1.8.Final/netty-codec-http-4.1.8.Final.jar:/root/.m2/repository
> /io/netty/netty-handler-proxy/4.1.8.Final/netty-handler-proxy-4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-codec-socks/4.1.8.Final/netty-codec-socks-4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-handler/4.1.8.Final/netty-handler-4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-buffer/4.1.8.Final/netty-buffer-
> 4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-common/4.1.8.Final/netty-common-4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-transport/4.1.8.Final/netty-transport-4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-resolver/4.1.8.Final/netty-resolver-4.1.8.Final.jar:/root/.m2/repository/io/netty/netty-codec/4.1.8.
> Final/netty-codec-4.1.8.Final.jar:/root/.m2/repository/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.jar:/root/.m2/repository/io/grpc/grpc-all/1.2.0/grpc-all-1.2.0.jar:/root/.m2/repository/io/grpc/grpc-okhttp/1.2.0/grpc-okhttp-1.2.0.jar:/root/.m2/repository/com/squareup/okhttp/okhttp/2.5.0/okhttp-2.5.0.jar:/root/.m2/repository/com
> /squareup/okio/okio/1.6.0/okio-1.6.0.jar:/root/.m2/repository/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.jar:/root/.m2/repository/io/grpc/grpc-protobuf-nano/1.2.0/grpc-protobuf-nano-1.2.0.jar:/root/.m2/repository/com/google/protobuf/nano/protobuf-javanano/3.0.0-alpha-5/protobuf-javanano-3.0.0-alpha-5.jar:/root
> /.m2/repository/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.jar:/root/.m2/repository/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.jar:/root/.m2/repository/com/google/code/gson/gson/2.7/gson-2.7.jar:/root/.m2/repository/com/google/cloud/bigtable/bigtable-protos/0.9.6/bigtable-protos-0.9.6.jar:/roo
> t/.m2/repository/com/google/cloud/bigtable/bigtable-client-core/0.9.6/bigtable-client-core-0.9.6.jar:/root/.m2/repository/com/google/auth/google-auth-library-appengine/0.6.0/google-auth-library-appengine-0.6.0.jar:/root/.m2/repository/com/google/appengine/appengine-api-1.0-sdk/1.9.34/appengine-api-1.0-sdk-1.9.34.jar:/root/.m
> 2/repository/io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.jar:/root/.m2/repository/com/google/api/grpc/grpc-google-common-protos/0.1.0/grpc-google-common-protos-0.1.0.jar:/root/.m2/repository/io/netty/netty-tcnative-boringssl-static/1.1.33.Fork26/netty-tcnative-boringssl-static-1.1.33.Fork26.jar:/root/.m2/repo
> sitory/com/google/api-client/google-api-client/1.22.0/google-api-client-1.22.0.jar:/root/.m2/repository/com/google/oauth-client/google-oauth-client/1.22.0/google-oauth-client-1.22.0.jar:/root/.m2/repository/com/google/apis/google-api-services-bigquery/v2-rev295-1.22.0/google-api-services-bigquery-v2-rev295-1.22.0.jar:/root/.
> m2/repository/com/google/http-client/google-http-client/1.22.0/google-http-client-1.22.0.jar:/root/.m2/repository/com/google/cloud/bigdataoss/util/1.4.5/util-1.4.5.jar:/root/.m2/repository/com/google/api-client/google-api-client-java6/1.22.0/google-api-client-java6-1.22.0.jar:/root/.m2/repository/com/google/api-client/google
> -api-client-jackson2/1.22.0/google-api-client-jackson2-1.22.0.jar:/root/.m2/repository/com/google/oauth-client/google-oauth-client-java6/1.22.0/google-oauth-client-java6-1.22.0.jar:/root/.m2/repository/com/google/auth/google-auth-library-oauth2-http/0.6.1/google-auth-library-oauth2-http-0.6.1.jar:/root/.m2/repository/com/goo
> gle/auth/google-auth-library-credentials/0.6.1/google-auth-library-credentials-0.6.1.jar:/root/.m2/repository/org/apache/avro/avro/1.8.1/avro-1.8.1.jar:/root/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/root/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapp
> er-asl-1.9.13.jar:/root/.m2/repository/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.jar:/root/.m2/repository/org/tukaani/xz/1.5/xz-1.5.jar:/root/.m2/repository/com/google/apis/google-api-services-pubsub/v1-rev10-1.22.0/google-api-services-pubsub-v1-rev10-1.22.0.jar:/root/.m2/repository/com/google/guava/guava/20.0/g
> uava-20.0.jar:/root/.m2/repository/com/google/cloud/datastore/datastore-v1-proto-client/1.4.0/datastore-v1-proto-client-1.4.0.jar:/root/.m2/repository/com/google/cloud/datastore/datastore-v1-protos/1.3.0/datastore-v1-protos-1.3.0.jar:/root/.m2/repository/joda-time/joda-time/2.4/joda-time-2.4.jar:/root/.m2/repository/org/slf4
> j/slf4j-api/1.7.14/slf4j-api-1.7.14.jar:/root/.m2/repository/org/slf4j/slf4j-jdk14/1.7.14/slf4j-jdk14-1.7.14.jar:/root/.m2/repository/org/apache/beam/beam-runners-core-construction-java/0.7.0-SNAPSHOT/beam-runners-core-construction-java-0.7.0-SNAPSHOT.jar:/root/.m2/repository/org/apache/beam/beam-runners-core-java/0.7.0-SNAP
> SHOT/beam-runners-core-java-0.7.0-SNAPSHOT.jar:/root/.m2/repository/org/apache/beam/beam-runners-apex/0.7.0-SNAPSHOT/beam-runners-apex-0.7.0-SNAPSHOT.jar:/root/.m2/repository/org/apache/apex/apex-common/3.5.0/apex-common-3.5.0.jar:/root/.m2/repository/org/apache/apex/apex-api/3.5.0/apex-api-3.5.0.jar:/root/.m2/repository/com
> /datatorrent/netlet/1.3.0/netlet-1.3.0.jar:/root/.m2/repository/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar:/root/.m2/repository/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar:/root/.m2/repository/org/objenesis/objenesis/2.1/objenesis-2.1.jar:/root/.m2/repository/javax/validation/validation-api/1.1.0.Final/v
> alidation-api-1.1.0.Final.jar:/root/.m2/repository/com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar:/root/.m2/repository/org/apache/apex/malhar-library/3.4.0/malhar-library-3.4.0.jar:/root/.m2/repository/org/eclipse/jetty/jetty-servlet/8.1.10.v20130312/jetty-servlet-8.1.10.v20130312.jar:/root/.m2/repository/org/eclipse/jet
> ty/jetty-security/8.1.10.v20130312/jetty-security-8.1.10.v20130312.jar:/root/.m2/repository/org/eclipse/jetty/jetty-server/8.1.10.v20130312/jetty-server-8.1.10.v20130312.jar:/root/.m2/repository/org/eclipse/jetty/jetty-continuation/8.1.10.v20130312/jetty-continuation-8.1.10.v20130312.jar:/root/.m2/repository/com/sun/mail/jav
> ax.mail/1.5.0/javax.mail-1.5.0.jar:/root/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/root/.m2/repository/com/sun/jersey/jersey-client/1.9/jersey-client-1.9.jar:/root/.m2/repository/javax/jms/jms-api/1.1-rev-1/jms-api-1.1-rev-1.jar:/root/.m2/repository/org/apache/activemq/activemq-client/5.8.0/activemq-
> client-5.8.0.jar:/root/.m2/repository/org/apache/geronimo/specs/geronimo-jms_1.1_spec/1.1.1/geronimo-jms_1.1_spec-1.1.1.jar:/root/.m2/repository/org/fusesource/hawtbuf/hawtbuf/1.9/hawtbuf-1.9.jar:/root/.m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
> :/root/.m2/repository/com/github/tony19/named-regexp/0.2.3/named-regexp-0.2.3.jar:/root/.m2/repository/org/codehaus/janino/commons-compiler/2.7.8/commons-compiler-2.7.8.jar:/root/.m2/repository/org/eclipse/jetty/jetty-websocket/8.1.10.v20130312/jetty-websocket-8.1.10.v20130312.jar:/root/.m2/repository/org/eclipse/jetty/jetty
> -util/8.1.10.v20130312/jetty-util-8.1.10.v20130312.jar:/root/.m2/repository/org/eclipse/jetty/jetty-io/8.1.10.v20130312/jetty-io-8.1.10.v20130312.jar:/root/.m2/repository/org/eclipse/jetty/jetty-http/8.1.10.v20130312/jetty-http-8.1.10.v20130312.jar:/root/.m2/repository/commons-beanutils/commons-beanutils/1.8.3/commons-beanut
> ils-1.8.3.jar:/root/.m2/repository/it/unimi/dsi/fastutil/7.0.6/fastutil-7.0.6.jar:/root/.m2/repository/org/apache/apex/apex-shaded-ning19/1.0.0/apex-shaded-ning19-1.0.0.jar:/root/.m2/repository/org/apache/apex/apex-engine/3.5.0/apex-engine-3.5.0.jar:/root/.m2/repository/org/apache/bval/bval-jsr303/0.5/bval-jsr303-0.5.jar:/ro
> ot/.m2/repository/org/apache/bval/bval-core/0.5/bval-core-0.5.jar:/root/.m2/repository/org/apache/apex/apex-bufferserver/3.5.0/apex-bufferserver-3.5.0.jar:/root/.m2/repository/com/sun/jersey/contribs/jersey-apache-client4/1.9/jersey-apache-client4-1.9.jar:/root/.m2/repository/jline/jline/2.11/jline-2.11.jar:/root/.m2/reposit
> ory/org/apache/ant/ant/1.9.2/ant-1.9.2.jar:/root/.m2/repository/org/apache/ant/ant-launcher/1.9.2/ant-launcher-1.9.2.jar:/root/.m2/repository/net/engio/mbassador/1.1.9/mbassador-1.1.9.jar:/root/.m2/repository/net/lingala/zip4j/zip4j/1.3.2/zip4j-1.3.2.jar:/root/.m2/repository/commons-codec/commons-codec/1.10/commons-codec-1.1
> 0.jar:/root/.m2/repository/org/apache/xbean/xbean-asm5-shaded/4.3/xbean-asm5-shaded-4.3.jar:/root/.m2/repository/org/jctools/jctools-core/1.1/jctools-core-1.1.jar:/root/.m2/repository/org/apache/httpcomponents/httpclient/4.3.5/httpclient-4.3.5.jar:/root/.m2/repository/org/apache/httpcomponents/httpcore/4.3.2/httpcore-4.3.2.j
> ar:/root/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar}
> {code}
> The application never gets submitted to Yarn.
> If the {{org.slf4j}} dependency is replaced by {{ch.qos.logback}} then the application
submission to yarn succeeds.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message