beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-3287) Go SDK support for portable pipelines
Date Tue, 20 Mar 2018 01:25:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-3287?focusedWorklogId=82116&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82116
]

ASF GitHub Bot logged work on BEAM-3287:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 20/Mar/18 01:24
            Start Date: 20/Mar/18 01:24
    Worklog Time Spent: 10m 
      Work Description: herohde commented on issue #4888: [BEAM-3287] Add Go support for universal
runners, incl Flink
URL: https://github.com/apache/beam/pull/4888#issuecomment-374440447
 
 
   @aljoscha Thanks! Yes! That is how I'm testing this stuff. I'm currently hitting an issue
with the artifacts on the Flink side:
   
   ```
   $ go run examples/wordcount/wordcount.go --runner=flink --output=/tmp/foo --endpoint=localhost:3000
   [{6: KV<string,int>/GW/KV<bytes,int[varintz]>}]
   [{10: KV<int,string>/GW/KV<int[varintz],bytes>}]
   2018/03/19 18:19:09 No container image specified. Using dev image: 'herohde-docker-apache.bintray.io/beam/go:latest'
   2018/03/19 18:19:09 Prepared job with id: go-job-1521508749032988000_2100233503
   2018/03/19 18:19:09 Cross-compiling /Users/herohde/go/src/github.com/apache/beam/sdks/go/examples/wordcount/wordcount.go
as /var/folders/s2/97strbs55_353t_t7r24yf94009w4s/T/beam-go-1521508749041446000
   2018/03/19 18:19:25 Staged binary artifact with token: /private/tmp/beam-artifact-staging/go-job-1521508749032988000_2100233503
   2018/03/19 18:19:25 Submitted job: go-job-1521508749032988000_2100233503_-2090543569
   2018/03/19 18:19:25 Job state: RUNNING
   2018/03/19 18:19:25 Job state: FAILED
   2018/03/19 18:19:25 Failed to execute job: job go-job-1521508749032988000_2100233503_-2090543569
failed
   exit status 1
   
   ```
   
   Flink runner logs (info):
   ```
   [...]
   [grpc-default-executor-1] INFO org.apache.beam.runners.flink.FlinkRunner - PipelineOptions.filesToStage
was not specified. Defaulting to files from the classpath: will stage 1 files. Enable logging
at DEBUG level to see which files will be staged.
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkRunner - Executing pipeline
using FlinkRunner.
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkRunner - Translating
pipeline to Flink program.
   [grpc-default-executor-2] WARN org.apache.beam.runners.flink.FlinkJobInvocation - addMessageObserver()
not yet implemented.
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment
- Using portability layer
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment
- Creating the required Batch Execution Environment.
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkBatchPipelineTranslator
-  enterCompositeTransform- 
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkBatchPipelineTranslator
- |    visitPrimitiveTransform- Impulse
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkBatchPipelineTranslator
- |    visitPrimitiveTransform- 
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkBatchPipelineTranslator
- |    visitPrimitiveTransform- CoGBK
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkBatchPipelineTranslator
- |    visitPrimitiveTransform- 
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkBatchPipelineTranslator
- |    visitPrimitiveTransform- CoGBK
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkBatchPipelineTranslator
-  leaveCompositeTransform- 
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkRunner - Registering
pipeline artifacts in Flink program.
   [flink-runner-job-server] ERROR org.apache.beam.runners.flink.FlinkRunner - Artifact registration
failed
   java.lang.IllegalMonitorStateException
   	at java.lang.Object.wait(Native Method)
   	at java.lang.Object.wait(Object.java:502)
   	at org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.loadStagedArtifacts(FlinkPipelineExecutionEnvironment.java:350)
   	at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:121)
   	at org.apache.beam.runners.flink.FlinkJobInvocation.lambda$start$0(FlinkJobInvocation.java:61)
   	at org.apache.beam.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
   	at org.apache.beam.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
   	at org.apache.beam.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   	at java.lang.Thread.run(Thread.java:748)
   [flink-runner-job-server] ERROR org.apache.beam.runners.flink.FlinkJobInvocation - Error
during job invocation go-job-1521508749032988000_2100233503_-2090543569.
   java.lang.RuntimeException: Artifact registration failed
   	at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:124)
   	at org.apache.beam.runners.flink.FlinkJobInvocation.lambda$start$0(FlinkJobInvocation.java:61)
   	at org.apache.beam.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
   	at org.apache.beam.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
   	at org.apache.beam.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   	at java.lang.Thread.run(Thread.java:748)
   Caused by: java.lang.IllegalMonitorStateException
   	at java.lang.Object.wait(Native Method)
   	at java.lang.Object.wait(Object.java:502)
   	at org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.loadStagedArtifacts(FlinkPipelineExecutionEnvironment.java:350)
   	at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:121)
   	... 7 more
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 82116)
    Time Spent: 1h 20m  (was: 1h 10m)

> Go SDK support for portable pipelines
> -------------------------------------
>
>                 Key: BEAM-3287
>                 URL: https://issues.apache.org/jira/browse/BEAM-3287
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-go
>            Reporter: Henning Rohde
>            Assignee: Henning Rohde
>            Priority: Major
>              Labels: portability
>          Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> The Go SDK should participate in the portability framework, incl. job submission w/ a
docker container image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message