Alex/Gleb can someone of you please add the detailed instructions that worked for you in some section of cwiki.
I have the impression that this will benefit us all at some point.

Thanks,



On Tue, Feb 18, 2020 at 9:46 AM Alex Van Boxel <alex@vanboxel.be> wrote:
Thanks everyone. This really helped a lot. I used Gleb's tip to make it work. Successfully validated my Pull Requests against Dataflow!

 _/
_/ Alex Van Boxel


On Mon, Feb 17, 2020 at 11:55 PM Brian Hulette <bhulette@google.com> wrote:
I think if you update past [1] this will go away. We had to build a new worker to use with builds on master after [2]. You should be fine running from master as long as you aren't using a commit between those two (merge commits are 6818560 and bde3031, respectively). Setting the Dataflow worker jar would work too.

On Mon, Feb 17, 2020 at 2:14 AM Gleb Kanterov <gleb@spotify.com> wrote:
You need to pass custom Dataflow worker jar. One of the ways of doing that is adding it as a dependency, and using following code snippet:

opts.setDataflowWorkerJar(
  BatchDataflowWorker.class
      .getProtectionDomain()
      .getCodeSource()
      .getLocation()
      .toString());
opts.setWorkerHarnessContainerImage("");

Coming with the disclaimer that it isn't for production :)

On Mon, Feb 17, 2020 at 8:34 AM Alex Van Boxel <alex@vanboxel.be> wrote:
Yes, running it manually with the normal parameters as I do for production Dataflow. I'm probably a bit ignorant on that, and I probably need to provide my own worker.

Thanks for the hint... I'll dive into that.

 _/
_/ Alex Van Boxel


On Mon, Feb 17, 2020 at 8:16 AM Reuven Lax <relax@google.com> wrote:
Are you running things manually? This probably means you are using an out-of-date Dataflow worker. I believe that all tests on Jenkins will build the Dataflow worker from head to prevent exactly this problem.

On Sun, Feb 16, 2020 at 11:10 PM Alex Van Boxel <alex@vanboxel.be> wrote:
Digging further in the traces, it seems like a result of changes to the model:

Caused by: java.lang.ClassNotFoundException: org.apache.beam.model.pipeline.v1.StandardWindowFns$SessionsPayload$Enum

I see changes by Lukasz Cwik. Will this be a problem for the release?

 _/
_/ Alex Van Boxel


On Sun, Feb 16, 2020 at 12:11 PM Alex Van Boxel <alex@vanboxel.be> wrote:
Hey,

I'm testing my own PR's against Dataflow, something I've done in the past with success seem to fail now. I get this error:

java.lang.NoClassDefFoundError: Could not initialize class org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.construction.WindowingStrategyTranslation

Am I doing something wrong?

 _/
_/ Alex Van Boxel