beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismaël Mejía <ieme...@gmail.com>
Subject Re: Custom 2.20 failing on Dataflow: what am I doing wrong?
Date Wed, 19 Feb 2020 16:52:10 GMT
Alex/Gleb can someone of you please add the detailed instructions that
worked for you in some section of cwiki.
I have the impression that this will benefit us all at some point.

Thanks,



On Tue, Feb 18, 2020 at 9:46 AM Alex Van Boxel <alex@vanboxel.be> wrote:

> Thanks everyone. This really helped a lot. I used Gleb's tip to make it
> work. Successfully validated my Pull Requests against Dataflow!
>
>  _/
> _/ Alex Van Boxel
>
>
> On Mon, Feb 17, 2020 at 11:55 PM Brian Hulette <bhulette@google.com>
> wrote:
>
>> I think if you update past [1] this will go away. We had to build a new
>> worker to use with builds on master after [2]. You should be fine running
>> from master as long as you aren't using a commit between those two (merge
>> commits are 6818560 and bde3031, respectively). Setting the Dataflow worker
>> jar would work too.
>>
>> [1] https://github.com/apache/beam/pull/10861
>> [2] https://github.com/apache/beam/pull/10790
>>
>> On Mon, Feb 17, 2020 at 2:14 AM Gleb Kanterov <gleb@spotify.com> wrote:
>>
>>> You need to pass custom Dataflow worker jar. One of the ways of doing
>>> that is adding it as a dependency, and using following code snippet:
>>>
>>> opts.setDataflowWorkerJar(
>>>   BatchDataflowWorker.class
>>>       .getProtectionDomain()
>>>       .getCodeSource()
>>>       .getLocation()
>>>       .toString());
>>> opts.setWorkerHarnessContainerImage("");
>>>
>>> Coming with the disclaimer that it isn't for production :)
>>>
>>> On Mon, Feb 17, 2020 at 8:34 AM Alex Van Boxel <alex@vanboxel.be> wrote:
>>>
>>>> Yes, running it manually with the normal parameters as I do for
>>>> production Dataflow. I'm probably a bit ignorant on that, and I
>>>> probably need to provide my own worker.
>>>>
>>>> Thanks for the hint... I'll dive into that.
>>>>
>>>>  _/
>>>> _/ Alex Van Boxel
>>>>
>>>>
>>>> On Mon, Feb 17, 2020 at 8:16 AM Reuven Lax <relax@google.com> wrote:
>>>>
>>>>> Are you running things manually? This probably means you are using an
>>>>> out-of-date Dataflow worker. I believe that all tests on Jenkins will
build
>>>>> the Dataflow worker from head to prevent exactly this problem.
>>>>>
>>>>> On Sun, Feb 16, 2020 at 11:10 PM Alex Van Boxel <alex@vanboxel.be>
>>>>> wrote:
>>>>>
>>>>>> Digging further in the traces, it seems like a result of changes
to
>>>>>> the model:
>>>>>>
>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>> org.apache.beam.model.pipeline.v1.StandardWindowFns$SessionsPayload$Enum
>>>>>>
>>>>>> I see changes by Lukasz Cwik. Will this be a problem for the release?
>>>>>>
>>>>>>  _/
>>>>>> _/ Alex Van Boxel
>>>>>>
>>>>>>
>>>>>> On Sun, Feb 16, 2020 at 12:11 PM Alex Van Boxel <alex@vanboxel.be>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey,
>>>>>>>
>>>>>>> I'm testing my own PR's against Dataflow, something I've done
in the
>>>>>>> past with success seem to fail now. I get this error:
>>>>>>>
>>>>>>> java.lang.NoClassDefFoundError: Could not initialize class
>>>>>>> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.construction.WindowingStrategyTranslation
>>>>>>>
>>>>>>>    1.
>>>>>>>
>>>>>>>
>>>>>>> Am I doing something wrong?
>>>>>>>
>>>>>>>  _/
>>>>>>> _/ Alex Van Boxel
>>>>>>>
>>>>>>

Mime
View raw message