flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Itamar Syn-Hershko <ita...@bigdataboutique.com>
Subject Re: PubSub source throwing grpc errors
Date Sun, 19 Jan 2020 11:24:54 GMT
Definitely (though will be hijacking my own thread).

The root issue is that it seems impossible to load custom Jackson Modules,
specifically in our case JodaModule
<https://github.com/FasterXML/jackson-datatype-joda> and KotlinModule
<https://github.com/FasterXML/jackson-module-kotlin>. Since Jackson is
shadowed in Flink, those modules need to be recompiled to return the
shadowed data-types; and using jackson directly instead of the shaded
version doesn't seem to be possible.

Any hints on how to approach this?

On Wed, Jan 15, 2020 at 6:37 PM Till Rohrmann <trohrmann@apache.org> wrote:

> Hi Itamar,
>
> could you share a bit more details about the serialization problem. Which
> class is not serializable and where does it originate from?
>
> Cheers,
> Till
>
> On Tue, Jan 14, 2020 at 9:47 PM Itamar Syn-Hershko <
> itamar@bigdataboutique.com> wrote:
>
>> Thanks!
>>
>> I was able to track this down. Essentially it was a deserialization error
>> which propagated and might have prevented the channel from closing down
>> properly. This could be considered as a problem, but I'm not further down
>> the rabbit hole chasing down a solution for the original deserialization
>> issue.
>>
>> Thanks for the help!
>>
>> On Tue, Jan 14, 2020 at 8:26 PM Till Rohrmann <trohrmann@apache.org>
>> wrote:
>>
>>> Hi Itamar,
>>>
>>> for further debugging it would be helpful to get the full logs of Flink
>>> and more information about your environment. Since I'm not too
>>> familiar with Flink's PubSub connector, I have pulled in Richard (original
>>> author), Becket and Robert (both helped with reviewing and merging this
>>> connector). They might know what's going on.
>>>
>>> The problem looks a bit similar to [1]. Maybe it would help to upgrade
>>> to a newer google-cloud-pubsub version than 1.62.0. I assume that the
>>> others might know more about it.
>>>
>>> [1] https://github.com/googleapis/google-cloud-java/issues/3648
>>>
>>> Cheers,
>>> Till
>>>
>>> On Mon, Jan 13, 2020 at 12:19 PM Itamar Syn-Hershko <
>>> itamar@bigdataboutique.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> We are trying to use the PubSub source with a very minimal and basic
>>>> Flink application as a POC, and getting the following error consistently
>>>> every couple of seconds. What am I missing?
>>>>
>>>> ```
>>>> io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference
>>>> cleanQueue
>>>> SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=5, target=
>>>> pubsub.googleapis.com:443} was not shutdown properly!!! ~*~*~*
>>>>     Make sure to call shutdown()/shutdownNow() and wait until
>>>> awaitTermination() returns true.
>>>> java.lang.RuntimeException: ManagedChannel allocation site
>>>> at
>>>> io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:103)
>>>> at
>>>> io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
>>>> at
>>>> io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
>>>> at
>>>> io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:419)
>>>> at
>>>> org.apache.flink.streaming.connectors.gcp.pubsub.DefaultPubSubSubscriberFactory.getSubscriber(DefaultPubSubSubscriberFactory.java:55)
>>>> at
>>>> org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.createAndSetPubSubSubscriber(PubSubSource.java:178)
>>>> at
>>>> org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.open(PubSubSource.java:100)
>>>> at
>>>> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
>>>> at
>>>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
>>>> at
>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:532)
>>>> at
>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:396)
>>>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
>>>> at java.base/java.lang.Thread.run(Thread.java:834)
>>>> ```
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>>
>>>> [image: logo] <https://bigdataboutique.com/>
>>>> Itamar Syn-Hershko
>>>>
>>>>
>>>> itamar@bigdataboutique.com
>>>> https://bigdataboutique.com
>>>> <https://www.linkedin.com/in/itamar-syn-hershko-78b25013>
>>>> <https://twitter.com/synhershko>
>>>> <https://www.youtube.com/channel/UCBHr7lM2u6SCWPJvcKug-Yg>
>>>>
>>>
>>
>> --
>>
>> [image: logo] <https://bigdataboutique.com/>
>> Itamar Syn-Hershko
>>
>>
>> itamar@bigdataboutique.com
>> https://bigdataboutique.com
>> <https://www.linkedin.com/in/itamar-syn-hershko-78b25013>
>> <https://twitter.com/synhershko>
>> <https://www.youtube.com/channel/UCBHr7lM2u6SCWPJvcKug-Yg>
>>
>

-- 

[image: logo] <https://bigdataboutique.com/>
Itamar Syn-Hershko


itamar@bigdataboutique.com
https://bigdataboutique.com
<https://www.linkedin.com/in/itamar-syn-hershko-78b25013>
<https://twitter.com/synhershko>
<https://www.youtube.com/channel/UCBHr7lM2u6SCWPJvcKug-Yg>

Mime
View raw message