beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nigel Kilmer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2629) Adding timestamp and id attributes to pubsub messages with no other attributes causes an exception.
Date Tue, 18 Jul 2017 19:51:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092077#comment-16092077
] 

Nigel Kilmer commented on BEAM-2629:
------------------------------------

Looks like the cause of this issue is that PubsubIO.FormatPayloadUsingCoder creates the PubsubMessage
with an ImmutableMap.of() for the attributes. PubsubJsonClient is unable to add the timestamp
and id attributes to this immutable map. See: https://github.com/apache/beam/blob/6a7eeeb932ffb33c876c0d11d7f325c47606d8de/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java#L984

I think it makes more sense to fix PubsubJsonClient to create a mutable copy of the attributes
rather than trying to ensure that PubsubIO format functions always create pubsub messages
with mutable attribute maps. I've implemented the former in my pull request.

> Adding timestamp and id attributes to pubsub messages with no other attributes causes
an exception.
> ---------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-2629
>                 URL: https://issues.apache.org/jira/browse/BEAM-2629
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-gcp
>    Affects Versions: 2.2.0
>            Reporter: Nigel Kilmer
>            Assignee: Nigel Kilmer
>
> I'm trying to use PubsubIO.writeProtos along with a timestamp and id attribute. When
PubsubIO tries to write the messages to my topic, it throws a PipelineExecutionException with
this cause:
> Caused by: java.lang.UnsupportedOperationException
> 	at java.util.AbstractMap.put(AbstractMap.java:209)
> 	at org.apache.beam.sdk.io.gcp.pubsub.PubsubJsonClient.publish(PubsubJsonClient.java:148)
> 	at org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSink$WriterFn.publishBatch(PubsubUnboundedSink.java:249)
> 	at org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSink$WriterFn.processElement(PubsubUnboundedSink.java:285)
> Looks like the problem is that PubsubJsonClient attempts to add the timestamp and id
attributes to a map which may be empty and immutable if no other attributes are specified
on the message. I will submit a pull request with a fix for this bug shortly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message