flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sathi Chowdhury <Sathi.Chowdh...@elliemae.com>
Subject Re: Flink errors out and job fails--IOException from CollectSink.open()
Date Fri, 14 Apr 2017 00:44:14 GMT
Hi Ted, Sorry for my big font earlier…was not intended ☺

I am on flink 1.2.0
I built flink-connector-kinesis_2.10-1.2.0.jar from source and included in the fatjar I am
Followed this http://www.kidder.io/2017/02/15/flink-kinesis-streaming-connector/

From code I read a kinesis stream using

consumerConfig.setProperty(ConsumerConfigConstants.AWS_REGION, region);
consumerConfig.setProperty(ConsumerConfigConstants.DEFAULT_STREAM_INITIAL_POSITION, "LATEST");
consumerConfig.setProperty(ConsumerConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO");
consumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "10");
consumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_RETRIES, "200");
consumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "2000");
DataStream<Map<String, Object>> stream = env.addSource(new FlinkKinesisConsumer<>(inputStream,
new MyJsonDeserializationSchema(), consumerConfig));

While I push the json event to the Kinesis stream intermittently I see this NPE and flink
job fails

2017-04-14 00:31:54,672 WARN  org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer
 - Error while closing Kinesis data fetcher


               at org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.shutdownFetcher(KinesisDataFetcher.java:472)

               at org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.cancel(FlinkKinesisConsumer.java:246)

               at org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.close(FlinkKinesisConsumer.java:256)

               at org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:43)

               at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.dispose(AbstractUdfStreamOperator.java:127)

               at org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:442)

               at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:343)

               at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)

               at java.lang.Thread.run(Thread.java:745)


From: Ted Yu <yuzhihong@gmail.com>
Date: Thursday, April 13, 2017 at 5:02 PM
To: Sathi Chowdhury <Sathi.Chowdhury@elliemae.com>
Cc: "user@flink.apache.org" <user@flink.apache.org>
Subject: Re: Flink errors out and job fails--IOException from CollectSink.open()

Can you give us a bit more information ?

release of flink
snippet of your code

=============Notice to Recipient: This e-mail transmission, and any documents, files or previous
e-mail messages attached to it may contain information that is confidential or legally privileged,
and intended for the use of the individual or entity named above. If you are not the intended
recipient, or a person responsible for delivering it to the intended recipient, you are hereby
notified that you must not read this transmission and that any disclosure, copying, printing,
distribution or use of any of the information contained in or attached to this transmission
is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately
notify the sender by telephone or return e-mail and delete the original transmission and its
attachments without reading or saving in any manner. Thank you. =============
View raw message