flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From prateekarora <prateek.arora...@gmail.com>
Subject Re: Getting java.lang.Exception when try to fetch data from Kafka
Date Wed, 27 Apr 2016 17:11:06 GMT
Thanks for the response .

can you please suggest some link or example to write own
DeserializationSchema ?

Regards
Prateek

On Tue, Apr 26, 2016 at 11:06 AM, rmetzger0 [via Apache Flink User Mailing
List archive.] <ml-node+s2336050n6464h14@n4.nabble.com> wrote:

> Hi Prateek,
>
> sorry for the late response. Can you try implementing your own
> DeserializationSchema, where you deserialize the String key manually (just
> call the "new String(byte[]) constructor).
>
> The TypeInformationKeyValueSerializationSchema[String, byte] is
> generating deserializers with Flink's internal serializer stack (these
> assume that the data has been serialized by Flink as well). I think Flink's
> StringSerializer does some fancy optimizations and is not compatible with
> the standard String() format.
>
>
>
> On Tue, Apr 26, 2016 at 6:34 PM, prateek arora <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=6464&i=0>> wrote:
>
>> Hi Robert ,
>>
>> Hi
>>
>> I have java program to send data into kafka topic. below is code for this
>> :
>>
>> private Producer<String, byte[]> producer = null
>>
>> Serializer<String> keySerializer = new StringSerializer();
>> Serializer<byte[]> valueSerializer = new ByteArraySerializer();
>> producer = new KafkaProducer<String, byte[]>(props, keySerializer,
>> valueSerializer);
>>
>> ProducerRecord<String, byte[]> imageRecord;
>> imageRecord = new ProducerRecord<String, byte[]>(streamInfo.topic,
>>                         Integer.toString(messageKey), imageData);
>>
>> producer.send(imageRecord);
>>
>>
>> then trying to fetch data in Apache flink .
>>
>> Regards
>> Prateek
>>
>> On Mon, Apr 25, 2016 at 2:42 AM, Robert Metzger <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=6464&i=1>> wrote:
>>
>>> Hi Prateek,
>>>
>>> were the messages written to the Kafka topic by Flink, using the
>>> TypeInformationKeyValueSerializationSchema ? If not, maybe the Flink
>>> deserializers expect a different data format of the messages in the topic.
>>>
>>> How are the messages written into the topic?
>>>
>>>
>>> On Fri, Apr 22, 2016 at 10:21 PM, prateekarora <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=6464&i=2>> wrote:
>>>
>>>>
>>>> Hi
>>>>
>>>> I am sending data using kafkaProducer API
>>>>
>>>>                        imageRecord = new ProducerRecord<String,
>>>> byte[]>(topic,messageKey, imageData);
>>>>                         producer.send(imageRecord);
>>>>
>>>>
>>>> And in flink program  try to fect data using FlinkKafkaConsumer08 .
>>>> below
>>>> are the sample code .
>>>>
>>>>     def main(args: Array[String]) {
>>>>           val env = StreamExecutionEnvironment.getExecutionEnvironment
>>>>           val properties = new Properties()
>>>>           properties.setProperty("bootstrap.servers",
>>>> "<IPADDRESS>:9092")
>>>>           properties.setProperty("zookeeper.connect",
>>>> "<IPADDRESS>:2181")
>>>>           properties.setProperty("group.id", "test")
>>>>
>>>>           val readSchema = new
>>>>
>>>> TypeInformationKeyValueSerializationSchema[String,Array[Byte]](classOf[String],classOf[Array[Byte]],
>>>>
>>>> env.getConfig).asInstanceOf[KeyedDeserializationSchema[(String,Array[Byte])]]
>>>>
>>>>           val stream : DataStream[(String,Array[Byte])]  =
>>>> env.addSource(new
>>>> FlinkKafkaConsumer08[(String,Array[Byte])]("a-0",readSchema,
>>>> properties))
>>>>
>>>>           stream.print
>>>>           env.execute("Flink Kafka Example")
>>>>   }
>>>>
>>>>
>>>> but getting  below error :
>>>>
>>>> 16/04/22 13:43:39 INFO ExecutionGraph: Source: Custom Source -> Sink:
>>>> Unnamed (1/4) (d7a151560f6eabdc587a23dc0975cb84) switched from RUNNING
>>>> to
>>>> FAILED
>>>> 16/04/22 13:43:39 INFO ExecutionGraph: Source: Custom Source -> Sink:
>>>> Unnamed (2/4) (d43754a27e402ed5b02a73d1c9aa3125) switched from RUNNING
>>>> to
>>>> CANCELING
>>>>
>>>> java.lang.Exception
>>>>     at
>>>>
>>>> org.apache.flink.streaming.connectors.kafka.internals.LegacyFetcher.run(LegacyFetcher.java:222)
>>>>     at
>>>>
>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer08.run(FlinkKafkaConsumer08.java:316)
>>>>     at
>>>>
>>>> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:78)
>>>>     at
>>>>
>>>> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:56)
>>>>     at
>>>>
>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:225)
>>>>     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
>>>>     at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: java.io.EOFException
>>>>     at
>>>>
>>>> org.apache.flink.runtime.util.DataInputDeserializer.readUnsignedByte(DataInputDeserializer.java:298)
>>>>     at
>>>> org.apache.flink.types.StringValue.readString(StringValue.java:771)
>>>>     at
>>>>
>>>> org.apache.flink.api.common.typeutils.base.StringSerializer.deserialize(StringSerializer.java:69)
>>>>     at
>>>>
>>>> org.apache.flink.api.common.typeutils.base.StringSerializer.deserialize(StringSerializer.java:28)
>>>>     at
>>>>
>>>> org.apache.flink.streaming.util.serialization.TypeInformationKeyValueSerializationSchema.deserialize(TypeInformationKeyValueSerializationSchema.java:105)
>>>>     at
>>>>
>>>> org.apache.flink.streaming.util.serialization.TypeInformationKeyValueSerializationSchema.deserialize(TypeInformationKeyValueSerializationSchema.java:39)
>>>>     at
>>>>
>>>> org.apache.flink.streaming.connectors.kafka.internals.LegacyFetcher$SimpleConsumerThread.run(LegacyFetcher.java:657)
>>>>
>>>>
>>>> Regards
>>>> Prateek
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Getting-java-lang-Exception-when-try-to-fetch-data-from-Kafka-tp6365.html
>>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>>> archive at Nabble.com.
>>>>
>>>
>>>
>>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Getting-java-lang-Exception-when-try-to-fetch-data-from-Kafka-tp6365p6464.html
> To unsubscribe from Getting java.lang.Exception when try to fetch data
> from Kafka, click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=6365&code=cHJhdGVlay5hcm9yYTJrNkBnbWFpbC5jb218NjM2NXwtOTMxMzI0ODYx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Getting-java-lang-Exception-when-try-to-fetch-data-from-Kafka-tp6365p6516.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Mime
View raw message