hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HUDI-2320) Add support ByteArrayDeserializer in AvroKafkaSource
Date Sun, 29 Aug 2021 09:07:00 GMT

    [ https://issues.apache.org/jira/browse/HUDI-2320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17406381#comment-17406381
] 

ASF GitHub Bot commented on HUDI-2320:
--------------------------------------

hudi-bot edited a comment on pull request #3502:
URL: https://github.com/apache/hudi/pull/3502#issuecomment-901822779


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "61dfee3f3db371b4c4ec55f630d2a5b40467ea12",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1809",
       "triggerID" : "61dfee3f3db371b4c4ec55f630d2a5b40467ea12",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b1de704c4a618aea6026eca4e832888d78b8b6b2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1934",
       "triggerID" : "b1de704c4a618aea6026eca4e832888d78b8b6b2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 61dfee3f3db371b4c4ec55f630d2a5b40467ea12 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1809)

   * b1de704c4a618aea6026eca4e832888d78b8b6b2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1934)

   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Add support ByteArrayDeserializer in AvroKafkaSource
> ----------------------------------------------------
>
>                 Key: HUDI-2320
>                 URL: https://issues.apache.org/jira/browse/HUDI-2320
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: DeltaStreamer
>            Reporter: 董可伦
>            Assignee: 董可伦
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 0.10.0
>
>
> When the 'value.serializer' of Kafka Avro Producer is 'org.apache.kafka.common.serialization.ByteArraySerializer',Use
the following configuration
> {code:java}
> --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
> --schemaprovider-class org.apache.hudi.utilities.schema.JdbcbasedSchemaProvider \
> --hoodie-conf "hoodie.deltastreamer.source.kafka.value.deserializer.class=org.apache.kafka.common.serialization.ByteArrayDeserializer"
> {code}
> For now,It will throw an exception::
> {code:java}
> java.lang.ClassCastException: [B cannot be cast to org.apache.avro.generic.GenericRecord{code}
> After support ByteArrayDeserializer,Use the configuration above,It works properly.And
there is no need to provide 'schema.registry.url',For example, we can use the JdbcbasedSchemaProvider
to get the sourceSchema



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message