spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dongjinleekr <>
Subject [GitHub] spark pull request #22282: [SPARK-23539][SS] Add support for Kafka headers i...
Date Thu, 30 Aug 2018 12:28:21 GMT
GitHub user dongjinleekr opened a pull request:

    [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming

    ## What changes were proposed in this pull request?
    This update adds support for Kafka Headers functionality in Structured Streaming.
    ## How was this patch tested?
    With following unit tests:
    - KafkaRelationSuite: "default starting and ending offsets with headers" (new)
    - KafkaSinkSuite: "batch - write to kafka" (updated)

You can merge this pull request into a Git repository by running:

    $ git pull feature/SPARK-23539

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22282
commit ddd08612ef8bdb173f974059d2dc6311e1c7d9a3
Author: Lee Dongjin <dongjin@...>
Date:   2018-08-26T13:19:52Z

    Remove unused methods from UnsafeArrayData

commit 2af13899ab052cc7b52c25b57f154b78a2c45b2a
Author: Lee Dongjin <dongjin@...>
Date:   2018-08-26T13:25:25Z

    Implement UnsafeArrayData#fromBinaryArray

commit a8e5c5c0f478a795af1236771236da2074093f3e
Author: Lee Dongjin <dongjin@...>
Date:   2018-08-27T12:22:18Z

    Implement UnsafeArrayData#fromStringArray

commit 2ca181046cf1102aed14f4957e11e4dd901ba3c7
Author: Lee Dongjin <dongjin@...>
Date:   2018-08-27T13:28:58Z

    Implement UnsafeMapData#of

commit d0d746d99d0a19ecbb2dc098589adbfd1ef0b5ae
Author: Lee Dongjin <dongjin@...>
Date:   2018-08-29T13:25:57Z

    Allow empty UnsafeArrayData: does not throw IllegalArgumentException on empty or null
array argument anymore.

commit b459cf3f391d6e4ee9cb77a7b5ed510d027d9ddd
Author: Lee Dongjin <dongjin@...>
Date:   2018-08-28T07:50:59Z

    Fix invalid formatting: UnsafeArraySuite

commit f077c5d75a83df3541a95a628726e1d74af8c153
Author: Lee Dongjin <dongjin@...>
Date:   2018-08-28T07:51:30Z

    Implemenet kafka headers functionality

commit 6b4d7754d01e4211d05c83746c848fdfd873f229
Author: Lee Dongjin <dongjin@...>
Date:   2018-08-29T09:39:55Z

    Add KafkaTestUtils#{sendMessage, sendMessages(String, Array[(String, String, Array[(String,
String)])], Option[Int])}

commit c7fb9819989056da4910e2cfc81af332cd603d41
Author: Lee Dongjin <dongjin@...>
Date:   2018-08-29T13:24:09Z

    Extend KafkaRelationSuite, KafkaSinkSuite to test headers functionality

commit dd2d9390478e4c69b01a7c699e28bfe923ef0db1
Author: Lee Dongjin <dongjin@...>
Date:   2018-08-30T10:50:38Z

    Minor refinements

commit 229aac85442b03736fc850cae2c3b26becaedade
Author: Lee Dongjin <dongjin@...>
Date:   2018-08-30T12:21:30Z

    Specify #selectExpr on KafkaSourceSuiteBase's 'Kafka column types' test



To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message