kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andy Muir (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-8629) Kafka Streams Apps to support small native images through GraalVM
Date Thu, 04 Jul 2019 06:42:00 GMT
Andy Muir created KAFKA-8629:

             Summary: Kafka Streams Apps to support small native images through GraalVM
                 Key: KAFKA-8629
                 URL: https://issues.apache.org/jira/browse/KAFKA-8629
             Project: Kafka
          Issue Type: Improvement
          Components: streams
    Affects Versions: 2.3.0
         Environment: OSX
Linux on Docker
            Reporter: Andy Muir

I'm investigating using [GraalVM|http://example.com/] to help with reducing docker image
size and required resources for a simple Kafka Streams microservice. To this end, I'm looking
at running a microservice which:
1) consumes from a Kafka topic (XML)
2) Transforms into JSON
3) Produces to a new Kafka topic.

The Kafka Streams app running in the JVM works fine.

When I attempt to build it to a GraalVM native image (binary executable which does not require
the JVM, hence smaller image size and less resources), I encountered a few [incompatibilities|https://github.com/oracle/graal/blob/master/substratevm/LIMITATIONS.md] with
the source code in Kafka.

I've implemented a workaround for each of these in a fork (link to follow) to help establish
if it is feasible. I don't intend (at this stage) for the changes to be applied to the broker
- I'm only after Kafka Streams for now. I'm not sure whether it'd be a good idea for the broker
itself to run as a native image!

There were 2 issues getting the native image with kafka streams:
1) Some Reflection use cases using MethodHandle
2) Anything JMX

To work around these issues, I have:
1) Replaced use of MethodHandle with alternatives
2) Commented out the JMX code in a few places

While the first may be sustainable, I'd expect that the 2nd option should be put behind a
configuration switch to allow the existing code to be used by default and turning off JMX
if configured.

*I haven't created a PR for now, as I'd like feedback to decide if it is going to be feasible
to carry this forwards.*

This message was sent by Atlassian JIRA

View raw message