activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (ARTEMIS-1478) ActiveMQ Artemis to Kafka Bridge
Date Wed, 25 Oct 2017 12:16:00 GMT


ASF GitHub Bot commented on ARTEMIS-1478:

Github user gemmellr commented on a diff in the pull request:
    --- Diff: docs/user-manual/en/ ---
    @@ -0,0 +1,180 @@
    +# Apache ActiveMQ Kafka Bridge
    +The function of a bridge is to consume messages from a source queue in Apache ActiveMQ
    +and forward them to a target topic, on a remote Apache Kafka server.
    +By pairing Apache ActiveMQ Artemis and Apache Kafka with the bridge you could have a
hybrid broker setup, 
    +having a data flow with CORE, AMQP, MQTT clients, as well as now Kafka clients also.

    +Taking and giving the best features of both broker technologies when and where needed
for a flow of data.
    +![ActiveMQ Artemis Kafka Bridge Clients](images/activemq-kafka-bridge-clients.png)
    +The intent is this will be a two way bridge, but currently the flow is a single direction

    +from Apache ActiveMQ Artemis to Apache Kafka
    +The source and target servers are remote making bridging suitable
    +for reliably sending messages from one artemis cluster to kafka, 
    +for instance across a WAN, to the cloud, or internet and where the connection may be
    +The bridge has built in resilience to failure so if the target server
    +connection is lost, e.g. due to network failure, the bridge will retry
    +connecting to the target until it comes back online. When it comes back
    +online it will resume operation as normal.
    +![ActiveMQ Artemis Kafka Bridge Diagram](images/activemq-kafka-bridge-diagram.png)
    +In summary, Apache ActiveMQ Kafka Bridge is a way to reliably connect separate 
    +Apache ActiveMQ Artemis server and Apache Kafka server together.
    +## Configuring Kakfa Bridges
    +Bridges are configured in `broker.xml`.  
    +Let's kick off
    +with an example (this is actually from the kafka bridge test example):
    +    <connector-services>
    +         <connector-service name="my-kafka-bridge">
    +            <factory-class>org.apache.activemq.artemis.integration.kafka.bridge.KafkaProducerBridgeFactory</factory-class>
    +            <param key="bootstrap.servers" value="kafka-1.domain.local:9092,kafka-2.domain.local:9092,kafka-3.domain.local:9092"
    +            <param key="queue-name" value="my.artemis.queue" />
    +            <param key="kafka-topic" value="my_kafka_topic" />
    +         </connector-service>
    +    </connector-services>
    +In the above example we have shown the required parameters to
    +configure for a kakfa bridge. See below for a complete list of available configuration
    +### Serialization
    +By default the CoreMessageSerializer is used.
    +#### CoreMessageSerializer
    +Default but can be explicitly set using
    +    <param key="value.serializer" value="org.apache.activemq.artemis.integration.kafka.protocol.core.CoreMessageSerializer"
    +This maps the Message properties to Record headers.
    +And then maps the payload binary as the Record value, encoding TextMessage
    +This makes it easy to consume from Kafka using default deserializers
    +TextMessage using StringDeserializer and
    +ByteMessage using BytesDeserializer.
    +Also to note, if you serialized your objects using binary serialization like Apache Avro,
Apache Thrift etc. into a byte array for the ByteMessage, 
    +as that byte array is preserved in the record value part as is, you could deserialize
using an equivalent deserializer 
    +direct to back into Avro Record / Thrift Struct. 
    +Also supplied are some Apache Kafka deserializers allowing you to consume from Apache
    +and get a more familiar CoreMessage or JMSMessage, that your consumers can use:
    +You can get these Apache Kafka, Serializers/Deserializers via maven using the following
GAV coordinates:
    +    <depedency>
    +       <groupId>org.apache.activemq</groupId>
    +       <artifactId>artemis-kafka-core-protocol</artifactId>
    +       <version>2.4.0-SNAPSHOT</version>
    --- End diff --
    I'd guess version is just going to go stale, and be wrong for any given release docs since
its SNAPSHOT. Might be best to use a placeholder value.

> ActiveMQ Artemis to Kafka Bridge
> --------------------------------
>                 Key: ARTEMIS-1478
>                 URL:
>             Project: ActiveMQ Artemis
>          Issue Type: Bug
>            Reporter: Michael Andre Pearce
>            Assignee: Michael Andre Pearce
> To create a service connector to bridge messages in similar fashion as a core bridge
to kafka. 
> The first scope of this is to provide an Artemis to Kafka bridge, but the end goal (another
jira later) to provide the return leg.
> This essentially would mean for a data flow clients can connect and produce to a data
flow on all the supported protocols (amqp, core, mqtt etc) and like wise consume, but also
on the same data flow you can have kafka consumers. 

This message was sent by Atlassian JIRA

View raw message