activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From michaelandrepearce <...@git.apache.org>
Subject [GitHub] activemq-artemis pull request #1607: ARTEMIS-1478 - ActiveMQ Artemis to Kafk...
Date Wed, 25 Oct 2017 14:09:41 GMT
Github user michaelandrepearce commented on a diff in the pull request:

    https://github.com/apache/activemq-artemis/pull/1607#discussion_r146868998
  
    --- Diff: docs/user-manual/en/kafka-bridges.md ---
    @@ -0,0 +1,180 @@
    +# Apache ActiveMQ Kafka Bridge
    +
    +The function of a bridge is to consume messages from a source queue in Apache ActiveMQ
Artemis, 
    +and forward them to a target topic, on a remote Apache Kafka server.
    +
    +By pairing Apache ActiveMQ Artemis and Apache Kafka with the bridge you could have a
hybrid broker setup, 
    +having a data flow with CORE, AMQP, MQTT clients, as well as now Kafka clients also.

    +Taking and giving the best features of both broker technologies when and where needed
for a flow of data.
    +
    +![ActiveMQ Artemis Kafka Bridge Clients](images/activemq-kafka-bridge-clients.png)
    +
    +The intent is this will be a two way bridge, but currently the flow is a single direction

    +from Apache ActiveMQ Artemis to Apache Kafka
    +
    +
    +The source and target servers are remote making bridging suitable
    +for reliably sending messages from one artemis cluster to kafka, 
    +for instance across a WAN, to the cloud, or internet and where the connection may be
unreliable.
    +
    +The bridge has built in resilience to failure so if the target server
    +connection is lost, e.g. due to network failure, the bridge will retry
    +connecting to the target until it comes back online. When it comes back
    +online it will resume operation as normal.
    +
    +![ActiveMQ Artemis Kafka Bridge Diagram](images/activemq-kafka-bridge-diagram.png)
    +
    +In summary, Apache ActiveMQ Kafka Bridge is a way to reliably connect separate 
    +Apache ActiveMQ Artemis server and Apache Kafka server together.
    +
    +## Configuring Kakfa Bridges
    +
    +Bridges are configured in `broker.xml`.  
    +Let's kick off
    +with an example (this is actually from the kafka bridge test example):
    +
    +
    +    <connector-services>
    +         <connector-service name="my-kafka-bridge">
    +            <factory-class>org.apache.activemq.artemis.integration.kafka.bridge.KafkaProducerBridgeFactory</factory-class>
    +            <param key="bootstrap.servers" value="kafka-1.domain.local:9092,kafka-2.domain.local:9092,kafka-3.domain.local:9092"
/>
    +            <param key="queue-name" value="my.artemis.queue" />
    +            <param key="kafka-topic" value="my_kafka_topic" />
    +         </connector-service>
    +    </connector-services>
    +
    +In the above example we have shown the required parameters to
    +configure for a kakfa bridge. See below for a complete list of available configuration
options. 
    +
    +### Serialization
    +By default the CoreMessageSerializer is used.
    +
    +#### CoreMessageSerializer
    +Default but can be explicitly set using
    +           
    +    <param key="value.serializer" value="org.apache.activemq.artemis.integration.kafka.protocol.core.CoreMessageSerializer"
/>
    +
    +This maps the Message properties to Record headers.
    +And then maps the payload binary as the Record value, encoding TextMessage
    +
    +This makes it easy to consume from Kafka using default deserializers
    +TextMessage using StringDeserializer and
    +ByteMessage using BytesDeserializer.
    +
    +Also to note, if you serialized your objects using binary serialization like Apache Avro,
Apache Thrift etc. into a byte array for the ByteMessage, 
    +as that byte array is preserved in the record value part as is, you could deserialize
using an equivalent deserializer 
    +direct to back into Avro Record / Thrift Struct. 
    +
    +Also supplied are some Apache Kafka deserializers allowing you to consume from Apache
Kafka 
    +and get a more familiar CoreMessage or JMSMessage, that your consumers can use:
    +
    +`org.apache.activemq.artemis.integration.kafka.protocol.core.jms.CoreJmsMessageDeserializer`
    +`org.apache.activemq.artemis.integration.kafka.protocol.core.CoreMessageDeserializer`
    +
    +You can get these Apache Kafka, Serializers/Deserializers via maven using the following
GAV coordinates:
    +
    +    <depedency>
    +       <groupId>org.apache.activemq</groupId>
    +       <artifactId>artemis-kafka-core-protocol</artifactId>
    +       <version>2.4.0-SNAPSHOT</version>
    +    </depedency>
    +
    +
    +#### AMQPMessageSerializer
    +Can be set by using:
    +    
    +    <param key="value.serializer" value="org.apache.activemq.artemis.integration.kafka.protocol.amqp.AMQPMessageSerializer"
/>
    +
    +
    +This encodes the whole message into amqp binary protocol into the Record value.
    --- End diff --
    
    will capitalise 


---

Mime
View raw message