Mailing-List: contact issues-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@activemq.apache.org
Date: Wed, 25 Oct 2017 12:16:00 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: issues@activemq.apache.org
Message-ID: <JIRA.13111623.1508829485000.55168.1508933760810@Atlassian.JIRA>
In-Reply-To: <JIRA.13111623.1508829485000@Atlassian.JIRA>
References: <JIRA.13111623.1508829485000@Atlassian.JIRA> <JIRA.13111623.1508829485955@jira-lw-us.apache.org>
Subject: [jira] [Commented] (ARTEMIS-1478) ActiveMQ Artemis to Kafka Bridge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 25 Oct 2017 12:16:09 -0000


    [ https://issues.apache.org/jira/browse/ARTEMIS-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218483#comment-16218483 ] 

ASF GitHub Bot commented on ARTEMIS-1478:
-----------------------------------------

Github user gemmellr commented on a diff in the pull request:

    https://github.com/apache/activemq-artemis/pull/1607#discussion_r146821726
  
    --- Diff: docs/user-manual/en/kafka-bridges.md ---
    @@ -0,0 +1,180 @@
    +# Apache ActiveMQ Kafka Bridge
    +
    +The function of a bridge is to consume messages from a source queue in Apache ActiveMQ Artemis, 
    +and forward them to a target topic, on a remote Apache Kafka server.
    +
    +By pairing Apache ActiveMQ Artemis and Apache Kafka with the bridge you could have a hybrid broker setup, 
    +having a data flow with CORE, AMQP, MQTT clients, as well as now Kafka clients also. 
    +Taking and giving the best features of both broker technologies when and where needed for a flow of data.
    +
    +![ActiveMQ Artemis Kafka Bridge Clients](images/activemq-kafka-bridge-clients.png)
    +
    +The intent is this will be a two way bridge, but currently the flow is a single direction 
    +from Apache ActiveMQ Artemis to Apache Kafka
    +
    +
    +The source and target servers are remote making bridging suitable
    +for reliably sending messages from one artemis cluster to kafka, 
    +for instance across a WAN, to the cloud, or internet and where the connection may be unreliable.
    +
    +The bridge has built in resilience to failure so if the target server
    +connection is lost, e.g. due to network failure, the bridge will retry
    +connecting to the target until it comes back online. When it comes back
    +online it will resume operation as normal.
    +
    +![ActiveMQ Artemis Kafka Bridge Diagram](images/activemq-kafka-bridge-diagram.png)
    +
    +In summary, Apache ActiveMQ Kafka Bridge is a way to reliably connect separate 
    +Apache ActiveMQ Artemis server and Apache Kafka server together.
    +
    +## Configuring Kakfa Bridges
    +
    +Bridges are configured in `broker.xml`.  
    +Let's kick off
    +with an example (this is actually from the kafka bridge test example):
    +
    +
    +    <connector-services>
    +         <connector-service name="my-kafka-bridge">
    +            <factory-class>org.apache.activemq.artemis.integration.kafka.bridge.KafkaProducerBridgeFactory</factory-class>
    +            <param key="bootstrap.servers" value="kafka-1.domain.local:9092,kafka-2.domain.local:9092,kafka-3.domain.local:9092" />
    +            <param key="queue-name" value="my.artemis.queue" />
    +            <param key="kafka-topic" value="my_kafka_topic" />
    +         </connector-service>
    +    </connector-services>
    +
    +In the above example we have shown the required parameters to
    +configure for a kakfa bridge. See below for a complete list of available configuration options. 
    +
    +### Serialization
    +By default the CoreMessageSerializer is used.
    +
    +#### CoreMessageSerializer
    +Default but can be explicitly set using
    +           
    +    <param key="value.serializer" value="org.apache.activemq.artemis.integration.kafka.protocol.core.CoreMessageSerializer" />
    +
    +This maps the Message properties to Record headers.
    +And then maps the payload binary as the Record value, encoding TextMessage
    +
    +This makes it easy to consume from Kafka using default deserializers
    +TextMessage using StringDeserializer and
    +ByteMessage using BytesDeserializer.
    +
    +Also to note, if you serialized your objects using binary serialization like Apache Avro, Apache Thrift etc. into a byte array for the ByteMessage, 
    +as that byte array is preserved in the record value part as is, you could deserialize using an equivalent deserializer 
    +direct to back into Avro Record / Thrift Struct. 
    +
    +Also supplied are some Apache Kafka deserializers allowing you to consume from Apache Kafka 
    +and get a more familiar CoreMessage or JMSMessage, that your consumers can use:
    +
    +`org.apache.activemq.artemis.integration.kafka.protocol.core.jms.CoreJmsMessageDeserializer`
    +`org.apache.activemq.artemis.integration.kafka.protocol.core.CoreMessageDeserializer`
    +
    +You can get these Apache Kafka, Serializers/Deserializers via maven using the following GAV coordinates:
    +
    +    <depedency>
    +       <groupId>org.apache.activemq</groupId>
    +       <artifactId>artemis-kafka-core-protocol</artifactId>
    +       <version>2.4.0-SNAPSHOT</version>
    --- End diff --
    
    I'd guess version is just going to go stale, and be wrong for any given release docs since its SNAPSHOT. Might be best to use a placeholder value.


> ActiveMQ Artemis to Kafka Bridge
> --------------------------------
>
>                 Key: ARTEMIS-1478
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-1478
>             Project: ActiveMQ Artemis
>          Issue Type: Bug
>            Reporter: Michael Andre Pearce
>            Assignee: Michael Andre Pearce
>
> To create a service connector to bridge messages in similar fashion as a core bridge to kafka. 
> The first scope of this is to provide an Artemis to Kafka bridge, but the end goal (another jira later) to provide the return leg.
> This essentially would mean for a data flow clients can connect and produce to a data flow on all the supported protocols (amqp, core, mqtt etc) and like wise consume, but also on the same data flow you can have kafka consumers. 
> http://activemq.2283324.n4.nabble.com/Kafka-ServiceConnector-td4731430.html#a4731436


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)