Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B6F44200C0B for ; Sat, 24 Dec 2016 07:55:56 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id B56C4160B2B; Sat, 24 Dec 2016 06:55:56 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7BEBE160B3C for ; Sat, 24 Dec 2016 07:55:54 +0100 (CET) Received: (qmail 26052 invoked by uid 500); 24 Dec 2016 06:55:53 -0000 Mailing-List: contact commits-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@incubator.apache.org Delivered-To: mailing list commits@incubator.apache.org Received: (qmail 26013 invoked by uid 99); 24 Dec 2016 06:55:53 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Dec 2016 06:55:53 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 65101DFC47; Sat, 24 Dec 2016 06:55:53 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: yukon@apache.org To: commits@incubator.apache.org Date: Sat, 24 Dec 2016 06:55:57 -0000 Message-Id: In-Reply-To: <89450a5629b9420ab70a09300cb7b8e7@git.apache.org> References: <89450a5629b9420ab70a09300cb7b8e7@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [05/10] incubator-rocketmq-site git commit: Add content to git repo. archived-at: Sat, 24 Dec 2016 06:55:56 -0000 http://git-wip-us.apache.org/repos/asf/incubator-rocketmq-site/blob/5479307c/content/categories/index.html ---------------------------------------------------------------------- diff --git a/content/categories/index.html b/content/categories/index.html new file mode 100644 index 0000000..ed4c234 --- /dev/null +++ b/content/categories/index.html @@ -0,0 +1,440 @@ + + + + + + + + + + + + + + + +Posts by Category - Apache RocketMQ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+
+
+ + + + + +
+ +
+ + +
+ + +
+ + + +
+ + +
+

+ +

+ A fast, low latency, reliable, scalable, distributed MOM. +

+ +
+ +
+ + +
+
+ + +
+ + +
+ +

Posts by Category

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Maven

+ + + + + +
+
+ +

+ + Mastering Component Compatible Dependency + + +

+ +

+ + + + + 11 minute read +

+ +

This article mainly includes three parts.at first,I will introduce compatibility principle(more details see here) briefly.followed by a detailed elaborating ...

+
+
+ + + +

RocketMQ

+ + + + + +
+ +
+ + +
+
+ +
+ +
+ + + + + + + + + http://git-wip-us.apache.org/repos/asf/incubator-rocketmq-site/blob/5479307c/content/collection-archive/index.html ---------------------------------------------------------------------- diff --git a/content/collection-archive/index.html b/content/collection-archive/index.html new file mode 100644 index 0000000..3d01a0f --- /dev/null +++ b/content/collection-archive/index.html @@ -0,0 +1,468 @@ + + + + + + + + + + + + + + + +Posts by Collection - Apache RocketMQ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+
+
+ + + + + +
+ +
+ + +
+ + +
+ + + +
+ + +
+

+ +

+ A fast, low latency, reliable, scalable, distributed MOM. +

+ +
+ +
+ + +
+
+ + +
+ + +
+ +

Posts by Collection

+ + + + + + + +

docs

+ + + + + + + + + +
+
+ +

+ + Quick Start + + +

+ +

How to quickly install and setup Apache RocketMQ. +

+
+
+ + + + + + + +
+
+ +

+ + Motivation + + +

+ +

In the early stages, we constructed our distributed messaging middleware on the basis of ActiveMQ 5.x(less than 5.3). Our international business uses it for ...

+
+
+ + + + + + + +
+ +
+ + + + + + + +
+
+ +

+ + CLI Admin Tool + + +

+ +

RocketMQ provides a CLI admin tool belt to query, manage and diagnose various issues. + +

+
+
+ + + + + + + +
+ +
+ + + + + + +

pets

+ + + + + + + + +

portfolio

+ + + + + + + + + + + + + + + +

recipes

+ + + + + +
+
+ +
+ +
+ + + + + + + + + http://git-wip-us.apache.org/repos/asf/incubator-rocketmq-site/blob/5479307c/content/community/index.html ---------------------------------------------------------------------- diff --git a/content/community/index.html b/content/community/index.html new file mode 100644 index 0000000..d65a7ca --- /dev/null +++ b/content/community/index.html @@ -0,0 +1,455 @@ + + + + + + + + + + + + + + + +Community - Apache RocketMQ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+
+
+ + + + + + + + + + +
+ +
+

+ + Community + + +

+ +

Apache RocketMQ is a low latency, reliable, scalable, distributed, easy to use message-oriented middleware, especially for processing large amounts of streaming data.
Latest release v3.5.8

+

+ + + +

Try it Now

+ +
+ + +
+ + +
+
+ + + + + +
+ +
+ + + + + +
+
+ + +
+ + + +
+

Who’s Involved in RocketMQ ?  

+ +
+ + + +
+
+
+ + +
+ +
+ + + + + +
+
+ +
+ Alibaba Group +
+ + +
+ + + + + +
+
+
+ + + + +
+
+ +
+ China Telecom Group +
+ + +
+ + + + + +
+
+
+ + + + +
+
+ +
+ YeahMobi Group +
+ + +
+ + + + + +
+
+
+ + +
+ +
+ + + + + +
+
+ +
+ Egov Group +
+ + +
+ + + + + +
+
+
+ + + + +
+
+ +
+ China Telecom Group +
+ + +
+ + + + + +
+
+
+ + + + +
+
+ +
+ Media Group +
+ + +
+ + + + + +
+
+
+ + +
+ +
+ + + + + +
+
+ +
+ Decent Group +
+ + +
+ + + + + +
+
+
+ + + + +
+
+ +
+ Yunpan Group +
+ + +
+ + + + + +
+
+
+ + + + +
+
+ +
+ Shihai Group +
+ + +
+ + + + + +
+
+
+ + +
+ +
+
+
+ +
+ +
+ + + + + + + + + http://git-wip-us.apache.org/repos/asf/incubator-rocketmq-site/blob/5479307c/content/docs/cli-admin-tool/index.html ---------------------------------------------------------------------- diff --git a/content/docs/cli-admin-tool/index.html b/content/docs/cli-admin-tool/index.html new file mode 100644 index 0000000..e8a150e --- /dev/null +++ b/content/docs/cli-admin-tool/index.html @@ -0,0 +1,509 @@ + + + + + + + + + + + + + + + +CLI Admin Tool - Apache RocketMQ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+
+
+ + + + + +
+ +
+ + + + + + + + + + + + + +
+ + +
+ + + + + +
+ +
+

CLI Admin Tool +

+ +
+ + +
+

RocketMQ provides a CLI admin tool belt to query, manage and diagnose various issues.

+ + + +

Prerequisite

+ +

Make sure you have walked through Quick Start and Core Concept sections.

+ +

How To Get it

+ +

The admin tool is shipped along with RocketMQ. Either you download a pre-built binary version or build from source by yourself, you already have it.

+ +

In case you have source code, the rocketmq-tools module contains its source code.

+ +

How to use

+ +

The Admin Tool is very easy to use. Here, for demonstration purpose, *nix environment is assumed.

+ +

Change directory to ${PACKAGE}/bin, command bash mqadmin, you should see the following help menu pops out.

+ +
The most commonly used mqadmin commands are:
+   updateTopic          Update or create topic
+   deleteTopic          Delete topic from broker and NameServer.
+   updateSubGroup       Update or create subscription group
+   deleteSubGroup       Delete subscription group from broker.
+   updateBrokerConfig   Update broker's config
+   updateTopicPerm      Update topic perm
+   topicRoute           Examine topic route info
+   topicStatus          Examine topic Status info
+   topicClusterList     get cluster info for topic
+   brokerStatus         Fetch broker runtime status data
+   queryMsgById         Query Message by Id
+   queryMsgByKey        Query Message by Key
+   queryMsgByUniqueKey  Query Message by Unique key
+   queryMsgByOffset     Query Message by offset
+   queryMsgByUniqueKey  Query Message by Unique key
+   printMsg             Print Message Detail
+   sendMsgStatus        send msg to broker.
+   brokerConsumeStats   Fetch broker consume stats data
+   producerConnection   Query producer's socket connection and client version
+   consumerConnection   Query consumer's socket connection, client version and subscription
+   consumerProgress     Query consumers's progress, speed
+   consumerStatus       Query consumer's internal data structure
+   cloneGroupOffset     clone offset from other group.
+   clusterList          List all of clusters
+   topicList            Fetch all topic list from name server
+   updateKvConfig       Create or update KV config.
+   deleteKvConfig       Delete KV config.
+   wipeWritePerm        Wipe write perm of broker in all name server
+   resetOffsetByTime    Reset consumer offset by timestamp(without client restart).
+   updateOrderConf      Create or update or delete order conf
+   cleanExpiredCQ       Clean expired ConsumeQueue on broker.
+   cleanUnusedTopic     Clean unused topic on broker.
+   startMonitoring      Start Monitoring
+   statsAll             Topic and Consumer tps stats
+   syncDocs             Synchronize wiki and issue to github.com
+   allocateMQ           Allocate MQ
+   checkMsgSendRT       check message send response time
+   clusterRT            List All clusters Message Send RT
+
+See 'mqadmin help <command>' for more information on a specific command.
+
+
+ +

As you see, the most commonly used commands are listed with a brief description. To get detailed manual of each command, bash mqadmin help <command>. For example, command bash mqadmin help clusterList will bring out the following help text:

+ +
usage: mqadmin clusterList [-h] [-i <arg>] [-m] [-n <arg>]
+ -h,--help                Print help
+ -i,--interval <arg>      specify intervals numbers, it is in seconds
+ -m,--moreStats           Print more stats
+ -n,--namesrvAddr <arg>   Name server address list, eg: 192.168.0.1:9876;192.168.0.2:9876
+
+
+ +

The help text lists possible options and interpretation of each option.

+ +

Command Manual

+ + +
+ +
+ + + + + +

Updated:

+ +
+ + + + + + +
+ + +
+ + + +
+ +
+ +
+ + + + + + + + + http://git-wip-us.apache.org/repos/asf/incubator-rocketmq-site/blob/5479307c/content/docs/cluster-deployment/index.html ---------------------------------------------------------------------- diff --git a/content/docs/cluster-deployment/index.html b/content/docs/cluster-deployment/index.html new file mode 100644 index 0000000..0b35350 --- /dev/null +++ b/content/docs/cluster-deployment/index.html @@ -0,0 +1,512 @@ + + + + + + + + + + + + + + + +Cluster Configuration and Deployment - Apache RocketMQ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+
+
+ + + + + +
+ +
+ + + + + + + + + + + + + +
+ + +
+ + + + + +
+ +
+

Cluster Configuration and Deployment +

+ +
+ + +
+

Prerequisite

+ +

Before starting this section, make sure you have read Quick Start section, in which core concepts and components of RocketMQ are introduced.

+ + + +

Introduction

+ +

This section is to introduce deployment solution that may be considered production ready. Generally speaking, we are deploying a resilient RocketMQ cluster having no single point of failure.

+ +

Name Server

+ +

To ensure the cluster can still operate normally when name server instance crashes, two or more name server instances are recommended. As long as there is one name server instance alive, the whole cluster remains serving.

+ +

Name server follows share-nothing design paradigm. Brokers send heartbeat data to all name servers. Producers and consumers may query meta data from any one of name servers available while sending / consuming messages.

+ +

Broker

+ +

Brokers can be divided into two categories according to their roles: master and slave. Master brokers can read and write while slave brokers can only read messages replicated from master. Master brokers have brokerId=0; brokerId of slave is non-zero.

+ +

Brokers may also be grouped by the brokerName property. Slave brokers find their master through brokerName. One master broker along with zero, one or multiple slave brokers is called a broker set.

+ +

In serious scenarios, we should have at least two broker sets. Each topic, as you guess, resides in two or more broker sets.

+ +

Disk Flush Type: SYNC_FLUSH / ASYNC_FLUSH

+ +

Each broker can be configured to be sync or async in terms of IO flushing strategy. To put it simply, if your business tolerates none message loss at extreme scenarios like OS crash or power loss, SYNC_FLUSH mode is recommended. Broker of SYNC_FLUSH mode would flush each message it receives onto disk before acknowledging producer. Obviously, throughout of this mode is highly dependent on IOPS of your hard disk.

+ +

Broker of ASYNC_FLUSH mode, on the other hand, takes advantage of group-committing: only flushes dirty page caches to disk periodically or when count of dirty page caches reaches certain amount. Hence, the latter yields better performance.

+ +

Replication: Sync / Async Broker

+ +

Similar to many replication system, sync brokers await until commit log is replicated to a slave before acknowledging. Async brokers, instead, return immediately after messages are processed on master.

+ +

Configuration

+ +

There are three pre-built configurations shipped with the distribution of RocketMQ under conf folder for your reference:

+ +
    +
  1. 2m-2s-sync
    2. +
  2. 2m-2s-async
    3. +
  3. 2m-noslave
    4. +
+ +

Note: all configurations uses ASYNC_FLUSH.

+ +

Deployment Example

+ +

For example, We want to have a cluster with 2 name servers, 2 broker sets with general purpose: +for this example, we choose 2m-2s-sync +Assuming binary RocketMQ is at /home/rocketmq/dist

+ +
    +
  1. +

    Start up two name servers as is shown in Quick Start guide. Assume their IPs are 192.168.0.2 and 192.168.0.3.

    +
    2. +
  2. +

    Start brokers

    + +

    cd /home/rocketmq/dist/bin

    + +

    bash mqbroker -c ../conf/2m-2s-sync/broker-a.properties -n 192.168.0.2:9876,192.168.0.3:9876

    + +

    bash mqbroker -c ../conf/2m-2s-sync/broker-a-s.properties -n 192.168.0.2:9876,192.168.0.3:9876

    + +

    bash mqbroker -c ../conf/2m-2s-sync/broker-b.properties -n 192.168.0.2:9876,192.168.0.3:9876

    + +

    bash mqbroker -c ../conf/2m-2s-sync/broker-b-s.properties -n 192.168.0.2:9876,192.168.0.3:9876

    +
    3. +
  3. +

    Verify

    + +

    Execute the following command to verify

    + +

    bash mqadmin clusterList

    +
    4. +
+ + +
+ +
+ + + + + +

Updated:

+ +
+ + + + + + +
+ + +
+ + + +
+ +
+ +
+ + + + + + + + + http://git-wip-us.apache.org/repos/asf/incubator-rocketmq-site/blob/5479307c/content/docs/core-concept/index.html ---------------------------------------------------------------------- diff --git a/content/docs/core-concept/index.html b/content/docs/core-concept/index.html new file mode 100644 index 0000000..b4ace00 --- /dev/null +++ b/content/docs/core-concept/index.html @@ -0,0 +1,548 @@ + + + + + + + + + + + + + + + +Core Concept - Apache RocketMQ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+
+
+ + + + + +
+ +
+ + + + + + + + + + + + + +
+ + +
+ + + + + +
+ +
+

Core Concept +

+ +
+ + +
+

RocketMQ model

+ +

From the above model, we can look deeper into some topics about messaging system design:

+ + + +
    +
  • Consumer Concurrency
    • +
  • Consumer Hot Issues
    • +
  • Consumer Load Balance
    • +
  • Message Router
    • +
  • Connection Multiplex
    • +
  • Canary Deployments
    • +
+ +

Producer

+ +

A producer sends messages generated by the business application systems to brokers. RocketMQ provides multiple paradigms of sending: synchronous, asynchronous and one-way.

+ +

Producer Group

+ +

Producers of the same role are grouped together. A different producer instance of the same producer group may be contacted by a broker to commit or roll back a transaction in case the original producer crashed after starting the transaction.

+ +

Warning: Considering the provided producer is sufficiently powerful at sending messages, only one instance is allowed per producer group and process to avoid unnecessarily initializing of producer instances.

+ +

Consumer

+ +

A Consumer pulls messages from brokers and feeds them into application. In perspective of user application, two types of consumers are provided:

+ +

PullConsumer

+ +

Pull consumer actively pulls messages from brokers. Once batches of messages are pulled, user application initiates consuming process.

+ +

PushConsumer

+ +

Push consumer, on the other hand, encapsulates message pulling, consuming progress maintaining and other effortful work inside, leaving a callback interface to end user to implement which will be executed on message arrival.

+ +

Consumer Group

+ +

Similar to previously mentioned producer group, consumers of the exactly same role are grouped together and named Consumer Group.

+ +

Consumer Group is a great concept with which achieving goals of load-balance and fault-tolerance, in terms of message consuming, is super easy.

+ +

Warning: consumer instances of a consumer group must have exactly same topic subscription(s).

+ +

Topic

+ +

Topic is a category to which producers deliver messages and from which consumers pull messages. Topics have very loose relation with producers and consumers. Specifically, a topic may have zero, one or multiple producers that sends messages to it; conversely, a producer can sends messages of different topics. In consumer’s view, a topic may be subscribed by zero, one or multiple consumer groups; and a consumer group, in the same paradigm, may subscribe one or multiple topics as long as instances of this group keep their subscription consistent as emphasized in the previous section.

+ +

Message

+ +

Message is the envelope of your information to deliver. A message must be specified with a topic, which can be interpreted as address of your letter to mail to. A message may also have an optional tag set. Extra key-value pairs may also be included. For example, you may set a business key for your message and look up the message on broker server to diagnose issues during development.

+ +

Message Queue

+ +

Topic, internally, is logically partitioned into one or more sub-topics. We call these sub-topics “message queues”. This concept plays a major role in implementing valuable features, including fail-over, maximum concurrency, etc.

+ +

Tag

+ +

Tag, which can be thought as sub-topic, provides an extra flexibility for user. Through introducing tag, messages with different purposes from the same business module may have the same topic yet different tag. It would be helpful to keep your code clean and coherent.

+ +

Broker

+ +

Broker is the major role of the RocketMQ system. It receives messages sent from producers, store them and being prepared to serve pull requests from consumers. It also stores message consuming related meta data, including consumer groups, consuming progress offsets and topic / queue info.

+ +

Name Server

+ +

Name server serves as the routing information provider. Producer/Consumer clients look up topics to find broker list to read from and write to.

+ +

Message Model

+ +
    +
  • +

    Clustering

    +
    • +
  • +

    Broadcasting

    +
    • +
+ +

Message Order

+ +

When DefaultMQPushConsumer is employed, you may decide to consume messages orderly or concurrently.

+ +
    +
  • Orderly
    • +
+ +

Consuming messages orderly means messages are consumed the same order they are sent by producers for each message queue. If you are dealing with scenario that global order is mandatory, make sure the topic you use has only one message queue.

+ +

Warn: If consuming orderly is specified, the maximum concurrency of message consuming is the number of message queues subscribed by the consumer group.

+ +
    +
  • Concurrently
    • +
+ +

When consuming concurrently, maximum concurrency of message consuming is only limited by thread pool specified for each consumer client.

+ +

Warn: Message order is no longer guaranteed in this mode.

+ + + +
+ +
+ + + + + +

Updated:

+ +
+ + + + + + +
+ + +
+ + + +
+ +
+ +
+ + + + + + + + +