From commits-return-43722-archive-asf-public=cust-asf.ponee.io@qpid.apache.org Thu Feb 8 22:14:57 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id D1B091807A6 for ; Thu, 8 Feb 2018 22:14:55 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id C13C6160C5D; Thu, 8 Feb 2018 21:14:55 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B2006160C6A for ; Thu, 8 Feb 2018 22:14:52 +0100 (CET) Received: (qmail 89568 invoked by uid 500); 8 Feb 2018 21:14:51 -0000 Mailing-List: contact commits-help@qpid.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@qpid.apache.org Delivered-To: mailing list commits@qpid.apache.org Received: (qmail 89348 invoked by uid 99); 8 Feb 2018 21:14:51 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Feb 2018 21:14:51 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id D9508E944D; Thu, 8 Feb 2018 21:14:49 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: orudyy@apache.org To: commits@qpid.apache.org Date: Thu, 08 Feb 2018 21:14:53 -0000 Message-Id: <4dbddf132369492380d8b81efeded4e1@git.apache.org> In-Reply-To: <271a997bf3864edb80636b74e3eba227@git.apache.org> References: <271a997bf3864edb80636b74e3eba227@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [05/32] qpid-site git commit: QPID-8094: Update site for Qpid Broker-J release 7.0.1 http://git-wip-us.apache.org/repos/asf/qpid-site/blob/42d76007/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime-Memory.html.in ---------------------------------------------------------------------- diff --git a/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime-Memory.html.in b/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime-Memory.html.in new file mode 100644 index 0000000..3a63cf1 --- /dev/null +++ b/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime-Memory.html.in @@ -0,0 +1,138 @@ +
9.11. Memory
Prev Chapter 9. Runtime Next

9.11. Memory

9.11.1. Introduction

+ Understanding how the Qpid broker uses memory is essential to running a high performing and reliable service. + A wrongly configured broker can exhibit poor performance or even crash with an OutOfMemoryError. + Unfortunately, memory usage is not a simple topic and thus requires some in depth explanations. + This page should give the required background information to make informed decisions on how to configure your broker. +

+ Section 9.11.2, “Types of Memory” explains the two different kinds of Java memory most relevant to the broker. + Section 9.11.3, “Memory Usage in the Broker” goes on to explain which parts of the broker use what kind of memory. + Section 9.11.4, “Low Memory Conditions” explains what happens when the system runs low on memory. + Section 9.11.5, “Defaults” lays out the default settings of the Qpid broker. + Finally, Section 9.11.6, “Memory Tuning the Broker” gives some advice on tuning your broker. +

9.11.2. Types of Memory

+ While Java has a couple of different internal memory types we will focus on the two types that are relevant to the Qpid broker. + Both of these memory types are taken from the same physical memory (RAM). +

9.11.2.1. Heap

+ Normally, all objects are allocated from Java's heap memory. + Once, nothing references an object it is cleaned up by the Java Garbage Collector and it's memory returned to the heap. + This works fine for most use cases. + However, when interacting with other parts of the operating system using Java's heap is not ideal. + This is where the so called direct memory comes into play. +

9.11.2.2. Direct

+ The world outside of the JVM, in particular the operating system (OS), does not know about Java heap memory and uses other structures like C arrays. + In order to interact with these systems Java needs to copy data between its own heap memory and these native structures. + This can become a bottle neck when there is a lot of exchange between Java and the OS like in I/O (both disk and network) heavy applications. + Java's solution to this is to allow programmers to request ByteBuffers from so called direct memory. + This is an opaque structure that might have an underlying implementation that makes it efficient to interact with the OS. + Unfortunately, the GC is not good at tracking direct memory and in general it is inadvisable to use direct memory for regular objects. +

9.11.3. Memory Usage in the Broker

+ This section lists some note worthy users of memory within the broker and where possible lists their usage of heap and direct memory. + Note that to ensure smooth performance some heap memory should remain unused by the application and be reserved for the JVM to do house keeping and garbage collection. + Some guides advise to reserve up to 30% of heap memory for the JVM. +

9.11.3.1. Broker

+ The broker itself uses a moderate amount of heap memory (≈15 MB). + However, each connection and session comes with a heap overhead of about 17 kB and 15 kB respectively. + In addition, each connection reserves 512 kB direct memory for network I/O. +

9.11.3.2. Virtual Hosts

+ The amount of memory a Virtual Host uses depends on its type. + For a JSON Virtual Host Node with a BDB Virtual Host the heap memory usage is approximately 2 MB. + However, each BDB Virtual Hosts has a mandatory cache in heap memory which has an impact on performance. + See below for more information. +

9.11.3.3. Messages

+ Messages and their headers are kept in direct memory and have an additional overhead of approximately 1 kB heap memory each. + This means that most brokers will want to have more direct memory than heap memory. + When many small messages accumulate on the broker the 1 kB heap memory overhead can become a limiting factor. +

+ When the broker is running low on direct memory + it will evict enqueued messages from memory and flow them to disk. + For persistent messages this only means freeing the direct memory representation because they always have an on-disk representation to guard against unexpected failure (e.g., a power cut). + For transient messages this implies additional disk I/O. + After being flowed to disk messages need to be re-read from disk before delivery. +

Please, note that messages from uncommitted transactions are not + flowed to disk as part of + running into low direct memory conditions, + as they are not enqueued yet. The Connection has its own threshold for + keeping messages from uncommitted transactions in memory. Only when Connection threshold + is breached, the uncommitted messages on the connection are + flowed to disk.

9.11.3.4. Message Store

Berkeley DB (BDB)

+ The broker can use Oracle's BDB JE (BDB) as a message store to persist messages by writing them to a database. + BDB uses a mandatory cache for navigating and organising its database structure. + Sizing and tuning this cache is a topic of its own and would go beyond the scope of this guide. + Suffice to say that by default Qpid uses 5% of heap memory for BDB caches (each Virtual Host uses a separate cache) or 10 MB per BDB store, whichever is greater. + See the official webpage especially this page for more information. + For those interested, Qpid uses EVICT_LN as its default JE cacheMode. +

+ Note that due to licensing concerns Qpid does not ship the BDB JE jar files. +

Derby

+ TODO +

9.11.3.5. HTTP Management

+ Qpid uses Jetty for the HTTP Management (both REST and Web Management Console). + When the management plugin is loaded it will allocate the memory it needs and should not require more memory during operation and can thus be largely ignored. +

9.11.4. Low Memory Conditions

9.11.4.1. Low on Heap Memory

+ When the broker runs low on heap memory performance will degrade because the JVM will trigger full garbage collection (GC) events in a struggle to free memory. + These full GC events are also called stop-the-world events as they completely halt the execution of the Java application. + Stop-the-world-events may take any where from a couple of milliseconds up to several minutes. + Should the heap memory demands rise even further the JVM will eventually throw an OutOfMemoryError which will cause the broker to shut down. +

9.11.4.2. Low on Direct Memory

+ When the broker detects that it uses 75% of available direct memory it will start flowing incoming transient messages to disk and reading them back before delivery. + This will prevent the broker from running out of direct memory but may degrade performance by requiring disk I/O. +

9.11.5. Defaults

+ By default Qpid uses these settings: +

  • + 0.5 GB heap memory +
  • + 1.5 GB direct memory +
  • + 5% of heap reserved for the BDB JE cache. +
  • + Start flow-to-disk at 75% direct memory utilisation. +

+ As an example, this would accommodate a broker with 50 connections, each serving 5 sessions, and each session having 1000 messages of 1 kB on queues in the broker. + This means a total of 250 concurrent sessions and a total of 250000 messages without flowing messages to disk. +

9.11.6. Memory Tuning the Broker

9.11.6.1. Java Tuning

+ Most of these options are implementation specific. It is assumed you are using Oracle Java 1.8. +

+

9.11.6.2. Qpid Tuning

  • + The system property qpid.broker.bdbTotalCacheSize sets the total amount of heap memory (in bytes) allocated to BDB caches. +
  • + The system property broker.flowToDiskThreshold sets the threshold (in bytes) for flowing transient messages to disk. + Should the broker use more than direct memory it will flow incoming messages to disk. + Should utilisation fall beneath the threshold it will stop flowing messages to disk. +
  • + The system property connection.maxUncommittedInMemorySize sets the threshold (in bytes) + for total messages sizes (in bytes) from connection uncommitted transactions when messages are hold in memory. + If threshold is exceeded, all messages from connection in-flight transactions are flowed to disk including + those arriving after breaching the threshold. +

9.11.6.3. Formulae

+ We developed a simple formula which estimates the minimum memory usage of the broker under certain usage. + These are rough estimate so we strongly recommend testing your configuration extensively. + Also, if your machine has more memory available by all means use more memory as it can only improve the performance and stability of your broker. + However, remember that both heap and direct memory are served from your computer's physical memory so their sum should never exceed the physically available RAM (minus what other processes use). +

+

+ memoryheap = 15 MB + 20 kB * Nsessions + 1.5 kB * Nmessages + 100 kB * Nconnections +

+

+

+ memorydirect = 2 MB + (200 B + averageSizemsg *2)* Nmessages + 1MB * Nconnections +

+

+ Where N denotes the total number of connections/sessions/messages on the broker. Furthermore, for direct memory only the messages that have not been flowed to disk are relevant. +

Note

The formulae assume the worst case in terms of memory usage: persistent messages and TLS connections. Transient messages consume less heap memory than persistent and plain connections consume less direct memory than TLS + connections. +

9.11.6.4. Things to Consider

Performance

+ Choosing a smaller direct memory size will lower the threshold for flowing transient messages to disk when messages accumulate on a queue. + This can have impact on performance in the transient case where otherwise no disk I/O would be involved. +

+ Having too little heap memory will result in poor performance due to frequent garbage collection events. See Section 9.11.4, “Low Memory Conditions” for more details. +

OutOfMemoryError

+ Choosing too low heap memory can cause an OutOfMemoryError which will force the broker to shut down. + In this sense the available heap memory puts a hard limit on the number of messages you can have in the broker at the same time. +

+ If the Java runs out of direct memory it also throws a OutOfMemoryError resulting the a broker shutdown. + Under normal circumstances this should not happen but needs to be considered when deviating from the default configuration, especially when changing the flowToDiskThreshold. +

+ If you are sending very large messages you should accommodate for this by making sure you have enough direct memory. +

Prev Up Next
9.10. Connection Limits Home Chapter 10. High Availability
\ No newline at end of file http://git-wip-us.apache.org/repos/asf/qpid-site/blob/42d76007/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime-Message-Compression.html.in ---------------------------------------------------------------------- diff --git a/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime-Message-Compression.html.in b/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime-Message-Compression.html.in new file mode 100644 index 0000000..b9c382f --- /dev/null +++ b/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime-Message-Compression.html.in @@ -0,0 +1,15 @@ +
9.9. Message Compression
Prev Chapter 9. Runtime Next

9.9. Message Compression

The Apache Qpid Broker-J supports[10] message compression. This feature works in co-operation with Qpid + Clients implementing the same feature.

Once the feature is enabled (using Broker context variable + broker.messageCompressionEnabled), the Broker will advertise support for the + message compression feature to the client at connection time. This allows clients to opt to turn + on message compression, allowing message payload sizes to be reduced.

If the Broker has connections from clients who have message compression enabled and others who + do not, it will internally, on-the-fly, decompress compressed messages when sending to clients + without support and conversely, compress uncomressed messages when sending to clients who do.

The Broker has a threshold below which it will not consider compressing a message, this is + controlled by Broker content variable + (connection.messageCompressionThresholdSize) and expresses a size in bytes.

This feature may have a beneficial effect on performance by:

  • Reducing the number of bytes transmitted over the wire, both between Client and Broker, and + in the HA case, Broker to Broker, for replication purposes.

  • Reducing storage space when data is at rest within the Broker, both on disk and in + memory.

Of course, compression and decompression is computationally expensive. Turning on the feature + may have a negative impact on CPU utilization on Broker and/or Client. Also for small messages + payloads, message compression may increase the message size. It is recommended to test the feature + with representative data.


[10] Message compression is not yet supported for the 1.0 + protocol.

Prev Up Next
9.8. Background Recovery Home 9.10. Connection Limits
\ No newline at end of file http://git-wip-us.apache.org/repos/asf/qpid-site/blob/42d76007/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime-Producer-Transaction-Timeout.html.in ---------------------------------------------------------------------- diff --git a/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime-Producer-Transaction-Timeout.html.in b/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime-Producer-Transaction-Timeout.html.in new file mode 100644 index 0000000..5f21b25 --- /dev/null +++ b/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime-Producer-Transaction-Timeout.html.in @@ -0,0 +1,60 @@ +
9.3. Producer Transaction Timeout
Prev Chapter 9. Runtime Next

9.3. Producer Transaction Timeout

9.3.1. General Information

Note

+ This feature is not implemented for producing applications using AMQP 1.0. If a long running transaction is suspected + examine the connection statistic oldestTransactionStartTime to confirm if this is the case. +

The transaction timeout mechanism is used to control broker resources when clients + producing messages using transactional sessions hang or otherwise become unresponsive, or simply + begin a transaction and keep using it without ever calling Session#commit().

Users can choose to configure an idleWarn or openWarn threshold, after which the identified + transaction should be logged as a WARN level alert as well as (more importantly) an idleClose or + openClose threshold after which the transaction and the connection it applies to will be + closed.

This feature is particularly useful in environments where the owner of the broker does not + have full control over the implementation of clients, such as in a shared services + deployment.

The following section provide more details on this feature and its use.

9.3.2. Purpose

This feature has been introduced to address the scenario where an open transaction on the + broker holds an open transaction on the persistent store. This can have undesirable consequences + if the store does not time out or close long-running transactions, such as with BDB. This can can + result in a rapid increase in disk usage size, bounded only by available space, due to growth of + the transaction log.

9.3.3. Scope

Note that only MessageProducer clients will be affected by a transaction timeout, since store + transaction lifespan on a consumer only spans the execution of the call to Session#commit() and + there is no scope for a long-lived transaction to arise.

It is also important to note that the transaction timeout mechanism is purely a JMS + transaction timeout, and unrelated to any other timeouts in the Qpid client library and will have + no impact on any RDBMS your application may utilise.

9.3.4. Effect

Full details of configuration options are provided in the sections that follow. This section + gives a brief overview of what the Transaction Timeout feature can do.

9.3.4.1. Broker Logging and Connection Close

When the openWarn or idleWarn specified threshold is exceeded, the broker will log a WARN + level alert with details of the connection and channel on which the threshold has been exceeded, + along with the age of the transaction.

When the openClose or idleClose specified threshold value is exceeded, the broker will + throw an exception back to the client connection via the ExceptionListener, log the + action and then close the connection.

The example broker log output shown below is where the idleWarn threshold specified is + lower than the idleClose threshold and the broker therefore logs the idle transaction 3 times + before the close threshold is triggered and the connection closed out.

CHN-1008 : Idle Transaction : 13,116 ms
+CHN-1008 : Idle Transaction : 14,116 ms
+CHN-1008 : Idle Transaction : 15,118 ms
+CHN-1003 : Close
+

The second example broker log output shown below illustrates the same mechanism operating + on an open transaction.

+CHN-1007 : Open Transaction : 12,406 ms
+CHN-1007 : Open Transaction : 13,406 ms
+CHN-1007 : Open Transaction : 14,406 ms
+CHN-1003 : Close
+

9.3.4.2. Client Side Effect

After a Close threshold has been exceeded, the trigger client will receive this exception + on its exception + listener, prior to being disconnected:

org.apache.qpid.AMQConnectionClosedException: Error: Idle transaction timed out + [error code 506: resource error]

Any later attempt to use the connection will result in this exception being thrown:

Producer: Caught an Exception: javax.jms.IllegalStateException: Object org.apache.qpid.client.AMQSession_0_8@129b0e1 has been closed
+    javax.jms.IllegalStateException: Object org.apache.qpid.client.AMQSession_0_8@129b0e1 has been closed
+    at org.apache.qpid.client.Closeable.checkNotClosed(Closeable.java:70)
+    at org.apache.qpid.client.AMQSession.checkNotClosed(AMQSession.java:555)
+    at org.apache.qpid.client.AMQSession.createBytesMessage(AMQSession.java:573)
+

Thus clients must be able to handle this case successfully, reconnecting where required and + registering an exception listener on all connections. This is critical, and must be communicated + to client applications by any broker owner switching on transaction timeouts.

9.3.5. Configuration

9.3.5.1. Configuration

The transaction timeouts can be specified when a new virtualhost is created or an exiting + virtualhost is edited.

We would recommend that only warnings are configured at first, which should allow broker + administrators to obtain an idea of the distribution of transaction lengths on their systems, + and configure production settings appropriately for both warning and closure. Ideally + establishing thresholds should be achieved in a representative UAT environment, with clients and + broker running, prior to any production deployment.

It is impossible to give suggested values, due to the large variation in usage depending on + the applications using a broker. However, clearly transactions should not span the expected + lifetime of any client application as this would indicate a hung client.

When configuring warning and closure timeouts, it should be noted that these only apply to + message producers that are connected to the broker, but that a timeout will cause the connection + to be closed - this disconnecting all producers and consumers created on that connection.

This should not be an issue for environments using Mule or Spring, where connection + factories can be configured appropriately to manage a single MessageProducer object per JMS + Session and Connection. Clients that use the JMS API directly should be aware that sessions + managing both consumers and producers, or multiple producers, will be affected by a single + producer hanging or leaving a transaction idle or open, and closed, and must take appropriate + action to handle that scenario.

Prev Up Next
9.2. Disk Space Management Home 9.4. Handing Undeliverable Messages
\ No newline at end of file http://git-wip-us.apache.org/repos/asf/qpid-site/blob/42d76007/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime.html.in ---------------------------------------------------------------------- diff --git a/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime.html.in b/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime.html.in new file mode 100644 index 0000000..3b31501 --- /dev/null +++ b/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Runtime.html.in @@ -0,0 +1,97 @@ +
Chapter 9. Runtime
Prev   Next

Chapter 9. Runtime

Table of Contents

9.1. Logging
9.1.1. Concepts
9.1.2. Default Configuration
9.1.3. Loggers
9.1.4. Inclusion Rules
9.1.5. Logging Management
9.2. Disk Space Management
9.2.1. Disk quota-based flow control
9.3. Producer Transaction Timeout
9.3.1. General Information
9.3.2. Purpose
9.3.3. Scope
9.3.4. Effect
9.3.5. Configuration
9.4. Handing Undeliverable Messages
9.4.1. Introduction
9.4.2. Maximum Delivery Count
9.4.3. Alternate Binding
9.5. Closing client connections on unroutable mandatory messages
9.5.1. Summary
9.5.2. Configuring + closeWhenNoRoute +
9.6. Flow to Disk
9.7. Consumers
9.7.1. Priority
9.8. Background Recovery
9.9. Message Compression
9.10. Connection Limits
9.11. Memory
9.11.1. Introduction
9.11.2. Types of Memory
9.11.3. Memory Usage in the Broker
9.11.4. Low Memory Conditions
9.11.5. Defaults
9.11.6. Memory Tuning the Broker

9.1. Logging

This section describes the flexible logging capabilities of the Ap ache Qpid Broker-J.

+

  • The Broker is capable of sending logging events to a variety of destinations including + plain files, remote syslog daemons, and an in-memory buffer (viewable from Management). + The system is also open for extension meaning it is possible to produce a plugin to log to + a bespoke destination.

  • Logging can be dynamically configured at runtime. For instance, it is possible to + temporarily increase the logging verbosity of the system whilst a problem is investigated + and then revert later, all without the need to restart the Broker.

  • Virtualhosts can be configured to generate their own separate log, and the Broker is + capable of generating a log either inclusive or exclusive of virtualhost events.

  • Logs are accessible over Management, removing the need for those operating the Broker + to have shell level access.

+

In the remainder of this section you will first find a description of the concepts used in + the logging subsystem. Next, you find a description of the default configuration. The section + then concludes with a in-depth description of the loggers themselves and how they may be + configured.

9.1.1. Concepts

The logging subsystem uses two concepts:

+

  • A Logger is responsible for production of a log. The Broker + ships a variety of loggers, for instance, a file logger, which is capable of writing a + log file to the file system, a Syslog Logger capable of writing to a remote syslog + daemon and console logger capable of writing to stdout or stderr.

    Loggers are attached at two points within the Broker Model; the Broker itself and + the virtualhosts. Loggers attached at the Broker can capture log events for the system + as a whole, or can exclude events related to virtualhosts.

    Loggers attached to a virtualhost capture log events relating to that virtualhost + only.

    The Broker and virtualhosts can have zero or more Loggers. If no loggers are + configured, no logging is generated at all.

  • Inclusion rules govern what appears within a log. Inclusion + rules are associated with Loggers. This means it is possible for different Loggers to + have different contents.

    A Logger with no inclusion rules will produce an empty log.

+

9.1.2. Default Configuration

The default configuration is designed to be suitable for use without change in small + production environments. It has the following characteristics:

+

  • The Broker generates a single log file qpid.log. This logfile is + rolled automatically when the file reaches 100MB. A maximum history of one file is + retained. On restart the the log will be appended to.

    The log contains:

    • All operational logging events. See Appendix C, Operational Logging.

    • Log events from Qpid itself deemed informational or + higher.

    • Log events from Qpid's dependencies (such as Derby or Jetty) that are + deemed warning or higher.

    +

    The default location for the log file is + ${QPID_WORK}/log/qpid.log.

  • The Broker also caches the last 4096 log events in a memory cache. By default, the + memory logger logs the same things the file logger does.

+

The configuration can be customised at runtime using Management. This makes it possible to + investigate unusual conditions without the need to restart the Broker. + For instance, you may alter the logging level so that a verbose log is produced whilst an + investigation is in progress and revert the setting later, all without the need to restart the + Broker.

9.1.3. Loggers

Loggers are responsible for the writing of a log. The log includes log events that match a + Logger's inclusion rules.

Loggers are associated with either the Broker or a virtualhost. Virtualhost loggers write + only log events related to that virtualhost. Broker Loggers write log events from the Broker + as a whole. Optionally a Broker Logger can be configured to exclude log events coming from + virtualhosts. These abilities can be usefully exploited together in managed service scenarios + to produce separate logs for separate user groups.

Loggers can be added or removed at runtime, without restarting the Broker. However changes + to a Logger's configuration such as filenames and rolling options don't take effect until the + next restart. Changes to a Logger's inclusion rules take effect immediately.

All loggers allow the log event layout to be customised. Loggers understand Logback Classic + Pattern Layouts.

The following sections describes each Logger implementation in detail.

9.1.3.1. FileLogger

A FileLogger - writes a log file to the filesystem. The name and + location of the log file, the rolling configuration, and compression options can be + configured.

The roll daily option, if enabled, will cause the log file will be + rolled at midnight local time. The rolled over file will have a suffix in the form + yyyy-mm-dd. In roll daily mode, maximum number of rolled + files controls the maximum number of days to be retained. + Older files will be deleted.

The maximum file size option limits the size of any one log file. + Once a log file reaches the given size, it will be rolled. The rolled over file will have + the numeric suffix, beginning at 1. If the log file rolls again, first + the existing file with the suffix .1 is renamed to .2 + and so forth. If roll daily is not in use, maximum number of rolled + files governs the number of rolled files that will be + retained.

Roll on restart governs whether the log file is rolled when the + Broker is restarted. If not ticked, the Broker will append to the existing log file until it + needs to be rolled.

9.1.3.2. ConsoleLogger

ConsoleLogger - writes a log file standard out or standard + error.

9.1.3.3. SyslogLogger

SyslogLogger - writes a log file to a syslog daemon using the + USER facility. The hostname and port number of the syslog daemon can be + configured.

Log entries can be prefixed with a string. This string defaults to include the word + Qpid and the name of the Broker or virtualhost. This serves to + distinguish the logging generated by this Qpid instance, from other Qpid instances, or other + applications using the USER.

9.1.3.4. MemoryLogger

MemoryLogger - writes a log file to a circular in-memory buffer. By + default the circular buffer holds the last 4096 log events. The contents of the buffer can + be viewed via Management. See Figure 9.3, “Viewing a memory logger”

9.1.4. Inclusion Rules

A Logger has one or more inclusion rules. These + govern what appears in the log. A Logger with no inclusion rules will log nothing.

Inclusion rules can be added, removed or changed at runtime. Changes take place + immediately.

+

  • The Name And Level inclusion rule accepts log events that match + a given log event source name and have a level that equals or + exceeds the specified value.

    The log event source name refers to the fully qualified class name from which the + event originates. These names permit a trailing wild card .*. For + instance a source name of org.apache.qpid.* will match all events + from classes in the package org.apache.qpid and any sub packages + beneath.

    The Level governs the level of the events that will be included + in the log. It may take one of the following values: ERROR, WARN, INFO, DEBUG, TRACE + where ERROR is considered the highest and TRACE the lowest. In addition, there are two + special values: OFF and ALL, the former excludes all log events whereas the latter will + include everything. When considering whether a logging event should be included in the + log, the logging event must have a level that matches that of the inclusion rule or be + higher, otherwise the log event will not appear in the log.

+

9.1.5. Logging Management

The logging subsystem can be completely managed from the Web Management Console or the + REST API. You can:

  • Add, remove, or change the configuration of Loggers.

  • Add, remove, or change the Inclusion Rules.

  • For FileLoggers, download the log file and rolled log files associated with + the Logger.

  • For MemoryLoggers, view the last n log + events

+

The figure that follows shows a FileLogger. The attributes area shows the configuration + of the Logger. The inclusion rule table shows the rules that are associated with the Logger. + The area towards the bottom of the tab allows the log files to be downloaded to the browser. +

Figure 9.1. Viewing a file logger

Viewing a file logger


+

The figure below shows the editing of the level of an inclusion rule.

Figure 9.2. Editing an inclusion rule

Editing an inclusion rule


+

The figure below shows a Memory Logger. Note that the Memory Logger provides access to + the cached message via the viewer towards the bottom on the tab.

Figure 9.3. Viewing a memory logger

Viewing a memory logger


+

Prev   Next
8.4. Configuration Encryption Home 9.2. Disk Space Management
\ No newline at end of file http://git-wip-us.apache.org/repos/asf/qpid-site/blob/42d76007/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Security-AccessControlProviders.html.in ---------------------------------------------------------------------- diff --git a/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Security-AccessControlProviders.html.in b/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Security-AccessControlProviders.html.in new file mode 100644 index 0000000..753d59c --- /dev/null +++ b/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Security-AccessControlProviders.html.in @@ -0,0 +1,221 @@ +
8.3. Access Control Providers
Prev Chapter 8. Security Next

8.3. Access Control Providers

+ The Access Control Provider governs the actions that a user may perform. +

There are two points within the hierarchy that enforce access control: the Broker itself and at each Virtual + Host. When an access decision needs to be made, the nearest control point configured with a provider is consulted + for a decision. The example, when making a decision about the ability to say, consume from, a Queue, if the + Virtual Host is configured with Access Control Provider it is consulted. Unless a decision is made, the decision + is delegated to the Access Control Provider configured at the Broker. +

Access Control Providers are configured with a list of ACL rules. The rules determine to which objects + the user has access and what actions the user may perform on those objects. Rules are ordered and are considered + top to bottom. The first matching rule makes the access decision. +

+ ACL rules may be written in terms of user or group names. A rule written in terms of a group name applies to the + user if he is a member of that group. Groups information is obtained from the + Authentication Providers + and + Group Providers. Writing ACL in terms of groups is + recommended. +

+ The Access Control Providers can be configured using + REST Management interfaces + and Web Management Console. +

8.3.1. Types

There are currently two types of Access Control Provider implementing ACL rules. +

  • RulesBased - a provider that stores the rules-set within + the Broker's or VirtualHost's configuration. When used with HA, the Virtualhost + rules automatically propagated to all nodes participating within the HA group.

  • +

    ACLFile - an older provider that references an externally provided + ACL file (or data url). This provider is deprecated.

    +

+

8.3.2.  + ACL Rules +

+ An ACL rule-set is an ordered list of ACL rules.

An ACL rule comprises matching criteria that determines if a rule applies to a situation and a decision + outcome. The rule produces an outcome only if the all matching criteria are satisfied. +

Matching criteria is composed of an ACL object type (e.g. QUEUE), an ACL action + (e.g. UPDATE) and other properties that further refine if a match is made. These properties + restrict the match based on additional criteria such as name or IP address. ACL Object type ALL + matches any object. Likewise ACL Action ALL matches any action. +

Let's look at some examples.

+      ACL ALLOW alice CREATE QUEUE              # Grants alice permission to create all queues.
+      ACL DENY bob CREATE QUEUE name="myqueue"  # Denies bob permission to create a queue called "myqueue"
+

+ As discussed, the ACL rule-set is considered in order with the first matching rule taking precedence over all those + that follow. In the following example, if the user bob tries to create an exchange "myexch", the action + will be allowed by the first rule. The second rule will never be considered. + 

+      ACL ALLOW bob ALL EXCHANGE
+      ACL DENY bob CREATE EXCHANGE name="myexch"  # Dead rule
+

+ If the desire is to allow bob to create all exchanges except "myexch", order of the rules must be reversed: + 

+      ACL DENY bob CREATE EXCHANGE name="myexch"
+      ACL ALLOW bob ALL EXCHANGE
+

+ If a rule-set fails to make a decision, the result is configurable. By default, the RuleBased + provider defers the decision allowing another provider further up the hierarchy to make a decision (i.e. allowing + the VirtualHost control point to delegate to the Broker). In the case of the ACLFile provider, by default, its + rule-set implicit have a rule denying all operations to all users. It is as if the rule-set ends with + ACL DENY ALL ALL. If no access control provider makes a decision the default is to + deny the action. +

+ When writing a new ACL, a useful approach is to begin with an rule-set containing only + 

ACL DENY-LOG ALL ALL

at the Broker control point which will cause the Broker to + deny all operations with details of the denial logged. Build up the ACL rule by rule, gradually working through + the use-cases of your system. Once the ACL is complete, consider switching the DENY-LOG actions to DENY. +

+ ACL rules are very powerful: it is possible to write very granular rules specifying many broker objects and their + properties. Most projects probably won't need this degree of flexibility. A reasonable approach is to choose to apply permissions + at a certain level of abstractions and apply them consistently across the whole system. +

8.3.3.  + Syntax +

+ ACL rules follow this syntax: + 

+     ACL {permission} {<group-name>|<user-name>|ALL} {action|ALL} [object|ALL] [property="<property-value>"]
+

+ Comments may be introduced with the hash (#) character and are ignored. Long lines can be broken with the slash (\) character. + 

+      # A comment
+      ACL ALLOW admin CREATE ALL # Also a comment
+      ACL DENY guest \
+      ALL ALL   # A broken line
+

Table 8.1. List of ACL permission

ALLOW

Allow the action

ALLOW-LOG

Allow the action and log the action in the log

DENY

Deny the action

DENY-LOG

Deny the action and log the action in the log


Table 8.2. List of ACL actions

< td>

Action

Description

Supported object types

Supported properties

CONSUME

Applied when subscriptions are created

QUEUE

name, autodelete, temporary, durable, exclusive, alternate, owner, virtualhost_name

PUBLISH

Applied on a per message basis on publish message transfers

EXCHANGE

name, routingkey, virtualhost_name

CREATE

Applied when an object is created, such as bindings, queues, exchanges

VIRTUALHOSTNODE, VIRTUALHOST, EXCHANGE, QUEUE, USER, GROUP

see properties on the corresponding object type

ACCESS

Applied when a connection is made for messaging or management

VIRTUALHOST, MANAGEMENT

name (for VIRTUALHOST only)

BIND

Applied when queues are bound to exchanges

EXCHANGE

name, routingKey, queue_name, virtualhost_name, temporary, durable

UNBIND

Applied when queues are unbound from exchanges

EXCHANGE

name, routingKey, queue_name, virtualhost_name, temporary, durable

DELETE

Applied when objects are deleted

VIRTUALHOSTNODE, VIRTUALHOST, EXCHANGE, QUEUE, USER, GROUP

see pr operties on the corresponding object type

PURGE +

Applied when the contents of a queue is purged

QUEUE

UPDATE

Applied when an object is updated

VIRTUALHOSTNODE, VIRTUALHOST, EXCHANGE, QUEUE, USER, GROUP

see EXCHANGE and QUEUE properties

CONFIGURE

Applied when a Broker/Port/Authentication Provider/Access Control Provider/BrokerLogger is created/update/deleted.

BROKER

ACCESS_LOGS

Allows/denies the specific user to download log file(s).

BROKER, VIRTUALHOST

name (for VIRTUALHOST only)

SHUTDOWN

Allows/denies the specific user to shutdown the Broker.

BROKER

INVOKE

Allows/denies the specific user to invoke the named operation.

BROKER, VIRTUALHOSTNODE, VIRTUALHOST, EXCHANGE, QUEUE, USER, GROUP

method_name, name and virtualhost_name


Table 8.3. List of ACL objects

Object type

Description

Supported actions

Supported properties

Allowed in Virtualhost ACLs?

VIRTUALHOSTNODE

A virtualhostnode or remote replication node

ALL, CREATE, UPDATE, DELETE, INVOKE

name

No

VIRTUALHOST

A virtualhost

ALL, CREATE, UPDATE, DELETE, ACCESS, ACCESS_LOGS, INVOKE

name

No

QUEUE

A queue

ALL, CREATE, DELETE, PURGE, CONSUME, UPDATE, INVOKE

name, autodelete, temporary, durable, exclusive, alternate, owner, virtualhost_name

Yes

EXCHANGE

An exchange

ALL, ACCESS, CREATE, DELETE, BIND, UNBIND, PUBLISH, UPDATE, INVOKE

name, autodelete, temporary, durable, type, virtualhost_name, queue_name(only for BIND and UNBIND), routingkey(only for BIND and UNBIND, PUBLISH)

Yes

USER

A user

ALL, CREATE, DELETE, UPDATE, INVOKE

name

No

GROUP

A group

ALL, CREATE, DELETE, UPDATE, INVOKE

name

No

BROKER

The broker

ALL, CONFIGURE, ACCESS_LOGS, INVOKE

No


Table 8.4. List of ACL properties

name

String. Object name, such as a queue name or exchan ge name.

durable

Boolean. Indicates the object is durable

routingkey

String. Specifies routing key

autodelete

Boolean. Indicates whether or not the object gets deleted when the connection is closed

exclusive

Boolean. Indicates the presence of an exclusive flag

temporary

Boolean. Indicates the presence of an temporary flag

type

String. Type of object, such as topic, or fanout

alternate

String. Name of the alternate exchange

queue_name

String. Name of the queue (used only when the object is EXCHANGE for BIND and UNBIND actions)

component

String. component name

from_network +

+ Comma-separated strings representing IPv4 address ranges. +

+

+ Intended for use in ACCESS VIRTUALHOST rules to apply firewall-like restrictions. +

+

+ The rule matches if any of the address ranges match the IPv4 address of the messaging client. + The address ranges are specified using either Classless Inter-Domain Routing notation + (e.g. 192.168.1.0/24; see RFC 4632) + or wildcards (e.g. 192.169.1.*). +

+
from_hostname +

+ Comma-separated strings representing hostnames, specified using Perl-style regular + expressions, e.g. .*\.example\.company\.com +

+

+ Intended for use in ACCESS VIRTUALHOST rules to apply firewall-like restrictions. +

+

+ The rule matches if any of the patterns match the hostname of the messaging client. +

+

+ To look up the client's hostname, Qpid uses Java's DNS support, which internally caches its results. +

+

+ You can modify the time-to-live of cached results using the *.ttl properties described on the + Java Networking + Properties page. +

+

+ For example, you can either set system property sun.net.inetaddr.ttl from the command line + (e.g. export QPID_OPTS="-Dsun.net.inetaddr.ttl=0") or networkaddress.cache.ttl in + $JAVA_HOME/lib/security/java.security. The latter is preferred because it is JVM + vendor-independent. +

+
virtualhost_name +

+ String. A name of virtual host to which the rule is applied. +

+
method_name +

+ String. The name of the method. A trailing wildcard (*) is permitted. Used with INVOKE ACL action. +

+
attribute_names +

+ Specifies attribute name criteria. Used by UPDATE ACL actions only. Rules with this criteria will match + if and only if the set of attributes being updated Comma separated list of attribute names . This criteria + will match if all attributes included within the update appear in the set described by + attribute_names. +

+

8.3.4.  + Worked Examples +

+ Here are some example ACLs illustrating common use cases. +

8.3.4.1.  + Worked example 1 - Management rights +

+ Suppose you wish to permission two users: a user operator must be able to perform all + Management operations, and a user 'readonly' must be enable to perform only read-only actions. Neither + operator nor readonly should be allowed to connect clients for + messaging. +

Example 8.1. Worked example 1 - Management rights

+          # Deny operator/readonly permission to connect for messaging.
+          ACL DENY-LOG operator ACCESS VIRTUALHOST
+          ACL DENY-LOG readonly ACCESS VIRTUALHOST
+          # Give operator permission to perfom all actions
+          ACL ALLOW operator ALL ALL
+          # Give readonly access permission to virtualhost. (Read permission for all objects implicit)
+          ACL ALLOW readonly ACCESS MANAGEMENT
+          ...
+          ... rules for other users
+          ...
+

8.3.4.2.  + Worked example 2 - Simple Messaging +

+ Suppose you wish to permission a system for application messaging. User publisher + needs permission to publish to appqueue and consumer needs permission to consume + from the same queue object. We also want operator to be able to inspect messages + and delete messages in case of the need to intervene. This example assumes that the queue exists on + the Broker. +

+ We use this ACL to illustrate separate Broker and Virtualhost access control providers. +

+ The following ACL rules are given to the Broker. +

Example 8.2. Worked example 2 - Simple Messaging - Broker ACL rules

+# This gives the operate permission to delete messages on all queues on all virtualhost
+ACL ALLOW operator ACCESS MANAGEMENT
+ACL ALLOW operator INVOKE QUEUE method_name="deleteMessages"
+ACL ALLOW operator INVOKE QUEUE method_name="getMessage*"
+

+ And the following ACL rule-set is applied to the Virtualhost. The default outcome of the + Access Control Provider must be DEFERED. This means that if a request for + access is made for which there are no matching rules, the decision will be deferred to the + Broker so it can make a decision instead. +

Example 8.3. Worked example 2 - Simple Messaging - Virtualhost ACL rules

+# Configure the rule-set to DEFER decisions that have no matching rules.
+CONFIG DEFAULTDEFER=TRUE
+# Allow client and server to connect to the virtual host.
+ACL ALLOW publisher ACCESS VIRTUALHOST
+ACL ALLOW consumer ACCESS VIRTUALHOST
+
+ACL ALLOW publisher PUBLISH EXCHANGE name="" routingKey="appqueue"
+ACL ALLOW consumer CONSUME QUEUE name="appqueue"
+# In some addressing configurations, the Qpid JMS AMQP 0-x client, will declare the queue as a side effect of creating the consumer.
+# The following line allows for this.  For the Qpid JMS AMQP 1.0 client, this is not required.
+ACL ALLOW consumer CREATE QUEUE name="appqueue"
+

8.3.4.3.  + Worked example 3 - firewall-like access control +

+ This example illustrates how to set up an ACL that restricts the IP addresses and hostnames + of messaging clients that can access a virtual host. +

Example 8.4. Worked example 3 - firewall-like access control

+          ################
+          # Hostname rules
+          ################
+
+          # Allow messaging clients from company1.com and company1.co.uk to connect
+          ACL ALLOW all ACCESS VIRTUALHOST from_hostname=".*\.company1\.com,.*\.company1\.co\.uk"
+
+          # Deny messaging clients from hosts within the dev subdomain
+          ACL DENY-LOG all ACCESS VIRTUALHOST from_hostname=".*\.dev\.company1\.com"
+
+          ##################
+          # IP address rules
+          ##################
+
+          # Deny access to all users in the IP ranges 192.168.1.0-192.168.1.255 and 192.168.2.0-192.168.2.255,
+          # using the notation specified in RFC 4632, "Classless Inter-domain Routing (CIDR)"
+          ACL DENY-LOG messaging-users ACCESS VIRTUALHOST \
+          from_network="192.168.1.0/24,192.168.2.0/24"
+
+          # Deny access to all users in the IP ranges 192.169.1.0-192.169.1.255 and 192.169.2.0-192.169.2.255,
+          # using wildcard notation.
+          ACL DENY-LOG messaging-users ACCESS VIRTUALHOST \
+          from_network="192.169.1.*,192.169.2.*"
+

Prev Up Next
8.2. Group Providers Home 8.4. Configuration Encryption
\ No newline at end of file http://git-wip-us.apache.org/repos/asf/qpid-site/blob/42d76007/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Security-Configuration-Encryption.html.in ---------------------------------------------------------------------- diff --git a/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Security-Configuration-Encryption.html.in b/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Security-Configuration-Encryption.html.in new file mode 100644 index 0000000..f13dfe9 --- /dev/null +++ b/input/releases/qpid-broker-j-7.0.1/book/Java-Broker-Security-Configuration-Encryption.html.in @@ -0,0 +1,20 @@ +
8.4. Configuration Encryption
Prev Chapter 8. Security Next

8.4. Configuration Encryption

The Broker is capable of encrypting passwords and other security items stored in the + Broker's configuration. This is means that items such as keystore/truststore passwords, JDBC + passwords, and LDAP passwords can be stored in the configure in a form that is difficult to + read.

The Broker ships with an encryptor implementation called AESKeyFile. This + uses a securely generated random key of 256bit[9] to encrypt the secrets stored within a key + file. Of course, the key itself must be guarded carefully, otherwise the passwords encrypted + with it may be compromised. For this reason, the Broker ensures that the file's permissions + allow the file to be read exclusively by the user account used for running the Broker.

Important

If the keyfile is lost or corrupted, the secrets will be irrecoverable.

8.4.1. Configuration

The AESKeyFile encyptor provider is enabled/disabled via the Broker attributes within the + Web Management Console. On enabling the provider, any existing passwords within the + configuration will be automatically rewritten in the encrypted form.

Note that passwords stored by the Authentication Providers PlainPasswordFile and. + PlainPasswordFile + with the external password files are not encrypted by the key. Use the + Scram Authentication Managers instead; these make use of the Configuration Encryption when + storing the users' passwords.

8.4.2. Alternate Implementations

If the AESKeyFile encryptor implementation does not meet the needs of + the user, perhaps owing to the security standards of their institution, the + ConfigurationSecretEncrypter interface is designed as an extension point. + Users may implement their own implementation of ConfigurationSecretEncrypter perhaps to employ + stronger encryption or delegating the storage of the key to an Enterprise Password + Safe.


[9] Java Cryptography Extension (JCE) + Unlimited Strength required

Prev Up Next
8.3. Access Control Providers Home Chapter 9. Runtime
\ No newline at end of file --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscribe@qpid.apache.org For additional commands, e-mail: commits-help@qpid.apache.org