qpid-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Qpid: Cluster Design Note (page edited)
Date Thu, 08 Mar 2007 22:06:00 GMT
<html>
<head>
    <base href="http://cwiki.apache.org/confluence" />
    <style type="text/css">
    <!--
    body, p, td, table, tr, .bodytext, .stepfield {
	font-family: Verdana, arial, sans-serif;
	font-size: 11px;
	line-height: 16px;
	color: #000000;
	font-weight: normal;
}
#PageContent {
	text-align: left;
	background-color: #fff;
	padding: 0px;
	margin: 0px;
    padding-bottom:20px;
}
/*
** when this stylesheet is used for the Tiny MCE Wysiwyg editor's edit area, we can't
** use an id=PageContent or class=wiki-content, so we must
** set the body style to that used for PageContent, and p to that used for wiki-content.
*/

body {
	margin: 0px;
	padding: 0px;
	text-align: center;
    background-color: #f0f0f0;
}

@media print {

body {
    background-color: #fff;
}

}

.monospaceInput {
    font:12px monospace
}

.wiki-content p, .commentblock p {
    margin: 16px 0px 16px 0px;
    padding: 0px;
}

.wiki-content-preview {
    padding: 5px;
    border-left: 1px solid #3c78b5;
    border-right: 1px solid #3c78b5;
}

ul, ol {
    margin-top: 2px;
    margin-bottom: 2px;
    padding-top: 0px;
    padding-bottom: 0px;
}

pre {
    padding: 0px;
    margin-top: 5px;
    margin-left: 15px;
    margin-bottom: 5px;
    margin-right: 5px;
    text-align: left;
}

.helpheading {
    font-weight: bold;
    background-color: #D0D9BD;
        border-bottom: 1px solid #3c78b5;
        padding: 4px 4px 4px 4px;
        margin: 0px;
        margin-top: 10px;
}
.helpcontent {
        padding: 4px 4px 20px 4px;
    background-color: #f5f7f1;
}

.code {
 	border: 1px dashed #3c78b5;
    font-size: 11px;
	font-family: Courier;
    margin: 10px;
	line-height: 13px;
}

.focusedComment {
    background: #ffffce;
}

.commentBox, .focusedComment {
    padding: 10px;
    margin: 5px 0 5px 0;
    border: 1px #bbb solid;
}

.codeHeader {
    background-color: #f0f0f0;
 	border-bottom: 1px dashed #3c78b5;
    padding: 3px;
	text-align: center;
}

.codeContent {
    text-align: left;
    background-color: #f0f0f0;
    padding: 3px;
}

.preformatted {
 	border: 1px dashed #3c78b5;
    font-size: 11px;
	font-family: Courier;
    margin: 10px;
	line-height: 13px;
}

.preformattedHeader {
    background-color: #f0f0f0;
 	border-bottom: 1px dashed #3c78b5;
    padding: 3px;
	text-align: center;
}

.preformattedContent {
    background-color: #f0f0f0;
    padding: 3px;
}

.panel {
 	border: 1px dashed #3c78b5;
    margin: 10px;
    margin-top: 0px;
}

.panelHeader {
    background-color: #f0f0f0;
 	border-bottom: 1px dashed #3c78b5;
    padding: 3px;
	text-align: center;
}

.panelContent {
    background-color: #f0f0f0;
    padding: 5px;
}

.anonymousAlert {
    background-color: #f0f0f0;
 	border: 1px dashed red;
    font-size: 11px;
    padding: 10px 5px 10px 5px;
    margin: 4px;
	line-height: 13px;
}

.lockAlert {
    background-color: #f0f0f0;
    width: 50%;
 	border: 1px dashed red;
    font-size: 11px;
    padding: 10px 5px 10px 5px;
    margin: 4px;
	line-height: 13px;
}


.code-keyword {
  color: #000091;
  background-color: inherit;
}

.code-object {
  color: #910091;
  background-color: inherit;
}

.code-quote {
  color: #009100;
  background-color: inherit;
}

.code-comment {
  color: #808080;
  background-color: inherit;
}


.code-xml .code-keyword {
  color: inherit;
  font-weight: bold;
}

.code-tag {
  color: #000091;
  background-color: inherit;
}

.breadcrumbs {
    background-color: #f0f0f0;
 	border-color: #3c78b5;
	border-width: 1px 0px 1px 0px;
	border-style: solid;
    font-size: 11px;
    padding: 3px 0px 3px 0px;
}

.navmenu {
    border: 1px solid #ccc;
}

.menuheading {
    font-weight: bold;
    background-color: #f0f0f0;
 	border-bottom: 1px solid #3c78b5;
	padding: 4px 4px 2px 4px;
}

.menuitems {
	padding: 4px 4px 20px 4px;
}

.rightpanel {
    border-left: 1px solid #ccc;
    border-bottom: 1px solid #ccc;
}

#helpheading {
    text-align: left;
    font-weight: bold;
    background-color: #D0D9BD;
 	border-bottom: 1px solid #3c78b5;
	padding: 4px 4px 4px 4px;
	margin: 0px;
}
#helpcontent {
	padding: 4px 4px 4px 4px;
    background-color: #f5f7f1;
}
.helptab-unselected {
    font-weight: bold;
	padding: 5px;
    background-color: #f5f7f1;
}
.helptab-selected {
    font-weight: bold;
    background-color: #D0D9BD;
	padding: 5px;
}
.helptabs {
    margin: 0px;
    background-color: #f5f7f1;
	padding: 5px;
}
.infopanel-heading {
    font-weight: bold;
	padding: 4px 0px 2px 0px;
}

.pagebody {
}

.pageheader {
	padding: 5px 5px 5px 0px;
 	border-bottom: 1px solid #3c78b5;
}

.pagetitle {
	font-size: 22px;
	font-weight: bold;
	font-family: Arial, sans-serif;
	color: #003366;
}

.newpagetitle {
    color: #ccc !important;
}

.steptitle {
	font-size: 18px;
	font-weight: bold;
	font-family: Arial, sans-serif;
	color: #003366;
	margin-bottom: 7px;
}

.substeptitle {
    font-size: 12px;
    font-weight: bold;
    font-family: Arial, sans-serif;
    color: #003366;
    margin: 2px 4px 4px 4px;
    padding: 2px 4px 1px 4px;
}

.stepdesc {
    font-family: Verdana, arial, sans-serif;
	font-size: 11px;
	line-height: 16px;
	font-weight: normal;
    color: #666666;
    margin-top: 7px;
    margin-bottom: 7px;
}

.steplabel {
    font-weight: bold;
    margin-right: 4px;
    color: black;
    float: left;
    width: 15%;
    text-align: right;
}

.stepfield {
    background: #f0f0f0;
    padding: 5px;
}

.submitButtons{
    margin-top:5px;
    text-align:right;
}

.formtitle {
	font-size: 12px;
	font-weight: bold;
	font-family: Arial, sans-serif;
	color: #003366;
}

.sectionbottom {
    border-bottom: 1px solid #3c78b5;
}

.topRow {
    border-top: 2px solid #3c78b5;
}

.tabletitle {
	font-size: 14px;
	font-weight: bold;
	font-family: Arial, sans-serif;
    padding: 3px 0px 2px 0px;
    margin: 8px 4px 2px 0px;
	color: #003366;
	border-bottom: 2px solid #3c78b5;
}
.pagesubheading {
    color: #666666;
    font-size: 10px;
    padding: 0px 0px 5px 0px;
}

HR {
	color: 3c78b5;
	height: 1;
}

A:link, A:visited, A:active, A:hover {
	color: #003366;
}

h1 A:link, h1 A:visited, h1 A:active {
	text-decoration: none;
}

h1 A:hover {
    border-bottom: 1px dotted #003366;
}

.wiki-content > :first-child, .commentblock > :first-child {
    margin-top: 3px;
}

.logocell {
    padding: 10px;
}

input {
	font-family: verdana, geneva, arial, sans-serif;
	font-size: 11px;
	color: #000000;
}

textarea, textarea.editor {
	font-family: verdana, geneva, arial, sans-serif;
	font-size: 11px;
	color: #333333;
}

/* use logoSpaceLink instead.
.spacenametitle {
	font: 21px/31px Impact, Arial, Helvetica;
    font-weight: 100;
    color: #999999;
	margin: 0px;
}
.spacenametitle img {
  margin: 0 0 -4px 0;
}
.spacenametitle a {
    text-decoration: none;
    color: #999999;
}
.spacenametitle a:visited {
    text-decoration: none;
    color: #999999;
}*/

.spacenametitle-printable {
	font: 20px/25px Impact, Arial, Helvetica;
    font-weight: 100;
    color: #999999;
	margin: 0px;
}
.spacenametitle-printable a {
    text-decoration: none;
    color: #999999;
}
.spacenametitle-printable a:visited {
    text-decoration: none;
    color: #999999;
}

.blogDate {
	font-weight: bold;
	text-decoration: none;
	color: black;
}

.blogSurtitle {
    background: #f0f0f0;
 	border: 1px solid #ddd;
	padding: 3px;
	margin: 1px 1px 10px 1px;
}

.blogHeading {
    font-size: 20px;
    line-height: normal;
    font-weight: bold;
    padding: 0px;
    margin: 0px;
}

.blogHeading a {
   text-decoration: none;
   color: black;
}

.endsection {
	align: right;
	color: #666666;
	margin-top: 10px;
}
.endsectionleftnav {
	align: right;
	color: #666666;
	margin-top: 10px;
}

h1 {
	font-size: 24px;
	line-height: normal;
	font-weight: bold;
	background-color: #f0f0f0;
	color: #003366;
 	border-bottom: 1px solid #3c78b5;
	padding: 2px;
	margin: 36px 0px 4px 0px;
}

h2 {
	font-size: 18px;
	line-height: normal;
	font-weight: bold;
	background-color: #f0f0f0;
 	border-bottom: 1px solid #3c78b5;
	padding: 2px;
	margin: 27px 0px 4px 0px;
}

h3 {
	font-size: 14px;
	line-height: normal;
	font-weight: bold;
	background-color: #f0f0f0;
	padding: 2px;
	margin: 21px 0px 4px 0px;
}

h4 {
	font-size: 12px;
	line-height: normal;
	font-weight: bold;
	background-color: #f0f0f0;
	padding: 2px;
	margin: 18px 0px 4px 0px;
}

h4.search {
	font-size: 12px;
	line-height: normal;
	font-weight: normal;
	background-color: #f0f0f0;
	padding: 4px;
	margin: 18px 0px 4px 0px;
}

h5 {
	font-size: 10px;
	line-height: normal;
	font-weight: bold;
	background-color: #f0f0f0;
	padding: 2px;
	margin: 14px 0px 4px 0px;
}

h6 {
	font-size: 8px;
	line-height: normal;
	font-weight: bold;
	background-color: #f0f0f0;
	padding: 2px;
	margin: 14px 0px 4px 0px;
}

.smallfont {
    font-size: 10px;
}
.descfont {
    font-size: 10px;
    color: #666666;
}
.smallerfont {
    font-size: 9px;
}
.smalltext {
    color: #666666;
    font-size: 10px;
}
.smalltext a {
    color: #666666;
}
.smalltext-blue {
    color: #3c78b5;
    font-size: 10px;
}
.surtitle {
    margin-left: 1px;
    margin-bottom: 5px;
    font-size: 14px;
    color: #666666;
}

/* css hack found here:  http://www.fo3nix.pwp.blueyonder.co.uk/tutorials/css/hacks/ */
.navItemOver { font-size: 10px; font-weight: bold; color: #ffffff; background-color: #003366; cursor: hand; voice-family: '\'}\''; voice-family:inherit; cursor: pointer;}
.navItemOver a { color: #ffffff; background-color:#003366; text-decoration: none; }
.navItemOver a:visited { color: #ffffff; background-color:#003366; text-decoration: none; }
.navItemOver a:hover { color: #ffffff; background-color:#003366; text-decoration: none; }
.navItem { font-size: 10px; font-weight: bold; color: #ffffff; background-color: #3c78b5; }
.navItem a { color: #ffffff; text-decoration: none; }
.navItem a:hover { color: #ffffff; text-decoration: none; }
.navItem a:visited { color: #ffffff; text-decoration: none; }

div.padded { padding: 4px; }
div.thickPadded { padding: 10px; }
h3.macrolibrariestitle {
    margin: 0px 0px 0px 0px;
}

div.centered { text-align: center; margin: 10px; }
div.centered table {margin: 0px auto; text-align: left; }

.tableview table {
    margin: 0;
}

.tableview th {
    text-align: left;
    color: #003366;
    font-size: 12px;
    padding: 5px 0px 0px 5px;
    border-bottom: 2px solid #3c78b5;
}
.tableview td {
    text-align: left;
    border-color: #ccc;
    border-width: 0px 0px 1px 0px;
    border-style: solid;
    margin: 0;
    padding: 4px 10px 4px 5px;
}

.grid {
    margin: 2px 0px 5px 0px;
    border-collapse: collapse;
}
.grid th  {
    border: 1px solid #ccc;
    padding: 2px 4px 2px 4px;
    background: #f0f0f0;
    text-align: center;
}
.grid td  {
    border: 1px solid #ccc;
    padding: 3px 4px 3px 4px;
}
.gridHover {
	background-color: #f9f9f9;
}

td.infocell {
    background-color: #f0f0f0;
}
.label {
	font-weight: bold;
	color: #003366;
}

label {
	font-weight: bold;
	color: #003366;
}

.error {
	background-color: #fcc;
}

.errorBox {
	background-color: #fcc;
    border: 1px solid #c00;
    padding: 5px;
    margin: 5px;
}

.errorMessage {
	color: #c00;
}

.success {
	background-color: #dfd;
}

.successBox {
	background-color: #dfd;
    border: 1px solid #090;
    padding: 5px;
    margin-top:5px;
    margin-bottom:5px;
}

blockquote {
	padding-left: 10px;
	padding-right: 10px;
	margin-left: 5px;
	margin-right: 0px;
	border-left: 1px solid #3c78b5;
}

table.confluenceTable
{
    margin: 5px;
    border-collapse: collapse;
}

/* Added as a temporary fix for CONF-4223. The table elements appear to be inheriting the border: none attribute from the sectionMacro class */
table.confluenceTable td.confluenceTd
{
    border-width: 1px;
    border-style: solid;
    border-color: #ccc;
    padding: 3px 4px 3px 4px;
}

/* Added as a temporary fix for CONF-4223. The table elements appear to be inheriting the border: none attribute from the sectionMacro class */
table.confluenceTable th.confluenceTh
{
    border-width: 1px;
    border-style: solid;
    border-color: #ccc;
    padding: 3px 4px 3px 4px;
    background-color: #f0f0f0;
    text-align: center;
}

td.confluenceTd
{
    border-width: 1px;
    border-style: solid;
    border-color: #ccc;
    padding: 3px 4px 3px 4px;
}

th.confluenceTh
{
    border-width: 1px;
    border-style: solid;
    border-color: #ccc;
    padding: 3px 4px 3px 4px;
    background-color: #f0f0f0;
    text-align: center;
}

DIV.small {
	font-size: 9px;
}

H1.pagename {
	margin-top: 0px;
}

IMG.inline  {}

.loginform {
    margin: 5px;
    border: 1px solid #ccc;
}

/* The text how the "This is a preview" comment should be shown. */
.previewnote { text-align: center;
                font-size: 11px;
                    color: red; }

/* How the preview content should be shown */
.previewcontent { background: #E0E0E0; }

/* How the system messages should be shown (DisplayMessage.jsp) */
.messagecontent { background: #E0E0E0; }

/* How the "This page has been modified..." -comment should be shown. */
.conflictnote { }

.createlink {
    color: maroon;
}
a.createlink {
    color: maroon;
}
.templateparameter {
    font-size: 9px;
    color: darkblue;
}

.diffadded {
    background: #ddffdd;
    padding: 1px 1px 1px 4px;
	border-left: 4px solid darkgreen;
}
.diffdeleted {
    color: #999;
    background: #ffdddd;
    padding: 1px 1px 1px 4px;
	border-left: 4px solid darkred;
}
.diffnochange {
    padding: 1px 1px 1px 4px;
	border-left: 4px solid lightgrey;
}
.differror {
    background: brown;
}
.diff {
    font-family: lucida console, courier new, fixed-width;
	font-size: 12px;
	line-height: 14px;
}
.diffaddedchars {
    background-color:#99ff99;
    font-weight:bolder;
}
.diffremovedchars {
    background-color:#ff9999;
    text-decoration: line-through;
    font-weight:bolder;
}

.greybackground {
    background: #f0f0f0
}

.greybox {
 	border: 1px solid #ddd;
	padding: 3px;
	margin: 1px 1px 10px 1px;
}

.borderedGreyBox {
    border: 1px solid #cccccc;
    background-color: #f0f0f0;
    padding: 10px;
}

.greyboxfilled {
 	border: 1px solid #ddd;
    background: #f0f0f0;
    padding: 3px;
	margin: 1px 1px 10px 1px;
}

.navBackgroundBox {
    padding: 5px 5px 5px 5px;
    font-size: 22px;
	font-weight: bold;
	font-family: Arial, sans-serif;
	color: white;
    background: #3c78b5;
    text-decoration: none;
}

.previewBoxTop {
	background-color: #f0f0f0;
    border-width: 1px 1px 0px 1px;
    border-style: solid;
    border-color: #3c78b5;
    padding: 5px;
    margin: 5px 0px 0px 0px;
    text-align: center;
}
.previewContent {
    background-color: #fff;
 	border-color: #3c78b5;
	border-width: 0px 1px 0px 1px;
	border-style: solid;
	padding: 10px;
	margin: 0px;
}
.previewBoxBottom {
	background-color: #f0f0f0;
    border-width: 0px 1px 1px 1px;
    border-style: solid;
    border-color: #3c78b5;
    padding: 5px;
    margin: 0px 0px 5px 0px;
    text-align: center;
}

.functionbox {
    background-color: #f0f0f0;
 	border: 1px solid #3c78b5;
	padding: 3px;
	margin: 1px 1px 10px 1px;
}

.functionbox-greyborder {
    background-color: #f0f0f0;
 	border: 1px solid #ddd;
	padding: 3px;
	margin: 1px 1px 10px 1px;
}

.search-highlight {
    background-color: #ffffcc;
}

/* normal (white) background */
.rowNormal {
    background-color: #ffffff;
 }

/* alternate (pale yellow) background */
.rowAlternate {
    background-color: #f7f7f7;
}

/* used in the list attachments table */
.rowAlternateNoBottomColor {
    background-color: #f7f7f7;
}

.rowAlternateNoBottomNoColor {
}

.rowAlternateNoBottomColor td {
    border-bottom: 0px;
}

.rowAlternateNoBottomNoColor td {
    border-bottom: 0px;
}

/* row highlight (grey) background */
.rowHighlight {
    background-color: #f0f0f0;

}

TD.greenbar {FONT-SIZE: 2px; BACKGROUND: #00df00; BORDER: 1px solid #9c9c9c; PADDING: 0px; }
TD.redbar {FONT-SIZE: 2px; BACKGROUND: #df0000; BORDER: 1px solid #9c9c9c; PADDING: 0px; }
TD.darkredbar {FONT-SIZE: 2px; BACKGROUND: #af0000; BORDER: 1px solid #9c9c9c; PADDING: 0px; }

TR.testpassed {FONT-SIZE: 2px; BACKGROUND: #ddffdd; PADDING: 0px; }
TR.testfailed {FONT-SIZE: 2px; BACKGROUND: #ffdddd; PADDING: 0px; }

.toolbar  {
    margin: 0px;
    border-collapse: collapse;
}

.toolbar td  {
    border: 1px solid #ccc;
    padding: 2px 2px 2px 2px;
    color: #ccc;
}

td.noformatting {
    border-width: 0px;
    border-style: none;
    text-align: center;
	padding: 0px;
}

.commentblock {
    margin: 12px 0 12px 0;
}

/*
 * Divs displaying the license information, if necessary.
 */
.license-eval, .license-none, .license-nonprofit {
    border-top: 1px solid #bbbbbb;
    text-align: center;
    font-size: 10px;
    font-family: Verdana, Arial, Helvetica, sans-serif;
}

.license-eval, .license-none {
    background-color: #ffcccc;
}

.license-eval b, .license-none b {
    color: #990000
}

.license-nonprofit {
    background-color: #ffffff;
}

/*
 * The shadow at the bottom of the page between the main content and the
 * "powered by" section.
 */
.bottomshadow {
    height: 12px;
    background-image: url("$req.contextPath/images/border/border_bottom.gif");
    background-repeat: repeat-x;
}

/*
 * Styling of the operations box
 */
.navmenu .operations li, .navmenu .operations ul {
    list-style: none;
    margin-left: 0;
    padding-left: 0;
}

.navmenu .operations ul {
    margin-bottom: 9px;
}

.navmenu .label {
    font-weight: inherit;
}

/*
 * Styling of ops as a toolbar
 */
.toolbar div {
    display: none;
}

.toolbar .label {
    display: none;
}

.toolbar .operations {
    display: block;
}

.toolbar .operations ul {
    display: inline;
    list-style: none;
    margin-left: 10px;
    padding-left: 0;
}

.toolbar .operations li {
    list-style: none;
    display: inline;
}

/* list page navigational tabs */
#foldertab {
padding: 3px 0px 3px 8px;
margin-left: 0;
border-bottom: 1px solid #3c78b5;
font: bold 11px Verdana, sans-serif;
}

#foldertab li {
list-style: none;
margin: 0;
display: inline;
}

#foldertab li a {
padding: 3px 0.5em;
margin-left: 3px;
border: 1px solid #3c78b5;
border-bottom: none;
background: #3c78b5;
text-decoration: none;
}

#foldertab li a:link { color: #ffffff; }
#foldertab li a:visited { color: #ffffff; }

#foldertab li a:hover {
color: #ffffff;
background: #003366;
border-color: #003366;
}

#foldertab li a.current {
background: white;
border-bottom: 1px solid white;
color: black;
}

#foldertab li a.current:link { color: black; }
#foldertab li a.current:visited { color: black; }
#foldertab li a.current:hover {
background: white;
border-bottom: 1px solid white;
color: black;
}

/* alphabet list */
ul#squaretab {
margin-left: 0;
padding-left: 0;
white-space: nowrap;
font: bold 8px Verdana, sans-serif;
}

#squaretab li {
display: inline;
list-style-type: none;
}

#squaretab a {
padding: 2px 6px;
border: 1px solid #3c78b5;
}

#squaretab a:link, #squaretab a:visited {
color: #fff;
background-color: #3c78b5;
text-decoration: none;
}

#squaretab a:hover {
color: #ffffff;
background-color: #003366;
border-color: #003366;
text-decoration: none;
}

#squaretab li a#current {
background: white;
color: black;
}

.blogcalendar * {
    font-family:verdana, arial, sans-serif;
    font-size:x-small;
    font-weight:normal;
    line-height:140%;
    padding:2px;
}


table.blogcalendar {
    border: 1px solid #3c78b5;
}

.blogcalendar th.calendarhead, a.calendarhead {
    font-size:x-small;
    font-weight:bold;
    padding:2px;
    text-transform:uppercase;
    background-color: #3c78b5;
    color: #ffffff;
    letter-spacing: .3em;
    text-transform: uppercase;
}

.calendarhead:visited {color: white;}
.calendarhead:active {color: white;}
.calendarhead:hover {color: white;}

.blogcalendar th {
    font-size:x-small;
    font-weight:bold;
    padding:2px;
    background-color:#f0f0f0;
}

.blogcalendar td {
    font-size:x-small;
    font-weight:normal;
}

.searchGroup { padding: 0 0 10px 0; background: #f0f0f0; }
.searchGroupHeading { font-size: 10px; font-weight: bold; color: #ffffff; background-color: #3c78b5; padding: 2px 4px 1px 4px; }
.searchItem { padding: 1px 4px 1px 4px; }
.searchItemSelected { padding: 1px 4px 1px 4px; font-weight: bold; background: #ddd; }

/* permissions page styles */
.permissionHeading {
    border-bottom: #bbb; border-width: 0 0 1px 0; border-style: solid; font-size: 16px; text-align: left;
}
.permissionTab {
    border-width: 0 0 0 1px; border-style: solid; background: #3c78b5; color: #ffffff; font-size: 10px;
}
.permissionSuperTab {
    border-width: 0 0 0 1px; border-style: solid; background: #003366; color: #ffffff;
}
.permissionCell {
    border-left: #bbb; border-width: 0 0 0 1px; border-style: solid;
}

/* warning panel */
.warningPanel { background: #FFFFCE; border:#F0C000 1px solid; padding: 8px; margin: 10px; }
/* alert panel */
.alertPanel { background: #FFCCCC; border:#C00 1px solid; padding: 8px; margin: 10px; }
/* info panel */
.infoPanel { background: #D8E4F1; border:#3c78b5 1px solid; padding: 8px; margin: 10px; }

/* side menu highlighting (e.g. space content screen) */
.optionPadded { padding: 2px; }
.optionSelected { background-color: #ffffcc; padding: 2px; border: 1px solid #ddd; margin: -1px; }
.optionSelected a { font-weight: bold; text-decoration: none; color: black; }

/* information macros */
.noteMacro { border-style: solid; border-width: 1px; border-color: #F0C000; background-color: #FFFFCE; text-align:left; margin-top: 5px; margin-bottom: 5px}
.warningMacro { border-style: solid; border-width: 1px; border-color: #c00; background-color: #fcc; text-align:left; margin-top: 5px; margin-bottom: 5px}
.infoMacro { border-style: solid; border-width: 1px; border-color: #3c78b5; background-color: #D8E4F1; text-align:left; margin-top: 5px; margin-bottom: 5px}
.tipMacro { border-style: solid; border-width: 1px; border-color: #090; background-color: #dfd; text-align:left; margin-top: 5px; margin-bottom: 5px}
.informationMacroPadding { padding: 5px 0 0 5px; }

table.infoMacro td, table.warningMacro td, table.tipMacro td, table.noteMacro td, table.sectionMacro td {
    border: none;
}

table.sectionMacroWithBorder td.columnMacro { border-style: dashed; border-width: 1px; border-color: #cccccc;}

.pagecontent
{
    padding: 10px;
    text-align: left;
}

/* styles for links in the top bar */
.topBarDiv a:link {color: #ffffff;}
.topBarDiv a:visited {color: #ffffff;}
.topBarDiv a:active {color: #ffffff;}
.topBarDiv a:hover {color: #ffffff;}
.topBarDiv {color: #ffffff;}

.topBar {
    background-color: #003366;
}


/* styles for extended operations */
.greyLinks a:link {color: #666666; text-decoration:underline;}
.greyLinks a:visited {color: #666666; text-decoration:underline;}
.greyLinks a:active {color: #666666; text-decoration:underline;}
.greyLinks a:hover {color: #666666; text-decoration:underline;}
.greyLinks {color: #666666; display:block; padding: 10px}

.logoSpaceLink {color: #999999; text-decoration: none}
.logoSpaceLink a:link {color: #999999; text-decoration: none}
.logoSpaceLink a:visited {color: #999999; text-decoration: none}
.logoSpaceLink a:active {color: #999999; text-decoration: none}
.logoSpaceLink a:hover {color: #003366; text-decoration: none}

/* basic panel (basicpanel.vmd) style */
.basicPanelContainer {border: 1px solid #3c78b5; margin-top: 2px; margin-bottom: 8px; width: 100%}
.basicPanelTitle {padding: 5px; margin: 0px; background-color: #f0f0f0; color: black; font-weight: bold;}
.basicPanelBody {padding: 5px; margin: 0px}

.separatorLinks a:link {color: white}
.separatorLinks a:visited {color: white}
.separatorLinks a:active {color: white}

.greynavbar {background-color: #f0f0f0; border-top: 1px solid #3c78b5; margin-top: 2px}

div.headerField {
    float: left;
    width: auto;
    height: 100%;
}

.headerFloat {
    margin-left: auto;
    width: 50%;
}

.headerFloatLeft {
    float: left;
    margin-right: 20px;
    margin-bottom: 10px;
}

#headerRow {
    padding: 10px;
}

div.license-personal {
   background-color: #003366;
   color: #ffffff;
}

div.license-personal a {
   color: #ffffff;
}

.greyFormBox {
    border: 1px solid #cccccc;
    padding: 5px;
}

/* IE automatically adds a margin before and after form tags. Use this style to remove that */
.marginlessForm {
    margin: 0px;
}

.openPageHighlight {
    background-color: #ffffcc;
    padding: 2px;
    border: 1px solid #ddd;
}

.editPageInsertLinks, .editPageInsertLinks a
{
    color: #666666;
    font-weight: bold;
    font-size: 10px;
}

/* Style for label heatmap. */
.top10 a {
    font-weight: bold;
    font-size: 2em;
    color: #003366;
}
.top25 a {
    font-weight: bold;
    font-size: 1.6em;
    color: #003366;
}
.top50 a {
    font-size: 1.4em;
    color: #003366;
}
.top100 a {
    font-size: 1.2em;
    color: #003366;
}

.heatmap {
    list-style:none;
    width: 95%;
    margin: 0px auto;
}

.heatmap a {
    text-decoration:none;
}

.heatmap a:hover {
    text-decoration:underline;
}

.heatmap li {
    display: inline;
}

.minitab {
padding: 3px 0px 3px 8px;
margin-left: 0;
margin-top: 1px;
margin-bottom: 0px;
border-bottom: 1px solid #3c78b5;
font: bold 9px Verdana, sans-serif;
text-decoration: none;
float:none;
}
.selectedminitab {
padding: 3px 0.5em;
margin-left: 3px;
margin-top: 1px;
border: 1px solid #3c78b5;
background: white;
border-bottom: 1px solid white;
color: #000000;
text-decoration: none;
}
.unselectedminitab {
padding: 3px 0.5em;
margin-left: 3px;
margin-top: 1px;
border: 1px solid #3c78b5;
border-bottom: none;
background: #3c78b5;
color: #ffffff;
text-decoration: none;
}

a.unselectedminitab:hover {
color: #ffffff;
background: #003366;
border-color: #003366;
}

a.unselectedminitab:link { color: white; }
a.unselectedminitab:visited { color: white; }

a.selectedminitab:link { color: black; }
a.selectedminitab:visited { color: black; }

.linkerror { background-color: #fcc;}

a.labelOperationLink:link {text-decoration: underline}
a.labelOperationLink:active {text-decoration: underline}
a.labelOperationLink:visited {text-decoration: underline}
a.labelOperationLink:hover {text-decoration: underline}

a.newLabel:link {background-color: #ddffdd}
a.newLabel:active {background-color: #ddffdd}
a.newLabel:visited {background-color: #ddffdd}
a.newLabel:hover {background-color: #ddffdd}

ul.square {list-style-type: square}

.inline-control-link {
    background: #ffc;
    font-size: 9px;
    color: #666;
    padding: 2px;
    text-transform: uppercase;
    text-decoration: none;
}


.inline-control-link a:link {text-decoration: none}
.inline-control-link a:active {text-decoration: none}
.inline-control-link a:visited {text-decoration: none}
.inline-control-link a:hover {text-decoration: none}

.inline-control-link {
    background: #ffc;
    font-size: 9px;
    color: #666;
    padding: 2px;
    text-transform: uppercase;
    text-decoration: none;
    cursor: pointer;
}

div.auto_complete {
    width: 350px;
    background: #fff;
}
div.auto_complete ul {
    border: 1px solid #888;
    margin: 0;
    padding: 0;
    width: 100%;
    list-style-type: none;
}
div.auto_complete ul li {
    margin: 0;
    padding: 3px;
}
div.auto_complete ul li.selected {
    background-color: #ffb;
}
div.auto_complete ul strong.highlight {
    color: #800;
    margin: 0;
    padding: 0;
}

/******* Edit Page Styles *******/
.toogleFormDiv{
    border:1px solid #A7A6AA;
    background-color:white;
    padding:5px;
    margin-top: 5px;
}

.toogleInfoDiv{
    border:1px solid #A7A6AA;
    background-color:white;
    display:none;
    padding:5px;
    margin-top: 10px;
}

.inputSection{
    margin-bottom:20px;
}

#editBox{
   border:1px solid lightgray;
   background-color:#F0F0F0;
}

/******* Left Navigation Theme Styles ********/
.leftnav li a {
    text-decoration:none;
    color:white;
    margin:0px;
    display:block;
    padding:2px;
    padding-left:5px;
    background-color: #3c78b5;
    border-top:1px solid #3c78b5;
}

.leftnav li a:active {color:white;}
.leftnav li a:visited {color:white;}
.leftnav li a:hover {background-color: #003366; color:white;}

/* Added by Shaun during i18n */
.replaced
{
    background-color: #33CC66;
}

.topPadding
{
    margin-top: 20px;
}

/* new form style */
.form-block {
    padding: 6px;
}
.form-error-block {
    padding: 6px;
    background: #fcc;
    border-top: #f0f0f0 1px solid;
    border-bottom: #f0f0f0 1px solid;
    margin-bottom: 6px;
    padding: 0 12px 0 12px;
}
.form-element-large {
    font-size: 16px;
    font-weight: bold;
    font-family: Arial, sans-serif;
    color: #003366;
}

.form-element-small {
    font-size: 12px;
    font-weight: bold;
    font-family: Arial, sans-serif;
    color: #003366;
}

.form-header {
    background: lightyellow;
    border-top: #f0f0f0 1px solid;
    border-bottom: #f0f0f0 1px solid;
    margin-bottom: 6px;
    padding: 0 12px 0 12px;
}
.form-header p, .form-block p, .form-error-block p {
    line-height: normal;
    margin: 12px 0 12px 0;
}
.form-example {
    color: #888;
    font-size: 11px;
}
.form-divider {
    border-bottom: #ccc 1px solid;
    margin-bottom: 6px;
}
.form-buttons {
    margin-top: 6px;
    border-top: #ccc 1px solid;
    border-bottom: #ccc 1px solid;
    background: #f0f0f0;
    padding: 10px;
    text-align: center;
}
.form-buttons input {
    width: 100px;
}
.form-block .error {
    padding: 6px;
    margin-bottom: 6px;
}
    -->
    </style>
</head>
<body>

<div id="PageContent">
<table class="pagecontent" border="0" cellpadding="0" cellspacing="0" width="100%"><tr>
<td valign="top" class="pagebody">

    <div class="pageheader">
        <span class="pagetitle">
            Page Edited :
            <a href="http://cwiki.apache.org/confluence/display/qpid">qpid</a> :
            <a href="http://cwiki.apache.org/confluence/display/qpid/Cluster+Design+Note">Cluster Design Note</a>
        </span>
    </div>

     <p>
        <a href="http://cwiki.apache.org/confluence/display/qpid/Cluster+Design+Note">Cluster Design Note</a>
        has been edited by             <a href="http://cwiki.apache.org/confluence/display/~aconway">Alan Conway</a>
            <span class="smallfont">(Mar 08, 2007)</span>.
     </p>
    
     <p>
                 <a href="http://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=31318&originalVersion=8&revisedVersion=9">(View changes)</a>
     </p>

    <span class="label">Content:</span><br/>
    <div class="greybox wiki-content"><h1><a name="ClusterDesignNote-ClusterDesignNote"></a>Cluster Design Note</h1>

<p>A Qpid <em>cluster</em> is a group of brokers co-operating to provide the<br/>
illusion of a single "virtual broker" with some extra qualities:</p>
<ul>
	<li>The cluster continues to provide service as long as at least one member survives.</li>
	<li>If a client is disconnected unexpectedly from a cluster member it can fail-over to another cluster member, giving the impression of uninterupted service.</li>
</ul>


<p>This design discusses clustering at the AMQP protocol layer,<br/>
i.e. members of a cluster have distinct AMQP addresses and AMQP<br/>
protocol commands are exchanged to negotiate reconnection and<br/>
failover. "Transparent" failover in this context means transparent to<br/>
the application, the AMQP client will be aware of disconnects and must<br/>
take action to fail over.</p>

<p>More precisely we define <em>transparent failover</em> to mean this: In the<br/>
event of failover, the sequence of protocol commands sent and received<br/>
by the client on each channel <em>excluding failover-related commands</em> is<br/>
identical to the sequence that would have been sent/received if no<br/>
failure had occured.</p>

<p>Given this definition the failover component of an AMQP client library<br/>
can simply hide all failover-related commands from the application<br/>
(indeed from the rest of the library) without breaking any semantics.</p>

<h1><a name="ClusterDesignNote-Requirements"></a>Requirements</h1>

<p><b>TODO</b>: Define levels of reliability we want to provide - survive one<br/>
node failure, survive multiple node failures, survive total failure,<br/>
network partitions etc. Does durable/non-durable message distinction<br/>
mean anything in a reliable cluster? I.e. can we lose non-durable<br/>
messages on a node failure? Can we lose them on orderly shutdown or<br/>
total failure?</p>

<p><b>TODO</b>: The requirements triangle. Concrete performance data.</p>

<p>Clients only need to use standard AMQP to talk to a cluster. They need<br/>
to understand some AMQP extensions to fail-over or resume sessions.<br/>
We will also use AMQP for point-to-point communication within the<br/>
cluster.</p>

<p>Ultimately we may propose extensions to AMQP spec but for the initial<br/>
implementation we can use existing extension points:</p>
<ul class="alternate" type="square">
	<li>Field table parameters to various AMQP methods (declare() arguments etc.)</li>
	<li>Field table in message headers.</li>
	<li>System exchanges and queues.</li>
</ul>


<h1><a name="ClusterDesignNote-AbstractModelandTerms"></a>Abstract Model and Terms</h1>

<p>A quick re-cap of AMQP terminology and introduction to some new terms:</p>

<p>A <em>broker</em> is a container for 3 types of <em>broker components</em>: <em>queues</em>,<br/>
<em>exchanges</em> and <em>bindings</em>. Broker components represent resources<br/>
available to multiple clients, and are not affected by clients<br/>
connecting and disconnecting. <em>Persistent</em> broker components are<br/>
unaffected by shut-down and re-start of a broker.</p>

<p>A <em>client</em> uses the components contained in a broker via the AMQP<br/>
protocol. The <em>client components</em> are <em>connection</em>, <em>channel</em>, <em>consumer</em> and<br/>
<em>session</em>[<span class="nobr"><a href="/confluence/pages/createpage.action?spaceKey=qpid&amp;title=Footnote%28The+%22session%22+concept+is+not+fully+defined+in+AMQP+0-8+or+0-9+but+is+under+discussion.+This+design+note+will+define+a+session+that+may+be+proposed+to+AMQP.%29&amp;linkCreation=true&amp;fromPageId=31318" title="Create Page: Footnote(The &quot;session&quot; concept is not fully defined in AMQP 0-8 or 0-9 but is under discussion. This design note will define a session that may be proposed to AMQP.)" class="createlink">Footnote&#40;The "session" concept is not fully defined in AMQP 0&#45;8 or 0&#45;9 but is under discussion. This design note will define a session that may be proposed to AMQP.&#41;<sup><img class="rendericon" src="/confluence/images/icons/plus.gif" height="7" width="7" align="absmiddle" alt="" border="0"/></sup></a></span>. Client components represent the relationship between a client<br/>
and a broker.</p>

<p><b>TODO</b>: &#95;Where do transactions fit in the model? They are also a kind of<br/>
relationship components but Dtx transactions may span more than just<br/>
client/broker.&#95;</p>

<p>A client's interaction with a unclustered <em>individual broker</em>[<span class="error">&#91;Footnote(An individual broker by this definition is really a broker behind a single AMQP address. Such a broker might in fact be a cluster using technologies like TCP failover/load balancing. This is outside the scope of this design, which focusses on clustering at the AMQP protocol layer, where cluster members have separate AMQP addresses.)&#93;</span> is<br/>
defined by AMQP 0-8/0-9: create a connection to the brokers <em>address</em>,<br/>
create channels, exchange AMQP commands (which may create consumers),<br/>
disconnect. After a disconnect the client can reconnect to the same<br/>
broker address. Broker components created by the previous connection<br/>
are preserved but client components are not. In the event of a<br/>
disorderly disconnect the outcome of commands in flight can only be<br/>
determined by their effects on broker components.</p>

<p>A <em>broker cluster</em> (or just <em>cluster</em>) is a "virtual" broker implemented<br/>
by several <em>member brokers</em> (or just <em>members</em>.) A cluster has many AMQP<br/>
addresses - the addresses of all its members - all semantically<br/>
equivalent for client connection [<span class="nobr"><a href="/confluence/pages/createpage.action?spaceKey=qpid&amp;title=Footnote%28They+may+not+be+equivalent+on+other+grounds%2C+e.g.+network+distance+from+client%2C+load+etc.%29&amp;linkCreation=true&amp;fromPageId=31318" title="Create Page: Footnote(They may not be equivalent on other grounds, e.g. network distance from client, load etc.)" class="createlink">Footnote&#40;They may not be equivalent on other grounds, e.g. network distance from client, load etc.&#41;<sup><img class="rendericon" src="/confluence/images/icons/plus.gif" height="7" width="7" align="absmiddle" alt="" border="0"/></sup></a></span>. The cluster members co-operate to<br/>
ensure:</p>
<ul>
	<li>all broker components are available via any cluster member.</li>
	<li>all broker components remain available if a member fails, provided at least one member remains active.</li>
	<li>clients disconnected by member failure or network failure can reconnect to another member and resume their <em>session</em>.</li>
</ul>


<p>A <em>session</em> is an identity for a collection of client components (i.e. a<br/>
client-broker relationship) that can outlive a single connection. AMQP<br/>
0-9 provides some support for sessions in the `resume` command.</p>

<p>If a connection is closed in an orderly manner by either end, the<br/>
session is also closed. However if there is an abrupt disconnect with<br/>
no Connection.close command, the session remains viable for some<br/>
(possibly long) timeout period. The client can reconnect to a failover<br/>
candidate and resume.</p>

<p>A session is like an extended connection: if you cut out the failover<br/>
commands the conversation on a session is exactly the conversation the<br/>
client would have had on a single connection to an individual broker<br/>
with no failures.</p>

<p>If a connection is in a session, events in the AMQP.0-8 spec that are<br/>
triggered by closing the connection (e.g. deleting auto-delete queues)<br/>
are instead trigged by the close (or timeout) of the session.</p>

<p>Note the session concept could also be used in the absence of<br/>
clustering to allow a client to disconnect and resume a long-running<br/>
session with the same broker. This is outside the scope of this<br/>
design.</p>

<h1><a name="ClusterDesignNote-ClusterState"></a>Cluster State</h1>

<p>We have to consider several kinds of state:</p>
<ul class="alternate" type="square">
	<li>Wiring: existence of exchanges, Queues and bindings between them.</li>
	<li>Content: messages on queues and references under construction.</li>
	<li>Cluster membership: list of members</li>
	<li>Conversational state: Sessions, channels, prefetch windows, consumers, acks, etc.</li>
</ul>


<p>Wiring and content needs to be replicated in a fault tolerant way<br/>
among the brokers so all brokers can provide access to all<br/>
resources. Persistent data also needs to be stored persistently for<br/>
recovery from total failure or deliberate shutdown.</p>

<p>Cluster membership needs to be replicated among brokers and passed to<br/>
the client. The client needs to know which members are candidates for<br/>
failover.</p>

<p>Conversational state relates to a client-broker relationship:</p>
<ul class="alternate" type="square">
	<li>session identity.</li>
	<li>open channels, channel attributes (qos).</li>
	<li>consumers.</li>
	<li>commands "in flight" - sent but not acknowledged.</li>
	<li>acknowledged command history.</li>
</ul>


<p>To resume successfully the converstaional state must be re-established<br/>
on both sidess. There are several options about how much state is<br/>
stored where. We'll outline the solution that minimizes broker-side<br/>
replication, but it's not clear yet if this is the optimal choice.</p>

<p>To minimize converstaional replication on the broker, the broker must<br/>
replicate at least:</p>
<ul>
	<li>session identities: to recognize the session on reconnect.</li>
	<li>history of acknowleded commands: to ignore duplicates resent by client.</li>
	<li>commands in flight: to resend to client and/or handle client responses.</li>
</ul>


<p>Everything else can be re-established by the client:</p>
<ul>
	<li>re-open the channels.</li>
	<li>re-set qos settings.</li>
	<li>re-create consumers.</li>
	<li>ignore incoming duplicates during reconnect.</li>
	<li>re-send all unacknowledged commands.</li>
</ul>


<p>Note that all of the above can be accomplished with normal AMQP commands.</p>

<p>The broker could replicate more converstaional state to relieve the<br/>
client from re-creating it. It's not clear yet where the best tradeoff<br/>
lies since it's not possible to have 0 conversational state on the<br/>
broker. Minimizing the brokers role seems like a good approach since<br/>
replicating data affects the whole cluster, keeping it on the<br/>
client affects only one client-broker connection.</p>

<h1><a name="ClusterDesignNote-Replication"></a>Replication</h1>

<p>The different types of state are replicated in different ways to<br/>
strike the best performance/reliability balance.</p>

<h2><a name="ClusterDesignNote-ClustermembershipandWiring"></a>Cluster membership and Wiring</h2>

<p>Membership changes are replicated to entire cluster symmetrically<br/>
using a virtual synchrony protocol.</p>

<p>Connected clients are also notified of changes to the set of fail-over<br/>
candidates for that client. Clients are notified over AMQP by binding<br/>
a queue to a special system exchange.</p>

<p>Wiring is low volume so and can also be replicated via virtual<br/>
synchrony. Cluster membership + wiring make up the common "picture"<br/>
that every member has of the cluster.</p>

<h2><a name="ClusterDesignNote-QueueContent"></a>Queue Content</h2>

<p>Message content is too high volume to replicate to the entire cluster.<br/>
To limit the extent of replication each queue or reference plays<br/>
exactly one of the following roles in each broker:</p>
<ul class="alternate" type="square">
	<li>Primary: content owner, all modifications are done by primary.</li>
	<li>Proxy: Forwards client requests to the primary.</li>
	<li>Backup: A proxy that also receives a replica of the content and can take over if the primary fails.</li>
	<li>Fragment: special case for shared queues, see below.</li>
</ul>


<p>A single cluster member may contain a mix of primary, backup and proxy<br/>
queues. (TODO: mechanism for establishing primary, backup etc.)</p>

<p>The client is unaware of the distinction, it sees an identical picture<br/>
regardless of what broker it connects to.</p>

<p><b>TODO</b>: Ordering issues with proxys and put-back messages (reject,<br/>
transaction rollback) or selectors.</p>

<h2><a name="ClusterDesignNote-FragmentedSharedQueues"></a>Fragmented Shared Queues</h2>

<p>A shared queue has reduced ordering requirements and increased<br/>
distribution requirements. "Fragmenting" a shared queue is a special<br/>
type of replication where each of a set of brokers holds a disjoint<br/>
subset (fragment) of the messages on the queue.  The idea is to<br/>
distribute load over independent fragments hosted in separate brokers.</p>

<p>Each fragment's content is replicated to backups independently just<br/>
like a normal queue.</p>

<p>The appearance of a single queue is created by collaboration between<br/>
fragments. Fragments store incomging messges in the local queue, and<br/>
server local consumers from the local queue whenever possible. Only<br/>
when a fragment cannot satisfy its consumers does it consume messages<br/>
from other fragments in the group.</p>

<p>Proxies to a fragmented queue will consume from the "nearest" fragment<br/>
if possible.</p>

<p><b>TODO</b>: Proxies can play a more active role. Ordering guarantees, we can<br/>
provide "same producer to same consumer preserves order" since<br/>
messages from the same producer always go on the same fragment<br/>
queue. May break down in the presence of failover unless we remember<br/>
which fragment received messges from the client and proxy to the same<br/>
one on the failover replica.</p>

<h2><a name="ClusterDesignNote-ConversationalState"></a>Conversational State</h2>

<p>The minimum converstaional state a broker needs to replicate for failover is</p>
<ul>
	<li>session identities: to recognize the session on reconnect.</li>
	<li>history of processed request-ids: to ignore duplicates resent by client.</li>
	<li>all requests in flight: to resend to client and/or handle client responses.</li>
	<li>open incoming references.</li>
</ul>


<p>Session identities and processed requiest history is very low volume<br/>
and could be replicated with virtual synchrony. However commands in flight<br/>
and open incoming references are probably too large to be replicated this<br/>
way.</p>

<p>Enter the <em>session backup</em>: the broker directly connected to a client is<br/>
the <em>session primary</em>. The session primary has one or more <em>session backup</em><br/>
brokers. Conversational state is replicated only to the session<br/>
backups, not to the entire group. On failure the client must reconnect<br/>
to one of the session backups, other members will not accept the<br/>
session.</p>

<h2><a name="ClusterDesignNote-Sharedstorage"></a>Shared storage</h2>

<p>In the case of high-performance, reliable shared storage (e.g. GFS)<br/>
queue content can be stored instead of replicated to backups. In that<br/>
case the conversations with content backups can be dehydrated, on<br/>
failover the backup can recover data from shared store.</p>

<p>For commodity hardware cases we need a solution with no shared store<br/>
as described above.</p>

<p><b>TODO</b>: Would we wan to use shared store for conversational state?</p>

<h1><a name="ClusterDesignNote-ClientBrokerProtocol"></a>Client-Broker Protocol</h1>

<p><b>TODO</b>: How it looks in client protocol terms - system queues &amp;<br/>
exchanges, connect argument tables.</p>

<p>Membership updates, choosing a replica, reconnection,<br/>
building conversational state =&gt; client stores representation of state<br/>
unacked messages &amp; duplicates, hiding failover.</p>

<h1><a name="ClusterDesignNote-BrokerBrokerProtocol"></a>Broker-Broker Protocol</h1>

<p>Broker-broker communication uses normal AMQP over specially identified<br/>
connections and channels (identified in the connection negotiation<br/>
argument table.)</p>

<p><b>Proxying</b>: Proxies simply forward methods between client and primary<br/>
and create consumers on behalf of the client. <b>TODO</b>: &#95;any issues with<br/>
message put-back and transactions?&#95;</p>

<p><b>Queue/fragment replication</b>: Use AMQP transfer commands to transfer content<br/>
to backup(s). <b>TODO</b>: propagating transactions.</p>

<p><b>Session replication</b>:<br/>
 must replicate a command (and get confirmation it was<br/>
replicated) before responding. However this can be done in async<br/>
streams - forward commands to replica</p>

<p>Session replication: AMQP on special connections. Primary forwards all<br/>
outgoing requests and incoming responses to session backup. Backup can<br/>
track the primary request/response tables and retransmit messages.</p>

<p>Note the<br/>
dehydrated requests to the session backup should reference the content<br/>
backup so the backup can recover content in a failure. Alternatively<br/>
content could be sent to both - what's the tradeoff?</p>

<h1><a name="ClusterDesignNote-PersistenceandRecovery"></a>Persistence and Recovery</h1>


<h2><a name="ClusterDesignNote-Usecasesforpersistence"></a>Use cases for persistence</h2>

<p><b>Durable messages</b>: must be stored to disk across broker shutdowns.</p>

<p><b>Flow-to-disk</b>: to cope with memory overflow on full queues.</p>

<p><b>Reliability</b>: recover after a crash.</p>

<p>We need a common solution - it would make no sense to be writing the<br/>
same message to disk multiple times for different purposes. The<br/>
solution may be tunable to offer better performance for individual<br/>
cases but should cover the case where all 3 are required.</p>

<h2><a name="ClusterDesignNote-Competingfailuremodes%3A"></a>Competing failure modes:</h2>

<p><b>Tibco</b>: fast when running clean but performance over time has GC<br/>
"spikes" Single journal for all queues. "holes" in log have to be<br/>
garbage collected to re-use the log. 1 slow consumer affects everyone<br/>
because it causes fragmentation of the log.</p>

<p><b>MQ</b>: write to journal, write journal to DB, read from DB.<br/>
Consistent &amp; reliable but slow.</p>

<p><b>Street homegrown solutions</b>: transient MQ with home grown<br/>
persistence. Can we get more design details for these solutions?</p>

<h2><a name="ClusterDesignNote-Persistenceapproach"></a>Persistence approach</h2>

<p>Try to avoid the problems with:</p>
<ul>
	<li>journal-per-queue</li>
	<li>multiple segment files per journal</li>
	<li>GC to "snapshot" (per queue? per broker? think about queue migration.)</li>
	<li>Disk thrashing - how do we distribute the parts to avoid it?</li>
</ul>


<p>Need to think about how we consolidate journal into a "snapshot" or<br/>
"database" to reclaim space.</p>

<p><b>No write on fast consume</b>: Optimization - if we can deliver (and get<br/>
ack) faster than we write then no need to write. How does this<br/>
interact with HA?</p>

<p><b>Async journalling</b>: writing to client, writing to journal, acks from<br/>
client, acks from journal are separate async streams? So if we get<br/>
client ack before the journalling stream has written the journal we<br/>
cancel the write? But what kind of ack info do we need? Need a diagram<br/>
of interactions, failure points and responses at each point. Start<br/>
simple and optimize, but dont rule out optimizations.</p>

<h2><a name="ClusterDesignNote-Whataboutpersistencefreereliability%3F"></a>What about persistence-free reliability?</h2>

<p>Is memory-only replication with no disk a viable option for high-speed<br/>
transient message flow? We will lose messages in total failure or<br/>
multiple failures where all backups fail, but we can survive single<br/>
failures and will run a lot faster than diskful.</p>

<h1><a name="ClusterDesignNote-Virtualsynchrony"></a>Virtual synchrony</h1>

<p><b>TODO</b>: Wiring &amp; membership via virtual synchrony<br/>
<b>TODO</b>: journaling, speed. Will file-per-q really help with disk burnout?</p>

<h1><a name="ClusterDesignNote-Configuration"></a>Configuration</h1>

<p>Connection issues: potentiall for big N*N connection web if primaries<br/>
and backups are not grouped.</p>

<p>Simplifying patterns:</p>
<ul>
	<li>Virtual hosts as units of replication.</li>
	<li>Backup rings: all primary components in a broker use the same backup broker and vice-versa. Backups form rings.</li>
	<li>Disk management issues?</li>
	<li>Shared storage issues?</li>
</ul>


<h2><a name="ClusterDesignNote-Dynamicclusterconfiguration"></a>Dynamic cluster configuration</h2>

<ul class="alternate" type="square">
	<li>Failover: the primary use case.</li>
	<li>Add node: backup, proxy, primary case?</li>
	<li>Redirect clients from loaded broker (pretend failure)</li>
	<li>Move queue primary from loaded broker/closer to consumers?</li>
	<li>Re-start after failover.</li>
</ul>


<p><b>Issue:</b> unit of failover/redirect is connection/channel but "working<br/>
set" of queues and exchanges is unrelated. Use virtual host as unit<br/>
for failover/relocation? It's also a queue namespace...</p>

<p>If a queue moves we have to redirect its <em>consumers</em>, can't redirect<br/>
entire channels&#33; Channels in the same session may move between<br/>
connections. Or rather we depend on broker to proxy?</p>

<p>Backups: chained backups rather than multi-backup? Ring backup?<br/>
What about split brain, elections, quorums etc.</p>

<p>Should new backups acquire state from primary, from disk or possibly<br/>
both? Depends on GFS/SAN vs. commodity hw?</p>

<h1><a name="ClusterDesignNote-OpenQuestions"></a>Open Questions</h1>

<p>Issues: double failure in backup ring: A &#45;&gt; B &#45;&gt; C. Simultaneous<br/>
failure of A and B. C doesn't have the replica data to take over for A.</p>

<p>Java/C+&#43; interworking - is there a requirement? Fail over from C+&#43; to Java?<br/>
Common persistence formats?</p>

<h1><a name="ClusterDesignNote-Implementationbreakdown."></a>Implementation breakdown.</h1>

<p>The following are independently useful units of work that combine to<br/>
give the full story:</p>

<p><b>Proxy Queues</b>: Useful in federation. Pure-AMQP proxies for exchanges<br/>
might also be useful but are not needed for current purpose as we will<br/>
use virtual synchrony to replicate wiring.</p>

<p><b>Fragmented queues</b>: Over pure AMQP (no VS) useful by itself for unreliable<br/>
high volume shared queue federation.</p>

<p><b>Virtual Synchrony Cluster</b>: Multicast membership and total ordering<br/>
protocol for brokers. Not useful alone, but useful with proxies and/or<br/>
fragments for dynamic federations.</p>

<p><b>Primary-backup replication</b>: Over AMQP, no persistence. Still offers some<br/>
level of reliability in a simple primary-backup pair.</p>

<p><b>Persistence</b>: Useful on its own for flow-to-disk and durable messages.<br/>
Must meet the performance requirements of reliable journalling.</p>

<hr />
<p>&lt;muse:fn-sep?/&gt;  Exclusive or auto-delete queues are deleted on disconnect, we'll return to this point.</p></div>


</td></tr></table></div>
<p>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
    <tr>
        <td height="12" background="http://cwiki.apache.org/confluence/images/border/border_bottom.gif"><img src="http://cwiki.apache.org/confluence/images/border/spacer.gif" width="1" height="1" border="0"/></td>
    </tr>
</table>

<div class="smalltext">
    Powered by
    <a href="http://www.atlassian.com/software/confluence/default.jsp?clicked=footer" class="smalltext">Atlassian Confluence</a>
    (Version: 2.2.9 Build:#527 Sep 07, 2006)
    -
    <a href="http://jira.atlassian.com/secure/BrowseProject.jspa?id=10470" class="smalltext">Bug/feature request</a><br/>
    <br>
    <a href="http://cwiki.apache.org/confluence/users/viewnotifications.action">Unsubscribe or edit your notifications preferences</a>

</div>

</body>
</html>


Mime
View raw message