incubator-accumulo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From build...@apache.org
Subject svn commit: r803212 - in /websites/staging/accumulo/trunk/content/accumulo/example: ./ wikisearch.html
Date Tue, 24 Jan 2012 19:38:52 GMT
Author: buildbot
Date: Tue Jan 24 19:38:51 2012
New Revision: 803212

Log:
Staging update by buildbot for accumulo

Added:
    websites/staging/accumulo/trunk/content/accumulo/example/
    websites/staging/accumulo/trunk/content/accumulo/example/wikisearch.html

Added: websites/staging/accumulo/trunk/content/accumulo/example/wikisearch.html
==============================================================================
--- websites/staging/accumulo/trunk/content/accumulo/example/wikisearch.html (added)
+++ websites/staging/accumulo/trunk/content/accumulo/example/wikisearch.html Tue Jan 24 19:38:51
2012
@@ -0,0 +1,423 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<html>
+<head>
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one or more
+    contributor license agreements.  See the NOTICE file distributed with
+    this work for additional information regarding copyright ownership.
+    The ASF licenses this file to You under the Apache License, Version 2.0
+    (the "License"); you may not use this file except in compliance with
+    the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE- 2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+-->
+  <link href="/accumulo/css/accumulo.css" rel="stylesheet" type="text/css">
+  <title>The wikipedia search example explained, with performance numbers.</title>
+  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+  <script type="text/javascript">
+
+  var _gaq = _gaq || [];
+  _gaq.push(['_setAccount', 'UA-21103458-6']);
+  _gaq.push(['_trackPageview']);
+
+  (function() {
+    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
+    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
+    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
+  })();
+
+  </script>
+</head>
+
+<body>
+  <div id="banner">&nbsp;
+  </div>
+
+  <div id="navigation">
+  <h1 id="project">Project</h1>
+<ul>
+<li><a href="/accumulo">Home</a></li>
+<li><a href="http://incubator.apache.org/projects/accumulo.html">Incubator page</a></li>
+<li><a href="/accumulo/downloads/downloads.html">Downloads</a></li>
+<li><a href="/accumulo/notable_features.html">Features</a></li>
+<li><a href="http://www.apache.org/licenses/LICENSE-2.0">License</a></li>
+</ul>
+<h1 id="community">Community</h1>
+<ul>
+<li><a href="/accumulo/get_involved.html">Get Involved</a></li>
+<li><a href="/accumulo/mailing_list.html">Mailing Lists</a></li>
+<li><a href="https://issues.apache.org/jira/secure/ConfigureReport.jspa?versionId=-2&amp;selectedProjectId=12312121&amp;reportKey=com.sourcelabs.jira.plugin.report.contributions%3Acontributionreport">People</a></li>
+</ul>
+<h1 id="development">Development</h1>
+<ul>
+<li><a href="/accumulo/source.html">Source Code</a></li>
+<li><a href="https://issues.apache.org/jira/browse/accumulo">Issues</a></li>
+<li><a href="https://builds.apache.org/view/A-F/view/Accumulo/">Builds</a></li>
+</ul>
+<h1 id="documentation">Documentation</h1>
+<ul>
+<li><a href="/accumulo/user_manual_1.3-incubating">Manual v1.3</a></li>
+<li><a href="/accumulo/user_manual_1.3-incubating/examples.html">Examples v1.3</a></li>
+<li><a href="/accumulo/user_manual_1.4-incubating">Manual v1.4</a>
+<!-- - klzzwxh:0006 -->
+<!-- - Javadoc -->
+<!-- - Examples --></li>
+<li><a href="/accumulo/screenshots.html">Screenshots</a></li>
+<li><a href="/accumulo/papers.html">Papers</a></li>
+</ul>
+<!--
+# Development
+ - Source code
+ - Building
+-->
+
+<h1 id="asf_links">ASF links</h1>
+<ul>
+<li><a href="http://www.apache.org">Apache Software Foundation</a></li>
+<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+<li><a href="http://www.apache.org/foundation/sponsorship.html">Become a Sponsor</a></li>
+</ul>
+  </div>
+
+  <div id="bannertext">
+    <img id="logo" alt="Apache Accumulo (Incubating)" src="/accumulo/images/accumulo-logo.png"/>
+  </div>
+
+  <div id="content">
+    <h1 class="title">The wikipedia search example explained, with performance numbers.</h1>
+    <h2 id="accumulo-query-performance">Accumulo Query Performance</h2>
+<h2 id="sample-application">Sample Application</h2>
+<p>Starting with release 1.4, Accumulo includes an example web application that provides
a flexible,  scalable search over the articles of Wikipedia, a widely available medium-sized
corpus.</p>
+<p>The example uses an indexing technique helpful for doing multiple logical tests
against content.  In this case, we can perform a word search on Wikipedia articles.   The
sample application takes advantage of 3 unique capabilities of Accumulo:</p>
+<ol>
+<li>Extensible iterators that operate within the distributed region servers of the
key-value store</li>
+<li>Custom aggregators which can efficiently condense information during the various
life-cycles of the log-structured merge tree </li>
+<li>Custom load balancing, which ensures that a table is evenly distributed on all
region servers</li>
+</ol>
+<p>In the example, Accumulo tracks the cardinality of all terms as elements are ingested.
 If the cardinality is small enough, it will track the set of documents by term directly.
 For example:</p>
+<table>
+<tr>
+<th>Row (word)</th>
+<th colspan="2">Value (count, document list)</th>
+</tr><tr>
+<td>Octopus
+<td align="right">2
+<td>[Document 57, Document 220]
+</tr><tr>
+<td>Other
+<td align="right">172849
+<td>[]
+</tr><tr>
+<td>Ostrich
+<td align="right">1
+<td>[Document 901]
+</tr>
+</table>
+
+<p>Searches can be optimized to focus on low-cardinality terms.  To create these counts,
the example installs “aggregators” which are used to combine inserted values.  The
ingester just writes simple  “(Octopus, 1, Document 57)” tuples.  The region servers
then used the installed aggregators to merge the cells as the data is re-written, or queried.
 This reduces the in-memory locking required to update high-cardinality terms, and defers
aggregation to a later time, where it can be done more efficiently.</p>
+<p>The example also creates a reverse word index to map each word to the document in
which it appears. But it does this by choosing an arbitrary partition for the document.  The
article, and the word index for the article are grouped together into the same partition.
 For example:</p>
+<table>
+<tr>
+<th>Row (partition)
+<th>Column Family
+<th>Column Qualifier
+<th>Value
+<tr>
+<td>1
+<td>D
+<td>Document 57
+<td>“smart Octopus”
+<tr>
+<td>1
+<td>Word, Octopus
+<td>Document 57
+<td>
+<tr>
+<td>1
+<td>Word, smart
+<td>Document 57
+<td>
+<tr>
+<td>...
+<td>
+<td>
+<td>
+<tr>
+<td>2
+<td>D
+<td>Document 220
+<td>“big Octopus”
+<tr>
+<td>2
+<td>Word, big
+<td>Document 220
+<td>
+<tr>
+<td>2
+<td>Word, Octopus
+<td>Document 220
+</table>
+
+<p>Of course, there would be large numbers of documents in each partition, and the
elements of those documents would be interlaced according to their sort order.</p>
+<p>By dividing the index space into partitions, the multi-word searches can be performed
in parallel across all the nodes.  Also, by grouping the document together with its index,
a document can be retrieved without a second request from the client.  The query “octopus”
and “big” will be performed on all the servers, but only those partitions for which
the low-cardinality term “octopus” can be found by using the aggregated reverse
index information.  The query for a document is performed by extensions provided in the example.
 These extensions become part of the region servers iterator stack.  By cloning the underlying
iterators, the query extensions can seek to specific words within the index, and when it finds
a matching document, it can then seek to the document location and retrieve the contents.</p>
+<p>We loaded the example on a  cluster of 10 servers, each with 12 cores, and 32G RAM,
6 500G drives.  Accumulo tablet servers were allowed a maximum of 3G of working memory, of
which 2G was dedicated to caching file data.</p>
+<p>Following the instructions in the example, the Wikipedia XML data for articles was
loaded for English, Spanish and German languages into 10 partitions.  The data is not partitioned
by language: multiple languages were used to get a larger set of test data.  The data load
took around 8 hours, and has not been optimized for scale.  Once the data was loaded, the
content was compacted which took about 35 minutes.</p>
+<p>The example uses the language-specific tokenizers available from the Apache Lucene
project for Wikipedia data.</p>
+<p>Original files:</p>
+<table>
+<tr>
+<th>Articles
+<th>Compressed size
+<th>Filename
+<tr>
+<td>1.3M
+<td>2.5G
+<td>dewiki-20111120-pages-articles.xml.bz2
+<tr>
+<td>3.8M
+<td>7.9G
+<td>enwiki-20111115-pages-articles.xml.bz2
+<tr>
+<td>0.8M
+<td>1.4G
+<td>eswiki-20111112-pages-articles.xml.bz2
+</table>
+
+<p>The resulting tables:</p>
+<div class="codehilite"><pre><span class="o">&gt;</span> <span
class="n">du</span> <span class="o">-</span><span class="n">p</span>
<span class="n">wiki</span><span class="o">.*</span>
+      <span class="mi">47</span><span class="p">,</span><span
class="mi">325</span><span class="p">,</span><span class="mi">680</span><span
class="p">,</span><span class="mi">634</span> <span class="p">[</span><span
class="n">wiki</span><span class="p">]</span>
+       <span class="mi">5</span><span class="p">,</span><span
class="mi">125</span><span class="p">,</span><span class="mi">169</span><span
class="p">,</span><span class="mi">305</span> <span class="p">[</span><span
class="n">wikiIndex</span><span class="p">]</span>
+                 <span class="mi">413</span> <span class="p">[</span><span
class="n">wikiMetadata</span><span class="p">]</span>
+       <span class="mi">5</span><span class="p">,</span><span
class="mi">521</span><span class="p">,</span><span class="mi">690</span><span
class="p">,</span><span class="mi">682</span> <span class="p">[</span><span
class="n">wikiReverseIndex</span><span class="p">]</span>
+</pre></div>
+
+
+<p>Roughly a 6:1 increase in size.</p>
+<p>We performed the following queries, and repeated the set 5 times.  The query language
is much more expressive than what is shown below.  The actual query specified that these words
were to be found in the body of the article.  Regular expressions, searches within titles,
negative tests, etc are available.</p>
+<table>
+<tr>
+<th>Query
+<th colspan="5">Samples (seconds)
+<th>Matches
+<tr>
+<td>“old” and “man” and “sea”
+<td>4.07
+<td>3.79
+<td>3.65
+<td>3.85
+<td>3.67
+<td>22956
+<tr>
+<td>“paris” and “in” and “the” and “spring”
+<td>3.06
+<td>3.06
+<td>2.78
+<td>3.02
+<td>2.92
+<td>10755
+<tr>
+<td>“rubber” and “duckie” and “ernie”
+<td>0.08
+<td>0.08
+<td>0.1
+<td>0.11
+<td>0.1
+<td>6
+<tr>
+<td>“fast”  and ( “furious” or “furriest”) 
+<td>1.34
+<td>1.33
+<td>1.3
+<td>1.31
+<td>1.31
+<td>2973
+<tr>
+<td>“slashdot” and “grok”
+<td>0.06
+<td>0.06
+<td>0.06
+<td>0.06
+<td>0.06
+<td>14
+<tr>
+<td>“three” and “little” and “pigs”
+<td>0.92
+<td>0.91
+<td>0.9
+<td>1.08
+<td>0.88
+<td>2742
+</table>
+
+<p>Because the terms are tested together within the region server, even fairly high-cardinality
terms such as “old,” “man,” and “sea” can be tested efficiently,
without needing to return to the client, or make distributed calls between servers to perform
the intersection between terms.</p>
+<p>Accumulo supports caching index information, which is turned on by default, and
for the non-index blocks of a file, which is not. After turning on data block caching for
the wiki table:</p>
+<table>
+<tr>
+<th>Query
+<th colspan="5">Samples (seconds)
+<tr>
+<td>“old” and “man” and “sea”
+<td>2.47
+<td>2.48
+<td>2.51
+<td>2.48
+<td>2.49
+</tr><tr>
+<td>“paris” and “in” and “the” and “spring”
+<td>1.33
+<td>1.42
+<td>1.6
+<td>1.61
+<td>1.47
+</tr><tr>
+<td>“rubber” and “duckie” and “ernie”
+<td>0.07
+<td>0.08
+<td>0.07
+<td>0.07
+<td>0.07
+</tr><tr>
+<td>“fast” and ( “furious” or “furriest”) 
+<td>1.28
+<td>0.78
+<td>0.77
+<td>0.79
+<td>0.78
+</tr><tr>
+<td>“slashdot” and “grok”
+<td>0.04
+<td>0.04
+<td>0.04
+<td>0.04
+<td>0.04
+</tr><tr>
+<td>“three” and “little” and “pigs”
+<td>0.55
+<td>0.32
+<td>0.32
+<td>0.31
+<td>0.27
+</tr>
+<table>
+
+For comparison, these are the cold start lookup times (restart Accumulo, and drop the operating
system disk cache):
+
+<table>
+<tr>
+<th>Query
+<th>Sample
+<tr>
+<td>“old” and “man” and “sea”
+<td>13.92
+<tr>
+<td>“paris” and “in” and “the” and “spring”
+<td>8.46
+<tr>
+<td>“rubber” and “duckie” and “ernie”
+<td>2.96
+<tr>
+<td>“fast” and ( “furious” or “furriest”) 
+<td>6.77
+<tr>
+<td>“slashdot” and “grok”
+<td>4.06
+<tr>
+<td>“three” and “little” and “pigs”
+<td>8.13
+</table>
+
+<p>Random Query Load</p>
+<p>Random queries were generated using common english words.  A uniform random sample
of 3 to 5 words taken from the 10000 most common words in the Project Gutenberg's online text
collection were joined with “and”.  Words containing anything other than letters
(such as contractions) were not used.  A client was started simultaneously on each of the
10 servers and each ran 100 random queries (1000 queries total).</p>
+<table>
+<tr>
+<th>Time
+<th>Count
+<tr>
+<td>41.97
+<td>440743
+<tr>
+<td>41.61
+<td>320522
+<tr>
+<td>42.11
+<td>347969
+<tr>
+<td>38.32
+<td>275655
+</table>
+
+<p>Query Load During Ingest</p>
+<p>The English wikipedia data was re-ingested on top of the existing, compacted data.
The following  query samples were taken in 5 minute intervals while ingesting 132 articles/second:</p>
+<table>
+<tr>
+<th>Query
+<th colspan="5">Samples (seconds)
+<tr>
+<td>“old” and “man” and “sea”
+<td>4.91
+<td>3.92
+<td>11.58
+<td>9.86
+<td>10.21
+<tr>
+<td>“paris” and “in” and “the” and “spring”
+<td>5.03
+<td>3.37
+<td>12.22
+<td>3.29
+<td>9.46
+<tr>
+<td>“rubber” and “duckie” and “ernie”
+<td>4.21
+<td>2.04
+<td>8.57
+<td>1.54
+<td>1.68
+<tr>
+<td>“fast”  and ( “furious” or “furriest”) 
+<td>5.84
+<td>2.83
+<td>2.56
+<td>3.12
+<td>3.09
+<tr>
+<td>“slashdot” and “grok”
+<td>5.68
+<td>2.62
+<td>2.2
+<td>2.78
+<td>2.8
+<tr>
+<td>“three” and “little” and “pigs”
+<td>7.82
+<td>3.42
+<td>2.79
+<td>3.29
+<td>3.3
+</table>
+  </div>
+
+  <div id="footer">
+    <div class="copyright">
+      <p>
+        Copyright &copy; 2011 The Apache Software Foundation, Licensed under
+        the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version
2.0</a>.
+        <br />
+        Apache and the Apache feather logos are trademarks of The Apache Software Foundation.
+      </p>
+    </div> 
+    <a alt="Apache Incubator" href="http://incubator.apache.org">
+      <img id="asf-logo" alt="Apache Incubator" src="/accumulo/images/apache-incubator-logo.png"
width="150"/>
+    </a>
+
+  </div>
+
+</body>
+</html>



Mime
View raw message