X-Mailer: ASF-Git Admin Mailer Subject: [18/50] [abbrv] accumulo-website git commit: Jekyll build from gh-pages:358b7b4 archived-at: Tue, 22 Nov 2016 19:41:03 -0000 http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/c0655661/release/accumulo-1.5.1/index.html ---------------------------------------------------------------------- diff --git a/release/accumulo-1.5.1/index.html b/release/accumulo-1.5.1/index.html new file mode 100644 index 0000000..247b3df --- /dev/null +++ b/release/accumulo-1.5.1/index.html @@ -0,0 +1,408 @@ + + + + + + + + + + + + +Apache Accumulo 1.5.1 + + + + + + + + + + + + +

+ + +

+ +

Apache Accumulo 1.5.1

+ +

06 Mar 2014

+ +

Apache Accumulo 1.5.1 is a maintenance release on the 1.5 version branch. +This release contains changes from over 200 issues, comprised of bug fixes +(client side and server side), new test cases, and updated Hadoop support +contributed by over 30 different contributors and committers. +As this is a maintenance release, Apache Accumulo 1.5.1 has no client API +incompatibilities over Apache Accumulo 1.5.0 and requires no manual upgrade +process. Users of 1.5.0 are strongly encouraged to update as soon as possible +to benefit from the improvements.

+ +

Notable Improvements

+ +

While new features are typically not added in a bug-fix release as 1.5.1, the +community does create a variety of improvements that are API compatible. Contained +here are some of the more notable improvements.

+ +

PermGen Leak from Client API

+ +

Accumulo’s client code creates background threads that users presently cannot +stop through the API. This is quick to cause problems when invoking the Accumulo +API in application containers such as Apache Tomcat or JBoss and repeatedly +redeploying an application. ACCUMULO-2128 introduces a static utility, +org.apache.accumulo.core.util.CleanUp, that users can invoke as part of a +teardown hook in their container that will stop these threads and avoid +the eventual OutOfMemoryError “PermGen space”.

+ +

Prefer IPv4 when starting Accumulo processes

+ +

While Hadoop does not support IPv6 networks, attempting to run on a +system that does not have IPv6 completely disabled can cause strange failures. +ACCUMULO-2262 invokes the JVM-provided configuration parameter at process +startup to prefer IPv4 over IPv6.

+ +

Memory units in configuration

+ +

In previous versions, units of memory had to be provided as upper-case (e.g. ‘2G’, not ‘2g’). +Additionally, a non-intuitive error was printed when a lower-case unit was provided. +ACCUMULO-1933 allows lower-case memory units in all Accumulo configurations.

+ +

Apache Thrift maximum frame size

+ +

Apache Thrift is used as the internal RPC service. ACCUMULO-2360 allows +users to configure the maximum frame size an Accumulo server will read. This +prevents non Accumulo client from connecting and causing memory exhaustion.

+ +

MultiTableBatchWriter concurrency

+ +

The MultiTableBatchWriter is a class which allows multiple tables to be written to +from a single object that maintains a single buffer for caching Mutations across all tables. This is desirable +as it greatly simplifies the JVM heap usage from caching Mutations across +many tables. Sadly, in Apache Accumulo 1.5.0, concurrent access to a single MultiTableBatchWriter +heavily suffered from synchronization issues. ACCUMULO-1833 introduces a fix +which alleviates the blocking and idle-wait that previously occurred when multiple threads accessed +a single MultiTableBatchWriter instance concurrently.

+ +

Hadoop Versions

+ +

Since Apache Accumulo 1.5.0 was released, Apache Hadoop 2.2.0 was also released +as the first generally available (GA) Hadoop 2 release. This was a very exciting release +for a number of reasons, but this also caused additional effort on Accumulo’s part to +ensure that Apache Accumulo continues to work across multiple Hadoop versions. Apache Accumulo 1.5.1 +should function with any recent Hadoop 1 or Hadoop 2 without any special steps, tricks or instructions +required.

+ +

Notable Bug Fixes

+ +

As with any Apache Accumulo release, we have numerous bug fixes that have been fixed. Most +are very subtle and won’t affect the common user; however, some notable bugs were resolved +as a part of 1.5.1 that are rather common.

+ +

Failure of ZooKeeper server in quorum kills connected Accumulo services

+ +

Apache ZooKeeper provides a number of wonderful features that Accumulo uses to accomplish +a variety of tasks, most notably a distributed locking service. Typically, multiple ZooKeeper +servers are run to provide resilience against a certain number of node failures. ACCUMULO-1572 +resolves an issue where Accumulo processes would kill themselves when the ZooKeeper server they +were communicating with died instead of failing over to another ZooKeeper server in the quorum.

+ +

Monitor table state isn’t updated

+ +

The Accumulo Monitor contains a column for the state of each table in the Accumulo instance. +The previous resolution was to restart the Monitor process when it got in this state. +ACCUMULO-1920 resolves an issue where the Monitor would not see updates from ZooKeeper.

+ +

Two locations for the same extent

+ +

The !METADATA table is the brains behind the data storage for each table, tracking information +like which files comprise a Tablet, and which TabletServers are hosting which Tablets. ACCUMULO-2057 +fixes an issue where the !METADATA table contained multiple locations (hosting server) for +a single Tablet.

+ +

Deadlock on !METADATA tablet unload

+ +

Tablets are unloaded, typically, when a shutdown request is issued. ACCUMULO-1143 resolves +a potential deadlock issue when a merging-minor compaction is issued to flush in-memory data +to disk before unloading a Tablet.

+ +

Other notable fixes

+ +

ACCUMULO-1800 Fixed deletes made via the Proxy.
ACCUMULO-1994 Fixed ranges in the Proxy.
ACCUMULO-2234 Fixed offline map reduce over non default HDFS location.
ACCUMULO-1615 Fixed service accumulo-tserver stop.
ACCUMULO-1876 Fixed issues depending on Accumulo using Apache Ivy.
ACCUMULO-2261 Duplicate locations for a Tablet.
ACCUMULO-2037 Tablets assigned to previous location.
ACCUMULO-1821 Avoid recovery on recovering Tablets.
ACCUMULO-2078 Incorrectly computed ACCUMULO_LOG_HOST in example configurations.
ACCUMULO-1985 Configuration to bind Monitor on all network interfaces.
ACCUMULO-1999 Allow ‘0’ to signify random port for the Master.
ACCUMULO-1630 Fixed GC to interpret any IP/hostname.

+ +

Known Issues

+ +

When using Accumulo 1.5 and Hadoop 2, Accumulo will call hsync() on HDFS. +Calling hsync improves durability by ensuring data is on disk (where other older +Hadoop versions might lose data in the face of power failure); however, calling +hsync frequently does noticably slow writes. A simple work around is to increase +the value of the tserver.mutation.queue.max configuration parameter via accumulo-site.xml.

+ +

A value of “4M” is a better recommendation, and memory consumption will increase by +the number of concurrent writers to that TabletServer. For example, a value of 4M with +50 concurrent writers would equate to approximately 200M of Java heap being used for +mutation queues.

+ +

For more information, see ACCUMULO-1950 and this comment.

+ +

Documentation

+ +

The following documentation updates were made:

+ +

Testing

+ +

Below is a list of all platforms that 1.5.1 was tested against by developers. Each Apache Accumulo release +has a set of tests that must be run before the candidate is capable of becoming an official release. That list includes the following:

+ +

Successfully run all unit tests
Successfully run all functional test (test/system/auto)
Successfully complete two 24-hour RandomWalk tests (LongClean module), with and without “agitation”
Successfully complete two 24-hour Continuous Ingest tests, with and without “agitation”, with data verification
Successfully complete two 72-hour Continuous Ingest tests, with and without “agitation”

+ +

Each unit and functional test only runs on a single node, while the RandomWalk and Continuous Ingest tests run +on any number of nodes. Agitation refers to randomly restarting Accumulo processes and Hadoop Datanode processes, +and, in HDFS High-Availability instances, forcing NameNode failover.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

OS	Hadoop	Nodes	ZooKeeper	HDFS High-Availability	Tests
CentOS 6.5	HDP 2.0 (Apache 2.2.0)	6	HDP 2.0 (Apache 3.4.5)	Yes (QJM)	All required tests
CentOS 6.4	CDH 4.5.0 (2.0.0+cdh4.5.0)	7	CDH 4.5.0 (3.4.5+cdh4.5.0)	Yes (QJM)	Unit, functional and 24hr Randomwalk w/ agitation
CentOS 6.4	CDH 4.5.0 (2.0.0+cdh4.5.0)	7	CDH 4.5.0 (3.4.5+cdh4.5.0)	Yes (QJM)	2x 24/hr continuous ingest w/ verification
CentOS 6.3	Apache 1.0.4	1	Apache 3.3.5	No	Local testing, unit and functional tests
RHEL 6.4	Apache 2.2.0	10	Apache 3.4.5	No	Functional tests

+ + + +

View all releases in the archive

+ +

+ + + + + +

+ + http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/c0655661/release/accumulo-1.5.2/index.html ---------------------------------------------------------------------- diff --git a/release/accumulo-1.5.2/index.html b/release/accumulo-1.5.2/index.html new file mode 100644 index 0000000..016ff9e --- /dev/null +++ b/release/accumulo-1.5.2/index.html @@ -0,0 +1,365 @@ + + + + + + + + + + + + +Apache Accumulo 1.5.2 + + + + + + + + + + + + +

+ + +

+ +

Apache Accumulo 1.5.2

+ +

19 Sep 2015

+ +

Apache Accumulo 1.5.2 is a maintenance release on the 1.5 version branch. +This release contains changes from over 100 issues, comprised of bug fixes +(client side and server side), new test cases, and updated Hadoop support +contributed by over 30 different contributors and committers. +As this is a maintenance release, Apache Accumulo 1.5.2 has no client API +incompatibilities over Apache Accumulo 1.5.0 and 1.5.1 and requires no manual upgrade +process. Users of 1.5.0 or 1.5.1 are strongly encouraged to update as soon as possible +to benefit from the improvements.

+ +

Users who are new to Accumulo are encouraged to use a 1.6 release as opposed +to the 1.5 line as development has already shifted towards the 1.6 line. For those +who cannot or do not want to upgrade to 1.6, 1.5.2 is still an excellent choice +over earlier versions in the 1.5 line.

+ +

Performance Improvements

+ +

Apache Accumulo 1.5.2 includes a number of performance-related fixes over previous versions.

+ +

Write-Ahead Log sync performance

+ +

The Write-Ahead Log (WAL) files are used to ensure durability of updates made to Accumulo. +A sync is called on the file in HDFS to make sure that the changes to the WAL are persisted +to disk, which allows Accumulo to recover in the case of failure. ACCUMULO-2766 fixed +an issue where an operation against a WAL would unnecessarily wait for multiple syncs, slowing +down the ingest on the system.

+ +

Minor-Compactions not aggressive enough

+ +

On a system with ample memory provided to Accumulo, long hold-times were observed which +blocks the ingest of new updates. Trying to free more server-side memory by running minor +compactions more frequently increased the overall throughput on the node. These changes +were made in ACCUMULO-2905.

+ +

HeapIterator optimization

+ +

Iterators, a notable feature of Accumulo, are provided to users as a server-side programming +construct, but are also used internally for numerous server operations. One of these system iterator +is the HeapIterator which implements a PriorityQueue of other Iterators. One way this iterator is +used is to merge multiple files in HDFS to present a single, sorted stream of Key-Value pairs. ACCUMULO-2827 +introduces a performance optimization to the HeapIterator which can improve the speed of the +HeapIterator in common cases.

+ +

Write-Ahead log sync implementation

+ +

In Hadoop-2, two implementations of sync are provided: hflush and hsync. Both of these +methods provide a way to request that the datanodes write the data to the underlying +medium and not just hold it in memory (the fsync syscall). While both of these methods +inform the Datanodes to sync the relevant block(s), hflush does not wait for acknowledgement +from the Datanodes that the sync finished, where hsync does. To provide the most reliable system +“out of the box”, Accumulo defaults to hsync so that your data is as secure as possible in +a variety of situations (notably, unexpected power outages).

+ +

The downside is that performance tends to suffer because waiting for a sync to disk is a very +expensive operation. ACCUMULO-2842 introduces a new system property, tserver.wal.sync.method, +that lets users to change the HDFS sync implementation from hsync to hflush. Using hflush instead +of hsync may result in about a 30% increase in ingest performance.

+ +

For users upgrading from Hadoop-1 or Hadoop-0.20 releases, hflush is the equivalent of how +sync was implemented in these older versions of Hadoop and should give comparable performance.

+ +

Server-side mutation queue size

+ +

When users desire writes to be as durable as possible, using hsync, the ingest performance +of the system can be improved by increasing the tserver.mutation.queue.max property. The cost +of this change is that it will cause TabletServers to use additional memory per writer. In 1.5.1, +the value of this parameter defaulted to a conservative 256K, which resulted in sub-par ingest +performance.

+ +

1.5.2 and ACCUMULO-3018 increases this buffer to 1M which has a noticeable positive impact on +ingest performance with a minimal increase in TabletServer memory usage.

+ +

Notable Bug Fixes

+ +

Fixes MapReduce package name change

+ +

1.5.1 inadvertently included a change to RangeInputSplit which created an incompatibility +with 1.5.0. The original class has been restored to ensure that users accessing +the RangeInputSplit class do not have to alter their client code. See ACCUMULO-2586 for +more information

+ +

Add configurable maximum frame size to Apache Thrift proxy

+ +

The Thrift proxy server was subject to memory exhaustion, typically +due to bad input, where the server would attempt to allocate a very large +buffer and die in the process. ACCUMULO-2658 introduces a configuration +parameter, like ACCUMULO-2360, to prevent this error.

+ +

Offline tables can prevent tablet balancing

+ +

Before 1.5.2, when a table with many tablets was created, ingested into, and +taken offline, tablet balancing may have stoppped. This would happen if there +were tablet migrations for the table, because the migrations couldn’t occur. +The balancer will not run when there are outstanding migrations; therefore, a +system could become unbalanced. ACCUMULO-2694 introduces a fix to ensure +that offline tables do not block balancing and improves the server-side +logging.

+ +

MiniAccumuloCluster process management

+ +

MiniAccumuloCluster had a few issues which could cause deadlock or a method that +never returns. Most of these are related to management of the Accumulo processes +(ACCUMULO-2764, ACCUMULO-2985, and ACCUMULO-3055).

+ +

IteratorSettings not correctly serialized in RangeInputSplit

+ +

The Writable interface methods on the RangeInputSplit class accidentally omitted +calls to serialize the IteratorSettings configured for the Job. ACCUMULO-2962 +fixes the serialization and adds some additional tests.

+ +

Constraint violation causes hung scans

+ +

A failed bulk import transaction had the ability to create an infinitely retrying +loop due to a constraint violation. This directly prevents scans from completing, +but will also hang compactions. ACCUMULO-3096 fixes the issue so that the +constraint no longer hangs the entire system.

+ +

Documentation

+ +

The following documentation updates were made:

+ +

Testing

+ +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

OS	Hadoop	Nodes	ZooKeeper	HDFS High-Availability	Tests
Gentoo	Apache 2.6.0-SNAPSHOT	1	Apache 3.4.5	No	Unit and Functional Tests, ContinuousIngest w/ verification (1B entries)
CentOS 6	Apache 2.3.0	20	Apache 3.4.5	No	24/hr RandomWalk, 24/hr ContinuousIngest w/ verification w/ and w/o agitation (30B and 23B entries)

+ + + +

View all releases in the archive

+ +

+ + + + + +

+ + http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/c0655661/release/accumulo-1.5.3/index.html ---------------------------------------------------------------------- diff --git a/release/accumulo-1.5.3/index.html b/release/accumulo-1.5.3/index.html new file mode 100644 index 0000000..583fa4e --- /dev/null +++ b/release/accumulo-1.5.3/index.html @@ -0,0 +1,314 @@ + + + + + + + + + + + + +Apache Accumulo 1.5.3 + + + + + + + + + + + + +

+ + +

+ +

Apache Accumulo 1.5.3

+ +

25 Jun 2015

+ +

Apache Accumulo 1.5.3 is a bug-fix release for the 1.5 series. It is likely to be the last +1.5 release, with development shifting towards newer release lines. We recommend upgrading +to a newer version to continue to get bug fixes and new features.

+ +

In the context of Accumulo’s Semantic Versioning guidelines, +this is a “patch version”. This means that there should be no public API changes. Any +changes which were made were done in a backwards-compatible manner. Code that +runs against 1.5.2 should run against 1.5.3.

+ +

We’d like to thank all of the committers and contributors which had a part in +making this release, from code contributions to testing. Everyone’s efforts are +greatly appreciated.

+ +

Security Changes

+ +

SSLv3 disabled (POODLE)

+ +

Many Accumulo services were capable of enabling wire encryption using +SSL connectors. To be safe, ACCUMULO-3316 disables the problematic SSLv3 version by default which was +potentially susceptible to the man-in-the-middle attack. ACCUMULO-3317 also disables SSLv3 in the monitor, +so it will not accept SSLv3 client connections, when running it with https.

+ +

Notable Bug Fixes

+ +

SourceSwitchingIterator Deadlock

+ +

An instance of SourceSwitchingIterator, the Accumulo iterator which transparently manages +whether data for a tablet read from memory (the in-memory map) or disk (HDFS after a minor +compaction), was found deadlocked in a production system.

+ +

This deadlock prevented the scan and the minor compaction from ever successfully completing +without restarting the tablet server. ACCUMULO-3745 fixes the inconsistent synchronization +inside of the SourceSwitchingIterator to prevent this deadlock from happening in the future.

+ +

The only mitigation of this bug was to restart the tablet server that is deadlocked.

+ +

Table flush blocked indefinitely

+ +

While running the Accumulo RandomWalk distributed test, it was observed that all activity in +Accumulo had stopped and there was an offline Accumulo metadata table tablet. The system first +tried to flush a user tablet, but the metadata table was not online (likely due to the agitation +process which stops and starts Accumulo processes during the test). After this call, a call to +load the metadata tablet was queued but could not complete until the previous flush call. Thus, +a deadlock occurred.

+ +

This deadlock happened because the synchronous flush call could not complete before the load +tablet call completed, but the load tablet call couldn’t run because of connection caching we +perform in Accumulo’s RPC layer to reduce the quantity of sockets we need to create to send data. +ACCUMULO-3597 prevents this deadlock by forcing the use of a non-cached connection for the RPC +message requesting a metadata tablet to be loaded.

+ +

While this feature does result in additional network resources to be used, the concern is minimal +because the number of metadata tablets is typically very small with respect to the total number of +tablets in the system.

+ +

The only mitigation of this bug was to restart the tablet server that is hung.

+ +

RPC Connections not cached

+ +

It was observed that the underlying connection for invoking RPCs were not actually being cached, +despite it being requested that they should be cached. While this did not result in a noticed +performance impact, it was deficiency. ACCUMULO-3574 ensures that connections are cached when +it is requested that they are.

+ +

Deletes on Apache Thrift Proxy API ignored

+ +

A user noted that when trying to specify a delete using the Accumulo Thrift Proxy, the delete +was treated as an update. ACCUMULO-3474 fixes the Proxy server such that deletes are properly +respected as specified by the client.

+ +

Other Changes

+ +

Other changes for this version can be found in JIRA.

+ +

Testing

+ +

Each unit and functional test only runs on a single node, while the RandomWalk +and Continuous Ingest tests run on any number of nodes. Agitation refers to +randomly restarting Accumulo processes and Hadoop DataNode processes, and, in +HDFS High-Availability instances, forcing NameNode fail-over.

+ +

During testing, multiple Accumulo developers noticed some stability issues +with HDFS using Apache Hadoop 2.6.0 when restarting Accumulo processes and +HDFS datanodes. The developers investigated these issues as a part of the +normal release testing procedures, but were unable to find a definitive cause +of these failures. Users are encouraged to follow +ACCUMULO-2388 if they wish to follow any future developments. +One possible workaround is to increase the general.rpc.timeout in the +Accumulo configuration from 120s to 240s.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

OS	Hadoop	Nodes	ZooKeeper	HDFS High-Availability	Tests
Gentoo	2.6.0	1	3.4.5	No	Unit and Integration Tests
Centos 6.5	2.7.1	6	3.4.5	No	Continuous Ingest and Verify

+ + + +

View all releases in the archive

+ +

+ + + + + +

+ + http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/c0655661/release/accumulo-1.5.4/index.html ---------------------------------------------------------------------- diff --git a/release/accumulo-1.5.4/index.html b/release/accumulo-1.5.4/index.html new file mode 100644 index 0000000..7782556 --- /dev/null +++ b/release/accumulo-1.5.4/index.html @@ -0,0 +1,276 @@ + + + + + + + + + + + + +Apache Accumulo 1.5.4 + + + + + + + + + + + + +

+ + +

+ +

Apache Accumulo 1.5.4

+ +

21 Sep 2015

+ +

Apache Accumulo 1.5.4 is one more bug-fix release for the 1.5 series. Like 1.5.3 before it, this release contains a +very small changeset when considering the normal size of changes in a release.

+ +

This release contains no changes to the public API. As such, there are no concerns +for the compatibility of user code running against 1.5.3. All users are encourage to upgrade +immediately without concern of stability and compatibility.

+ +

A full list of changes is available via CHANGES.

+ +

We’d like to thank all of the committers and contributors which had a part in +making this release, from code contributions to testing. Everyone’s efforts are +greatly appreciated.

+ +

Correctness Bugs

+ +

Silent data-loss via bulk imported files

+ +

A user recently reported that a simple bulk-import application would occasionally lose some records. Through investigation, +it was found that when bulk imports into a table failed the initial assignment, the logic that automatically retries +the imports was incorrectly choosing the tablets to import the files into. ACCUMULO-3967 contains +more information on the cause and identification of the bug. The data-loss condition would only affect entire files. +If records from a file exist in Accumulo, it is still guaranteed that all records within that imported file were +successful.

+ +

As such, users who have bulk import applications using previous versions of Accumulo should verify that all of their +data was correctly ingested into Accumulo and immediately update to Accumulo 1.5.4.

+ +

Thanks to Edward Seidl for reporting this bug to us!

+ +

Server-side auditing changes

+ +

Thanks to James Mello for reporting and providing the fixes to the following server-side auditing issues.

+ +

Incorrect audit initialization

+ +

It was observed that the implementation used to audit user API requests on Accumulo server processes +was not being correctly initialized which caused audit messages to never be generated. This was rectified +in ACCUMULO-3939.

+ +

Missing audit implementations

+ +

It was also observed that some server-side API implementations did not include audit messages which resulted +in an incomplete historical picture on what operations a user might have invoked. The missing audits (and those +that were added) are described in ACCUMULO-3946.

+ +

Testing

+ +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

OS	Hadoop	Nodes	ZooKeeper	HDFS High-Availability	Tests
OSX	2.6.0	1	3.4.5	No	Unit and Functional Tests
Centos 6.5	2.7.1	6	3.4.5	No	Continuous Ingest and Verify (10B entries), Randomwalk (24hrs)

+ + + +

View all releases in the archive

+ +

+ + + + + +

+ +