X-Mailer: ASF-Git Admin Mailer Subject: [42/50] [abbrv] accumulo-website git commit: Jekyll build from gh-pages:a71ede8 archived-at: Tue, 22 Nov 2016 19:41:07 -0000 http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/59a6d65d/features/index.html ---------------------------------------------------------------------- diff --git a/features/index.html b/features/index.html new file mode 100644 index 0000000..2857088 --- /dev/null +++ b/features/index.html @@ -0,0 +1,649 @@ + + + + + + + + + + + + +Features + + + + + + + + + + + + +

+ + +

+ +

Features

+ + +

Table Design and Configuration
Integrity/Availability
Performance
Testing
Client API
Extensible Behaviors
General Administration
Internal Data Management
On-demand Data Management
Screenshots

+ +

Table Design and Configuration

+ +

Iterators

+ +

A server-side programming mechanism to encode functions such as filtering and +aggregation within the data management steps (scopes where data is read from or +written to disk) that happen in the tablet server.

+ +

Cell labels

+ +

An additional portion of the Key that sorts after the column qualifier and +before the timestamp. It is called column visibility and enables expressive +cell-level access control. Authorizations are passed with each query to control +what data is returned to the user. The column visibilities are boolean AND and +OR combinations of arbitrary strings (such as “(A&B)|C”) and authorizations +are sets of strings (such as {C,D}).

+ +

Constraints

+ +

Configurable conditions under which writes to a table will be rejected. +Constraints are written in Java and configurable on a per table basis.

+ +

Sharding

+ +

Through the use of specialized iterators, Accumulo can be a parallel sharded +document store. For example wikipedia could be stored and searched for +documents containing certain words.

+ +

Large Rows

+ +

When reading rows, there is no requirement that an entire row fits into memory.

+ +

Namespaces

+ +

In version 1.6.0, the concept of table “namespaces” was created to allow for logical +grouping and configuration of Accumulo tables. By default, tables are created in a +default namespace which is the empty string to preserve the feel for how tables operate +in previous versions. One application of table namespaces is placing the Accumulo root +and metadata table in an “accumulo” namespace to denote that these tables are used +internally by Accumulo.

+ +

Volume support

+ +

Accumulo 1.6.0 migrated away from configuration of HDFS by using a single HDFS host and +directory, to a collection of HDFS URIs (host and path) which allows Accumulo to operate +over multiple disjoint HDFS instances. This allows Accumulo to scale beyond the limits +of a single namenode. When used in conjunction with HDFS federation, multiple namenodes +can share a pool of datanodes.

+ +

Integrity/Availability

+ +

Master fail over

+ +

Multiple masters can be configured. Zookeeper locks are used to determine +which master is active. The remaining masters simply wait for the current +master to lose its lock. Current master state is held in the metadata table +and Zookeeper (see FATE).

+ +

Logical time

+ +

A mechanism to ensure that server set times never go backwards, even when time +across the cluster is incorrect. This ensures that updates and deletes are not +lost. If a tablet is served on machine with time a year in the future, then the +tablet will continue to issue new timestamps a year in the future even when it +moves to another server. In this case the timestamps preserve ordering, but +lose their meaning. In addition to logical time, Accumulo has master +authoritative time. The master averages the time of all of the tablet servers +and sends this back to the tablet servers. Tablet servers use this information +to adjust the timestamps they issue. So logical time ensures ordering is +always correct and master authoritative time tries to ensure that timestamps +are meaningful.

+ +

Logical Time for bulk import

+ +

Logical time as described above works with streaming (batch) ingest, where the +tablet server assigns the timestamp. Logical time is also important for bulk +imported data, for which the client code may be choosing a timestamp. Accumulo +uses specialized system iterators to lazily set times in a bulk imported +file. This mechanism guarantees that times set by unsynchronized multi-node +applications (such as those running on MapReduce) will maintain some semblance +of causal ordering. This mitigates the problem of the time being wrong on the +system that created the file for bulk import. These times are not set when the +file is imported, but whenever it is read by scans or compactions. At import, a +time is obtained and always used by the specialized system iterator to set that +time.

+ +

FATE

+ +

Fault Tolerant Executor. A framework for executing operations in a fault +tolerant manner. In the previous release, if the master process died in the +middle of creating a table it could leave the system in an inconsistent state. +With this new framework, if the master dies in the middle of create table it +will continue on restart. Also the client requesting the create table operation +will never know anything happened. The framework serializes work in Zookeeper +before attempting to do the work. Clients start a FATE transaction, seed it +with work, and then wait for it to finish. Most table operations are executed +using this framework. Persistent, per table, read-write locks are created in +Zookeeper to synchronize operations across process faults.

+ +

Scalable master

+ +

Stores its metadata in an Accumulo table and Zookeeper.

+ +

Isolation

+ +

Scans will not see data inserted into a row after the scan of that row begins.

+ +

Performance

+ +

Relative encoding

+ +

If consecutive keys have identical portions (row, colf, colq, or colvis), there +is a flag to indicate that a portion is the same as that of the previous key. +This is applied when keys are stored on disk and when transferred over the +network. Starting with 1.5, prefix erasure is supported. When its cost +effective, prefixes repeated in subsequent key fields are not repeated.

+ +

Native In-Memory Map

+ +

By default data written is stored outside of Java managed memory into a C++ STL +map of maps. It maps rows to columns to values. This hierarchical structure +improves performance of inserting a mutation with multiple column values in a +single row. A custom STL allocator is used to avoid the global malloc lock and +memory fragmentation.

+ +

Scan pipeline

+ +

A long running Accumulo scan will eventually cause multiple threads to start. +One server thread to read data from disk, one server thread to serialize and +send data, and one client thread to deserialize and read data. When pipelining +kicks in, it substantially increases scan speed while maintaining key order. It +does not activate for short scans.

+ +

Caching

+ +

Recently scanned data is cached into memory. There are separate caches for +indexes and data. Caching can be turned on and off for individual tables.

+ +

Multi-level RFile Index

+ +

RFiles store an index of the last key in each block. For large files the index +can become quite large. When the index is large a lot of memory is consumed and +files take a long time to open. To avoid this problem, RFiles have a +multi-level index tree. Index blocks can point to other index blocks or data +blocks. The entire index never has to be resident, even when the file is +written. When an index block exceeds the configurable size threshold, its +written out between data blocks. The size of index blocks is configurable on a +per table basis.

+ +

Binary search in RFile blocks

+ +

RFile uses its index to locate a block of key values. Once it reaches a block +it performs a linear scan to find a key on interest. Accumulo will generate +indexes of cached blocks in an adaptive manner. Accumulo indexes the blocks +that are read most frequently. When a block is read a few times, a small index +is generated. As a block is read more, larger indexes are generated making +future seeks faster. This strategy allows Accumulo to dynamically respond to +read patterns without precomputing block indexes when RFiles are written.

+ +

Testing

+ +

Mock

+ +

The Accumulo client API has a mock implementation that is useful writing unit +test against Accumulo. Mock Accumulo is in memory and in process.

+ +

Mini Accumulo Cluster

+ +

Mini Accumulo cluster is a set of utility code that makes it easy to spin up +a local Accumulo instance running against the local filesystem. Mini Accumulo +is slower than Mock Accumulo, but its behavior mirrors a real Accumulo +instance more closely.

+ +

Accumulo Maven Plugin

+ +

Using the Mini Accumulo Cluster in unit and integration tests is a great way for +developers to test their applications against Accumulo in an environment that is +much closer to physical deployments than Mock Accumulo provided. Accumulo 1.6.0 also +introduced a maven-accumulo-plugin which +can be used to start a Mini Accumulo Cluster instance as a part of the Maven +lifecycle that your application tests can use.

+ +

Functional Test

+ +

Small, system-level tests of basic Accumulo features run in a test harness, +external to the build and unit-tests. These tests start a complete Accumulo +instance, and require Hadoop and Zookeeper to be running. They attempt to +simulate the basic functions of Accumulo, as well as common failure conditions, +such as lost disks, killed processes, and read-only file systems.

+ +

Scale Test

+ +

A test suite that verifies data is not lost at scale. This test runs many +ingest clients that continually create linked lists containing 25 million +nodes. At some point the clients are stopped and a map reduce job is run to +ensure no linked list has a hole. A hole indicates data was lost by Accumulo. +The Agitator can be run in conjunction with this test to randomly kill tablet +servers. This test suite has uncovered many obscure data loss bugs. This test +also helps find bugs that impact uptime and stability when run for days or +weeks.

+ +

Random Walk Test

+ +

A test suite that looks for unexpected system states that may emerge in +plausible real-world applications. Application components are defined as test +nodes (such as create table, insert data, scan data, delete table, etc.), and +are programmed as Java classes that implement a specified interface. The nodes +are connected together in a graph specified in an XML document. Many processes +independently and concurrently execute a random walk of the test graphs. Some +of the test graphs have a concept of correctness and can verify data over time. +Other tests have no concept of data correctness and have the simple goal of +crashing Accumulo. Many obscure bugs have been uncovered by this testing +framework and subsequently corrected.

+ +

Client API

+ +

Batch Scanner

+ +

Takes a list of Ranges, batches them to the appropriate tablet servers, and +returns data as it is received (i.e. not in sorted order).

+ +

Batch Writer

+ +

Clients buffer writes in memory before sending them in batches to the +appropriate tablet servers.

+ +

Bulk Import

+ +

Instead of writing individual mutations to Accumulo, entire files of sorted key +value pairs can be imported. These files are moved into the Accumulo directory +and referenced by Accumulo. This feature is useful for ingesting a large amount +of data. This method of ingest usually offers higher throughput at the cost of +higher latency for data availability for scans. Usually the data is sorted +using map reduce and then bulk imported. This method of ingest also allows for +flexibility in resource allocation. The nodes running map reduce to sort data +could be different from the Accumulo nodes.

+ +

Map Reduce

+ +

Accumulo can be a source and/or sink for map reduce jobs.

+ +

Apache Thrift Proxy

+ +

The Accumulo client code contains a lot of complexity. For example, the +client code locates tablets, retries in the case of failures, and supports +concurrent reading and writing. All of this is written in Java. The thrift +proxy wraps the Accumulo client API with thrift, making this API easily +available to other languages like Python, Ruby, C++, etc.

+ +

Conditional Mutations

+ +

In version 1.6.0, Accumulo introduced ConditionalMutations +which allow users to perform efficient, atomic read-modify-write operations on rows. Conditions can +be defined using on equality checks of the values in a column or the absence of a column. For more +information on using this feature, users can reference the Javadoc for ConditionalMutation and +ConditionalWriter.

+ +

Lexicoders

+ +

Common boilerplate code that exists when interacting with Accumulo is the conversion +of Java objects to lexicographically sorted bytes, e.g. ensure that the byte representation +of the number 9 sorts before the byte representation of the number 11. Version 1.6.0 introduced +Lexicoders which have numerous implementations that support for efficient translation from common +Java primitives to byte arrays and vice versa. These classes can greatly reduce the burden in +re-implementing common programming mistakes in encoding.

+ +

Extensible Behaviors

+ +

Pluggable balancer

+ +

Users can provide a balancer plugin that decides how to distribute tablets +across a table. These plugins can be provided on a per table basis. This is +useful for ensuring a particular table’s tablets are placed optimally for +tables with special query needs. The default balancer randomly spreads each +table’s tablets across the cluster. It takes into account where a tablet was +previously hosted to leverage locality. When a tablet splits, the default +balancer moves one child to another tablet server. The assumption here is that +splitting tablets are being actively written to, so this keeps write load evenly +spread.

+ +

Pluggable memory manager

+ +

The plugin that decides when and what tablets to minor compact is configurable. +The default plugin compacts the largest tablet when memory is over a certain +threshold. It varies the threshold over time depending on minor compaction +speed. It flushes tablets that are not written to for a configurable time +period.

+ +

Pluggable logger assignment strategy

+ +

The plugin that decided which loggers should be assigned to which tablet +servers is configurable.

+ +

Pluggable compaction strategy

+ +

The plugin that decides which files should be chosen for major compaction is now +configurable. Given certain workloads, it may be known that once data is written, +it is very unlikely that more data will be written to it, and thus paying the penalty +to re-write a large file can be avoided. Implementations of this compaction strategy +can be used to optimize the data that compactions will write.

+ +

General Administration

+ +

Monitor page

+ +

A simple web server provides basic information about the system health and +performance. It displays table sizes, ingest and query statistics, server +load, and last-update information. It also allows the user to view recent +diagnostic logs and traces.

+ +

Tracing

+ +

It can be difficult to determine why some operations are taking longer than +expected. For example, you may be looking up items with very low latency, but +sometimes the lookups take much longer. Determining the cause of the delay is +difficult because the system is distributed, and the typical lookup is fast. +Accumulo has been instrumented to record the time that various operations take +when tracing is turned on. The fact that tracing is enabled follows all the +requests made on behalf of the user throughout the distributed infrastructure +of Accumulo, and across all threads of execution.

+ +

Online reconfiguration

+ +

System and per table configuration is stored in Zookeeper. Many, but not all, +configuration changes take effect while Accumulo is running. Some do not take +effect until server processes are restarted.

+ +

Table renaming

+ +

Tables can be renamed easily because Accumulo uses internal table IDs and +stores mappings between names and IDs in Zookeeper.

+ +

Internal Data Management

+ +

Locality groups

+ +

Groups columns within a single file. There is a default locality group so that +not all columns need be specified. The locality groups can be restructured +while the table is online and the changes will take effect on the next +compaction. A tablet can have files with different locality group +configurations. In this case scans may be suboptimal, but correct, until +compactions rewrite all files. After reconfiguring locality groups, a user can +force a table to compact in order to write all data into the new locality +groups. Alternatively, the change could be allowed to happen over time as +writes to the table cause compactions to happen.

+ +

Smart compaction algorithm

+ +

It is inefficient to merge small files with large files. Accumulo merges files +only if all files are larger than a configurable ratio (default is 3) +multiplied by the largest file size. If this cannot be done with all the +files, the largest file is removed from consideration, and the remaining files +are considered for compaction. This is done until there are no files to merge.

+ +

Merging Minor Compaction

+ +

When a max number of files per tablet is reached, minor compactions will merge +data from the in-memory map with the smallest file instead of creating new +files. This throttles ingest. In previous releases new files were just created +even if major compactions were falling behind and the number of tablets per file +was growing. Without this feature, ingest performance can roughly continue at a +constant rate, even as scan performance decreases because tablets have too many +files.

+ +

Loading jars using VFS

+ +

User written iterators are a useful way to manipulate data in data in Accumulo. +Before 1.5., users had to copy their iterators to each tablet server. Starting +with 1.5 Accumulo can load iterators from HDFS using Apache commons VFS.

+ +

Encryption

+ +

Still a work in progress, Accumulo 1.6.0 introduced encryption at rest (RFiles +and WriteAheadLogs) and wire encryption (Thrift over SSL) to provide enhance the +level of security that Accumulo provides. It is still a work in progress because +the intermediate files created by Accumulo when recovering from a TabletServer +failure are not encrypted.

+ +

On-demand Data Management

+ +

Compactions

+ +

Ability to force tablets to compact to one file. Even tablets with one file are +compacted. This is useful for improving query performance, permanently +applying iterators, or using a new locality group configuration. One example +of using iterators is applying a filtering iterator to remove data from a +table. Additionally, users can initiate a compaction with iterators only applied to +that compaction event.

+ +

Split points

+ +

Arbitrary split points can be added to an online table at any point in time. +This is useful for increasing ingest performance on a new table. It can also be +used to accommodate new data patterns in an existing table.

+ +

Tablet Merging

+ +

Tablet merging is a new feature. Merging of tablets can be requested in the +shell; Accumulo does not merge tablets automatically. If desired, the METADATA +tablets can be merged.

+ +

Table Cloning

+ +

Allows users to quickly create a new table that references an existing table’s +data and copies its configuration. A cloned table and its source table can be +mutated independently. Testing was the motivating reason behind this new +feature. For example to test a new filtering iterator, clone the table, add the +filter to the clone, and force a major compaction.

+ +

Import/Export Table

+ +

An offline tables metadata and files can easily be copied to another cluster and +imported.

+ +

Compact Range

+ +

Compact each tablet that falls within a row range down to a single file.

+ +

Delete Range

+ +

Added an operation to efficiently delete a range of rows from a table. Tablets +that fall completely within a range are simply dropped. Tablets overlapping the +beginning and end of the range are split, compacted, and then merged.

+ +

Screenshots

+ +

+The following is a screenshot of the Apache Accumulo™ monitor overview web page. Accumulo was running on an eleven node cluster. The continuous ingest test suite was running to generate load. Ten continuous ingest clients were running. +

+ +

+The following screenshot shows the monitor master page. This page gives information about individual tables within accumulo. Continuous ingest is running against the table ci. +

+ +

The data generated by the continuous ingest test suite looks like the following. The data is random, but forms many giant linked list. Each ingest client continually generates linked list containing 25 million nodes. In the shell session below a few scans are issued to start following the linked list that the first node in the table is part of.

+ +

root@test15 ci> scan 
+000000000135fbbe 074c:569c []    2db1de18-cd37-407d-b060-481a0a214c90:000000002b2bcfe1:2e15cb9f62fd22ab:fe6f84c7
+000000000d76dcc9 4f62:0647 []    65f401e8-4f93-498c-bffa-ecc21a8645fd:000000000e7c2bbc:6fad98d023b8f146:a2c4d59f
+0000000024e07be4 5e56:221d []    5283b4bd-4f18-4af9-a242-c8053b095e04:000000001e1ad441:12de1d82f4a7bf2f:ae231a6a
+000000003e13cb50 1d0a:415c []    5944afab-8407-4391-9994-91b313e545dd:0000000002d61c91:4890a8885e54b714:d542fafa
+00000000634d0aaa 310d:1e58 []    2db1de18-cd37-407d-b060-481a0a214c90:000000000c9e8018:4111345bc9afe2bb:4cbfd774
+000000007d378414 08d2:0066 []    15ab8578-faae-4ebb-9827-26728be78476:0000000052de5aac:06202f405c60726d:8ad4911a
+000000008c3615d8 3868:5f66 []    8f47568c-a383-4910-83c9-36b75ac66440:0000000067faf572:750d567f3dd0c5bf:5ea938fe
+0000000090be2a77 7f32:2ca4 []    8f47568c-a383-4910-83c9-36b75ac66440:000000005912be62:47ac0c7bb89013f1:dff13db9
+00000000a7ca8e58 360f:5763 []    a9f5e129-e260-4a0e-a763-c6e3ef7b3eb6:0000000030bff255:37c8226225c804ba:665f261f
+00000000f1000713 5079:46bd []    30af7f48-15ae-4ccb-81c9-4662ad74be92:0000000014d408e3:56a9dec63aed4d8f:0f2a44ec
+000000017e3ca989 56ac:062b []    30af7f48-15ae-4ccb-81c9-4662ad74be92:000000000a8926a1:0bd54c769871aa78:b8e61d77
+--------------------------------------- hit any key to continue or 'q' to quit -----------------------------------------
+root@test15 ci> scan -b 2e15cb9f62fd22ab -e 2e15cb9f62fd22ab
+2e15cb9f62fd22ab 1fc1:1486 []    2db1de18-cd37-407d-b060-481a0a214c90:000000002b1c8da1:53a9e4ba97e4a6e0:4a8bd624
+root@test15 ci> scan -b 53a9e4ba97e4a6e0 -e 53a9e4ba97e4a6e0
+53a9e4ba97e4a6e0 37f3:33a2 []    2db1de18-cd37-407d-b060-481a0a214c90:000000002b0d4b61:1c16a4ae1bef9f1f:f03c869b
+root@test15 ci> scan -b 1c16a4ae1bef9f1f -e 1c16a4ae1bef9f1f
+1c16a4ae1bef9f1f 07c8:7cd3 []    2db1de18-cd37-407d-b060-481a0a214c90:000000002afe0921:2e3ec09a8d2fdf52:8b82be48
+

+ + +

+ + + + + +

+ + http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/59a6d65d/feed.xml ---------------------------------------------------------------------- diff --git a/feed.xml b/feed.xml index 5e439ab..52cd0d1 100644 --- a/feed.xml +++ b/feed.xml @@ -6,8 +6,8 @@ https://accumulo.apache.org/ - Thu, 17 Nov 2016 08:58:59 -0500 - Thu, 17 Nov 2016 08:58:59 -0500 + Mon, 21 Nov 2016 11:23:52 -0500 + Mon, 21 Nov 2016 11:23:52 -0500 Jekyll v3.3.0 @@ -848,6 +848,14 @@ release contains changes from more than 150 issues, comprised of bug-fixes, performance improvements, build quality improvements, and more. See <a href="https://issues.apache.org/jira/browse/ACCUMULO/fixforversion/12329940">JIRA</a> for a complete list. +Below are resources for this release: + +<ul> + <li><a href="/1.7/accumulo_user_manual.html">User Manual</a></li> + <li><a href="/1.7/apidocs">Javadocs</a></li> + <li><a href="/1.7/examples">Examples</a></li> +</ul> + Users of any previous 1.7.x release are strongly encouraged to update as soon as possible to benefit from the improvements with very little concern in change of underlying functionality. Users of 1.6 or earlier that are seeking to @@ -1023,6 +1031,14 @@ release contains changes from 55 issues, comprised of bug-fixes, performance improvements, build quality improvements, and more. See <a href="https://issues.apache.org/jira/browse/ACCUMULO/fixforversion/12333674">JIRA</a> for a complete list. +Below are resources for this release: + +<ul> + <li><a href="/1.6/accumulo_user_manual.html">User Manual</a></li> + <li><a href="/1.6/apidocs">Javadocs</a></li> + <li><a href="/1.6/examples">Examples</a></li> +</ul> + Users of any previous 1.6.x release are strongly encouraged to update as soon as possible to benefit from the improvements with very little concern in change of underlying functionality. Users of 1.4 or 1.5 that are seeking to upgrade to 1.6 @@ -1163,6 +1179,14 @@ This release contains changes from 21 issues, comprised of bug-fixes, performance improvements and better test cases. See <a href="https://issues.apache.org/jira/browse/ACCUMULO/fixforversion/12332840">JIRA</a> for a complete list. +Below are resources for this release: + +<ul> + <li><a href="/1.6/accumulo_user_manual.html">User Manual</a></li> + <li><a href="/1.6/apidocs">Javadocs</a></li> + <li><a href="/1.6/examples">Examples</a></li> +</ul> + Users of any previous 1.6.x release are strongly encouraged to update as soon as possible to benefit from the improvements with very little concern in change of underlying functionality. Users of 1.4 or 1.5 that are seeking to upgrade @@ -1245,6 +1269,14 @@ HDFS High-Availability instances, forcing NameNode failover. Apache Accumulo 1.5.4 is one more bug-fix release for the 1.5 series. Like 1.5.3 before it, this release contains a very small changeset when considering the normal size of changes in a release. +Below are resources for this release: + +<ul> + <li><a href="/1.5/accumulo_user_manual.html">User Manual</a></li> + <li><a href="/1.5/apidocs">Javadocs</a></li> + <li><a href="/1.5/examples">Examples</a></li> +</ul> + This release contains no changes to the <a href="https://github.com/apache/accumulo/blob/1.7.0/README.md#api">public API</a>. As such, there are no concerns for the compatibility of user code running against 1.5.3. All users are encourage to upgrade immediately without concern of stability and compatibility. @@ -1340,8 +1372,17 @@ HDFS High-Availability instances, forcing NameNode fail-over. Apache Accumulo 1.5.2 is a maintenance release on the 1.5 version branch. This release contains changes from over 100 issues, comprised of bug fixes (client side and server side), new test cases, and updated Hadoop support -contributed by over 30 different contributors and committers. -As this is a maintenance release, Apache Accumulo 1.5.2 has no client API +contributed by over 30 different contributors and committers. + +Below are resources for this release: + +<ul> + <li><a href="/1.5/accumulo_user_manual.html">User Manual</a></li> + <li><a href="/1.5/apidocs">Javadocs</a></li> + <li><a href="/1.5/examples">Examples</a></li> +</ul> + +As this is a maintenance release, Apache Accumulo 1.5.2 has no client API incompatibilities over Apache Accumulo 1.5.0 and 1.5.1 and requires no manual upgrade process. Users of 1.5.0 or 1.5.1 are strongly encouraged to update as soon as possible to benefit from the improvements. http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/59a6d65d/get_involved.html ---------------------------------------------------------------------- diff --git a/get_involved.html b/get_involved.html index 26d8a56..a329ac2 100644 --- a/get_involved.html +++ b/get_involved.html @@ -93,13 +93,13 @@

@@ -163,7 +144,7 @@
Get Involved

You don’t need to be a software developer to contribute to -Apache Accumulo™. To be successful this project +Apache Accumulo. To be successful this project requires a huge range of different skills, levels of involvement and degrees of technical expertise. So, if you want to get involved in Apache Accumulo, there is almost certainly a role for you.
@@ -198,9 +179,9 @@ as easy as possible for people to get involved.
Apache Accumulo developers and community members hang out in the #accumulo channel on irc.freenode.net.
-
Developer’s Guide
+
Contributor Guide
-
See the developer’s guide for recommended practices in interacting with our source code. +
See the contributor guide for recommended practices in interacting with our source code. Take a look at Accumulo’s Ohloh page for an analysis of the code base.

Community Interaction
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/59a6d65d/git.html ---------------------------------------------------------------------- diff --git a/git.html b/git.html index 4dda960..d324e3a 100644 --- a/git.html +++ b/git.html @@ -1,705 +1,10 @@ - - - + - - - - - - - -Git - - - - - - - - - -
-
-
- - -
-
-
- Download
- - Releases -
 -
 1.8.0 (Latest)
 -
 1.7.2
 -
 1.6.6
 -
 Archive
 -
 -
- - Documentation -
 -
 User Manuals
 -
 Javadocs
 -
 Examples
 -
 Features
 -
 Screenshots
 -
 Papers & Presentations
 -
 Glossary
 -
 -
- - Community -
 -
 Get Involved
 -
 Mailing Lists
 -
 People
 -
 News Archive
 -
 Community Projects
 -
 Thanks
 -
 Governance
 -
 -
- - Development -
 -
 Issue Tracker
 -
 Pull Requests
 -
 Jenkins Builds
 -
 TravisCI Builds
 -
 -
 Guides
 -
 Source & Guide
 -
 Git Workflow
 -
 Versioning
 -
 Contrib Projects
 -
 Review Board
 -
 Making Releases
 -
 Verifying Releases
 -
 -
-
- - Apache Software Foundation -
 -
 Apache Homepage
 -
 License
 -
 Sponsorship
 -
 Security
 -
 Thanks
 -
 Code of Conduct
 -
 -
-
-
-
- - -
-
-
- -
- Visit the official site at: https://accumulo.apache.org -
-
- -
Git
- -
Git is an open source, distributed version control system -which has become very popular in large, complicated software projects due to -its efficient handling of multiple, simultaneously and independently developed -branches of source code.\
- -
Workflow Background
- -
Likely the most contested subject matter regarding switching an active team -from one SCM tool to another is a shift in the development paradigm.
- -
Some background, the common case, as is present with this team, is that -developers coming from a Subversion history are very light on merging being a -critical consideration on how to perform development. Because merging in -Subversion is typically of no consequence to the complete view of history, not -to mention that Subversion allows “merging” of specific revisions instead of -sub-trees. As such, a transition to Git typically requires a shift in mindset -in which a fix is not arbitrarily applied to trunk (whatever the main -development is called), but the fix should be applied to the earliest, non -end-of-life (EOL) version of the application.
- -
For example, say there is a hypothetical application which has just released -version-3 of their software and have shifted their development to a version-4 -WIP. Version-2 is still supported, while version-1 was just EOL’ed. Each -version still has a branch. A bug was just found which affects all released -versions of this application. In Subversion, considering the history in the -repository, it is of no consequence where the change is initially applied, -because a Subversion merge is capable of merging it to any other version. -History does not suffer because Subversion doesn’t have the capacities to -accurately track the change across multiple branches. In Git, to maintain a -non-duplicative history, it is imperative that the developer choose the correct -branch to fix the bug in. In this case, since version-1 is EOL’ed, version-2, -version-3 and the WIP version-4 are candidates. The earliest version is -obviously version-2; so, the change should be applied in version-2, merged to -version-3 and then the version-4 WIP.
- -
The importance of this example is making a best-attempt to preserve history -when using Git. While Git does have commands like cherry-pick which allow the -contents of a commit to be applied from one branch to another which are not -candidates to merge without conflict (which is typically the case when merging -a higher version into a lower version), this results in a duplication of that -commit in history when the two trees are eventually merged together. While Git -is smart enough to not attempt to re-apply the changes, history still contains -the blemish.
- -
The purpose of this extravagant example is to outline, in the trivial case, how -the workflow decided upon for development is very important and has direct -impact on the efficacy of the advanced commands bundled with Git.
- -
Proposed Workflow
- -
This is a summary of what has been agreed upon by vocal committers/PMC members -on dev@accumulo.apache.org. Enumeration of -every possible situation out of the scope of this document. This document -intends to lay the ground work to define the correct course of action regardless -of circumstances. Some concrete examples will be provided to ensure the -explanation is clear.
- -
-
-
Active development is performed concurrently over no less than two versions -of Apache Accumulo at any one point in time. As such, the workflow must -provide guidance on how and where changes should be made which apply to -multiple versions and how to ensure that such changes are contained in all -applicable versions.
-
-
-
Releases are considered extremely onerous and time-consuming so no emphasis -is placed on rapid iterations or development-cycles.
-
-
-
New features typically have lengthy development cycles in which more than -one developer contributes to the creation of the new feature from planning, -to implementation, to scale testing. Mechanisms/guidance should be provided -for multiple developers to teach contribute to a remote-shared resource.
-
-
-
The repository should be the irrefutable, sole source of information -regarding the development of Apache Accumulo, and, as such, should have -best-efforts given in creating a clean, readable history without any single -entity having to control all write access and changes (a proxy). In other -words, the developers are the ones responsible for ensuring that previous -releases are preserved to meet ASF policy, for not rewriting any public -history of code not yet officially released (also ASF policy relevant) and -for a best-effort to be given to avoid duplicative commits appearing in - history created by the application of multiple revisions which have - different identifying attributes but the same contents (git-rebase and - git-cherry-pick).
-
-
- -
The implementation
- -
Contributors
- -
Use the following steps, original derived from Apache Kafka’s simple -contributor -workflow.
- -
To be specific, let’s consider a contributor wanting to work on a fix for the -Jira issue ACCUMULO-12345 that affects 1.5.0 release.
- -
-
-
Ensure you configured Git with your information
- -
git config --global user.name 'My Name' && git config --global user.email - 'myname@mydomain.com'
-
-
-
Clone the Accumulo repository:
- -
git clone https://git-wip-us.apache.org/repos/asf/accumulo.git accumulo
-
-
-
or update your local copy:
- -
git fetch && git fetch --tags
-
-
-
For the given issue you intend to work on, choose the ‘lowest’ fixVersion -and create a branch for yourself to work in. This example is against the next release of 1.5
- -
git checkout -b ACCUMULO-12345-my-work origin/1.5
-
-
-
Make commits as you see fit as you fix the issue, referencing the issue name -in the commit message:
- -
git commit -av
- -
Please include the ticket number at the beginning of the log message, and - in the first line, as it’s easier to parse quickly. For example:
- -
ACCUMULO-2428 throw exception when rename fails after compaction
- -
Consider following the git log message format described in - Zach Holman’s talk - (specifically slides 78-98, beginning at 15:20 into the video). Essentially, - leave a short descriptive message in the first line, skip a line, and write - more detailed stuff there, if you need to. For example:
- -
ACCUMULO-2194 Add delay for randomwalk Security teardown
- -
If two Security randomwalk tests run back-to-back, the second test may see that the - table user still exists even though it was removed when the first test was torn down. - This can happen if the user drop does not propagate through Zookeeper quickly enough. - This commit adds a delay to the end of the Security test to give ZK some time.
-
-
-
Assuming others are developing against the version you also are, as you -work, or before you create your patch, rebase your branch against the remote -to lift your changes to the top of your branch. The branch specified here should be the same one you used in step 4.
- -
git pull --rebase origin 1.5
-
-
-
At this point, you can create a patch file from the upstream branch to -attach to the ACCUMULO-12345 Jira issue. The branch specified here should be teh same one you used in step 4.
- -
git format-patch --stdout origin/1.5 > ACCUMULO-12345.patch
-
-
- -
An alternative to creating a patch is submitting a request to pull your changes -from some repository, e.g. Github. Please include the repository and branch -name merge in the notice to the Jira issue, e.g.
- -
repo=git://github.com/<username>/accumulo.git branch=ACCUMULO-12345 -
-
- -
A second alternative is to use Github’s “Pull Requests” feature against the -Apache Accumulo account. Notifications of -new pull-requests from Github should automatically be sent to -dev@accumulo.apache.org.
- -
Ignoring specifics, contributors should be sure to make their changes against -the earlier version in which the fix is intended, git-rebase‘ing their -changes against the upstream branch so as to keep their changes co-located and -free of unnecessary merges.
- -
Developers
- -
Primary Development
- -
Primary development should take place in master which is to contain the most -recent, un-released version of Apache Accumulo. Branches exist for minor releases -for each previously released major version.
- -
Using long-lived branches that track a major release line simplifies management and -release practices. Developers are encouraged to make branches for their own purposes, -for large features, release candidates or whatever else is deemed useful.
- -
Reviewing contributor changes
- -
It is always the responsibility of committers to determine that a patch is -intended and able to be contributed. From the -new committer’s guide: -“Please take care to ensure that patches are original works which have been -clearly contributed to the ASF in public. In the case of any doubt (or when a -contribution with a more complex history is presented) please consult your -project PMC before committing it.”
- -
Extra diligence may be necessary when code is contributed via a pull request. -Committers should verify that the contributor intended to submit the code as a -Contribution under the Apache License. -When pulling the code, committers should also verify that the commits pulled match the -list of commits sent to the Accumulo dev list in the pull request.
- -
Patches
- -
Developers should use the following steps to apply patches from -contributors:
- -
-
-
Checkout the branch for the major version which the patch is intended:
- -
git checkout 1.5
-
-
-
Verify the changes introduced by the patch:
- -
git apply --stat ACCUMULO-12345.patch
-
-
-
Verify that the patch applies cleanly:
- -
git apply --check ACCUMULO-12345.patch
-
-
-
If all is well, apply the patch:
- -
git am --signoff < ACCUMULO-12345.patch
-
-
-
When finished, push the changes:
- -
git push origin 1.5
-
-
-
Merge where appropriate:
- -
git checkout master && git merge 1.5
-
-
- -
Github Pull-Requests
- -
If the contributor submits a repository and branch to pull -from, the steps are even easier:
- -
-
-
Add their repository as a remote to your repository
- -
git remote add some_name ${repository}
-
-
-
Fetch the refs from the given repository
- -
git fetch ${repository}
-
-
-
Merge in the given branch to your local branch
- -
git merge some_name/${branch}
-
-
-
Delete the remote:
- -
git remote remove some_name
-
-
- -
If the branch doesn’t fast-forward merge, you likely want to inform the -contributor to update their branch to avoid the conflict resolution and merge -commit. See the Git -manual -for more information on merging. When merging a pull-request, it’s best to not -include a signoff on the commit(s) as it changes the final commit ID in the -Accumulo repository. This also has the negative of not automatically closing -the Pull-Request when the changes are made public.
- -
Feature Branches
- -
Ease in creating and using feature branches is a desirable merit which Git -provides with little work. Feature branches are a great way in which developers -to work on multiple, long-running features concurrently, without worry of -mixing code ready for public-review and code needing more internal work. -Additionally, this creates an easily consumable series of commits in which -other developers can track changes, and see how the feature itself evolved.
- -
To prevent developers’ feature branches from colliding with one another, it was -suggested to impose a “hierarchy” in which shared feature branches are prefixed -with <apache_id>/ACCUMULO-<issue#>[-description].
- -
-
-
Create a branch off of master.
- -
git checkout <apache_id>/ACCUMULO-<issue#> master
-
-
-
Create the feature, commiting early and often to appropriately outline the -“story” behind the feature and it’s implementation.
-
-
-
As long as you have not collaborating with others, git-rebase your feature -branch against upstream changes in master
- -
git fetch && git rebase origin/master
-
-
-
If you are actively collaborating with others, you should be nice and not -change their history. Use git-merge instead.
- -
git fetch && git merge origin/master
-
-
-
Continue steps 2 through 4 until the feature is complete.
-
-
-
Depending on the nature, duration and history of the changes in your feature -branch, you can choose to:
- -
-
-
‘Simple’ Merge:
- -
git checkout master && git merge <apache_id>/ACCUMULO-<issue#>
-
-
-
Rebase and merge – keeps all feature-branch commits -co-located:
- -
git fetch && git rebase origin/master && git checkout master && git merge <apache_id>/ACCUMULO-<issue#>
-
-
-
Merge with squash – feature branch history is a mess, just make one commit -with the lump-sum of your feature branch changes:
- -
git checkout master && git merge --squash <apache_id>/ACCUMULO-<issue#>
-
-
-
-
- -
Changes which affect multiple versions (a.k.a. merging)
- -
Merging can be a very confusing topic for users switching to Git, but it can be -summarized fairly easily.
- -
-
-
Precondition: choose the right branch to start! (lowest, applicable version -for the change)
-
-
-
Get your changes fixed for that earliest version.
-
-
-
Switch to the next highest version which future minor versions will be -released (non-EOL major release series).
-
-
-
git-merge the branch from #1 into the current.
-
-
-
In the face of conflicts, use options from git-merge to help you.
- -
-
git checkout new-version && git merge --stat old-version
-
git checkout new-version && git merge --no-commit old-version
-
-
-
-
Treat your current branch as the branch from #2, and repeat from #2.
-
-
- -
When merging changes across major releases, there is always the possibility of -changes which are applicable/necessary in one release, but not in any future -releases, changes which are different in implementation due to API changes, or -any number of other cases. Whatever the actual case is, the developer who made -the first set of changes (you) is the one responsible for performing the merge -through the rest of the active versions. Even when the merge may results in a -zero-length change in content, this is incredibly important to record as you -are the one who knows that this zero-length change in content is correct!
- -
Release Management
- -
Releases, although not a day to day task, have their own unique steps which are -to be followed. Releases can be categorized in to minor and major releases.
- -
A minor release
- -
A minor release is some set of changes z' on top of a version x.y.z. -Typically, z' is simply z + 1, e.g. given a release named ‘1.6.0’, and the -next minor release is ‘1.6.1’. These changes for z' should not break any -client code which works against z and should absolutely not change the public -API.
- -
By convention, the branch containing the changes z' should be named -x.y (where the changes for z' are commits since x.y.z. The steps to take are as follows:
- -
-
Prepare the release candidate. Release -Guide, Maven -Instructions
-
Create a branch for the release candidate from the x.y branch, -named something like x.y.z'-RCN.
-
Test and Vote
-
Create a GPG-signed tag with the correct final name: x.y.z'
-
Push a delete of the remote branch x.y.z'-RCN
-
- -
This process is not firm and should not be viewed as requirements for making a release. -The release manager is encouraged to make use branches/tags in whichever way is best.
- -
A major release
- -
A major release is a release in which multiple new features are introduced -and/or the public API are modified. The major release y', even when the -client API is not modified, will typically contain many new features and -functionality above the last release series y. A major release also resets -the z value back to 0.
- -
The steps to create a new major release are very similar to a minor release:
- -
-
Prepare the release candidate. reference release instructions
-
Create a tag of the release candidate from the x.y branch, -named something like x.y.0-RCN.
-
Test and Vote
-
Create a GPG-signed tag with the correct final name: x.y.0
-
Push a delete of the remote branch x.y.0-RCN
-
- -
The infrastructure
- -
This section deals with the changes that must be requested through INFRA. As -with any substantial INFRA request, the VOTE and result from the mailing should -be referenced so INFRA knows that the request has been acknowledged. Likely, a -PMC member should be the one to submit the request.
- -
Repositories
- -
I believe that we will need multiple repositories to best align ourselves with -how we currently track “Accumulo” projects. The repositories follow:
- -
-
-
The main source tree. This will track the standard trunk, branches, tags -structure from Subversion for Apache Accumulo.
-
-
-
One repository for every project in -contrib: Accumulo-BSP, -Instamo Archetype, and the Wikisearch project. Each of these -are considered disjoint from one another, and the main source tree, so they -each deserve their own repository.
-
-
- -
Given the list of repositories that currently exist on the ASF -site and a brief search over INFRA -tickets, multiple repositories for a single Apache project is not an issue. -Having this list when making the initial ticket will likely reduce the amount -of work necessary in opening multiple INFRA tickets.
- -
Mirroring
- -
It should be noted in the INFRA requst that each repository will also need to -be configured to properly mirror to the ASF Github -account to provide the same functionality with current have via the git+svn -mirror. This should be noted in the INFRA request. Same change needs to be -applied for the Apache hosted mirror’ing.
- -
Mailing lists
- -
It should be noted in the INFRA request that commit messages should be sent to -commits@accumulo.apache.org. The subject -can be decided on using the provided -variables.
- -
Examples
- -
For the sake of clarity, some examples of common situations are included below.
- -
Releasing 1.6.0
- -
-
-
Branch from master to 1.6
- -
git checkout master && git branch 1.6
-
-
-
Tag 1.6.0-RC1 from the just created 1.6 branch
- -
git tag 1.6.0-RC1 1.6
-
-
-
Test, vote, etc. and tag from 1.6.0-RC1
- -
git -s tag 1.6.0 1.6.0-RC1
-
-
-
Delete the RC tag, if desired.
- -
git tag -d 1.6.0-RC1 && git push --delete origin 1.6.0-RC1
-
-
-
Ensure master contains all features and fixes from 1.6.0
- -
git checkout master && git merge 1.6
-
-
-
Update the project version in master to 1.7.0-SNAPSHOT
-
-
- - -
- - -
- -
- -
Copyright © 2011-2016 The Apache Software Foundation. Licensed under the Apache License, Version 2.0.
- -
- - -
-
-
- +Redirecting… + + +
Redirecting…
+Click here if you are not redirected. +

Features

Table Design and Configuration

Iterators

Cell labels

Constraints

Sharding

Large Rows

Namespaces

Volume support

Integrity/Availability

Master fail over

Logical time

Logical Time for bulk import

FATE

Scalable master

Isolation

Performance

Relative encoding

Native In-Memory Map

Scan pipeline

Caching

Multi-level RFile Index

Binary search in RFile blocks

Testing

Mock

Mini Accumulo Cluster

Accumulo Maven Plugin

Functional Test

Scale Test

Random Walk Test

Client API

Batch Scanner

Batch Writer

Bulk Import

Map Reduce

Apache Thrift Proxy

Conditional Mutations

Lexicoders

Extensible Behaviors

Pluggable balancer

Pluggable memory manager

Pluggable logger assignment strategy

Pluggable compaction strategy

General Administration

Monitor page

Tracing

Online reconfiguration

Table renaming

Internal Data Management

Locality groups

Smart compaction algorithm

Merging Minor Compaction

Loading jars using VFS

Encryption

On-demand Data Management

Compactions

Split points

Tablet Merging

Table Cloning

Import/Export Table

Compact Range

Delete Range

Screenshots

Get Involved

Developer’s Guide

Contributor Guide

Community Interaction

Git

Workflow Background

Proposed Workflow

The implementation

Contributors

Developers

Primary Development

Reviewing contributor changes

Patches

Github Pull-Requests

Feature Branches

Changes which affect multiple versions (a.k.a. merging)

Release Management