Return-Path: X-Original-To: apmail-accumulo-commits-archive@www.apache.org Delivered-To: apmail-accumulo-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A51211016E for ; Fri, 28 Mar 2014 00:51:00 +0000 (UTC) Received: (qmail 11773 invoked by uid 500); 28 Mar 2014 00:50:55 -0000 Delivered-To: apmail-accumulo-commits-archive@accumulo.apache.org Received: (qmail 11662 invoked by uid 500); 28 Mar 2014 00:50:53 -0000 Mailing-List: contact commits-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@accumulo.apache.org Delivered-To: mailing list commits@accumulo.apache.org Received: (qmail 11649 invoked by uid 99); 28 Mar 2014 00:50:53 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Mar 2014 00:50:53 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id 1640D8ABBA8; Fri, 28 Mar 2014 00:50:53 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: ctubbsii@apache.org To: commits@accumulo.apache.org Date: Fri, 28 Mar 2014 00:50:53 -0000 Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: [1/6] ACCUMULO-1487, ACCUMULO-1491 Stop packaging docs for monitor Repository: accumulo Updated Branches: refs/heads/master 0721f8dca -> 5655a044e http://git-wip-us.apache.org/repos/asf/accumulo/blob/a20e19fc/server/monitor/src/main/resources/docs/isolation.html ---------------------------------------------------------------------- diff --git a/server/monitor/src/main/resources/docs/isolation.html b/server/monitor/src/main/resources/docs/isolation.html deleted file mode 100644 index d0e77cc..0000000 --- a/server/monitor/src/main/resources/docs/isolation.html +++ /dev/null @@ -1,39 +0,0 @@ - - - -Accumulo Isolation - - - - -

Apache Accumulo Documentation : Isolation

- -

Scanning

- -

Accumulo supports the ability to present an isolated view of rows when scanning. There are three possible ways that a row could change in accumulo : -

a mutation applied to a table -
iterators executed as part of a minor or major compaction -
bulk import of new files -

-Isolation guarantees that either all or none of the changes made by these operations on a row are seen. Use the IsolatedScanner to obtain an isolated view of an accumulo table. When using the regular scanner it is possible to see a non isolated view of a row. For example if a mutation modifies three columns, it is possible that you will only see two of those modifications. With the isolated scanner either all three of the changes are seen or none. For an example of this try running the InterferenceTest example. - -

At this time there is no client side isolation support for the BatchScanner. You may consider using the WholeRowIterator with the BatchScanner to achieve isolation though. This drawback of doing this is that entire rows are read into memory on the server side. If a row is too big, it may crash a tablet server. The IsolatedScanner buffers rows on the client side so a large row will not crash a tablet server. - -

Iterators

When writing server side iterators for accumulo isolation is something to be aware of. A scan time iterator in accumulo reads from a set of data sources. While an iterator is reading data it has an isolated view. However, after it returns a key/value it is possible that accumulo may switch data sources and re-seek the iterator. This is done so that resources may be reclaimed. When the user does not request isolation this can occur after any key is returned. When a user request isolation this will only occur after a new row is returned, in which case it will re-seek to the very beginning of the next possible row. http://git-wip-us.apache.org/repos/asf/accumulo/blob/a20e19fc/server/monitor/src/main/resources/docs/lgroups.html ---------------------------------------------------------------------- diff --git a/server/monitor/src/main/resources/docs/lgroups.html b/server/monitor/src/main/resources/docs/lgroups.html deleted file mode 100644 index 0012ffb..0000000 --- a/server/monitor/src/main/resources/docs/lgroups.html +++ /dev/null @@ -1,42 +0,0 @@ - - - -Accumulo Locality Groups - - - - -

Apache Accumulo Documentation : Locality Groups

- -

Accumulo supports locality groups similar to those described in the Big Table paper. Locality groups allow vertical partitioning of data by column family. This allows user to configure their tables such that scans over a subset of column families are much faster. The Accumulo locality group model has the following features. - -

There is a default locality group that holds all column families not in a declared locality group. -
No requirement to declare locality groups or column families at table creation. -
Can change locality group configuration on the fly. -

- - -

When the locality group configuration for a table is changed it has no effect on existing data. All minor and major compactions that occur after the change will organize data into the new locality group structure. As data is written into a table, it will cause minor and major compactions to occur. Over time this will result in all data being organized according to the new locality groups. If all data must be reorganized into the new locality groups immediately, this can be accomplished by forcing a full major compaction of the table. Use the compact command in the shell to accomplish this. - -

There are two ways to manipulate locality groups, via the shell or through the Java API. From the shell use the getgroups and setgroups commands. Through the API, TableOperations has the methods setLocalityGroups() and getLocalityGroups(). - -

To limit scans to a set of locality groups, use the fetchColumnFamily() function on Scanner or BatchScanner. From the shell use scan with the -c option. - - - http://git-wip-us.apache.org/repos/asf/accumulo/blob/a20e19fc/server/monitor/src/main/resources/docs/metrics.html ---------------------------------------------------------------------- diff --git a/server/monitor/src/main/resources/docs/metrics.html b/server/monitor/src/main/resources/docs/metrics.html deleted file mode 100644 index 00f0a5b..0000000 --- a/server/monitor/src/main/resources/docs/metrics.html +++ /dev/null @@ -1,182 +0,0 @@ - - - -Accumulo Metrics - - - - -

Apache Accumulo Documentation : Metrics

- -As of version 1.2, metrics for the Master, Tablet Servers, and Loggers are available. A new configuration file, accumulo-metrics.xml, is located in the conf directory and can -be modified to turn metrics collection on or off, and to enable file logging if desired. This file can be modified at runtime and the changes will be seen after a few seconds. -Except where specified all time values are in milliseconds. -

Master Metrics

JMX Object Name: org.apache.accumulo.server.metrics:type=MasterMetricsMBean,name= <current thread name>

- - - - - - - - - - - - -

Method Name	Description
public long getPingCount();	Number of pings to tablet servers
public long getPingAvgTime();	Average time for each ping
public long getPingMinTime();	Minimum time for each ping
public long getPingMaxTime();	Maximum time for each ping
public String getTServerWithHighestPingTime();	tablet server with highest ping
public void reset();	Resets all counters to zero

Logging Server Metrics

JMX Object Name: org.apache.accumulo.server.metrics:type=LogWriterMBean,name= <current thread name>

- - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Method Name	Description
public long getCloseCount();	Number of closed log files
public long getCloseAvgTime();	Average time to close a log file
public long getCloseMinTime();	Minimum time to close a log file
public long getCloseMaxTime();	Maximum time to close a log file
public long getCopyCount();	Number of log files copied
public long getCopyAvgTime();	Average time to copy a log file
public long getCopyMinTime();	Minimum time to copy a log file
public long getCopyMaxTime();	Maximum time to copy a log file
public long getCreateCount();	Number of log files created
public long getCreateMinTime();	Minimum time to create a log file
public long getCreateMaxTime();	Maximum time to create a log file
public long getCreateAvgTime();	Average time to create a log file
public long getLogAppendCount();	Number of times logs have been appended
public long getLogAppendMinTime();	Minimum time to append to a log file
public long getLogAppendMaxTime();	Maximum time to append to a log file
public long getLogAppendAvgTime();	Average time to append to a log file
public long getLogFlushCount();	Number of log file flushes
public long getLogFlushMinTime();	Minimum time to flush a log file
public long getLogFlushMaxTime();	Maximum time to flush a log file
public long getLogFlushAvgTime();	Average time to flush a log file
public long getLogExceptionCount();	Number of log exceptions
public void reset();	Resets all counters to zero

Tablet Server Metrics

JMX Object Name: org.apache.accumulo.server.metrics:type=TabletServerMBean,name= <current thread name>

- - - - - - - - - - - - - - - - - - - - - -

Method Name	Description
public int getOnlineCount();	Number of tablets online
public int getOpeningCount();	Number of tablets that are being opened
public int getUnopenedCount();	Number or unopened tablets
public int getMajorCompactions();	Number of Major Compactions currently running
public int getMajorCompactionsQueued();	Number of Major Compactions yet to run
public int getMinorCompactions();	Number of Minor Compactions currently running
public int getMinorCompactionsQueued();	Number of Minor Compactions yet to run
public int getShutdownStage();	Current stage in the shutdown process
public long getEntries();	Number of entries in all the tablets
public long getEntriesInMemory();	Number of entries in memory on all tablet servers
public long getQueries();	Number of queries currently running on all the tablet servers
public long getIngest();	Number of entries currently being ingested on all the tablet servers
public long getTotalMinorCompactions();	Number of Minor Compactions completed
public double getHoldTime();	Number of seconds that ingest is waiting for memory to be freed on tablet servers
public String getName();	Address of the master

Tablet Server Minor Compaction Metrics

JMX Object Name: org.apache.accumulo.server.metrics:type=TabletServerMinCMetricsMBean,name= <current thread name>

- - - - - - - - - - - - - - - -

Method Name	Description
public long getMinorCompactionCount();	Number of completed Minor Compactions on all tablet servers
public long getMinorCompactionAvgTime();	Average time to complete Minor Compaction
public long getMinorCompactionMinTime();	Minimum time to complete Minor Compaction
public long getMinorCompactionMaxTime();	Maximum time to complete Minor Compaction
public long getMinorCompactionQueueCount();	Number of Minor Compactions yet to be run
public long getMinorCompactionQueueAvgTime();	Average time Minor Compaction is in the queue
public long getMinorCompactionQueueMinTime();	Minimum time Minor Compaction is in the queue
public long getMinorCompactionQueueMaxTime();	Maximum time Minor Compaction is in the queue
public void reset();	Resets all counters to zero

Tablet Server Scan Metrics

JMX Object Name: org.apache.accumulo.server.metrics:type=TabletServerScanMetricsMBean,name= <current thread name>

- - - - - - - - - - - - - - - -

Method Name	Description
public long getScanCount();	Number of scans completed
public long getScanAvgTime();	Average time for scan operation
public long getScanMinTime();	Minimum time for scan operation
public long getScanMaxTime();	Maximum time for scan operation
public long getResultCount();	Number of scans that returned a result
public long getResultAvgSize();	Average size of scan result
public long getResultMinSize();	Minimum size of scan result
public long getResultMaxSize();	Maximum size of scan result
public void reset();	Resets all counters to zero

Tablet Server Update Metrics

JMX Object Name: org.apache.accumulo.server.metrics:type=TabletServerUpdateMetricsMBean,name= <current thread name>

- - - - - - - - - - - - - - - - - - - - - - - - - -

Method Name	Description
public long getPermissionErrorCount();	Number of permission errors
public long getUnknownTabletErrorCount();	Number of unknown tablet errors
public long getMutationArrayAvgSize();	Average size of mutation array
public long getMutationArrayMinSize();	Minimum size of mutation array
public long getMutationArrayMaxSize();	Maximum size of mutation array
public long getCommitPrepCount();	Number of commit preparations
public long getCommitPrepMinTime();	Minimum time for commit preparation
public long getCommitPrepMaxTime();	Maximum time for commit preparatation
public long getCommitPrepAvgTime();	Average time for commit preparation
public long getConstraintViolationCount();	Number of constraint violations
public long getWALogWriteCount();	Number of writes to the Write Ahead Log
public long getWALogWriteMinTime();	Minimum time of a write to the Write Ahead Log
public long getWALogWriteMaxTime();	Maximum time of a write to the Write Ahead Log
public long getWALogWriteAvgTime();	Average time of a write to the Write Ahead Log
public long getCommitCount();	Number of commits
public long getCommitMinTime();	Minimum time for a commit
public long getCommitMaxTime();	Maximum time for a commit
public long getCommitAvgTime();	Average time for a commit
public void reset();	Resets all counters to zero

Thrift Server Metrics

JMX Object Name: org.apache.accumulo.server.metrics:type=ThriftMetricsMBean,name= <thread name>

- - - - - - - - - - - - - - - -

Method Name	Description
public long getIdleCount();	Number of times the Thrift server has been idle
public long getIdleMinTime();	Minimum amount of time the Thrift server has been idle
public long getIdleMaxTime();	Maximum amount of time the Thrift server has been idle
public long getIdleAvgTime();	Average time the Thrift server has been idle
public long getExecutionCount();	Number of calls processed by the Thrift server
public long getExecutionMinTime();	Minimum amount of time executing method
public long getExecutionMaxTime();	Maximum amount of time executing method
public long getExecutionAvgTime();	Average time executing methods
public void reset();	Resets all counters to zero

- - http://git-wip-us.apache.org/repos/asf/accumulo/blob/a20e19fc/server/monitor/src/main/resources/docs/timestamps.html ---------------------------------------------------------------------- diff --git a/server/monitor/src/main/resources/docs/timestamps.html b/server/monitor/src/main/resources/docs/timestamps.html deleted file mode 100644 index 52290c7..0000000 --- a/server/monitor/src/main/resources/docs/timestamps.html +++ /dev/null @@ -1,160 +0,0 @@ - - - -Accumulo Timestamps - - - - -

Apache Accumulo Documentation : Timestamps

- -

Everything inserted into accumulo has a timestamp. If the user does not -set it, then the system will set the timestamp. The timestamp is the last -thing accumulo sorts on. So when two keys have the same row, column family, -column qualifier, and column visibility then the timestamp of the two keys is -compared. - -

Timestamps are sorted in descending order, so the most recent data comes -first. When a table is created in accumulo, by default it has a versioning -iterator that only shows the most recent. In the example below two identical -things are inserted. The scan after that only shows the most recent version. -However when the versioning iterator configuration is changed, then both are -seen. When data is inserted with a lower timestamp than existing data, it will -fall behind the existing data and may not be seen depending on the versioning -settings. This is why the insert made with a timestamp of 500 is not seen in -the scan below. - -

-root@ac12> createtable foo
-root@ac12 foo> 
-root@ac12 foo> 
-root@ac12 foo> insert r1 cf1 cq1 value1                                   
-root@ac12 foo> insert r1 cf1 cq1 value2
-root@ac12 foo> scan -st
-r1 cf1:cq1 [] 1279906856203    value2
-root@ac12 foo> config -t foo -f iterator                                  
----------+---------------------------------------------+-----------------------------------------------------------------------------------------------------
-SCOPE    | NAME                                        | VALUE
----------+---------------------------------------------+-----------------------------------------------------------------------------------------------------
-table    | table.iterator.majc.vers .................. | 20,org.apache.accumulo.core.iterators.VersioningIterator
-table    | table.iterator.majc.vers.opt.maxVersions .. | 1
-table    | table.iterator.minc.vers .................. | 20,org.apache.accumulo.core.iterators.VersioningIterator
-table    | table.iterator.minc.vers.opt.maxVersions .. | 1
-table    | table.iterator.scan.vers .................. | 20,org.apache.accumulo.core.iterators.VersioningIterator
-table    | table.iterator.scan.vers.opt.maxVersions .. | 1
----------+---------------------------------------------+-----------------------------------------------------------------------------------------------------
-root@ac12 foo> config -t foo -s table.iterator.scan.vers.opt.maxVersions=3
-root@ac12 foo> config -t foo -s table.iterator.minc.vers.opt.maxVersions=3
-root@ac12 foo> config -t foo -s table.iterator.majc.vers.opt.maxVersions=3
-root@ac12 foo> scan -st
-r1 cf1:cq1 [] 1279906856203    value2
-r1 cf1:cq1 [] 1279906853170    value1
-root@ac12 foo> insert -t 600 r1 cf1 cq1 value3
-root@ac12 foo> insert -t 500 r1 cf1 cq1 value4
-root@ac12 foo> scan -st
-r1 cf1:cq1 [] 1279906856203    value2
-r1 cf1:cq1 [] 1279906853170    value1
-r1 cf1:cq1 [] 600    value3
-root@ac12 foo>
-
-

- -

Deletes are special keys in accumulo that get sorted along will all the other -data. When a delete key is inserted, accumulo will not show anything that has -a timestamp less than or equal to the delete key. In the example below an -insert is made with timestamp 5 and then a delete is inserted with timestamp 3. -The scan after that show that the delete marker does not hide the key. However -when a delete is inserted with timestamp 5, then nothing can be seen. Once a -delete marker is inserted, it is there until a full major compaction occurs. -That is why the insert made after the delete can not be seen. The insert after -the flush and compact commands can be seen because the delete marker is gone. -The flush forced a minor compaction and compact forced a full major compaction. - -

-root@ac12> createtable bar
-root@ac12 bar> insert -t 5 r1 cf1 cq1 val1
-root@ac12 bar> scan -st
-r1 cf1:cq1 [] 5    val1
-root@ac12 bar> delete -t 3 r1 cf1 cq1     
-root@ac12 bar> scan
-r1 cf1:cq1 []    val1
-root@ac12 bar> scan -st
-r1 cf1:cq1 [] 5    val1
-root@ac12 bar> delete -t 5 r1 cf1 cq1
-root@ac12 bar> scan -st              
-root@ac12 bar> insert -t 5 r1 cf1 cq1 val2
-root@ac12 bar> scan -st
-root@ac12 bar> flush -t bar
-23 14:01:36,587 [shell.Shell] INFO : Flush of table bar initiated...
-root@ac12 bar> compact -t bar
-23 14:02:00,042 [shell.Shell] INFO : Compaction of table bar scheduled for 20100723140200EDT
-root@ac12 bar> insert -t 5 r1 cf1 cq1 val1
-root@ac12 bar> scan
-r1 cf1:cq1 []    val1
-

- -

If two inserts are made into accumulo with the same row, column, and -timestamp, then the behavior is non-deterministic. - -

Accumulo 1.2 introduces the concept of logical time. This ensures that -timestamps set by accumulo always move forward. There have been many problems -caused by tablet servers with different system times. In the case where a -tablet servers time is in the future, tablets hosted on that tablet server and -then migrated will have future timestamps in their data. This can cause newer -keys to fall behind existing keys, which can result in seeing older data or not -seeing data if a new key falls behind on old delete. Logical time prevents -this by ensuring that accumulo set time stamps never go backwards, on a per -tablet basis. So if a tablet servers time is a year in the future, then any -tablet hosted there will generate timestamps a year in the future even when -later hosted on a server with correct time. Logical time can be configured on a -per table basis to either set time in millis or to use a per tablet counter. -The per tablet counter gives unique one up time stamps on a per mutation -basis. When using time in millis, if two things arrive within the same -millisecond then both receive the same timestamp. - -

The example below shows a table created using a per tablet counter for -timestamps. Two inserts are made, the first gets timestamp 0 the second 1. -After that the table is split into two tablets and two more inserts are made. -These inserts get the same timestamp because they are made on different -tablets. When the original tablet is split into two, the two child tablets -inherit the next timestamp of their parent and start from there. So do not -expect this configuration to offer unique timestamps across a table. Its only -purpose is to uniquely order events within a tablet. - -

-root@ac12 foo> createtable -tl logical
-root@ac12 logical> insert 000892 person name "John Doe"
-root@ac12 logical> insert 003042 person name "Jane Doe"
-root@ac12 logical> scan -st
-000892 person:name [] 0    John Doe
-003042 person:name [] 1    Jane Doe
-root@ac12 logical>
-root@ac12 logical> addsplits -t logical 002000
-root@ac12 logical> insert 003042 person address "123 Somewhere"
-root@ac12 logical> insert 000892 person address "123 Nowhere"  
-root@ac12 logical> scan -st
-000892 person:address [] 2    123 Nowhere
-000892 person:name [] 0    John Doe
-003042 person:address [] 2    123 Somewhere
-003042 person:name [] 1    Jane Doe
-root@ac12 logical> 
- 
-

- - -