accumulo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ktur...@apache.org
Subject svn commit: r1351452 - in /accumulo/branches/1.4.1: ./ src/ src/core/ src/server/ src/server/src/main/java/org/apache/accumulo/server/test/continuous/ test/system/continuous/
Date Mon, 18 Jun 2012 18:29:00 GMT
Author: kturner
Date: Mon Jun 18 18:29:00 2012
New Revision: 1351452

URL: http://svn.apache.org/viewvc?rev=1351452&view=rev
Log:
ACCUMULO-637 ACCUMULO-614 merge changes from 1.4 branch

Modified:
    accumulo/branches/1.4.1/   (props changed)
    accumulo/branches/1.4.1/CHANGES
    accumulo/branches/1.4.1/src/   (props changed)
    accumulo/branches/1.4.1/src/core/   (props changed)
    accumulo/branches/1.4.1/src/server/   (props changed)
    accumulo/branches/1.4.1/src/server/src/main/java/org/apache/accumulo/server/test/continuous/ContinuousIngest.java
    accumulo/branches/1.4.1/test/system/continuous/continuous-env.sh.example
    accumulo/branches/1.4.1/test/system/continuous/start-batchwalkers.sh
    accumulo/branches/1.4.1/test/system/continuous/start-ingest.sh
    accumulo/branches/1.4.1/test/system/continuous/start-scanners.sh
    accumulo/branches/1.4.1/test/system/continuous/start-stats.sh
    accumulo/branches/1.4.1/test/system/continuous/start-walkers.sh

Propchange: accumulo/branches/1.4.1/
------------------------------------------------------------------------------
  Merged /accumulo/trunk:r1350779
  Merged /accumulo/branches/1.4:r1346082-1346088,1346090-1346833,1346835-1349971,1349973-1351449

Modified: accumulo/branches/1.4.1/CHANGES
URL: http://svn.apache.org/viewvc/accumulo/branches/1.4.1/CHANGES?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/CHANGES (original)
+++ accumulo/branches/1.4.1/CHANGES Mon Jun 18 18:29:00 2012
@@ -1,3 +1,62 @@
+Release Notes - Apache Accumulo - Version 1.4.1
+
+** Bug
+    * [ACCUMULO-355] - functional tests should run under a non-hadoop user
+    * [ACCUMULO-398] - Table tablets not evenly spread.
+    * [ACCUMULO-428] - loggers are not closed when high hold times are detected
+    * [ACCUMULO-449] - Failed log copy is not restarted
+    * [ACCUMULO-468] - low-memory warning message is incorrect
+    * [ACCUMULO-487] - Batch Scanner can get stuck when external thread closes scanner
+    * [ACCUMULO-488] - InputFormats' RecordReaders should call Context.progress
+    * [ACCUMULO-496] - Cloudtrace.thrift needs to have it's namespace updated to the new
package
+    * [ACCUMULO-499] - Garbage collector should call Tracer.offNoFlush instead of Tracer.off
+    * [ACCUMULO-505] - "% of Used DFS" is overflowing on a large cluster
+    * [ACCUMULO-509] - default walog copy/sort uses replication of 1
+    * [ACCUMULO-512] - DEFAULT_MAX_LATENCY in AccumuloOutputFormat wrong units
+    * [ACCUMULO-516] - Column family search with sparse files is painfully long
+    * [ACCUMULO-520] - assigned tablets are not considered offline
+    * [ACCUMULO-521] - examples functional test is using the wrong class name
+    * [ACCUMULO-533] - make system iterators thread-safer
+    * [ACCUMULO-542] - with a large root tablet, tablet servers will get stuck loading other
metadata tablets
+    * [ACCUMULO-561] - Monitor scan mb/s does not show batch scan activity
+    * [ACCUMULO-562] - Multi term grep in shell fails
+    * [ACCUMULO-566] - monitor should display zero tablet servers as red, even if there's
only one tablet server configured
+    * [ACCUMULO-570] - dirlist example README is a little inconsistent
+    * [ACCUMULO-585] - Complain loudly when env file is missing
+    * [ACCUMULO-591] - Need to clear tablet location cache before computing input splits
in input format
+    * [ACCUMULO-592] - We lost the security policy example
+    * [ACCUMULO-601] - client fails to resolve master hostname when not using fully qualified
domain name
+    * [ACCUMULO-615] - accumulo runs fine on Hadoop 1.0
+    * [ACCUMULO-633] - FirstEntryInRowIterator is broken and has no test
+
+** Improvement
+    * [ACCUMULO-337] - Make logging an example file
+    * [ACCUMULO-364] - Security Policy file should be an example
+    * [ACCUMULO-399] - need a randomwalk node that checks tablet balance
+    * [ACCUMULO-459] - Accumulo init should fail fast
+    * [ACCUMULO-471] - allow wikisearch ingest to run on uncompressed input
+    * [ACCUMULO-480] - config.html should be more clear with regards to "needs restart"
+    * [ACCUMULO-489] - Input Format puts Base64 encoded passwords in Configuration, which
is world readable
+    * [ACCUMULO-494] - Document relationship between max walogs and max tserver memory
+    * [ACCUMULO-502] - Document what replaces the deprecated PerColumnIteratorConfig class.
+    * [ACCUMULO-503] - start-here does unecessary SSHs
+    * [ACCUMULO-510] - Rename scanner threads to identify table being scanned
+    * [ACCUMULO-550] - Colocate rfile index entries within file
+    * [ACCUMULO-554] - monitor should display red when time-since last report is higher than
a few minutes
+    * [ACCUMULO-573] - Fix grammar in shell commands
+    * [ACCUMULO-574] - Document reseek of iterators
+    * [ACCUMULO-637] - Make entries written configurable for continuous ingest
+
+** New Feature
+    * [ACCUMULO-49] - optionally monitor swappiness on every server
+    * [ACCUMULO-404] - Support running on-top of Kerberos-enabled HDFS
+    * [ACCUMULO-492] - Provide method for gathering system stats to API
+
+** Task
+    * [ACCUMULO-491] - Manual has UTF8 characters
+    * [ACCUMULO-546] - User manual inconsistencies
+
+
 Release notes for Apache Accumulo 1.4.0
 
 Major features added prior to Incubation

Propchange: accumulo/branches/1.4.1/src/
------------------------------------------------------------------------------
  Merged /accumulo/trunk/src:r1350779
  Merged /accumulo/branches/1.4/src:r1346082-1346088,1346090-1346833,1346835-1349971,1349973-1351449
  Merged /accumulo/trunk:r1350779

Propchange: accumulo/branches/1.4.1/src/core/
------------------------------------------------------------------------------
  Merged /accumulo/trunk/src/core:r1350779
  Merged /accumulo/trunk/core:r1350779
  Merged /accumulo/branches/1.4/src/core:r1346082-1346088,1346090-1346833,1346835-1349971,1349973-1351449

Propchange: accumulo/branches/1.4.1/src/server/
------------------------------------------------------------------------------
  Merged /accumulo/trunk/src/server:r1350779
  Merged /accumulo/branches/1.4/src/server:r1346082-1346088,1346090-1346833,1346835-1349971,1349973-1351449
  Merged /accumulo/trunk/server:r1350779

Modified: accumulo/branches/1.4.1/src/server/src/main/java/org/apache/accumulo/server/test/continuous/ContinuousIngest.java
URL: http://svn.apache.org/viewvc/accumulo/branches/1.4.1/src/server/src/main/java/org/apache/accumulo/server/test/continuous/ContinuousIngest.java?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/src/server/src/main/java/org/apache/accumulo/server/test/continuous/ContinuousIngest.java
(original)
+++ accumulo/branches/1.4.1/src/server/src/main/java/org/apache/accumulo/server/test/continuous/ContinuousIngest.java
Mon Jun 18 18:29:00 2012
@@ -32,6 +32,7 @@ import org.apache.accumulo.core.client.B
 import org.apache.accumulo.core.client.Connector;
 import org.apache.accumulo.core.client.Instance;
 import org.apache.accumulo.core.client.MutationsRejectedException;
+import org.apache.accumulo.core.client.TableExistsException;
 import org.apache.accumulo.core.client.ZooKeeperInstance;
 import org.apache.accumulo.core.data.Mutation;
 import org.apache.accumulo.core.data.Value;
@@ -67,11 +68,11 @@ public class ContinuousIngest {
     
     args = processOptions(args);
     
-    if (args.length != 13) {
+    if (args.length != 14) {
       throw new IllegalArgumentException(
           "usage : "
               + ContinuousIngest.class.getName()
-              + " [--debug <debug log>] <instance name> <zookeepers> <user>
<pass> <table> <min> <max> <max colf> <max colq> <max
mem> <max latency> <max threads> <enable checksum>");
+              + " [--debug <debug log>] <instance name> <zookeepers> <user>
<pass> <table> <num> <min> <max> <max colf> <max colq>
<max mem> <max latency> <max threads> <enable checksum>");
     }
     
     if (debugLog != null) {
@@ -89,16 +90,17 @@ public class ContinuousIngest {
     
     String table = args[4];
     
-    long min = Long.parseLong(args[5]);
-    long max = Long.parseLong(args[6]);
-    short maxColF = Short.parseShort(args[7]);
-    short maxColQ = Short.parseShort(args[8]);
-    
-    long maxMemory = Long.parseLong(args[9]);
-    long maxLatency = Integer.parseInt(args[10]);
-    int maxWriteThreads = Integer.parseInt(args[11]);
+    long num = Long.parseLong(args[5]);
+    long min = Long.parseLong(args[6]);
+    long max = Long.parseLong(args[7]);
+    short maxColF = Short.parseShort(args[8]);
+    short maxColQ = Short.parseShort(args[9]);
+    
+    long maxMemory = Long.parseLong(args[10]);
+    long maxLatency = Integer.parseInt(args[11]);
+    int maxWriteThreads = Integer.parseInt(args[12]);
     
-    boolean checksum = Boolean.parseBoolean(args[12]);
+    boolean checksum = Boolean.parseBoolean(args[13]);
     
     if (min < 0 || max < 0 || max <= min) {
       throw new IllegalArgumentException("bad min and max");
@@ -109,6 +111,11 @@ public class ContinuousIngest {
     String path = ZooUtil.getRoot(instance) + Constants.ZTRACERS;
     Tracer.getInstance().addReceiver(new ZooSpanClient(zooKeepers, path, localhost, "cingest",
1000));
     
+    if (!conn.tableOperations().exists(table))
+      try {
+        conn.tableOperations().create(table);
+      } catch (TableExistsException tee) {}
+
     BatchWriter bw = conn.createBatchWriter(table, maxMemory, maxLatency, maxWriteThreads);
     bw = Trace.wrapAll(bw, new CountSampler(1024));
     
@@ -133,7 +140,7 @@ public class ContinuousIngest {
     
     long lastFlushTime = System.currentTimeMillis();
     
-    while (true) {
+    out: while (true) {
       // generate first set of nodes
       for (int index = 0; index < flushInterval; index++) {
         long rowLong = genLong(min, max, r);
@@ -152,6 +159,8 @@ public class ContinuousIngest {
       }
       
       lastFlushTime = flush(bw, count, flushInterval, lastFlushTime);
+      if (count >= num)
+        break out;
       
       // generate subsequent sets of nodes that link to previous set of nodes
       for (int depth = 1; depth < maxDepth; depth++) {
@@ -165,6 +174,8 @@ public class ContinuousIngest {
         }
         
         lastFlushTime = flush(bw, count, flushInterval, lastFlushTime);
+        if (count >= num)
+          break out;
       }
       
       // create one big linked list, this makes all of the first inserts
@@ -175,7 +186,11 @@ public class ContinuousIngest {
         bw.addMutation(m);
       }
       lastFlushTime = flush(bw, count, flushInterval, lastFlushTime);
+      if (count >= num)
+        break out;
     }
+    
+    bw.close();
   }
   
   private static long flush(BatchWriter bw, long count, final int flushInterval, long lastFlushTime)
throws MutationsRejectedException {

Modified: accumulo/branches/1.4.1/test/system/continuous/continuous-env.sh.example
URL: http://svn.apache.org/viewvc/accumulo/branches/1.4.1/test/system/continuous/continuous-env.sh.example?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/test/system/continuous/continuous-env.sh.example (original)
+++ accumulo/branches/1.4.1/test/system/continuous/continuous-env.sh.example Mon Jun 18 18:29:00
2012
@@ -32,6 +32,9 @@ DEBUG_WALKER=off
 DEBUG_BATCH_WALKER=off
 DEBUG_SCANNER=off
 
+#the number of entries each client should write
+NUM=9223372036854775807
+
 #the minimum random row to generate
 MIN=0
 

Modified: accumulo/branches/1.4.1/test/system/continuous/start-batchwalkers.sh
URL: http://svn.apache.org/viewvc/accumulo/branches/1.4.1/test/system/continuous/start-batchwalkers.sh?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/test/system/continuous/start-batchwalkers.sh (original)
+++ accumulo/branches/1.4.1/test/system/continuous/start-batchwalkers.sh Mon Jun 18 18:29:00
2012
@@ -24,5 +24,5 @@ if [ "$DEBUG_BATCH_WALKER" = "on" ] ; th
 	DEBUG_OPT="--debug $CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_batch_walk.log";
 fi
 
-pssh -h batch_walkers.txt "nohup $ACCUMULO_HOME/bin/accumulo org.apache.accumulo.server.test.continuous.ContinuousBatchWalker
$DEBUG_OPT $INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $MIN $MAX $BATCH_WALKER_SLEEP $BATCH_WALKER_BATCH_SIZE
$BATCH_WALKER_THREADS >$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_batch_walk.out
2>$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_batch_walk.err &" < /dev/null
+pssh -h batch_walkers.txt "mkdir -p $CONTINUOUS_LOG_DIR; nohup $ACCUMULO_HOME/bin/accumulo
org.apache.accumulo.server.test.continuous.ContinuousBatchWalker $DEBUG_OPT $INSTANCE_NAME
$ZOO_KEEPERS $USER $PASS $TABLE $MIN $MAX $BATCH_WALKER_SLEEP $BATCH_WALKER_BATCH_SIZE $BATCH_WALKER_THREADS
>$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_batch_walk.out 2>$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_batch_walk.err &" < /dev/null
 

Modified: accumulo/branches/1.4.1/test/system/continuous/start-ingest.sh
URL: http://svn.apache.org/viewvc/accumulo/branches/1.4.1/test/system/continuous/start-ingest.sh?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/test/system/continuous/start-ingest.sh (original)
+++ accumulo/branches/1.4.1/test/system/continuous/start-ingest.sh Mon Jun 18 18:29:00 2012
@@ -24,5 +24,10 @@ if [ "$DEBUG_INGEST" = "on" ] ; then
 	DEBUG_OPT="--debug $CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_ingest.log";
 fi
 
-pssh -h ingesters.txt "nohup $ACCUMULO_HOME/bin/accumulo org.apache.accumulo.server.test.continuous.ContinuousIngest
$DEBUG_OPT $INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $MIN $MAX $MAX_CF $MAX_CQ $MAX_MEM
$MAX_LATENCY $NUM_THREADS $CHECKSUM >$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_ingest.out
2>$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_ingest.err &" < /dev/null
+#do this check so that script will work w/ 1.4.0 config file
+if [ -z $NUM ]; then
+	NUM=9223372036854775807
+fi
+
+pssh -h ingesters.txt "mkdir -p $CONTINUOUS_LOG_DIR; nohup $ACCUMULO_HOME/bin/accumulo org.apache.accumulo.server.test.continuous.ContinuousIngest
$DEBUG_OPT $INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $NUM $MIN $MAX $MAX_CF $MAX_CQ $MAX_MEM
$MAX_LATENCY $NUM_THREADS $CHECKSUM >$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_ingest.out
2>$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_ingest.err &" < /dev/null
 

Modified: accumulo/branches/1.4.1/test/system/continuous/start-scanners.sh
URL: http://svn.apache.org/viewvc/accumulo/branches/1.4.1/test/system/continuous/start-scanners.sh?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/test/system/continuous/start-scanners.sh (original)
+++ accumulo/branches/1.4.1/test/system/continuous/start-scanners.sh Mon Jun 18 18:29:00 2012
@@ -24,5 +24,5 @@ if [ "$DEBUG_SCANNER" = "on" ] ; then
 	DEBUG_OPT="--debug $CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_scanner.log";
 fi
 
-pssh -h scanners.txt "nohup $ACCUMULO_HOME/bin/accumulo org.apache.accumulo.server.test.continuous.ContinuousScanner
$DEBUG_SCANNERS $INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $MIN $MAX $SCANNER_SLEEP_TIME
$SCANNER_ENTRIES >$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_scanner.out 2>$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_scanner.err &" < /dev/null
+pssh -h scanners.txt "mkdir -p $CONTINUOUS_LOG_DIR; nohup $ACCUMULO_HOME/bin/accumulo org.apache.accumulo.server.test.continuous.ContinuousScanner
$DEBUG_SCANNERS $INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $MIN $MAX $SCANNER_SLEEP_TIME
$SCANNER_ENTRIES >$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_scanner.out 2>$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_scanner.err &" < /dev/null
 

Modified: accumulo/branches/1.4.1/test/system/continuous/start-stats.sh
URL: http://svn.apache.org/viewvc/accumulo/branches/1.4.1/test/system/continuous/start-stats.sh?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/test/system/continuous/start-stats.sh (original)
+++ accumulo/branches/1.4.1/test/system/continuous/start-stats.sh Mon Jun 18 18:29:00 2012
@@ -18,6 +18,8 @@
 
 . continuous-env.sh
 
+mkdir -p $CONTINUOUS_LOG_DIR
+
 CONFIG_OUT=$CONTINUOUS_LOG_DIR/`date +%Y%m%d%H%M%S`_`hostname`_config.out
 
 cat $ACCUMULO_HOME/conf/accumulo-env.sh > $CONFIG_OUT

Modified: accumulo/branches/1.4.1/test/system/continuous/start-walkers.sh
URL: http://svn.apache.org/viewvc/accumulo/branches/1.4.1/test/system/continuous/start-walkers.sh?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/test/system/continuous/start-walkers.sh (original)
+++ accumulo/branches/1.4.1/test/system/continuous/start-walkers.sh Mon Jun 18 18:29:00 2012
@@ -24,5 +24,5 @@ if [ "$DEBUG_WALKER" = "on" ] ; then
 	DEBUG_OPT="--debug $CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_walk.log";
 fi
 
-pssh -h walkers.txt "nohup $ACCUMULO_HOME/bin/accumulo org.apache.accumulo.server.test.continuous.ContinuousWalk
$DEBUG_OPT $INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $MIN $MAX $SLEEP_TIME >$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_walk.out 2>$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_walk.err
&" < /dev/null
+pssh -h walkers.txt "mkdir -p $CONTINUOUS_LOG_DIR; nohup $ACCUMULO_HOME/bin/accumulo org.apache.accumulo.server.test.continuous.ContinuousWalk
$DEBUG_OPT $INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $MIN $MAX $SLEEP_TIME >$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_walk.out 2>$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_walk.err
&" < /dev/null
 



Mime
View raw message