Return-Path: X-Original-To: apmail-incubator-accumulo-commits-archive@minotaur.apache.org Delivered-To: apmail-incubator-accumulo-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0C035950D for ; Mon, 19 Mar 2012 18:00:41 +0000 (UTC) Received: (qmail 39070 invoked by uid 500); 19 Mar 2012 18:00:41 -0000 Delivered-To: apmail-incubator-accumulo-commits-archive@incubator.apache.org Received: (qmail 39049 invoked by uid 500); 19 Mar 2012 18:00:41 -0000 Mailing-List: contact accumulo-commits-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: accumulo-dev@incubator.apache.org Delivered-To: mailing list accumulo-commits@incubator.apache.org Received: (qmail 39041 invoked by uid 99); 19 Mar 2012 18:00:40 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Mar 2012 18:00:40 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Mar 2012 18:00:36 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 243DD2388C6D; Mon, 19 Mar 2012 18:00:14 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1302575 - in /incubator/accumulo/branches/1.4.0rc: ./ src/assemble/ src/core/ src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/ingest/ src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/exa... Date: Mon, 19 Mar 2012 18:00:13 -0000 To: accumulo-commits@incubator.apache.org From: kturner@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20120319180014.243DD2388C6D@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: kturner Date: Mon Mar 19 18:00:13 2012 New Revision: 1302575 URL: http://svn.apache.org/viewvc?rev=1302575&view=rev Log: ACCUMULO-145 merge updates from 1.4 branch into 1.4.0rc branch Modified: incubator/accumulo/branches/1.4.0rc/ (props changed) incubator/accumulo/branches/1.4.0rc/src/assemble/build.sh incubator/accumulo/branches/1.4.0rc/src/core/ (props changed) incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/ingest/LRUOutputCombiner.java incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/output/BufferingRFileRecordWriter.java incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/output/SortingRFileOutputFormat.java incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/test/java/org/apache/accumulo/examples/wikisearch/ingest/WikipediaInputSplitTest.java incubator/accumulo/branches/1.4.0rc/src/server/ (props changed) Propchange: incubator/accumulo/branches/1.4.0rc/ ------------------------------------------------------------------------------ Merged /incubator/accumulo/branches/1.4:r1300713,1302533 Modified: incubator/accumulo/branches/1.4.0rc/src/assemble/build.sh URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4.0rc/src/assemble/build.sh?rev=1302575&r1=1302574&r2=1302575&view=diff ============================================================================== --- incubator/accumulo/branches/1.4.0rc/src/assemble/build.sh (original) +++ incubator/accumulo/branches/1.4.0rc/src/assemble/build.sh Mon Mar 19 18:00:13 2012 @@ -41,12 +41,13 @@ runAt() { run mvn -U -P distclean clean mvn rat:check COUNT=`grep '!????' target/rat.txt | wc -l` -EXPECTED=19 +EXPECTED=56 if [ "$COUNT" -ne $EXPECTED ] then fail expected $EXPECTED files missing licenses, but saw "$COUNT" fi +#need to run mvn package twice to properly build docs/config.html +run mvn package run mvn package javadoc:aggregate javadoc:jar source:jar runAt ./src/server/src/main/c++ make -run mvn package source:jar assembly:single -test -x /usr/bin/rpmbuild && run mvn -N rpm:rpm +run mvn assembly:single -N Propchange: incubator/accumulo/branches/1.4.0rc/src/core/ ------------------------------------------------------------------------------ Merged /incubator/accumulo/branches/1.4/src/core:r1300713,1302533 Modified: incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/ingest/LRUOutputCombiner.java URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/ingest/LRUOutputCombiner.java?rev=1302575&r1=1302574&r2=1302575&view=diff ============================================================================== --- incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/ingest/LRUOutputCombiner.java (original) +++ incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/ingest/LRUOutputCombiner.java Mon Mar 19 18:00:13 2012 @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package org.apache.accumulo.examples.wikisearch.ingest; import java.util.LinkedHashMap; @@ -6,14 +22,12 @@ import java.util.Map; public class LRUOutputCombiner extends LinkedHashMap { private static final long serialVersionUID = 1L; - - public static abstract class Fold - { + + public static abstract class Fold { public abstract Value fold(Value oldValue, Value newValue); } - public static abstract class Output - { + public static abstract class Output { public abstract void output(Key key, Value value); } Modified: incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/output/BufferingRFileRecordWriter.java URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/output/BufferingRFileRecordWriter.java?rev=1302575&r1=1302574&r2=1302575&view=diff ============================================================================== --- incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/output/BufferingRFileRecordWriter.java (original) +++ incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/output/BufferingRFileRecordWriter.java Mon Mar 19 18:00:13 2012 @@ -1,10 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package org.apache.accumulo.examples.wikisearch.output; import java.io.IOException; import java.util.HashMap; import java.util.Map; -import java.util.TreeMap; import java.util.Map.Entry; +import java.util.TreeMap; import org.apache.accumulo.core.conf.AccumuloConfiguration; import org.apache.accumulo.core.data.ColumnUpdate; @@ -31,12 +47,10 @@ final class BufferingRFileRecordWriter e private Map> buffers = new HashMap>(); private Map bufferSizes = new HashMap(); - - private TreeMap getBuffer(Text tablename) - { + + private TreeMap getBuffer(Text tablename) { TreeMap buffer = buffers.get(tablename); - if(buffer == null) - { + if (buffer == null) { buffer = new TreeMap(); buffers.put(tablename, buffer); bufferSizes.put(tablename, 0l); @@ -44,14 +58,11 @@ final class BufferingRFileRecordWriter e return buffer; } - private Text getLargestTablename() - { + private Text getLargestTablename() { long max = 0; Text table = null; - for(Entry e:bufferSizes.entrySet()) - { - if(e.getValue() > max) - { + for (Entry e : bufferSizes.entrySet()) { + if (e.getValue() > max) { max = e.getValue(); table = e.getKey(); } @@ -59,10 +70,9 @@ final class BufferingRFileRecordWriter e return table; } - private void flushLargestTable() throws IOException - { + private void flushLargestTable() throws IOException { Text tablename = getLargestTablename(); - if(tablename == null) + if (tablename == null) return; long bufferSize = bufferSizes.get(tablename); TreeMap buffer = buffers.get(tablename); @@ -98,7 +108,7 @@ final class BufferingRFileRecordWriter e @Override public void close(TaskAttemptContext arg0) throws IOException, InterruptedException { - while(size > 0) + while (size > 0) flushLargestTable(); } @@ -106,9 +116,9 @@ final class BufferingRFileRecordWriter e public void write(Text table, Mutation mutation) throws IOException, InterruptedException { TreeMap buffer = getBuffer(table); int mutationSize = 0; - for(ColumnUpdate update: mutation.getUpdates()) - { - Key k = new Key(mutation.getRow(),update.getColumnFamily(),update.getColumnQualifier(),update.getColumnVisibility(),update.getTimestamp(),update.isDeleted()); + for (ColumnUpdate update : mutation.getUpdates()) { + Key k = new Key(mutation.getRow(), update.getColumnFamily(), update.getColumnQualifier(), update.getColumnVisibility(), update.getTimestamp(), + update.isDeleted()); Value v = new Value(update.getValue()); // TODO account for object overhead mutationSize += k.getSize(); @@ -121,7 +131,7 @@ final class BufferingRFileRecordWriter e // TODO use a MutableLong instead bufferSize += mutationSize; bufferSizes.put(table, bufferSize); - + while (size >= maxSize) { flushLargestTable(); } Modified: incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/output/SortingRFileOutputFormat.java URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/output/SortingRFileOutputFormat.java?rev=1302575&r1=1302574&r2=1302575&view=diff ============================================================================== --- incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/output/SortingRFileOutputFormat.java (original) +++ incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/main/java/org/apache/accumulo/examples/wikisearch/output/SortingRFileOutputFormat.java Mon Mar 19 18:00:13 2012 @@ -1,3 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package org.apache.accumulo.examples.wikisearch.output; import java.io.IOException; @@ -14,9 +30,9 @@ import org.apache.hadoop.mapreduce.Recor import org.apache.hadoop.mapreduce.TaskAttemptContext; public class SortingRFileOutputFormat extends OutputFormat { - + // private static final Logger log = Logger.getLogger(SortingRFileOutputFormat.class); - + public static final String PATH_NAME = "sortingrfileoutputformat.path"; public static final String MAX_BUFFER_SIZE = "sortingrfileoutputformat.max.buffer.size"; Modified: incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/test/java/org/apache/accumulo/examples/wikisearch/ingest/WikipediaInputSplitTest.java URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/test/java/org/apache/accumulo/examples/wikisearch/ingest/WikipediaInputSplitTest.java?rev=1302575&r1=1302574&r2=1302575&view=diff ============================================================================== --- incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/test/java/org/apache/accumulo/examples/wikisearch/ingest/WikipediaInputSplitTest.java (original) +++ incubator/accumulo/branches/1.4.0rc/src/examples/wikisearch/ingest/src/test/java/org/apache/accumulo/examples/wikisearch/ingest/WikipediaInputSplitTest.java Mon Mar 19 18:00:13 2012 @@ -1,9 +1,24 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package org.apache.accumulo.examples.wikisearch.ingest; import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; import java.io.DataInput; -import java.io.DataOutput; import java.io.IOException; import java.io.ObjectInputStream; import java.io.ObjectOutputStream; @@ -17,20 +32,19 @@ import org.junit.Test; public class WikipediaInputSplitTest { @Test - public void testSerialization() throws IOException - { + public void testSerialization() throws IOException { Path testPath = new Path("/foo/bar"); - String [] hosts = new String [2]; + String[] hosts = new String[2]; hosts[0] = "abcd"; hosts[1] = "efgh"; - FileSplit fSplit = new FileSplit(testPath,1,2,hosts); - WikipediaInputSplit split = new WikipediaInputSplit(fSplit,7); + FileSplit fSplit = new FileSplit(testPath, 1, 2, hosts); + WikipediaInputSplit split = new WikipediaInputSplit(fSplit, 7); ByteArrayOutputStream baos = new ByteArrayOutputStream(); ObjectOutputStream out = new ObjectOutputStream(baos); split.write(out); - out.close(); + out.close(); baos.close(); - + ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray()); DataInput in = new ObjectInputStream(bais); @@ -45,12 +59,11 @@ public class WikipediaInputSplitTest { Assert.assertTrue(fSplit.getPath().equals(fSplit2.getPath())); Assert.assertTrue(fSplit.getStart() == fSplit2.getStart()); Assert.assertTrue(fSplit.getLength() == fSplit2.getLength()); - - String [] hosts2 = fSplit2.getLocations(); + + String[] hosts2 = fSplit2.getLocations(); Assert.assertEquals(hosts.length, hosts2.length); - for(int i = 0; i < hosts.length; i++) - { - Assert.assertEquals(hosts[i],hosts2[i]); + for (int i = 0; i < hosts.length; i++) { + Assert.assertEquals(hosts[i], hosts2[i]); } } } Propchange: incubator/accumulo/branches/1.4.0rc/src/server/ ------------------------------------------------------------------------------ Merged /incubator/accumulo/branches/1.4/src/server:r1300713,1302533