accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: exception thrown during minor compaction
Date Mon, 31 Aug 2015 22:52:55 GMT
An empty value seems to imply that you wrote some unexpected data to 
your table. The following code does work correctly (using 1.7.0).

     byte[] bytes = new LongLexicoder().encode(1l);
     System.out.println(Arrays.toString(bytes));
     System.out.println(new LongLexicoder().decode(bytes));

Can you check the code you're using to create Mutations and verify that 
you're passing in the bytes from the LongLexicoder encode() method as 
the Value?

You can try removing the SummingCombiner from your table which should 
let you scan the table to verify the records (and also let your 
compactions happen). Because you removed the VersioningIterator, it 
should preserve all of the entries you have in the table.

z11373 wrote:
> Hi,
> I attach a summing combiner into a newly created table. The snippet code is
> something like:
>
> 			EnumSet<IteratorScope>  iteratorScopes =
> EnumSet.allOf(IteratorScope.class);
>
> 			// first, remove versioning iterator since it will not work with combiner
> 			conn.tableOperations().removeIterator(tableName,
> 				VERS_ITERATOR_NAME,
> 				iteratorScopes);
> 			
> 			// create the combiner setting, in this case it is SummingCombiner, which
> will
> 			// sum value of all rows with same key (different timestamp is considered
> same)
> 			// and result in single row with that key and aggregate value
> 			IteratorSetting setting = new IteratorSetting(
> 				COMBINERS_PRIORITY,
> 				SUM_COMBINERS_NAME,
> 				SummingCombiner.class);
> 			
> 			// set the combiner to apply to all columns
> 			SummingCombiner.setCombineAllColumns(setting, true);
> 			
> 			// need to set encoding type, otherwise exception will be thrown during
> scan
> 			SummingCombiner.setEncodingType(setting, LongLexicoder.class);
> 			
> 			// attach the combiner to the table
> 			conn.tableOperations().attachIterator(tableName, setting,
> iteratorScopes);
>
>
> As you see from the code above, I use LongLexicoder class as the encoding
> type.
> The mutation I add for that table will be unique row id, a string for column
> family, empty column qualifier, and the value is "new
> LongLexicoder().encode(1L)", so basically the value is 1.
>
> It runs fine until at one point (and I can see rows are inserted into the
> table), but it hung then.
> Looking at the tablet server logs I found:
>
> 2015-08-31 17:59:42,371 [tserver.MinorCompactor] WARN : MinC failed (0) to
> create
> hdfs://<machine>:<port>/accumulo/tables/l/default_tablet/F00009gp.rf_tmp
> retrying ...
> java.lang.ArrayIndexOutOfBoundsException: 0
>          at
> org.apache.accumulo.core.client.lexicoder.ULongLexicoder.decode(ULongLexicoder.java:60)
>          at
> org.apache.accumulo.core.client.lexicoder.LongLexicoder.decode(LongLexicoder.java:33)
>          at
> org.apache.accumulo.core.client.lexicoder.LongLexicoder.decode(LongLexicoder.java:25)
>          at
> org.apache.accumulo.core.iterators.TypedValueCombiner$VIterator.hasNext(TypedValueCombiner.java:82)
>          at
> org.apache.accumulo.core.iterators.user.SummingCombiner.typedReduce(SummingCombiner.java:31)
>          at
> org.apache.accumulo.core.iterators.user.SummingCombiner.typedReduce(SummingCombiner.java:27)
>          at
> org.apache.accumulo.core.iterators.TypedValueCombiner.reduce(TypedValueCombiner.java:182)
>          at
> org.apache.accumulo.core.iterators.Combiner.findTop(Combiner.java:166)
>          at
> org.apache.accumulo.core.iterators.Combiner.next(Combiner.java:147)
>          at
> org.apache.accumulo.tserver.Compactor.compactLocalityGroup(Compactor.java:505)
>          at org.apache.accumulo.tserver.Compactor.call(Compactor.java:362)
>          at
> org.apache.accumulo.tserver.MinorCompactor.call(MinorCompactor.java:96)
>          at org.apache.accumulo.tserver.Tablet.minorCompact(Tablet.java:2072)
>          at org.apache.accumulo.tserver.Tablet.access$4400(Tablet.java:172)
>          at
> org.apache.accumulo.tserver.Tablet$MinorCompactionTask.run(Tablet.java:2159)
>          at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>          at
> org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>          at
> org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>          at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>          at java.lang.Thread.run(Thread.java:722)
>
>
> Looking at Accumulo's source code in ULongLexicoder.java, it looks like the
> array is empty, hence it throws exception, which is still happening even
> after I killed my app. I am thinking to stop Accumulo.
> Do you have any idea why the array is empty? I am thinking to experiment
> with String encoder instead of using LongLexicoder.
>
>
>    public Long decode(byte[] data) {
>
>      long l = 0;
>      int shift = 0;
>
>      if (data[0]<  0 || data[0]>  16)
>        throw new IllegalArgumentException("Unexpected length " + (0xff&
> data[0]));
>
>      ...
>    }
>
>
> Thanks,
> zainal
>
>
>
> --
> View this message in context: http://apache-accumulo.1065345.n5.nabble.com/exception-thrown-during-minor-compaction-tp15010.html
> Sent from the Developers mailing list archive at Nabble.com.

Mime
View raw message