incubator-drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tridib Samanta <tridib.sama...@live.com>
Subject RE: Aggregation sum() fails on parquet file
Date Fri, 31 Oct 2014 07:36:32 GMT
Nope. Here is exception. This only happens when I am running it against parquet file in HDFS.
 
org.apache.drill.common.exceptions.DrillRuntimeException: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
        at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:427)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:188) [drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.ScanBatch.buildSchema(ScanBatch.java:122) [drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.buildSchema(IteratorValidatorBatchIterator.java:80)
[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.buildSchema(HashAggBatch.java:97)
[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.buildSchema(IteratorValidatorBatchIterator.java:80)
[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.buildSchema(PartitionSenderRootExec.java:115)
[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:103)
[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:249)
[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_67]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_67]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
        at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78) ~[na:na]
        at org.xerial.snappy.SnappyNative.rawUncompress(Native Method) ~[na:na]
        at org.xerial.snappy.Snappy.uncompress(Snappy.java:467) ~[snappy-java-1.0.5-M3.jar:na]
        at parquet.hadoop.codec.SnappyDecompressor$SnappyDirectDecompressor.decompress(SnappyDecompressor.java:160)
~[parquet-hadoop-1.5.1-drill-r4.jar:na]
        at parquet.hadoop.CodecFactoryExposer.decompress(CodecFactoryExposer.java:88) ~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        at org.apache.drill.exec.store.parquet.columnreaders.PageReader.next(PageReader.java:212)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        at org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readPage(ColumnReader.java:198)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        at org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:141)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        at org.apache.drill.exec.store.parquet.columnreaders.VarLenBinaryReader.readFields(VarLenBinaryReader.java:56)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:411)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
        ... 11 common frames omitted

Thanks
Tridib
 
> From: danielrulez@gmail.com
> Subject: Re: Aggregation sum() fails on parquet file
> Date: Fri, 31 Oct 2014 09:30:03 +0200
> To: drill-user@incubator.apache.org
> 
> And did it solve your problem?
> 
> Daniel
> 
> > On 31 באוק׳ 2014, at 09:25, Tridib Samanta <tridib.samanta@live.com>
wrote:
> > 
> > Thanks. I am able to build and run 0.7.0-SNAPSHOT.
> > 
> >> From: altekrusejason@gmail.com
> >> Date: Thu, 30 Oct 2014 23:50:14 -0700
> >> Subject: Re: Aggregation sum() fails on parquet file
> >> To: drill-user@incubator.apache.org
> >> 
> >> Here is the wiki page on building from source. I am not much of a maven
> >> expert, but the dev team uses 'mvn install' to run builds, not sure how it
> >> differs from mvn package.
> >> 
> >> https://cwiki.apache.org/confluence/display/DRILL/Compiling+Drill+from+Source
> >> 
> >> If you want to poke around at all in the code when you start building it
> >> there are a number of articles linked from the main wiki page about setting
> >> up the source in an IDE and contributing back patches.
> >> 
> >> https://cwiki.apache.org/confluence/display/DRILL/Apache+Drill+Wiki
> >> 
> >> -Jason Altekruse
> >> 
> >> On Thu, Oct 30, 2014 at 11:27 PM, Tridib Samanta <tridib.samanta@live.com>
> >> wrote:
> >> 
> >>> Nope. I am running 0.5. I tried to build from latest master with 'mvn
> >>> package'. It failed with error .git not found. I wish there was a document
> >>> how to build from source.
> >>> 
> >>>> Date: Thu, 30 Oct 2014 23:14:17 -0700
> >>>> Subject: Re: Aggregation sum() fails on parquet file
> >>>> From: tnachen@gmail.com
> >>>> To: drill-user@incubator.apache.org
> >>>> 
> >>>> Hi there,
> >>>> 
> >>>> Are you running the latest master?
> >>>> 
> >>>> Tim
> >>>> 
> >>>> On Thu, Oct 30, 2014 at 10:57 PM, Tridib Samanta
> >>>> <tridib.samanta@live.com> wrote:
> >>>>> No clue. :(. Now I am struggling to run distinct on parquet file.
> >>>>> 
> >>>>>> From: danielrulez@gmail.com
> >>>>>> Subject: Re: Aggregation sum() fails on parquet file
> >>>>>> Date: Fri, 31 Oct 2014 07:14:43 +0200
> >>>>>> To: drill-user@incubator.apache.org
> >>>>>> 
> >>>>>> Hi,
> >>>>>> Any clue on that? I'm facing a similar problem
> >>>>>> 
> >>>>>> Thanks,
> >>>>>> Daniel
> >>>>>> 
> >>>>>>> On 30 באוק׳ 2014, at 21:28, Tridib Samanta <tridib.samanta@live.com>
> >>> wrote:
> >>>>>>> 
> >>>>>>> Hello,
> >>>>>>> I am running a query like: select sum(amount) from
> >>> myhdfs.parquet.`claim.parquet`;
> >>>>>>> This fails with the following error at sqlline:
> >>>>>>> 
> >>>>>>> Query failed: Failure while running fragment. Schema is
currently
> >>> null.  You must call buildSchema(SelectionVectorMode) before this container
> >>> can return a schema. [00314ea1-342a-4322-ac00-f8307c554733]
> >>>>>>> Error: exception while executing query: Failure while trying
to get
> >>> next result batch. (state=,code=0)
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Here is the stack trace from drillbits.log:
> >>>>>>> 
> >>>>>>> 2014-10-30 15:12:05,865
> >>> [3fb04943-c0e5-4801-a907-2bc3462a2997:frag:2:10] WARN
> >>> o.a.d.e.w.fragment.FragmentExecutor - Failure while closing out resources.
> >>>>>>> java.lang.IllegalStateException: Failure while trying to
close
> >>> child allocator: Child level allocators not closed. Fragment 2:10. Stack
> >>> trace:
> >>> org.apache.drill.exec.ops.OperatorContext.<init>(OperatorContext.java:41)
> >>> org.apache.drill.exec.physical.impl.ScanBatch.<init>(ScanBatch.java:90)
> >>> org.apache.drill.exec.store.parquet.ParquetScanBatchCreator.getBatch(ParquetScanBatchCreator.java:148)
> >>> org.apache.drill.exec.store.parquet.ParquetScanBatchCreator.getBatch(ParquetScanBatchCreator.java:54)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.visitOp(ImplCreator.java:62)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.visitOp(ImplCreator.java:39)
> >>> org.apache.drill.exec.physical.base.AbstractPhysicalVisitor.visitSubScan(AbstractPhysicalVisitor.java:113)
> >>> org.apache.drill.exec.store.parquet.ParquetRowGroupScan.accept(ParquetRowGroupScan.java:107)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:74)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.visitOp(ImplCreator.java:62)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.visitOp(ImplCreator.java:39)
> >>> org.apache.drill.exec.physical.base.AbstractPhysicalVisitor.visitIteratorValidator(AbstractPhysicalVisitor.java:196)
> >>> org.apache.drill.exec.physical.config.IteratorValidator.accept(IteratorValidator.java:34)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:74)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.visitOp(ImplCreator.java:62)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.visitOp(ImplCreator.java:39)
> >>> org.apache.drill.exec.physical.base.AbstractPhysicalVisitor.visitHashAggregate(AbstractPhysicalVisitor.java:93)
> >>> org.apache.drill.exec.physical.config.HashAggregate.accept(HashAggregate.java:62)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:74)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.visitOp(ImplCreator.java:62)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.visitOp(ImplCreator.java:39)
> >>> org.apache.drill.exec.physical.base.AbstractPhysicalVisitor.visitIteratorValidator(AbstractPhysicalVisitor.java:196)
> >>> org.apache.drill.exec.physical.config.IteratorValidator.accept(IteratorValidator.java:34)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:74)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.visitOp(ImplCreator.java:59)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.visitOp(ImplCreator.java:39)
> >>> org.apache.drill.exec.physical.base.AbstractPhysicalVisitor.visitSender(AbstractPhysicalVisitor.java:98)
> >>> org.apache.drill.exec.physical.base.AbstractPhysicalVisitor.visitHashPartitionSender(AbstractPhysicalVisitor.java:141)
> >>> org.apache.drill.exec.physical.config.HashPartitionSender.accept(HashPartitionSender.java:66)
> >>> org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:87)
> >>> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:94)
> >>> org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:250)
> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>>>>>               java.lang.Thread.run(Thread.java:745)
> >>>>>>>       at
> >>> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.close(TopLevelAllocator.java:215)
> >>> ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating]
> >>>>>>>       at
> >>> org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:271)
> >>> ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating]
> >>>>>>>       at
> >>> org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:139)
> >>> [drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating]
> >>>>>>>       at
> >>> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:107)
> >>> [drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating]
> >>>>>>>       at
> >>> org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:250)
> >>> [drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating]
> >>>>>>>       at
> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>> [na:1.7.0_67]
> >>>>>>>       at
> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>> [na:1.7.0_67]
> >>>>>>> 
> >>>>>>> Any help is greatly appreciated.
> >>>>>>> 
> >>>>>>> Thanks & Regards
> >>>>>>> Tridib
> >                         
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message