Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BFDB9200BFE for ; Mon, 16 Jan 2017 18:51:33 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id BE70A160B4D; Mon, 16 Jan 2017 17:51:33 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 985DA160B28 for ; Mon, 16 Jan 2017 18:51:32 +0100 (CET) Received: (qmail 90165 invoked by uid 500); 16 Jan 2017 17:51:31 -0000 Mailing-List: contact issues-help@carbondata.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@carbondata.incubator.apache.org Delivered-To: mailing list issues@carbondata.incubator.apache.org Received: (qmail 90156 invoked by uid 99); 16 Jan 2017 17:51:31 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Jan 2017 17:51:31 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D6A56C0CA9 for ; Mon, 16 Jan 2017 17:51:30 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1.199 X-Spam-Level: X-Spam-Status: No, score=-1.199 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id h_dwXcb6AA07 for ; Mon, 16 Jan 2017 17:51:29 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 8E69D5F610 for ; Mon, 16 Jan 2017 17:51:28 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id BBB3CE0309 for ; Mon, 16 Jan 2017 17:51:27 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 6756325284 for ; Mon, 16 Jan 2017 17:51:26 +0000 (UTC) Date: Mon, 16 Jan 2017 17:51:26 +0000 (UTC) From: "Ravindra Pesala (JIRA)" To: issues@carbondata.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CARBONDATA-243) Filters on dictionary column not always work. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 16 Jan 2017 17:51:33 -0000 [ https://issues.apache.org/jira/browse/CARBONDATA-243?page=3Dcom.atlas= sian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D= 15824367#comment-15824367 ]=20 Ravindra Pesala commented on CARBONDATA-243: -------------------------------------------- I think this issue got resolved in latest master. Please verify it once > Filters on dictionary column not always work. > --------------------------------------------- > > Key: CARBONDATA-243 > URL: https://issues.apache.org/jira/browse/CARBONDATA-243 > Project: CarbonData > Issue Type: Bug > Components: examples > Affects Versions: 0.2.0-incubating > Environment: Ubuntu 14.04 > Reporter: Yue Shang > Priority: Minor > > Filter on dictionary column not always work. > Table loaded contains 100 columns, 100M rows, and one of the column c100'= s cardinality is 10M. Dictionary include c100.=20 > DDL statement as follows: > CREATE TABLE IF NOT EXISTS big_table4 > ( > c100 Int, > c1 Int, c2 Int, c3 Int, c4 Int, c5 Int, c6 Int, c7 Int, c8 Int, c= 9 Int, c10 Int, c11 Int, c12 Int, c13 Int, c14 Int, c15 Int, c16 Int, c17 I= nt, c18 Int, c19 Int, c20 Int, > c21 Int, c22 Int, c23 Int, c24 Int, c25 Int, c26 Int, c27 Int, c2= 8 Int, c29 Int, c30 Int, c31 Int, c32 Int, c33 Int, c34 Int, c35 Int, c36 I= nt, c37 Int, c38 Int, c39 Int, c40 Int, > c41 Int, c42 Int, c43 Int, c44 Int, c45 Int, c46 Int, c47 Int, c4= 8 Int, c49 Int, c50 Int, c51 Int, c52 Int, c53 Int, c54 Int, c55 Int, c56 I= nt, c57 Int, c58 Int, c59 Int, c60 Int, > c61 Int, c62 Int, c63 Int, c64 Int, c65 Int, c66 Int, c67 Int, c6= 8 Int, c69 Int, c70 Int, c71 Int, c72 Int, c73 Int, c74 Int, c75 Int, c76 I= nt, c77 Int, c78 Int, c79 Int, c80 Int, > c81 Int, c82 Int, c83 Int, c84 Int, c85 Int, c86 Int, c87 Int, c8= 8 Int, c89 Int, c90 Int, c91 Int, c92 Int, c93 Int, c94 Int, c95 Int, c96 I= nt, c97 Int, c98 Int, c99 Int, > c101 String > ) > STORED BY 'carbondata' > TBLPROPERTIES ("DICTIONARY_INCLUDE"=3D"c101,c100, c1, c2,c3,c4,c5= ,c6,c7,c8,c9,c10,c11,c12,c13,c14, > c15") > I tried this query to make sure the value i'm querying is exist. > select c100 from big_table4 where c100 like '1234_'=20 > =E2=80=8B+-----+ > | c100| > +-----+ > |12340| > |12341| > |12342| > |12343| > |12344| > |12345| > |12346| > |12347| > |12348| > |12349| > +-----+ > =E2=80=8B > But when I tried: " select c100 from big_table4 where c100 like '12345'= =E2=80=8B ", I got a runtime error(Log is at the bottom of this mail). > But the most wield part is, some values can be queried while others not. > cc.sql("select c100 from big_table4 where c100 like '1000016'") > This query will get the exact correct answer. > Any idea about this?=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > log > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > =E2=80=8BERROR 14-09 17:16:52,024 - [Executor task launch worker-0][parti= tionID:table4;queryID:275884499876481_0]=20 > java.lang.NullPointerException > =09at org.apache.carbondata.scan.result.iterator.AbstractDetailQueryResul= tIterator.intialiseInfos(AbstractDetailQueryResultIterator.java:95) > =09at org.apache.carbondata.scan.result.iterator.AbstractDetailQueryResul= tIterator.(AbstractDetailQueryResultIterator.java:87) > =09at org.apache.carbondata.scan.result.iterator.DetailQueryResultIterato= r.(DetailQueryResultIterator.java:47) > =09at org.apache.carbondata.scan.executor.impl.DetailQueryExecutor.execut= e(DetailQueryExecutor.java:39) > =09at org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.(Carbon= ScanRDD.scala:193) > =09at org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD= .scala:174) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > =09at org.apache.spark.scheduler.Task.run(Task.scala:88) > =09at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:21= 4) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > ERROR 14-09 17:16:52,026 - Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.RuntimeException: Exception occurred in query execution.Please = check logs. > =09at scala.sys.package$.error(package.scala:27) > =09at org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.(Carbon= ScanRDD.scala:203) > =09at org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD= .scala:174) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > =09at org.apache.spark.scheduler.Task.run(Task.scala:88) > =09at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:21= 4) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > ERROR 14-09 17:16:52,038 - Task 0 in stage 0.0 failed 1 times; aborting j= ob > Exception in thread "main" org.apache.spark.SparkException: Job aborted d= ue to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failur= e: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.RuntimeExceptio= n: Exception occurred in query execution.Please check logs. > =09at scala.sys.package$.error(package.scala:27) > =09at org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.(Carbon= ScanRDD.scala:203) > =09at org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD= .scala:174) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > =09at org.apache.spark.scheduler.Task.run(Task.scala:88) > =09at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:21= 4) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > Driver stacktrace: > =09at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$= DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283) > =09at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply= (DAGScheduler.scala:1271) > =09at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply= (DAGScheduler.scala:1270) > =09at scala.collection.mutable.ResizableArray$class.foreach(ResizableArra= y.scala:59) > =09at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > =09at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.sca= la:1270) > =09at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFaile= d$1.apply(DAGScheduler.scala:697) > =09at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFaile= d$1.apply(DAGScheduler.scala:697) > =09at scala.Option.foreach(Option.scala:236) > =09at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGSche= duler.scala:697) > =09at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive= (DAGScheduler.scala:1496) > =09at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(D= AGScheduler.scala:1458) > =09at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(D= AGScheduler.scala:1447) > =09at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) > =09at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:5= 67) > =09at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824) > =09at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837) > =09at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850) > =09at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scal= a:215) > =09at org.apache.spark.sql.execution.Limit.executeCollect(basicOperators.= scala:207) > =09at org.apache.spark.sql.DataFrame$$anonfun$collect$1.apply(DataFrame.s= cala:1385) > =09at org.apache.spark.sql.DataFrame$$anonfun$collect$1.apply(DataFrame.s= cala:1385) > =09at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQL= Execution.scala:56) > =09at org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:1= 903) > =09at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1384) > =09at org.apache.spark.sql.DataFrame.head(DataFrame.scala:1314) > =09at org.apache.spark.sql.DataFrame.take(DataFrame.scala:1377) > =09at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178) > =09at org.apache.spark.sql.DataFrame.show(DataFrame.scala:401) > =09at org.apache.spark.sql.DataFrame.show(DataFrame.scala:362) > =09at org.apache.carbondata.examples.MyBigTest$.main(MyBigTest.scala:107) > =09at org.apache.carbondata.examples.MyBigTest.main(MyBigTest.scala) > =09at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > =09at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImp= l.java:57) > =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc= essorImpl.java:43) > =09at java.lang.reflect.Method.invoke(Method.java:606) > =09at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147= ) > Caused by: java.lang.RuntimeException: Exception occurred in query execut= ion.Please check logs. > =09at scala.sys.package$.error(package.scala:27) > =09at org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.(Carbon= ScanRDD.scala:203) > =09at org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD= .scala:174) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scal= a:38) > =09at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) > =09at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) > =09at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > =09at org.apache.spark.scheduler.Task.run(Task.scala:88) > =09at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:21= 4) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745)=E2=80=8B -- This message was sent by Atlassian JIRA (v6.3.4#6332)