Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 956FD200D30 for ; Mon, 30 Oct 2017 13:48:07 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 93BDE160BE4; Mon, 30 Oct 2017 12:48:07 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 8B22B1609EF for ; Mon, 30 Oct 2017 13:48:06 +0100 (CET) Received: (qmail 37681 invoked by uid 500); 30 Oct 2017 12:48:05 -0000 Mailing-List: contact issues-help@carbondata.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@carbondata.apache.org Delivered-To: mailing list issues@carbondata.apache.org Received: (qmail 37672 invoked by uid 99); 30 Oct 2017 12:48:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Oct 2017 12:48:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id F36861806A9 for ; Mon, 30 Oct 2017 12:48:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 4-gSN5SPiq5i for ; Mon, 30 Oct 2017 12:48:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 33E3760F1F for ; Mon, 30 Oct 2017 12:48:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 6A3F9E0662 for ; Mon, 30 Oct 2017 12:48:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 30C1B212FA for ; Mon, 30 Oct 2017 12:48:00 +0000 (UTC) Date: Mon, 30 Oct 2017 12:48:00 +0000 (UTC) From: "Ashwini K (JIRA)" To: issues@carbondata.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CARBONDATA-1654) NullPointerException when insert overwrite table MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 30 Oct 2017 12:48:07 -0000 [ https://issues.apache.org/jira/browse/CARBONDATA-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16224862#comment-16224862 ] Ashwini K commented on CARBONDATA-1654: --------------------------------------- pls share the table definition > NullPointerException when insert overwrite table > ------------------------------------------------ > > Key: CARBONDATA-1654 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1654 > Project: CarbonData > Issue Type: Bug > Components: data-load > Affects Versions: 1.2.0 > Environment: spark 2.1.1 carbondata 1.2.0 > Reporter: cen yuhai > Priority: Critical > > carbon.sql("insert overwrite table carbondata_table select * from hive_table where dt = '2017-10-10' ").collect > carbondata wanto find directory Segment_1, but there is Segment_2 > {code} > [Stage 0:> (0 + 504) / 504]17/10/28 19:11:28 WARN [org.glassfish.jersey.internal.Errors(191) -- SparkUI-174]: The following warnings have been detected: WARNING: The (sub)resource method stageData in org.apache.spark.status.api.v1.OneStageResource contains empty path annotation. > 17/10/28 19:25:20 ERROR [org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile(141) -- main]: main Exception occurred:File does not exist: hdfs://bipcluster/user/master/carbon/store/dm_test/carbondata_table/Fact/Part0/Segment_1 > 17/10/28 19:25:22 ERROR [org.apache.spark.sql.execution.command.LoadTable(143) -- main]: main > java.lang.NullPointerException > at org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile.isDirectory(AbstractDFSCarbonFile.java:88) > at org.apache.carbondata.core.util.CarbonUtil.deleteRecursive(CarbonUtil.java:364) > at org.apache.carbondata.core.util.CarbonUtil.access$100(CarbonUtil.java:93) > at org.apache.carbondata.core.util.CarbonUtil$2.run(CarbonUtil.java:326) > at org.apache.carbondata.core.util.CarbonUtil$2.run(CarbonUtil.java:322) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) > at org.apache.carbondata.core.util.CarbonUtil.deleteFoldersAndFiles(CarbonUtil.java:322) > at org.apache.carbondata.spark.load.CarbonLoaderUtil.recordLoadMetadata(CarbonLoaderUtil.java:331) > at org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$.updateStatus$1(CarbonDataRDDFactory.scala:595) > at org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$.loadCarbonData(CarbonDataRDDFactory.scala:1107) > at org.apache.spark.sql.execution.command.LoadTable.processData(carbonTableSchema.scala:1046) > at org.apache.spark.sql.execution.command.LoadTable.run(carbonTableSchema.scala:754) > at org.apache.spark.sql.execution.command.LoadTableByInsert.processData(carbonTableSchema.scala:651) > at org.apache.spark.sql.execution.command.LoadTableByInsert.run(carbonTableSchema.scala:637) > at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58) > at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56) > at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:67) > at org.apache.spark.sql.Dataset.(Dataset.scala:180) > at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:65) > at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:619) > at $line23.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(:36) > at $line23.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(:41) > at $line23.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(:43) > at $line23.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(:45) > at $line23.$read$$iw$$iw$$iw$$iw$$iw$$iw.(:47) > at $line23.$read$$iw$$iw$$iw$$iw$$iw.(:49) > at $line23.$read$$iw$$iw$$iw$$iw.(:51) > at $line23.$read$$iw$$iw$$iw.(:53) > at $line23.$read$$iw$$iw.(:55) > at $line23.$read$$iw.(:57) > at $line23.$read.(:59) > at $line23.$read$.(:63) > at $line23.$read$.() > at $line23.$eval$.$print$lzycompute(:7) > at $line23.$eval$.$print(:6) > at $line23.$eval.$print() > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786) > at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047) > at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638) > at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637) > at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31) > at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19) > at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637) > at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569) > at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565) > at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807) > at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681) > at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395) > at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:415) > at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:923) > at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909) > at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909) > at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97) > at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909) > at org.apache.spark.repl.Main$.doMain(Main.scala:69) > at org.apache.spark.repl.Main$.main(Main.scala:52) > at org.apache.spark.repl.Main.main(Main.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:186) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:211) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > 17/10/28 19:25:22 AUDIT [org.apache.spark.sql.execution.command.LoadTable(207) -- main]: [sh-hadoop-datanode-250-104.elenet.me][master][Thread-1]Dataload failure for dm_test.carbondata_table. Please check the logs > java.lang.NullPointerException > at org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile.isDirectory(AbstractDFSCarbonFile.java:88) > at org.apache.carbondata.core.util.CarbonUtil.deleteRecursive(CarbonUtil.java:364) > at org.apache.carbondata.core.util.CarbonUtil.access$100(CarbonUtil.java:93) > at org.apache.carbondata.core.util.CarbonUtil$2.run(CarbonUtil.java:326) > at org.apache.carbondata.core.util.CarbonUtil$2.run(CarbonUtil.java:322) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) > at org.apache.carbondata.core.util.CarbonUtil.deleteFoldersAndFiles(CarbonUtil.java:322) > at org.apache.carbondata.spark.load.CarbonLoaderUtil.recordLoadMetadata(CarbonLoaderUtil.java:331) > at org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$.updateStatus$1(CarbonDataRDDFactory.scala:595) > at org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$.loadCarbonData(CarbonDataRDDFactory.scala:1107) > at org.apache.spark.sql.execution.command.LoadTable.processData(carbonTableSchema.scala:1046) > at org.apache.spark.sql.execution.command.LoadTable.run(carbonTableSchema.scala:754) > at org.apache.spark.sql.execution.command.LoadTableByInsert.processData(carbonTableSchema.scala:651) > at org.apache.spark.sql.execution.command.LoadTableByInsert.run(carbonTableSchema.scala:637) > at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58) > at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56) > at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:67) > at org.apache.spark.sql.Dataset.(Dataset.scala:180) > at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:65) > at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:619) > ... 50 elided > {code} > {code} > [master@sh-hadoop-datanode-132.elenet.me ~]$ hadoop fs -ls /user/master/carbon/store/dm_test/carbondata_table/Fact/Part0 > Found 1 items > drwxr-xr-x - master hadoop 0 2017-10-28 19:25 /user/master/carbon/store/dm_test/carbondata_table/Fact/Part0/Segment_2 > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)