carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ravipesala <...@git.apache.org>
Subject [GitHub] carbondata pull request #1508: [CARBONDATA-1738] [PreAgg] Block direct inser...
Date Sun, 26 Nov 2017 12:13:37 GMT
Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1508#discussion_r153068092
  
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateListeners.scala
---
    @@ -49,13 +50,52 @@ object LoadPostAggregateListener extends OperationEventListener {
                                     carbonLoadModel.getTableName, "false")
             val childTableName = dataMapSchema.getRelationIdentifier.getTableName
             val childDatabaseName = dataMapSchema.getRelationIdentifier.getDatabaseName
    -        val selectQuery = dataMapSchema.getProperties.get("CHILD_SELECT QUERY")
    -        sparkSession.sql(s"insert into $childDatabaseName.$childTableName $selectQuery")
    +        val childDataFrame = sparkSession.sql(new CarbonSpark2SqlParser()
    +          .addPreAggLoadFunction(s"${dataMapSchema.getProperties.get("CHILD_SELECT QUERY")}
")).drop("preAggLoad")
    +        val headers = dataMapSchema.getChildSchema.getListOfColumns.asScala.map(_.getColumnName)
    +          .mkString(",")
    +        try {
    +          LoadTableCommand(Some(childDatabaseName),
    +            childTableName,
    +            null,
    +            Nil,
    +            Map("fileheader" -> headers),
    +            isOverwriteTable = false,
    +            dataFrame = Some(childDataFrame),
    +            internalOptions = Map(CarbonCommonConstants.IS_INTERNAL_LOAD_CALL -> "true")).run(sparkSession)
    +        } finally {
    +          CarbonSession.threadUnset(CarbonCommonConstants.CARBON_INPUT_SEGMENTS +
    +                                    carbonLoadModel.getDatabaseName + "." +
    +                                    carbonLoadModel.getTableName)
    +          CarbonSession.threadUnset(CarbonCommonConstants.VALIDATE_CARBON_INPUT_SEGMENTS
+
    +                                    carbonLoadModel.getDatabaseName + "." +
    +                                    carbonLoadModel.getTableName)
    --- End diff --
    
    Indentation is wrong, format properly


---

Mime
View raw message