carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anubhav100 <...@git.apache.org>
Subject [GitHub] carbondata pull request #1774: [CARBONDATA-2001] Unable to Save DataFrame As...
Date Sat, 13 Jan 2018 08:35:06 GMT
Github user anubhav100 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1774#discussion_r161367337
  
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala
---
    @@ -167,13 +167,20 @@ class CarbonDataFrameWriter(sqlContext: SQLContext, val dataFrame:
DataFrame) {
         val carbonSchema = schema.map { field =>
           s"${ field.name } ${ convertToCarbonType(field.dataType) }"
         }
    +  val isStreaming = if (options.isStreaming) Some("true") else None
    +
         val property = Map(
           "SORT_COLUMNS" -> options.sortColumns,
           "DICTIONARY_INCLUDE" -> options.dictionaryInclude,
           "DICTIONARY_EXCLUDE" -> options.dictionaryExclude,
    -      "TABLE_BLOCKSIZE" -> options.tableBlockSize
    -    ).filter(_._2.isDefined).map(p => s"'${p._1}' = '${p._2.get}'").mkString(",")
    +      "TABLE_BLOCKSIZE" -> options.tableBlockSize,
    +      "STREAMING" -> isStreaming
    +    )
    +      .filter(_._2.isDefined).
    --- End diff --
    
    using it directly will cause this test case to fail
    
    test("test datasource table with specified table path") {
        val path = "./source"
        df2.write
          .format("carbondata")
          .option("tableName", "carbon10")
          .option("tablePath", path)
          .mode(SaveMode.Overwrite)
          .save()
        assert(new File(path).exists())
        checkAnswer(
          sql("select count(*) from carbon10 where c3 > 500"), Row(500)
        )
        sql("drop table carbon10")
        assert(!new File(path).exists())
        assert(intercept[AnalysisException](
          sql("select count(*) from carbon10 where c3 > 500"))
          .message
          .contains("not found"))
      }
    
    this is because of some problem at parsing level giving tableproperties followed by table
path is not parsing correctlly a seperate jira is already created for it here is the link

    
    https://issues.apache.org/jira/browse/CARBONDATA-2005
    
    are we going to support it?
    
    thats why what i do is that if is streaming is false then do not include it in property
map make it none


---

Mime
View raw message