spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kailash Gupta (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-26379) Structured Streaming - Exception on adding column to Dataset
Date Sun, 16 Dec 2018 16:37:00 GMT
Kailash Gupta created SPARK-26379:
-------------------------------------

             Summary: Structured Streaming - Exception on adding column to Dataset
                 Key: SPARK-26379
                 URL: https://issues.apache.org/jira/browse/SPARK-26379
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 2.3.0
            Reporter: Kailash Gupta


While using withColumn to add a column to a structured streaming Dataset, I am getting following
exception: 

org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved
object, tree: 'timestamp

Following is sample code
{code:java}
final String path = "path_to_input_directory";

final StructType schema = new StructType(new StructField[] { new StructField("word", StringType,
false, Metadata.empty()), new StructField("count", DataTypes.IntegerType, false, Metadata.empty())
});

SparkSession sparkSession = SparkSession.builder().appName("StructuredStreamingIssue").master("local").getOrCreate();
Dataset<Row> words = sparkSession.readStream().option("sep", ",").schema(schema).csv(path);

Dataset<Row> wordsWithTimestamp = words.withColumn("timestamp", functions.current_timestamp());

// wordsWithTimestamp.explain(true);

StreamingQuery query = wordsWithTimestamp.writeStream().outputMode("update").option("truncate",
"false").format("console").trigger(Trigger.ProcessingTime("2 seconds")).start();

query.awaitTermination();{code}
Following are the contents of the file present at _path_
{code:java}
a,2
c,4
d,2
r,1
t,9
{code}
This seems working with 2.2.0 release, but not with 2.3.0 and 2.4.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message