spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pravin Agrawal (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-23077) Apache Structured Streaming: Unable to write streaming dataset into Hive?
Date Mon, 15 Jan 2018 11:42:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-23077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pravin Agrawal updated SPARK-23077:
-----------------------------------
    Issue Type: Bug  (was: Question)

> Apache Structured Streaming: Unable to write streaming dataset into Hive?
> -------------------------------------------------------------------------
>
>                 Key: SPARK-23077
>                 URL: https://issues.apache.org/jira/browse/SPARK-23077
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 2.2.0
>            Reporter: Pravin Agrawal
>            Priority: Minor
>
> Using Apache Spark 2.2: Structured Streaming, I am creating a program which reads data
from Kafka and write it to Hive.
> I am looking for writing bulk data incoming in Kafka topic @ 100 records/sec.
> **Hive Table Created:**
> CREATE TABLE demo_user( timeaa BIGINT, numberbb INT, decimalcc DOUBLE, stringdd STRING,
booleanee BOOLEAN ) STORED AS ORC ;
> **Insert via Manual Hive Query:**
> INSERT INTO TABLE demo_user (1514133139123, 14, 26.4, 'pravin', true);
> **Insert via spark structured streaming code:**
> SparkConf conf = new SparkConf();
>  conf.setAppName("testing");
>  conf.setMaster("local[2]");
>  conf.set("hive.metastore.uris", "thrift://localhost:9083");
>  SparkSession session = 
>  SparkSession.builder().config(conf).enableHiveSupport().getOrCreate();
> // workaround START: code to insert static data into hive
>  String insertQuery = "INSERT INTO TABLE demo_user (1514133139123, 14, 26.4, 'pravin',
true)";
>  session.sql(insertQuery);
>  // workaround END:
> // Solution START
>  Dataset<Row> dataset = readFromKafka(sparkSession); // private method reading
data from Kafka's 'xyz' topic
> // **My question here:**
>  // some code which writes dataset into hive table demo_user
>  // Solution END



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message