spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: thriftserver: access temp dataframe from in-memory of spark-shell
Date Wed, 14 Oct 2015 17:17:49 GMT
Yes, call startWithContext from the spark shell:
https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala#L56

On Wed, Oct 14, 2015 at 7:10 AM, <Saif.A.Ellafi@wellsfargo.com> wrote:

> Hi,
>
> Is it possible to load a spark-shell, in which we do any number of
> operations in a dataframe, then register it as a temporary table and get to
> see it through thriftserver?
> ps. or even better, submit a full job and store the dataframe in
> thriftserver in-memory before the job completes.
>
> I have been trying this without success, bee does not see the dataframes
> of the spark-shell’s hive context.
> If any of you confirms this possibility, I will try further ahead. So far
> it only seems to be able to manually read from persistent tables.
>
> Thanks for any insights,
> Saif
>
>

Mime
View raw message