spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: Merging two Spark SQL tables?
Date Mon, 25 Aug 2014 19:49:44 GMT
>
> SO I tried the above (why doesn't union or ++ have the same behavior
> btw?)


I don't think there is a good reason for this.  I'd open a JIRA.


> and it works, but is slow because the original Rdds are not
> cached and files must be read from disk.
>
> I also discovered you can recover the InMemoryCached versions of the
> Rdds using sqlContext.table("table1").
>

Yeah, this is an unfortunate consequence of the way we handle caching.
 I've opened this JIRA for the 1.2 roadmap:
https://issues.apache.org/jira/browse/SPARK-3212

Mime
View raw message