spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <>
Subject Re: PySpark - Hive Context Does Not Return Results but SQL Context Does for Similar Query.
Date Wed, 14 Oct 2015 21:23:12 GMT
No link to the original stack overflow so I can up my reputation? :)

This is likely not a difference between HiveContext/SQLContext, but instead
a difference between a table where the metadata is coming from the
HiveMetastore vs the SparkSQL Data Source API.  I would guess that if you
create the table the same way, the performance would be similar.

In the data source API we have spent a fair amount of time optimizing the
discovery and handling of many partitions, and in general I would say this
path is easier to use / faster.

Likely the problem with the hive table, is downloading all of the partition
metadata from the metastore and converting it to our internal format.  We
do this for all partitions, even though in this case you only want the
first ~20 rows.

On Wed, Oct 14, 2015 at 1:38 PM, charles.drotar <> wrote:

> I have duplicated my submission to stack overflow below since it is exactly
> the same question I would like to post here as well. Please don't judge me
> too harshly for my laziness
> <
> >
> *The questions I am concerned with are the same ones listed in the
> "QUESTIONS" section namely:*
> */1) Has anyone noticed anything similar to this?
> 2) What is happening on the backend that could be causing this consumption
> of resources and what could I do to avoid it?/*
> --
> View this message in context:
> Sent from the Apache Spark User List mailing list archive at
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

View raw message