spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yin Huai <yh...@databricks.com>
Subject Re: [VOTE] Release Apache Spark 1.5.0 (RC2)
Date Fri, 28 Aug 2015 22:36:56 GMT
-1

Found a problem on reading partitioned table. Right now, we may create a
SQL project/filter operator for every partition. When we have thousands of
partitions, there will be a huge number of SQLMetrics (accumulators), which
causes high memory pressure to the driver and then takes down the cluster
(long GC time causes different kinds of timeouts).

https://issues.apache.org/jira/browse/SPARK-10339

Will have a fix soon.

On Fri, Aug 28, 2015 at 3:18 PM, Jon Bender <jonathan.bender@gmail.com>
wrote:

> Marcelo,
>
> Thanks for replying -- after looking at my test again, I misinterpreted
> another issue I'm seeing which is unrelated (note I'm not using a pre-built
> binary, rather had to build my own with Yarn/Hive support, as I want to use
> it on an older cluster (CDH5.1.0)).
>
> I can start up a pyspark app on YARN, so I don't want to block this.  +1
>
> Best,
> Jonathan
>
> On Fri, Aug 28, 2015 at 2:34 PM, Marcelo Vanzin <vanzin@cloudera.com>
> wrote:
>
>> Hi Jonathan,
>>
>> Can you be more specific about what problem you're running into?
>>
>> SPARK-6869 fixed the issue of pyspark vs. assembly jar by shipping the
>> pyspark archives separately to YARN. With that fix in place, pyspark
>> doesn't need to get anything from the Spark assembly, so it has no
>> problems running on YARN. I just downloaded
>> spark-1.5.0-bin-hadoop2.6.tgz and tried that out, and pyspark works
>> fine on YARN for me.
>>
>>
>> On Fri, Aug 28, 2015 at 2:22 PM, Jonathan Bender
>> <jonathan.bender@gmail.com> wrote:
>> > -1 for regression on PySpark + YARN support
>> >
>> > It seems like this JIRA
>> https://issues.apache.org/jira/browse/SPARK-7733
>> > added a requirement for Java 7 in the build process.  Due to some quirks
>> > with the Java archive format changes between Java 6 and 7, using PySpark
>> > with a YARN uberjar seems to break when compiled with anything after
>> Java 6
>> > (see https://issues.apache.org/jira/browse/SPARK-1920 for reference).
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-5-0-RC2-tp13826p13890.html
>> > Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: dev-help@spark.apache.org
>> >
>>
>>
>>
>> --
>> Marcelo
>>
>
>

Mime
View raw message