spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Spark standalone hangup during shuffle flatMap or explode in cluster
Date Wed, 07 Oct 2015 17:37:51 GMT
-dev

Is r.getInt(ind) very large in some cases? I think there's not quite
enough info here.

On Wed, Oct 7, 2015 at 6:23 PM,  <Saif.A.Ellafi@wellsfargo.com> wrote:
> When running stand-alone cluster mode job, the process hangs up randomly
> during a DataFrame flatMap or explode operation, in HiveContext:
>
> -->> df.flatMap(r => for (n <- 1 to r.getInt(ind)) yield r)
>
> This does not happen either with SQLContext in cluster, or Hive/SQL in local
> mode, where it works fine.
>
> A couple minutes after the hangup, executors start dropping. I am attching
> the logs
> Saif
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message