spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <>
Subject Spark standalone hangup during shuffle flatMap or explode in cluster
Date Wed, 07 Oct 2015 17:23:20 GMT
When running stand-alone cluster mode job, the process hangs up randomly during a DataFrame
flatMap or explode operation, in HiveContext:

-->> df.flatMap(r => for (n <- 1 to r.getInt(ind)) yield r)

This does not happen either with SQLContext in cluster, or Hive/SQL in local mode, where it
works fine.

A couple minutes after the hangup, executors start dropping. I am attching the logs


View raw message