spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From freedafeng <freedaf...@yahoo.com>
Subject Re: What does NativeMethodAccessorImpl.java do?
Date Thu, 13 Aug 2015 20:12:14 GMT
Thanks Marcelo! 

The reason I was asking that question is that I was expecting my spark job
to be a "map only" job. In other words, it should finish after the
mapPartitions run for all partitions. This is because the job is only
mapPartitions() plus count() where mapPartitions only yield one integer for
each partition. The first stage running "count at
/root/workspace/**/mapred/aerospike_calculations.py:35" completed after
reasonably long time. I was expecting the job to complete right away after
the first stage is complete. To my surprise, the second stage calling
"collect at NativeMethodAccessorImpl.java:-2" runs super slow, about as slow
as the first stage. 

I want to know what the second stage is doing..

================================UI============================
Spark Stages
Total Duration: 8.2 h
Scheduling Mode: FIFO
Active Stages: 1
Completed Stages: 2
Failed Stages: 0
Active Stages (1)

Stage Id	Description	Submitted	Duration	Tasks: Succeeded/Total	Input	Shuffle
Read	Shuffle Write
2	
(kill) collect at NativeMethodAccessorImpl.java:-2 +details
2015/08/13 16:01:59	4.1 h	
360/2048	375.1 GB		
Completed Stages (2)

Stage Id	Description	Submitted	Duration	Tasks: Succeeded/Total	Input	Shuffle
Read	Shuffle Write
1	
count at /root/workspace/**/aerospike_calculations.py:35
2015/08/13 12:02:40	7.5 h	
2048/2048	1785.6 GB		
0	
first at SerDeUtil.scala:70 +details
2015/08/13 12:02:34	4 s	
1/1	839.0 MB		
Failed Stages (0)

Stage Id	Description	Submitted	Duration	Tasks: Succeeded/Total	Input	Shuffle
Read	Shuffle Write	Failure Reason




--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/What-does-NativeMethodAccessorImpl-java-do-tp13667p13684.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message