spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewan Higgs <>
Subject Terasort example
Date Tue, 11 Nov 2014 13:03:40 GMT
Hi all,
I saw that Reynold Xin had a Terasort example PR on Github[1]. It didn't 
appear to be similar to the Hadoop Terasort example, so I've tried to 
brush it into shape so it can generate Terasort files (teragen), sort 
the files (terasort) and validate the files (teravalidate). My branch is 
available here:

With this code, you can run the following:

# Generate 1M 100 byte records:
  ./bin/run-example terasort.TeraGen 100M ~/data/terasort_in

# Sort the file:
MASTER=local[4] ./bin/run-example terasort.TeraSort ~/data/terasort_in  

# Validate the file
MASTER=local[4] ./bin/run-example terasort.TeraValidate 
~/data/terasort_out  ~/data/terasort_validate

# Validate that an unsorted file is indeed not correctly sorted:

MASTER=local[4] ./bin/run-example terasort.TeraValidate 
~/data/terasort_in  ~/data/terasort_validate_bad

This matches the interface for the Hadoop version of Terasort, except I 
added the ability to use K,M,G,T for record sizes in TeraGen. This code 
therefore makes a good example of how to use Spark, how to read and 
write Hadoop files, and also a way to test some of the performance 
claims of Spark.

 > That's great, but why is this on the mailing list and not submitted 
as a PR?

I suspect there are some rough edges and I'd really appreciate reviews. 
I would also like to know if others can try it out on clusters and tell 
me if it's performing as it should.

For example, I find it runs fine on my local machine, but when I try to 
sort 100G of data on a cluster of 16 nodes, I get >2900 file splits. 
This really eats into the sort time.

Another issue is that in TeraValidate, to work around SPARK-1018 I had 
to clone each element. Does this /really/ need to be done? It's pretty lame.

In any event, I know the Spark 1.2 merge window closed on Friday but as 
this is only for the examples directory maybe we can slip it in if we 
can bash it into shape quickly enough?

Anyway, thanks to everyone on #apache-spark and #scala who helped me get 
through learning some rudimentary Scala to get this far.

Ewan Higgs


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message