hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "GOEKE, MATTHEW (AG/1000)" <matthew.go...@monsanto.com>
Subject Issue with MR code not scaling correctly with data sizes
Date Thu, 14 Jul 2011 22:14:29 GMT

I have a MR program that I feed in a list of IDs and it generates the unique comparison set
as a result. Example: if I have a list {1,2,3,4,5} then the resulting output would be {2x1,
3x2, 3x1, 4x3, 4x2, 4x1, 5x4, 5x3, 5x2, 5x1} or (n^2-n)/2 number of comparisons. My code works
just fine on smaller scaled sets (I can verify less than 1000 fairly easily) but fails when
I try to push the set to 10-20k IDs which is annoying when the end goal is 1-10 million.

The flow of the program is:
	1) Partition the IDs evenly, based on amount of output per value, into a set of keys equal
to the number of reduce slots we currently have
	2) Use the distributed cache to push the ID file out to the various reducers
	3) In the setup of the reducer, populate an int array with the values from the ID file in
distributed cache
	4) Output a comparison only if the current ID from the values iterator is greater than the
current iterator through the int array

I realize that this could be done many other ways but this will be part of an oozie workflow
so it made sense to just do it in MR for now. My issue is that when I try the larger sized
ID files it only outputs part of resulting data set and there are no errors to be found. Part
of me thinks that I need to tweak some site configuration properties, due to the size of data
that is spilling to disk, but after scanning through all 3 sites I am having issues pin pointing
anything I think could be causing this. I moved from reading the file from HDFS to using the
distributed cache for the join read thinking that might solve my problem but there seems to
be something else I am overlooking.

Any advice is greatly appreciated!

This e-mail message may contain privileged and/or confidential information, and is intended
to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the
sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail
by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, reading and archival
by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence
of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such
code transmitted by or accompanying
this e-mail or any attachment.

The information contained in this email may be subject to the export control laws and regulations
of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations
issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC).  As a recipient of this information you
are obligated to comply with all
applicable U.S. export laws and regulations.

View raw message