Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of matthew.goeke@monsanto.com
 designates 164.144.240.27 as permitted sender)
From: "GOEKE, MATTHEW (AG/1000)" <matthew.goeke@monsanto.com>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>,
	"mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
CC: "GOEKE, MATTHEW (AG/1000)" <matthew.goeke@monsanto.com>
Subject: Issue with MR code not scaling correctly with data sizes
Thread-Topic: Issue with MR code not scaling correctly with data sizes
Thread-Index: AQHMQiZLp07usybDG0aMcJD6BtgK4pTsKEWAgAAzQxA=
Date: Thu, 14 Jul 2011 22:14:29 +0000
Message-ID: 
 <E236CD55EC618B4884CA7273B31AA07807480E25@stlwexmbxprd04.na.ds.monsanto.com>
References: <4E1EE8C6.70604@orkash.com>
 <D8C70ABBE12144A6907E063995BCD067@china.huawei.com>
In-Reply-To: <D8C70ABBE12144A6907E063995BCD067@china.huawei.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

All,

I have a MR program that I feed in a list of IDs and it generates the uniqu=
e comparison set as a result. Example: if I have a list {1,2,3,4,5} then th=
e resulting output would be {2x1, 3x2, 3x1, 4x3, 4x2, 4x1, 5x4, 5x3, 5x2, 5=
x1} or (n^2-n)/2 number of comparisons. My code works just fine on smaller =
scaled sets (I can verify less than 1000 fairly easily) but fails when I tr=
y to push the set to 10-20k IDs which is annoying when the end goal is 1-10=
 million.

The flow of the program is:
	1) Partition the IDs evenly, based on amount of output per value, into a s=
et of keys equal to the number of reduce slots we currently have
	2) Use the distributed cache to push the ID file out to the various reduce=
rs
	3) In the setup of the reducer, populate an int array with the values from=
 the ID file in distributed cache
	4) Output a comparison only if the current ID from the values iterator is =
greater than the current iterator through the int array

I realize that this could be done many other ways but this will be part of =
an oozie workflow so it made sense to just do it in MR for now. My issue is=
 that when I try the larger sized ID files it only outputs part of resultin=
g data set and there are no errors to be found. Part of me thinks that I ne=
ed to tweak some site configuration properties, due to the size of data tha=
t is spilling to disk, but after scanning through all 3 sites I am having i=
ssues pin pointing anything I think could be causing this. I moved from rea=
ding the file from HDFS to using the distributed cache for the join read th=
inking that might solve my problem but there seems to be something else I a=
m overlooking.

Any advice is greatly appreciated!

Matt
This e-mail message may contain privileged and/or confidential information,=
 and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, ple=
ase notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use=
 of this e-mail by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, re=
ading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checki=
ng for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage =
caused by any such code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export contro=
l laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) an=
d sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC).  As a recipient of this =
information you are obligated to comply with all
applicable U.S. export laws and regulations.