Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
DomainKey-Status: good
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=G5IdC34n0+K9HVc3H7jiVzo2RSKBARl+LjowRWCPX4fN2OeBLGbAGXiIVcL4Eg9E
Message-ID: <451CC957.4040505@yahoo-inc.com>
Date: Fri, 29 Sep 2006 00:20:55 -0700
From: Benjamin Reed <breed@yahoo-inc.com>
User-Agent: Thunderbird 1.5.0.7 (X11/20060922)
MIME-Version: 1.0
To: hadoop-dev@lucene.apache.org
Subject: Re: Creating splits/tasks at the client
References: <451B820E.6080101@yahoo-inc.com> <451C10D8.8040800@apache.org>
In-Reply-To: <451C10D8.8040800@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

I please correct me if I'm reading the code incorrectly, but it seems
like submitJob puts the submitted job on the jobInitQueue which is
immediately dequeued by the JobInitThread and then initTasks() will get
the file splits and create Tasks. Thus, it doesn't seem like there is
any difference in memory foot print.

ben

Doug Cutting wrote:
>
> Right, so JobSubmissionProtocol.submitJob(String jobFile) could be
> altered to be submitJob(StringJobFile, Split[]).  The RPC system can
> handle reasonably large values like this, so I don't think that would
> be a problem.  But the memory impact on the JobTracker could become
> significant, since the splits for queued jobs would now be around. 
> This could be mitigated by writing the splits to a temporary file.
>
> The semantics would be subtly different: if you queue a job now, the
> file listing is done just before the job is executed, not when its
> submitted.  But programs shouldn't rely on that, so I don't think this
> is a big worry.
>
> Overall, I don't see any major problems with this.  It won't simplify
> things much.  We can remove the code which computes splits in a
> separate thread, but we'd have to add code to store splits to
> temporary files, so codesize is a wash.  And it would remove a
> potential reliability problem.