hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karam Singh (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3831) RM scalability runtime is worse than 0.20.204 by 14.2%
Date Tue, 07 Feb 2012 14:40:59 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202431#comment-13202431
] 

Karam Singh commented on MAPREDUCE-3831:
----------------------------------------

Settings for For RM scalability comparison  :- 
GridMix settings were for both Hadoop-0.20.204 and Hadoop-0.23 :

{code}
gridmix.client.submit.threads=10
gridmix.submit.multiplier=0.01
gridmix.client.pending.queue.depth=100
gridmix.job.type=SLEEPJOB
gridmix.sleep.maptask-only=false
gridmix.sleep.max-map-time=300000
gridmix.sleep.max-reduce-tim=300000
gridmix.sleep.fake-locations=3
gridmix.distributed-cache-emulation.enable=falseemulation.enable>
gridmix.compression-emulation.enable=false
{code}

>From JobHistroy Parsing and GridMix client found that :
	Runtime (seconds): 2473
	GridMix Simulation Time Spent: 41mins 8sec
	Workflow End: 2046 (From histroy parsing)

While looking at GridMix log and JobHistory files :
	1. Found that according to gm client last was completed was 
	{code}
	12/02/07 08:32:26 INFO gridmix.JobMonitor: GRIDMIX000029 job_1328600848949_1182) success.
	Whereas the 
	EndTime of Job is : 1328602818684 Tue, 07 Feb 2012 08:20:18
	{code}

	Which means somehow GridMix got Job completion event 12 minutes after the the actual job
got completed .

	2. Similarly acc. to JobHistory last Jo completed was :
		{code}
		job_1328600848949_1162: 1328603121882 Tue, 07 Feb 2012 08:25:21
		Whereas according to GridMix client log:
		12/02/07 08:32:08 INFO gridmix.JobMonitor: GRIDMIX000029 (job_1328600848949_1162) success
	{code}
	Which again means GridMix got job completion event nearly 7 minutes after the actual job
got finished. 

Whereas this problem does not exists with Hadoop-0.20.204
Seems that, Somehow in Hadoop-0.23, GridMix is getting job completion events long after the
actual job getting completed
                
> RM scalability runtime is worse than 0.20.204 by 14.2%
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-3831
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3831
>             Project: Hadoop Map/Reduce
>          Issue Type: Task
>          Components: benchmarks, resourcemanager
>    Affects Versions: 0.23.1
>            Reporter: Vinay Kumar Thota
>
> RM scalability runtime is worse than 0.20.204 by 14.2%
> Overall runtime against Fred in a 350 nodes cluster is 2155 secs.
> Overall runtime against .23 in 350 nodes cluster is 2462 secs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message