hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bhallamudi Venkata Siva Kamesh (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3971) Job History web services need to have limits on the number of itmes they can return.
Date Sat, 10 Mar 2012 18:30:58 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226923#comment-13226923
] 

Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-3971:
-----------------------------------------------------------

Hi Robert,
 Thanks for clarifying. Yes, [MAPREDUCE-3944|https://issues.apache.org/jira/browse/MAPREDUCE-3944]
eliminated JobHistory.getJob(JobId) invocation for each job and replaced CompletedJob with
PartialJob.

As I observed in the JobHistory#getAllJobsInternal uses a TreeMap to store jobId and its corresponding
PartialJob. But, *intermediateListCache* & *jobListCache* are implementations of SortedMap,
so the iterator returns jobs from most recent to least recent. Again we are storing these
Jobs in a TreeMap. So each put opearation takes O(lgn) time. Where as  put operatin of LinkedHashMap
takes O(1) time and guareentes retrival of the elements as per their insertion order so I
*think* we can replace TreeMap with LinkedHashMap.  Or is there anything am I missing?
                
> Job History web services need to have limits on the number of itmes they can return.
> ------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3971
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: mrv2
>    Affects Versions: 0.23.2
>            Reporter: Robert Joseph Evans
>
> The Job History web services canput a very large load on the job history server.  We
should put in a limit on the number of entries that can be returned by the web service, and
also add in the ability to modify the starting location in the list, so that all entries can
still be downlaoded.  Just not all at once.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message