hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3971) Job History web services need to have limits on the number of itmes they can return.
Date Thu, 08 Mar 2012 14:47:57 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225231#comment-13225231
] 

Robert Joseph Evans commented on MAPREDUCE-3971:
------------------------------------------------

Yes you are correct about the implementation of the /mapreduce/jobs API up until MAPREDUCE-3944.
 MAPREDUCE-3944 changed it so that /mapreduce/jobs will only return the equivalent of a PartialJob,
not a CompletedJob.  This reduces the load a lot, and is hopefully a temporary measure until
more issues can be addressed as part of MAPREDUCE-3973.

I am not sure of the best way to implement this, because it is something that we want to be
able to do consistently across APIs if need be.  But, it will probably be something like keeping
the limit that is already in place and adding in a start parameter as well.  We will probably
also need to add in fields to the returned list indicating the start index and end index of
the items returned, and possibly something else indicating the total number of objects that
can be returned.

The maximum limit would be in place regardless of what the limit is set in the URL.  

The real difficultly with this is that we now need to be able to maintain a guaranteed consistent
ordering of the values returned or the start will be useless.
                
> Job History web services need to have limits on the number of itmes they can return.
> ------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3971
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: mrv2
>    Affects Versions: 0.23.2
>            Reporter: Robert Joseph Evans
>
> The Job History web services canput a very large load on the job history server.  We
should put in a limit on the number of entries that can be returned by the web service, and
also add in the ability to modify the starting location in the list, so that all entries can
still be downlaoded.  Just not all at once.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message