hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4447) Provide a mechanism to represent complex filters and parse them at the REST layer
Date Wed, 27 Apr 2016 18:25:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15260663#comment-15260663

Varun Saxena commented on YARN-4447:

[~sjlee0], kindly review.
Apologise for the delay in updating this patch. I was away for a week and was busy with internal
work earlier so could not work on it.
I will add a few more test cases to test application fetching flow. But the main code can
be reviewed anyways.

Will give a small description of what has been done.
# Metric filters are of the form {{(((metric1 gt 23) AND (metric2 lt 40)) OR (metric5 eq 40))}}.
Comparison operators supported are as follows : 
#* gt(Greater than) / ge(Greater than or equals)
#* lt(Less than) / le(Less than or equals)
#* eq(Equals) / ne(Not equals. If key(metric/config/info) does not exist, this would mean
a match) / ene(Exists and not equals. Key must exist).
# Config and info filters are of the same form as metric filters except that only eq,ne and
ene comparison operators are supported for them.
# Event filters will take the form {{(((event1,event2) AND \!(event3,event4)) OR event5, event6)}}.
This means for an entity to match, event1 and event2 should exist and event3 and event4 should
exist. Or, event5 and event6 should exist. ! indicates not equals(non-existence). A not(\!)
should be followed by an opening bracket i.e. (
# Relation filters will take the same form as event filters. Here instead of each event we
will have type-entities expression i.e. of the form {{type1:entity1:entity2}}. This part of
the expression cannot contain spaces. Relation filters hence would be of the form {{type1:entity11:entity12
, type2:entity21 AND !(type3:entity31)}}.
Also, ene kind of case wont be supported here. If entity type does not exist, match wont occur.

# Metrics to retrieve and configs to retrieve will have similar format to above. However ANDs'
and ORs' do not make much sense here. Hence an expression of the form conf1,conf2 means return
configs conf1 and conf2. And expression of the form \!(conf1,conf2) means returns all configs
other than conf1 and conf2.
# Pls note that metric filters have not yet been supported for flow runs. They need to be
matched locally as summation for metrics happens in coprocessor. Can be done in a separate

Now coming to implementation, I had 2 options. To implement parsing logic in static methods
or encapsulate this logic in a class. Went with latter as it makes it easier to break code
into multiple methods without a need to pass several parameters to helper methods. And makes
the code cleaner IMO. This would mean that an extra object will have to be created everytime
though. Would like to know thoughts of others on the approach.
# There is a new interface named {{TimelineParser}} added which needs to be implemented for
parsing different expressions.
# There are 2 abstract classes added namely TimelineParserForCompareExpr(for expressions of
the form explained above for metric/config/info filters) and TimelineParserForEqualityExpr(for
expressions of the form explained above for event/relation filters). These classes will have
abstract methods, which will be implemented by concrete implementations, for deciding what
kind of filter needs to be constructed for filter list, how to parse the values and how to
set value, compareop, etc. to the filters.
# These abstract classes will then have concrete implementation for different filters. These
include TimelineParserForNumericFilters(for metric filters), TimelineParserForKVFilters(for
config/info filters), TimelineParserForExistFilters(used for filters which check for existence
such as event filters) and TimelineParserForRelationFilters(for relation filters).
# Some code between TimelineParserForCompareExpr and TimelineParserForEqualityExpr is similar.
It can be moved to another base abstract class. But this might make the code confusing. So
have left it as it is. Would like to know thoughts of others on this.

> Provide a mechanism to represent complex filters and parse them at the REST layer 
> ----------------------------------------------------------------------------------
>                 Key: YARN-4447
>                 URL: https://issues.apache.org/jira/browse/YARN-4447
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>              Labels: yarn-2928-1st-milestone
>         Attachments: YARN-4447-YARN-2928.01.patch

This message was sent by Atlassian JIRA

View raw message