hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3863) Support complex filters in TimelineReader
Date Mon, 29 Feb 2016 15:35:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172005#comment-15172005
] 

Varun Saxena commented on YARN-3863:
------------------------------------

Latest patch fixes the comments given by Sangjin.

# I have split TestHBaseTimelineStorage into TestHBaseTimelineStorageApps and TestHBaseTimelineStorageEntities.
Also moved the part related to loading apps and entities to a separate class.
# Methods to match filters in TimelineStorageUtils have been refactored for better readability
and to club common code together. Have passed on enum to decide which filter we are trying
to match. Suggestions for better name of this enum ?
# For separation of logic between Generic and Application entity reader, I have refactored
methods which were exclusively in GenericEntityReader but were being used by ApplicationEntityReader.
Now relevant logic will exist both the classes(with EntityColumnPrefix used in GenericEntityReader
and ApplicationColumnPrefix used in ApplicationEntityReader). Have tried to move some of the
common code used in these methods to utils classes.
# I have also moved previously written methods in GenericEntityReader to read relations, events,etc.
to TimelineStorageUtils. These methods were being used by ApplicationEntityReader as well
in addition to GenericEntityReader.
# Refactored createSingleColValueFiltersByRange() and  createHBaseSingleColValueFilter() so
that createSingleColValueFiltersByRange can call createHBaseSingleColValueFilter().
# Fixed javadoc related comments and made members final in the classes pointed out.
# Removed preconditions check for filters not being null. Now if filters are null, I create
TimelineEntityFilters object with default values in augmentParams.
# Used == instead of equals to match enums.
# Changed name of TimelineEqualityFilter and TimelineMultiValEqualityFilter to TimelineKeyValueFilter
and TimelineKeyValuesFilter respectively.
# There was a comment on why getCompoundColQualBytes is being used. I had missed using it
in previous patch. It is to be used for events. If say event to be fetched is UPDATE_APP with
associated info as info1 as key, ts as timestamp and val1 as value, then the column is of
the form {{e!UPDATE_APP=ts=info1}}. This kind of column within code is referred to as a compound
column. The part after the prefix {{e!}} i.e part with {{=}} as separator is what we want
to construct with getCompoundColQualBytes.
So, if we have to match event filters, the qualifier filter will have to be matched with prefix
{{e!UPDATE_APP=}}. The getCompoundColQualBytes bit for event filters comes in here.
# For the comments for TimelineReaderWebServicesUtils.java, kindy refer to the explanation
in the comment in which I have detailed what I have done. The changes here are required.

I could not quite get below comment. I did not make any change on line 448. Sangjin, can you
elaborate. Maybe you meant some other line.
{quote}
(HBaseTimelineWriterImpl.java)
l.448: it should simply be a else if
{quote}

cc [~sjlee0], [~djp]
As the patch is quite big, refer to comment https://issues.apache.org/jira/browse/YARN-3863?focusedCommentId=15169833&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15169833
to get details of what has been implemented.

> Support complex filters in TimelineReader
> -----------------------------------------
>
>                 Key: YARN-3863
>                 URL: https://issues.apache.org/jira/browse/YARN-3863
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: YARN-2928
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>              Labels: yarn-2928-1st-milestone
>         Attachments: YARN-3863-YARN-2928.v2.01.patch, YARN-3863-YARN-2928.v2.02.patch,
YARN-3863-YARN-2928.v2.03.patch, YARN-3863-feature-YARN-2928.wip.003.patch, YARN-3863-feature-YARN-2928.wip.01.patch,
YARN-3863-feature-YARN-2928.wip.02.patch, YARN-3863-feature-YARN-2928.wip.04.patch, YARN-3863-feature-YARN-2928.wip.05.patch
>
>
> Currently filters in timeline reader will return an entity only if all the filter conditions
hold true i.e. only AND operation is supported. We can support OR operation for the filters
as well. Additionally as primary backend implementation is HBase, we can design our filters
in a manner, where they closely resemble HBase Filters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message