hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3904) Refactor timelineservice.storage to add support to online and offline aggregation writers
Date Thu, 30 Jul 2015 23:12:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14648457#comment-14648457
] 

Zhijie Shen commented on YARN-3904:
-----------------------------------

[~gtCarrera9], thanks for the patch. Bellow are my comments:

bq. The two failed tests passed on my local machine, and the failures appeared to be irrelevant.
This said, we may still need to fix those intermittent test failures.

Do we plan to fix it in this patch?

Some high level comments:

1. As is also mentioned in YARN-3049, how about we refactoring reader/writer method signature
in a separate jira to avoid conflicts?

2.  I suggest moving the table creation stuff into TimelineSchemaCreator.

3. As HBase backend is accessed both directly and via Phoenix, it's good for us to cleanup
the configuration to say we're using the HBase backend (comparing to FS backend) instead of
specifically HBase or Phoenix writer/reader.

Other patch details:

1. Make OfflineAggregationWriter extend Service, such that you don't need to define init.

2. Now we're working towards a production standard patch. Would you please write some javadoc
to explain the schema of the aggregation tables like what we did for HBase tables.

3. The connection config should be moved to YarnConfiguration.

4. Why is info column family kept? I expect the aggregation table will only have metrics data

5. Let's also have a default PhoenixOfflineAggregationWriterImpl constructor to be used in
the production code.

6. {{Class.forName(DRIVER_CLASS_NAME);}} doesn't  need to be invoked every time we get a connection.

> Refactor timelineservice.storage to add support to online and offline aggregation writers
> -----------------------------------------------------------------------------------------
>
>                 Key: YARN-3904
>                 URL: https://issues.apache.org/jira/browse/YARN-3904
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: YARN-3904-YARN-2928.001.patch, YARN-3904-YARN-2928.002.patch, YARN-3904-YARN-2928.003.patch,
YARN-3904-YARN-2928.004.patch, YARN-3904-YARN-2928.005.patch
>
>
> After we finished the design for time-based aggregation, we can adopt our existing Phoenix
storage into the storage of the aggregated data. In this JIRA, I'm proposing to refactor writers
to add support to aggregation writers. Offline aggregation writers typically has less contextual
information. We can distinguish these writers by special naming. We can also use CollectorContexts
to model all contextual information and use it in our writer interfaces. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message