hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3904) Refactor timelineservice.storage to add support to online and offline aggregation writers
Date Tue, 04 Aug 2015 00:41:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14652862#comment-14652862

Sangjin Lee commented on YARN-3904:

I agree it is appealing to centralize table creations. After putting some thoughts here I
think what we really want is a centralized workflow for storage schema creations. That is
to say, when setting up a v2 timeline server, users can simply run data schema creator for
once to create necessary data storage schemas. With this in mind, I added Phoenix schema creation
into the existing data schema creator, with a separate option -p. However, I'm keeping the
SQL statements for table creation inside the writer file so that we also have a centralized
place for the Phoenix storage schema.

I'm fine with that approach.

We can definitely reuse this PreparedStatement (as well as the connections) after we integrated
the aggregation writer with the aggregator. My plan is to use this (relatively) stable writer
to unblock the future patch on flow and user level offline aggregation. After we have the
whole workflow, we can gradually add optimizations. Thoughts?

Yes, that sounds fine. Thanks!

> Refactor timelineservice.storage to add support to online and offline aggregation writers
> -----------------------------------------------------------------------------------------
>                 Key: YARN-3904
>                 URL: https://issues.apache.org/jira/browse/YARN-3904
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: YARN-3904-YARN-2928.001.patch, YARN-3904-YARN-2928.002.patch, YARN-3904-YARN-2928.003.patch,
YARN-3904-YARN-2928.004.patch, YARN-3904-YARN-2928.005.patch, YARN-3904-YARN-2928.006.patch,
> After we finished the design for time-based aggregation, we can adopt our existing Phoenix
storage into the storage of the aggregated data. In this JIRA, I'm proposing to refactor writers
to add support to aggregation writers. Offline aggregation writers typically has less contextual
information. We can distinguish these writers by special naming. We can also use CollectorContexts
to model all contextual information and use it in our writer interfaces. 

This message was sent by Atlassian JIRA

View raw message