hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vrushali C (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3031) [Storage abstraction] Create backing storage write interface for ATS writers
Date Wed, 25 Feb 2015 05:54:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336054#comment-14336054
] 

Vrushali C commented on YARN-3031:
----------------------------------

Hi [~zjshen],

Thanks for the prompt review, appreciate it! These are very good points you mention, let me
add some more context around why these are coded this way right now.

1. The reasoning behind having two more apis for writing metrics and events in addition to
the entity write is that, it would be good (efficient) to have the option to write a single
metric or a single event. For example, say a job has many custom metrics and one particular
metric is updated extremely frequently but not the others. We may want to write out only that
particular metric without having to look through/write all other metrics and other information
in that entity. Similarly for events. Perhaps we could do it differently that what is proposed
in the patch, but the functionality of writing them individually would help in performance
I believe. 

2. Having a separate write and aggregator API makes them independent of the order in which
the entity details and aggregation are invoked/stored and makes them independent of each other.
For instance, we may choose to invoke the aggregation at a different frequency (more slower)
than the regular entity writes. Hence two apis. 

3. The TimelineServiceWriteResponse has two error codes presently: NO_START_TIME and  IO_EXCEPTION.
We can of course add in more error codes as we proceed.  The reason I chose these two for
now is that each flow is inherently associated with a submit timestamp (run id of that flow).
In case, we don’t find that timestamp, it would be difficult to write the flow information
for that run to the store - I think an error should be thrown with an error code. The other
one, IO_EXCEPTION is what I thought would help accounting for write/put errors to the store-
we should be able to indicate that the write did not go through. We can rename these if these
names don’t sound meaningful. 

thanks
Vrushali

> [Storage abstraction] Create backing storage write interface for ATS writers
> ----------------------------------------------------------------------------
>
>                 Key: YARN-3031
>                 URL: https://issues.apache.org/jira/browse/YARN-3031
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Vrushali C
>         Attachments: Sequence_diagram_write_interaction.2.png, Sequence_diagram_write_interaction.png,
YARN-3031.01.patch, YARN-3031.02.patch
>
>
> Per design in YARN-2928, come up with the interface for the ATS writer to write to various
backing storages. The interface should be created to capture the right level of abstractions
so that it will enable all backing storage implementations to implement it efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message