pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-4714) Improve logging across multiple components with callerId
Date Wed, 18 May 2016 21:20:12 GMT

     [ https://issues.apache.org/jira/browse/PIG-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Daniel Dai updated PIG-4714:
----------------------------
    Attachment: PIG-4714-3.patch

bq. 1) pig.ats.enabled=true is the default and this code affects PigServer.
With the checking of YarnConfiguration.TIMELINE_SERVICE_ENABLED first, can we turn on ATS
by default if Yarn timeline service is enabled? It is rare when Yarn timeline service is on
and user want to keep Pig silent.

bq. 8) public static final String CALLER_ID = "pig.log.trace.id";
bq. This is not being set anywhere. Will always be null
This would be set by upstream component such as Ooize/WebHCat as a cross-project protocol.
Null is Ok meaning Pig is invoked directly.

Attach new patch to address all other comments.

> Improve logging across multiple components with callerId
> --------------------------------------------------------
>
>                 Key: PIG-4714
>                 URL: https://issues.apache.org/jira/browse/PIG-4714
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>             Fix For: 0.16.0
>
>         Attachments: PIG-4714-1.patch, PIG-4714-2.patch, PIG-4714-3.patch
>
>
> The idea is to add callerId to every component, so we can track the chain of application
which cause the underlining operation. A typical chain is Oozie->Pig->Tez->Hdfs.
With proper callerId logging, we can trace Hdfs operation back to Oozie workflow which triggers
it.
> The protocol we decided is every component log its immediate callerId. 
> For Pig, this includes passing Pig script ID to underlining components as callerId using
component specific API, log callerId of Pig and store it on ATS. More specific, it includes:
> 1. Generate a CallerId for each Pig script, pass it to Hdfs/Yarn/MapReduce/Tez which
Pig invokes
> 2. Pig caller passes "pig.log.trace.id" to Pig, Pig will publish it to ATS



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message