falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-2051) Falcon post-processing services are not getting invoked
Date Wed, 29 Jun 2016 04:59:45 GMT

    [ https://issues.apache.org/jira/browse/FALCON-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354540#comment-15354540
] 

Pallavi Rao commented on FALCON-2051:
-------------------------------------

[~me.venkatr], we were actually planning to get rid of post-processing altogether and move
any logic there in, as a service on Falcon Server. The reasons are as follows:
1. If the user workflow succeeds and succeeded-post-processing fails or hangs, the whole instance
gets flagged as failed and retries kick in. We have quite many such instances where there
is unnecessary delay because of post-processing failing.
2. post-processing takes up one container on YARN which can be avoided.

Two things that are happening in post-processing apart from notification:
1. The Hadoop counters - These are not directly from Hadoop. It expects a counter.txt in the
LogDir. I think this is broken as of now. Correct me if I'm wrong.
2. Log archival - Move oozie and task logs under staging dir. This logic can be easily moved
to Server side (listen to notification and do a hdfs copy).

Given this, lets not continue to depend on post processing. Thoughts?



> Falcon post-processing services are not getting invoked
> -------------------------------------------------------
>
>                 Key: FALCON-2051
>                 URL: https://issues.apache.org/jira/browse/FALCON-2051
>             Project: Falcon
>          Issue Type: Bug
>          Components: general
>    Affects Versions: 0.9
>            Reporter: Peeyush Bishnoi
>            Assignee: Venkatesan Ramachandran
>            Priority: Blocker
>             Fix For: 0.10
>
>
> With latest Falcon, none of the post-processing services like metadata, late data handling,
retry is getting invoked. Hence created a blocker issue to fix this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message