falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-221) Logmover is not copying all action level logs
Date Tue, 28 Jan 2014 09:30:38 GMT

    [ https://issues.apache.org/jira/browse/FALCON-221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13883943#comment-13883943
] 

Srikanth Sundarrajan commented on FALCON-221:
---------------------------------------------

These are my observations so far (specific to hadoop-1).

When JT retires the job when the user limit exceeds, JobInProgress object is stripped off
all the TaskCompletionEvents, at which point It is still possible to retrieve RunningJob handle
for a job, however the TaskCompletionEvents are empty. Subsequently the job is moved out of
retired cache, when the RunningJob is no longer available.

Falcon needs to consider the possibility of a workflow that runs for a long duration resulting
in scenarios where RunningJob / TaskCompletionEvents are inaccessible.

Proposed fix: We can use the job's tracking url which is active for a much longer duration
(where JT auto-redirects with HTTP 302 to history file), parse the contents of history and
move the contents of the task logs.

I am working on a patch along these lines. If there are any concerns please do chime in.

> Logmover is not copying all action level logs
> ---------------------------------------------
>
>                 Key: FALCON-221
>                 URL: https://issues.apache.org/jira/browse/FALCON-221
>             Project: Falcon
>          Issue Type: Bug
>          Components: archival
>    Affects Versions: 0.3
>            Reporter: Pracheer Agarwal
>            Priority: Minor
>
> Log mover copies the action level logs and oozie logs of a worklfow to hdfs. My workflow
has 6 actions. Logs of 2-3 actions are getting copied to hdfs. Logs for all the actions are
not available at hdfs.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message