hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4705) ATS 1.5 parse pipeline to consider handling open() events recoverably
Date Fri, 19 Feb 2016 19:02:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15154667#comment-15154667
] 

Steve Loughran commented on YARN-4705:
--------------------------------------

stack attached.

Looking at the code, I'm not sure what the retry policy here should be. FNFE, auth failures?
recoverable or not? it's hard to predict.

{code}
2016-02-19 18:46:32,641 [EntityLogPluginWorker #1] WARN  org.apache.hadoop.fs.FSInputChecker
(ChecksumFileSystem.java:<init>(157)) - Problem opening checksum file: file:/Users/stevel/spark-timeline-integration/target/tmp/ats/active/application_1111_0000/appattempt_1111_0000_000000/summarylog-appattempt_1111_0000_000000.
 Ignoring exception: 
java.io.EOFException
	at java.io.DataInputStream.readFully(DataInputStream.java:197)
	at java.io.DataInputStream.readFully(DataInputStream.java:169)
	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:149)
	at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:775)
	at org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:130)
	at org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:110)
	at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore$AppLogs.parseSummaryLogs(EntityGroupFSTimelineStore.java:584)
	at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore$AppLogs.parseSummaryLogs(EntityGroupFSTimelineStore.java:560)
	at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore$ActiveLogParser.run(EntityGroupFSTimelineStore.java:747)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
2016-02-19 18:46:32,688 [EntityLogPluginWorker #1] INFO  org.apache.hadoop.yarn.server.timeline.LogInfo
(LogInfo.java:parseForStore(112)) - Parsed 0 entities from file:/Users/stevel/Projects/spark-timeline-integration/target/tmp/ats/active/application_1111_0000/appattempt_1111_0000_000000/summarylog-appattempt_1111_0000_000000
in 50 msec
2016-02-19 18:46:32,689 [EntityLogPluginWorker #1] DEBUG org.apache.hadoop.yarn.server.timel
{code}

> ATS 1.5 parse pipeline to consider handling open() events recoverably
> ---------------------------------------------------------------------
>
>                 Key: YARN-4705
>                 URL: https://issues.apache.org/jira/browse/YARN-4705
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: timelineserver
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> During one of my own timeline test runs, I've been seeing a stack trace warning that
the CRC check failed in Filesystem.open() file; something the FS was ignoring.
> Even though its swallowed (and probably not the cause of my test failure), looking at
the code in {{LogInfo.parsePath()}} that it considers a failure to open a file as unrecoverable.

> on some filesystems, this may not be the case, i.e. if its open for writing it may not
be available for reading; checksums maybe a similar issue. 
> Perhaps a failure at open() should be viewed as recoverable while the app is still running?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message