hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4696) EntityGroupFSTimelineStore to work in the absence of an RM
Date Tue, 08 Mar 2016 17:08:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15185261#comment-15185261
] 

Steve Loughran commented on YARN-4696:
--------------------------------------

BTW, I managed to create an exception in scale tests; when the file:// fs did eventually flush,
a checksum exception was raised on the read
{code}
2016-03-08 12:01:21,991 [EntityLogPluginWorker #2] INFO  org.apache.hadoop.fs.FSInputChecker
(FSInputChecker.java:readChecksumChunk(308)) - Found checksum error: b[1802, 1802]=
org.apache.hadoop.fs.ChecksumException: Checksum error: file:/Users/stevel/Projects/Hortonworks/Projects/sparkwork/spark-timeline-integration/target/tmp/integration/ats/active/application_1111_0000/appattempt_1111_0000_000000/summarylog-appattempt_1111_0000_000000_1457438447660
at 13106176
	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:258)
	at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:299)
	at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:251)
	at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
	at java.io.DataInputStream.read(DataInputStream.java:149)
	at org.codehaus.jackson.impl.Utf8StreamParser.loadMore(Utf8StreamParser.java:172)
	at org.codehaus.jackson.impl.Utf8StreamParser._decodeEscaped(Utf8StreamParser.java:2515)
	at org.codehaus.jackson.impl.Utf8StreamParser._finishString2(Utf8StreamParser.java:1949)
	at org.codehaus.jackson.impl.Utf8StreamParser._finishString(Utf8StreamParser.java:1905)
	at org.codehaus.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276)
	at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:59)
	at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.mapObject(UntypedObjectDeserializer.java:218)
	at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:47)
	at org.codehaus.jackson.map.deser.std.MapDeserializer._readAndBind(MapDeserializer.java:319)
	at org.codehaus.jackson.map.deser.std.MapDeserializer.deserialize(MapDeserializer.java:263)
	at org.codehaus.jackson.map.deser.std.MapDeserializer.deserialize(MapDeserializer.java:33)
	at org.codehaus.jackson.map.deser.SettableBeanProperty$SetterlessProperty.deserializeAndSet(SettableBeanProperty.java:511)
	at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:697)
	at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580)
	at org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:217)
	at org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:194)
	at org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:30)
	at org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:299)
	at org.codehaus.jackson.map.deser.SettableBeanProperty$MethodProperty.deserializeAndSet(SettableBeanProperty.java:414)
	at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:697)
	at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580)
	at org.codehaus.jackson.map.MappingIterator.nextValue(MappingIterator.java:178)
	at org.codehaus.jackson.map.MappingIterator.next(MappingIterator.java:111)
	at org.apache.hadoop.yarn.server.timeline.EntityLogInfo.doParse(LogInfo.java:189)
	at org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:153)
	at org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:112)
	at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore$AppLogs.parseSummaryLogs(EntityGroupFSTimelineStore.java:589)
	at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore$AppLogs.parseSummaryLogs(EntityGroupFSTimelineStore.java:565)
	at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore$ActiveLogParser.run(EntityGroupFSTimelineStore.java:752)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
{code}

> EntityGroupFSTimelineStore to work in the absence of an RM
> ----------------------------------------------------------
>
>                 Key: YARN-4696
>                 URL: https://issues.apache.org/jira/browse/YARN-4696
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: YARN-4696-001.patch, YARN-4696-002.patch, YARN-4696-003.patch, YARN-4696-005.patch,
YARN-4696-006.patch, YARN-4696-007.patch, YARN-4696-008.patch, YARN-4696-009.patch, YARN-4696-010.patch,
YARN-4696-012.patch
>
>
> {{EntityGroupFSTimelineStore}} now depends on an RM being up and running; the configuration
pointing to it. This is a new change, and impacts testing where you have historically been
able to test without an RM running.
> The sole purpose of the probe is to automatically determine if an app is running; it
falls back to "unknown" if not. If the RM connection was optional, the "unknown" codepath
could be called directly, relying on age of file as a metric of completion
> Options
> # add a flag to disable RM connect
> # skip automatically if RM not defined/set to 0.0.0.0
> # disable retries on yarn client IPC; if it fails, tag app as unknown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message