hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-7052) TestFixedLengthInputFormat#testFormatCompressedIn is flaky
Date Wed, 14 Feb 2018 18:26:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364557#comment-16364557
] 

Jason Lowe commented on MAPREDUCE-7052:
---------------------------------------

Thanks for the report and the patch!  The unit test is unrelated, I filed MAPREDUCE-7053 to
track the failure which was introduced by MAPREDUCE-5124.  You may be interested in following
that.

I think the fixup is appropriate given the record length adjustment is tied to what the 19th
test is doing.  However I'd like to see the fixup occur near the point where they are needing
the value be > 1.  An issue now is that if someone updates the code to divide by 3 instead
of 2 they have to remember to find the fixup code 20+ lines above it and fix that, too.  It'd
be nice if the logic for handling test 19 was all in one place.

> TestFixedLengthInputFormat#testFormatCompressedIn is flaky
> ----------------------------------------------------------
>
>                 Key: MAPREDUCE-7052
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7052
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: client, test
>            Reporter: Peter Bacsko
>            Assignee: Peter Bacsko
>            Priority: Major
>         Attachments: MAPREDUCE-7052-001.patch
>
>
> Sometimes the test case TestFixedLengthInputFormat#testFormatCompressedIn can fail with
the following error:
> {noformat}
> java.lang.OutOfMemoryError: Requested array size exceeds VM limit
> 	at org.apache.hadoop.mapred.TestFixedLengthInputFormat.runRandomTests(TestFixedLengthInputFormat.java:322)
> 	at org.apache.hadoop.mapred.TestFixedLengthInputFormat.testFormatCompressedIn(TestFixedLengthInputFormat.java:90)
> {noformat}
> *Root cause:* under special circumstances, the following line can return a huge number:
> {noformat}
>           // Test a split size that is less than record len
>           numSplits = (int)(fileSize/Math.floor(recordLength/2));
> {noformat}
> For example, let {{seed}} be 2026428718. This causes {{recordLength}} to be 1 at iteration
19. {{Math.floor()}} returns negative Infinity, which becomes positve infinity after the divison.
Casting it to {{int}} yields {{Integer.MAX_VALUE}}. Eventually we get an OOME because the
test wants to create a huge {{InputSplit}} array.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message