flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2545) NegativeArraySizeException while creating hash table bloom filters
Date Fri, 28 Aug 2015 20:55:46 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14720556#comment-14720556
] 

ASF GitHub Bot commented on FLINK-2545:
---------------------------------------

Github user greghogan commented on the pull request:

    https://github.com/apache/flink/pull/1067#issuecomment-135885866
  
    I am currently running release-0.10.0-milestone-1.
    
    Debugging with Eclipse and looking at MutableHashTable.initTable, numBuckets is computed
as 16086. There are 63 memory segments with 256 buckets each = 16128 total buckets. The last
16128 - 16086 = 42 buckets are not initialized by initTable which terminates the inner loop
when bucket == numBuckets. Here is an example header dump from the last memory segment showing
the crossover from initialized to uninitialized data.
    
    offset, partition, status, count, next-pointer
    26880 10 0 0 -72340172838076673
    27008 11 0 0 -72340172838076673
    27136 12 0 0 -72340172838076673
    27264 13 0 0 -72340172838076673
    27392 0 -56 9 844425030795264
    27520 0 -56 9 -9191846839379296256
    27648 0 -56 9 10133099245469696
    27776 0 -56 9 12103424082444288
    
    Setting a breakpoint for MutableHashTable.buildBloomFilterForBucket for count < 0,
the last memory segment looked as follows (this is from a different execution, operation,
and thread).
    
    offset, partition, status, count, next-pointer
    26880 10 0 9 27584547767975936
    27008 11 0 9 -9208735337998712832
    27136 12 0 9 4503599694479360
    27264 13 0 9 -9219994337067139072
    27392 0 0 -32697 1161165883580435
    27520 0 3 -15328 18016855230957176
    27648 0 5 1388 -33740636012148672
    27776 0 6 25494 -17363350186618861
    
    MutableHashTable.buildBloomFilterForBucketsInPartition processed offset 27392 which happened
to match the partition number and bucket status even though it looks to be uninitialized.
    
    After changing MutableHashTable.initTable to initialize all buckets in all segments I
have not seen the bug reoccur.
    
    {code}
    for (int k = 0; k < bucketsPerSegment /* && bucket < numBuckets*/; k++,
bucket++) {
    }
    {code}
    
    I see at least three potential resolutions: 1) have MutableHashTable.initTable initialize
all buckets, 2) have MutableHashTable.buildBloomFilterForBucket skip uninitialized buckets,
or 3) I have not looked enough at MutableHashTable.getInitialTableSize but it is possible
to completely fill the last segment with usable buckets?


> NegativeArraySizeException while creating hash table bloom filters
> ------------------------------------------------------------------
>
>                 Key: FLINK-2545
>                 URL: https://issues.apache.org/jira/browse/FLINK-2545
>             Project: Flink
>          Issue Type: Bug
>          Components: Distributed Runtime
>    Affects Versions: master
>            Reporter: Greg Hogan
>            Assignee: Chengxiang Li
>
> The following exception occurred a second time when I immediately re-ran my application,
though after recompiling and restarting Flink the subsequent execution ran without error.
> java.lang.Exception: The data preparation for task '...' , caused an error: null
> 	at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:465)
> 	at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:354)
> 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:581)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NegativeArraySizeException
> 	at org.apache.flink.runtime.operators.hash.MutableHashTable.buildBloomFilterForBucket(MutableHashTable.java:1160)
> 	at org.apache.flink.runtime.operators.hash.MutableHashTable.buildBloomFilterForBucketsInPartition(MutableHashTable.java:1143)
> 	at org.apache.flink.runtime.operators.hash.MutableHashTable.spillPartition(MutableHashTable.java:1117)
> 	at org.apache.flink.runtime.operators.hash.MutableHashTable.insertBucketEntry(MutableHashTable.java:946)
> 	at org.apache.flink.runtime.operators.hash.MutableHashTable.insertIntoTable(MutableHashTable.java:868)
> 	at org.apache.flink.runtime.operators.hash.MutableHashTable.buildInitialTable(MutableHashTable.java:692)
> 	at org.apache.flink.runtime.operators.hash.MutableHashTable.open(MutableHashTable.java:455)
> 	at org.apache.flink.runtime.operators.hash.ReusingBuildSecondHashMatchIterator.open(ReusingBuildSecondHashMatchIterator.java:93)
> 	at org.apache.flink.runtime.operators.JoinDriver.prepare(JoinDriver.java:195)
> 	at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:459)
> 	... 3 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message