crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Surbhi Mungre (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-494) Unable to union large number of PCollections
Date Fri, 13 Feb 2015 14:58:11 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14320199#comment-14320199
] 

Surbhi Mungre commented on CRUNCH-494:
--------------------------------------

I tried this fix and it worked for 5000 PCollections. However, for 8000 PCollections I get
following stack overflow exception. 
{noformat}
java.lang.StackOverflowError
        at java.util.Arrays$ArrayList.size(Arrays.java:3362)
        at java.util.AbstractList$Itr.hasNext(AbstractList.java:339)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124)
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
        at com.google.common.collect.Iterators$3.hasNext(Iterators.java:124) 
        at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
{noformat}

Following unit test can reproduce it. I am guessing it is coming from Sets.union method in
PCollectionImpl class. 
https://gist.github.com/anonymous/a7f1ca3a91d2a4b49218


> Unable to union large number of PCollections 
> ---------------------------------------------
>
>                 Key: CRUNCH-494
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-494
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.3
>            Reporter: Surbhi Mungre
>            Assignee: Josh Wills
>            Priority: Minor
>             Fix For: 0.12.0
>
>         Attachments: CRUNCH-494.patch
>
>
> If you try to union large number of PCollections(~5K), then Crunch throws StackOverflowError
exception. 
> {noformat}
> java.lang.StackOverflowError
> 	at com.google.common.collect.AbstractIndexedListIterator.<init>(AbstractIndexedListIterator.java:68)
> 	at com.google.common.collect.AbstractIndexedListIterator.<init>(AbstractIndexedListIterator.java:54)
> 	at com.google.common.collect.Iterators$12.<init>(Iterators.java:1072)
> 	at com.google.common.collect.Iterators.forArray(Iterators.java:1072)
> 	at com.google.common.collect.RegularImmutableList.iterator(RegularImmutableList.java:68)
> 	at com.google.common.collect.RegularImmutableList.iterator(RegularImmutableList.java:31)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:291)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> 	at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> {noformat}
> Here is a simple test which can reproduce the issue. 
> https://gist.github.com/anonymous/22f08511604341d0ffda



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message