crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-601) Short PCollections in SparkPipeline get length null.
Date Sat, 20 Aug 2016 22:21:21 GMT


Micah Whitacre updated CRUNCH-601:
    Attachment: CRUNCH-601c.patch

Corrected LengthPObject spelling.

I don't quite follow the reasoning behind returning the parentSize value if newSize == 0 instead
of e.g. returning Math.max(1L, newSize)?

Mikael answered similar to my reasoning.  Size is typically used for two reasons:
* Making decisions about when the planner should write out data vs not
* Planning for skipping processing if there is no data to process.

Since small sizes like 1-100 won't matter much for the planner, really we just need to be
accurate about 0 or not.

That's the reason for looking at the parent size up the stack.

> Short PCollections in SparkPipeline get length null.
> ----------------------------------------------------
>                 Key: CRUNCH-601
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>          Components: Spark
>    Affects Versions: 0.13.0
>         Environment: Running in local mode on Mac as well as in a ubuntu 14.04 docker
>            Reporter: Mikael Goldmann
>            Assignee: Micah Whitacre
>            Priority: Minor
>         Attachments: CRUNCH-601.patch, CRUNCH-601b.patch, CRUNCH-601c.patch,
> I'll attach a file with a test that I would expect to pass but which fails.
> It creates five PCollection<String> of lengths 0, 1, 2, 3, 4 gets the lengths,
runs the pipeline and prints the lengths. Finally it asserts that all lengths are non-null.
> I would expect it to print lengths 0, 1, 2, 3, 4 and pass.
> What it does is print lengths null, null, null, 3, 4 and fail.
> I think the underlying reason is the use of getSize() on an unmaterialized object and
assuming that when the estimate that getSize() returns is 0, then the PCollection is guaranteed
to be empty, which is false in some cases.

This message was sent by Atlassian JIRA

View raw message