pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Szita (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4899) The number of records of input file is calculated wrongly in spark mode in multiquery case
Date Mon, 07 Nov 2016 19:17:58 GMT

    [ https://issues.apache.org/jira/browse/PIG-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15645133#comment-15645133
] 

Adam Szita commented on PIG-4899:
---------------------------------

I've created a fix for this and attached test in [^PIG-4899.patch]

>  The number of records of input file is calculated wrongly in spark mode in multiquery
case
> -------------------------------------------------------------------------------------------
>
>                 Key: PIG-4899
>                 URL: https://issues.apache.org/jira/browse/PIG-4899
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: Adam Szita
>             Fix For: spark-branch
>
>         Attachments: PIG-4899.patch
>
>
> sparkCounter to calucate the records of input file(LoadConverter#ToTupleFunction#apply)
will be executed multiple times in multiquery case. This will cause the input records number
is calculated wrongly. for example:
> {code}
> #--------------------------------------------------
> # Spark Plan                                  
> #--------------------------------------------------
> Spark node scope-534
> Split - scope-548
> |   |
> |   Store(hdfs://localhost:48350/tmp/temp649016960/tmp48836938:org.apache.pig.impl.io.InterStorage)
- scope-538
> |   |
> |   |---C: Filter[bag] - scope-495
> |       |   |
> |       |   Less Than or Equal[boolean] - scope-498
> |       |   |
> |       |   |---Project[int][1] - scope-496
> |       |   |
> |       |   |---Constant(5) - scope-497
> |   |
> |   Store(hdfs://localhost:48350/tmp/temp649016960/tmp804709981:org.apache.pig.impl.io.InterStorage)
- scope-546
> |   |
> |   |---B: Filter[bag] - scope-507
> |       |   |
> |       |   Equal To[boolean] - scope-510
> |       |   |
> |       |   |---Project[int][0] - scope-508
> |       |   |
> |       |   |---Constant(3) - scope-509
> |
> |---A: New For Each(false,false,false)[bag] - scope-491
>     |   |
>     |   Cast[int] - scope-483
>     |   |
>     |   |---Project[bytearray][0] - scope-482
>     |   |
>     |   Cast[int] - scope-486
>     |   |
>     |   |---Project[bytearray][1] - scope-485
>     |   |
>     |   Cast[int] - scope-489
>     |   |
>     |   |---Project[bytearray][2] - scope-488
>     |
>     |---A: Load(hdfs://localhost:48350/user/root/input:org.apache.pig.builtin.PigStorage)
- scope-481--------
> Spark node scope-540
> C: Store(hdfs://localhost:48350/user/root/output:org.apache.pig.builtin.PigStorage) -
scope-502
> |
> |---Load(hdfs://localhost:48350/tmp/temp649016960/tmp48836938:org.apache.pig.impl.io.InterStorage)
- scope-539--------
> Spark node scope-542
> D: Store(hdfs://localhost:48350/user/root/output2:org.apache.pig.builtin.PigStorage)
- scope-533
> |
> |---D: FRJoin[tuple] - scope-525
>     |   |
>     |   Project[int][0] - scope-522
>     |   |
>     |   Project[int][0] - scope-523
>     |   |
>     |   Project[int][0] - scope-524
>     |
>     |---Load(hdfs://localhost:48350/tmp/temp649016960/tmp48836938:org.apache.pig.impl.io.InterStorage)
- scope-541--------
> Spark node scope-545
> Store(hdfs://localhost:48350/tmp/temp649016960/tmp-2036144538:org.apache.pig.impl.io.InterStorage)
- scope-547
> |
> |---A1: New For Each(false,false,false)[bag] - scope-521
>     |   |
>     |   Cast[int] - scope-513
>     |   |
>     |   |---Project[bytearray][0] - scope-512
>     |   |
>     |   Cast[int] - scope-516
>     |   |
>     |   |---Project[bytearray][1] - scope-515
>     |   |
>     |   Cast[int] - scope-519
>     |   |
>     |   |---Project[bytearray][2] - scope-518
>     |
>     |---A1: Load(hdfs://localhost:48350/user/root/input2:org.apache.pig.builtin.PigStorage)
- scope-511-------
> {code}
> PhysicalOperator (LoadA) will be executed in LoadConverter#ToTupleFunction#apply for
more than the correct times because this is a multi-query case. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message