pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "liyunzhang_intel (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-5068) Set SPARK_REDUCERS by pig.properties not by system configuration
Date Thu, 24 Nov 2016 03:14:58 GMT

     [ https://issues.apache.org/jira/browse/PIG-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

liyunzhang_intel updated PIG-5068:
----------------------------------
    Attachment:     (was: PIG-5068_1.patch)

> Set SPARK_REDUCERS by pig.properties not by system configuration
> ----------------------------------------------------------------
>
>                 Key: PIG-5068
>                 URL: https://issues.apache.org/jira/browse/PIG-5068
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: liyunzhang_intel
>         Attachments: PIG-5068.patch, PIG-5068_1.patch
>
>
> In SparkUtil.java, we set the SPARK_REDUCERS by system configuration
> {code}
>     public static int getParallelism(List<RDD<Tuple>> predecessors,
>             PhysicalOperator physicalOperator) {
>         String numReducers = System.getenv("SPARK_REDUCERS");
>         if (numReducers != null) {
>             return Integer.parseInt(numReducers);
>         }
>         int parallelism = physicalOperator.getRequestedParallelism();
>         if (parallelism <= 0) {
>             // Parallelism wasn't set in Pig, so set it to whatever Spark thinks
>             // is reasonable.
>             parallelism = predecessors.get(0).context().defaultParallelism();
>         }
>         return parallelism;
>     }
> {code}
> It is better to set it by pig.properties



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message