pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mitesh Singh Jat (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-1917) NativeMapReduce does not Allow Configuration Parameters containing Spaces
Date Fri, 18 Mar 2011 10:03:29 GMT

     [ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mitesh Singh Jat updated PIG-1917:
----------------------------------

    Description: 
In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.

Considering a simple pig script with _wordcount_ as Native M-R job.

{code:title=nativeMR.pig|borderStyle=solid}
A = load 'input/WordCountInput.txt';
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray,
count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m
-Xss128k" inputDir outputDir`;
store B into 'output/WordCount' using PigStorage;
{code}

The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_
makes
_-Xss128k"_ as the first argument to _wordcount_
Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_
in Jobconf.

Physical plan, Logical plan, and M-R plan of pig show double quotes. 
For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
{code}
MapReduce node scope-13
hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m
-Xss128k" inputDir outputDir
--------
{code}

On executing above M-R job after copying  'input/WordCountInput.txt' into 'inputDir/', as
Hadoop Job, this ran successfully.
{code}
$ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m
-Xss128k" inputDir outputDir*
11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
11/03/18 14:23:30 INFO mapred.JobClient:  map 0% reduce 0%
11/03/18 14:23:37 INFO mapred.JobClient:  map 100% reduce 0%
11/03/18 14:23:49 INFO mapred.JobClient:  map 100% reduce 100%
11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
...
...

{code}

Whereas, when  _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed
successfully.

  was:
In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.

Considering a simple pig script with _wordcount_ as Native M-R job.

{code:pig|title=nativeMR.pig|borderStyle=solid}
A = load 'input/WordCountInput.txt';
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray,
count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m
-Xss128k" inputDir outputDir`;
store B into 'output/WordCount' using PigStorage;
{code}

The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_
makes
_-Xss128k"_ as the first argument to _wordcount_
Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_
in Jobconf.

Physical plan, Logical plan, and M-R plan of pig show double quotes. 
For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
{code}
MapReduce node scope-13
hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m
-Xss128k" inputDir outputDir
--------
{code}

On executing above M-R job after copying  'input/WordCountInput.txt' into 'inputDir/', as
Hadoop Job, this ran successfully.
{code}
$ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m
-Xss128k" inputDir outputDir*
11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
11/03/18 14:23:30 INFO mapred.JobClient:  map 0% reduce 0%
11/03/18 14:23:37 INFO mapred.JobClient:  map 100% reduce 0%
11/03/18 14:23:49 INFO mapred.JobClient:  map 100% reduce 100%
11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
...
...

{code}

Whereas, when  _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed
successfully.


> NativeMapReduce does not Allow Configuration Parameters containing Spaces 
> --------------------------------------------------------------------------
>
>                 Key: PIG-1917
>                 URL: https://issues.apache.org/jira/browse/PIG-1917
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Mitesh Singh Jat
>
> In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
> Considering a simple pig script with _wordcount_ as Native M-R job.
> {code:title=nativeMR.pig|borderStyle=solid}
> A = load 'input/WordCountInput.txt';
> B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray,
count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m
-Xss128k" inputDir outputDir`;
> store B into 'output/WordCount' using PigStorage;
> {code}
> The above pig script fails while launching native MAPREDUCE job. Because space after
_-Dmapred.child.java.opts="-Xmx1536m_ makes
> _-Xss128k"_ as the first argument to _wordcount_
> Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned
_"-Xmx1536m_ in Jobconf.
> Physical plan, Logical plan, and M-R plan of pig show double quotes. 
> For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
> {code}
> MapReduce node scope-13
> hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m
-Xss128k" inputDir outputDir
> --------
> {code}
> On executing above M-R job after copying  'input/WordCountInput.txt' into 'inputDir/',
as Hadoop Job, this ran successfully.
> {code}
> $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m
-Xss128k" inputDir outputDir*
> 11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
> 11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
> 11/03/18 14:23:30 INFO mapred.JobClient:  map 0% reduce 0%
> 11/03/18 14:23:37 INFO mapred.JobClient:  map 100% reduce 0%
> 11/03/18 14:23:49 INFO mapred.JobClient:  map 100% reduce 100%
> 11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
> ...
> ...
> {code}
> Whereas, when  _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed
successfully.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message