hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vincent Y. Shen" <vincent.y.s...@gmail.com>
Subject Re: mapred.tasktacker.reduce.tasks.maximum issue
Date Thu, 14 Nov 2013 15:53:21 GMT
Hmmmm i tried but still not working... 4 nodes 8 reducers.... all values
are final


On Thu, Nov 14, 2013 at 3:02 PM, Dieter De Witte <drdwitte@gmail.com> wrote:

> you could make the values final: add <final> true </final> to make sure
> they cannot be overridden..
>
>
> 2013/11/14 Vincent Y. Shen <vincent.y.shen@gmail.com>
>
>> Hi Dieter,
>>
>> Thanks a lot for prompt reply! very much appreciated.
>>
>> Indeed, I did change in the mapred-site.xml in ALL the nodes and
>> restarted the cluster 4 times but no luck...
>>
>>
>> Any other hint?
>>
>>
>>
>>
>>
>>
>> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <drdwitte@gmail.com>wrote:
>>
>>> Dear Vincent,
>>>
>>> You have to make sure that mapred-site is modified on every tasktracker
>>> node! Not just the master! Mapred-site can be customized if you have a
>>> heterogeneous cluster. I usually start the dfs, then copy all the
>>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>>> job.
>>>
>>> Regards, Dieter
>>>
>>>
>>> 2013/11/14 Vincent Y. Shen <vincent.y.shen@gmail.com>
>>>
>>> Hi Guys,
>>>>
>>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>>> but the default (2)
>>>>
>>>>
>>>>
>>>>
>>>> What I did do
>>>> ===========
>>>>
>>>> stop mapred
>>>> stop dfs
>>>>
>>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>>
>>>>
>>>> start dfs
>>>> start mapred
>>>>
>>>>
>>>> But the system still uses default values
>>>>
>>>>
>>>>
>>>>
>>>> Environment
>>>> =========
>>>>
>>>> Hadoop version: 1.1.2
>>>> Java version: 7
>>>>
>>>>
>>>> Mapred-site.xml
>>>> ============
>>>>
>>>> <property>
>>>>   <name>mapred.system.dir</name>
>>>>   <value>/hadoop/mapred/system</value>
>>>> </property>
>>>>
>>>>
>>>>
>>>> <property>
>>>>   <name>mapred.reduce.tasks</name>
>>>>   <value>50</value>
>>>> </property>
>>>>
>>>> <property>
>>>>   <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>>>   <value>15</value>
>>>> </property>
>>>>
>>>> <property>
>>>>   <name>mapred.tasktacker.map.tasks.maximum</name>
>>>>   <value>15</value>
>>>> </property>
>>>>
>>>>
>>>> <property>
>>>>   <name>mapred.child.java.opts</name>
>>>>   <value>-Xmx500m</value>
>>>> </property>
>>>>
>>>>
>>>>
>>>> <property>
>>>>   <name>mapred.job.tracker</name>
>>>>   <value>machine1.mydomain.com:54311</value>
>>>>
>>>> </property>
>>>>
>>>>
>>>>
>>>> Thanks guys for any pointers!!!!
>>>>
>>>>
>>>> Best regards,
>>>>
>>>>
>>>> Vincent.
>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message