hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Seigel <ja...@tynt.com>
Subject Re: Configuration for small Cluster
Date Mon, 02 May 2011 12:44:45 GMT
Do you see swapping on your data nodes with this config?

James

Sent from my mobile. Please excuse the typos.

On 2011-05-02, at 5:38 AM, baran cakici <barancakici@gmail.com> wrote:

> any comments???
>
> 2011/4/28 baran cakici <barancakici@gmail.com>
>
>> Hi Everyone,
>>
>> I have a Cluster with one Master(JobTracker and NameNode - Intel Core2Duo 2
>> GB Ram) and four Slaves(Datanode and Tasktracker - Celeron 2 GB Ram). My
>> Inputdata are between 2GB-10GB and I read Inputdata in MapReduce line by
>> line. Now, I try to accelerate my System(Benchmark), but I'm not sure, if my
>> Configuration is correctly. Can you please just look, if it is ok?
>>
>> -mapred-site.xml
>>
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>apple:9001</value>
>> </property>
>>
>> <property>
>> <name>mapred.child.java.opts</name>
>> <value>-Xmx512m -server</value>
>> </property>
>>
>> <property>
>> <name>mapred.job.tracker.handler.count</name>
>> <value>2</value>
>> </property>
>>
>> <property>
>> <name>mapred.local.dir</name>
>> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/local</value>
>> </property>
>>
>> <property>
>> <name>mapred.map.tasks</name>
>> <value>1</value>
>> </property>
>>
>> <property>
>> <name>mapred.reduce.tasks</name>
>> <value>4</value>
>> </property>
>>
>> <property>
>> <name>mapred.submit.replication</name>
>> <value>2</value>
>> </property>
>>
>> <property>
>> <name>mapred.system.dir</name>
>>
>> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/system</value>
>> </property>
>>
>> <property>
>> <name>mapred.tasktracker.indexcache.mb</name>
>> <value>10</value>
>> </property>
>>
>> <property>
>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> <value>1</value>
>> </property>
>>
>> <property>
>> <name>mapred.tasktracker.reduce.tasks.maximum</name>
>> <value>4</value>
>> </property>
>>
>> <property>
>> <name>mapred.temp.dir</name>
>> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/temp</value>
>> </property>
>>
>> <property>
>> <name>webinterface.private.actions</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>mapred.reduce.slowstart.completed.maps</name>
>> <value>0.01</value>
>> </property>
>>
>> -hdfs-site.xml
>>
>> <property>
>> <name>dfs.block.size</name>
>> <value>268435456</value>
>> </property>
>> PS: I extended dfs.block.size, because I won 50% better performance with
>> this change.
>>
>> I am waiting for your comments...
>>
>> Regards,
>>
>> Baran
>>

Mime
View raw message