hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Seigel <ja...@tynt.com>
Subject Re: Configuration for small Cluster
Date Mon, 02 May 2011 14:40:01 GMT
If you have windows and cygwin you probably don't have a lot if memory
left at 2 gig.

Pull up system monitor on the data nodes and check for free memory
when you have you jobs running. I bet it is quite low.

I am not a windows guy so I can't take you much farther.

James

Sent from my mobile. Please excuse the typos.

On 2011-05-02, at 8:32 AM, baran cakici <barancakici@gmail.com> wrote:

> yes, I am running under cygwin on my datanodes too. OS of Datanodes are
> Windows as well.
>
> What can I do exactly for a better Performance. I changed
> mapred.child.java.opts to default value.How can I solve this "swapping"
> problem?
>
> PS: I dont have a chance to get Slaves(Celeron 2GHz) with Liniux OS.
>
> thanks, both of you
>
> Regards,
>
> Baran
> 2011/5/2 Richard Nadeau <strouthos@gmail.com>
>
>> Are you running under cygwin on your data nodes as well? That is certain to
>> cause performance problems. As James suggested, swapping to disk is going
>> to
>> be a killer, running on Windows with Celeron processors only compounds the
>> problem. The Celeron processor is also sub-optimal for CPU intensive tasks
>>
>> Rick
>>
>> On Apr 28, 2011 9:22 AM, "baran cakici" <barancakici@gmail.com> wrote:
>>> Hi Everyone,
>>>
>>> I have a Cluster with one Master(JobTracker and NameNode - Intel Core2Duo
>> 2
>>> GB Ram) and four Slaves(Datanode and Tasktracker - Celeron 2 GB Ram). My
>>> Inputdata are between 2GB-10GB and I read Inputdata in MapReduce line by
>>> line. Now, I try to accelerate my System(Benchmark), but I'm not sure, if
>> my
>>> Configuration is correctly. Can you please just look, if it is ok?
>>>
>>> -mapred-site.xml
>>>
>>> <property>
>>> <name>mapred.job.tracker</name>
>>> <value>apple:9001</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.child.java.opts</name>
>>> <value>-Xmx512m -server</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.job.tracker.handler.count</name>
>>> <value>2</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.local.dir</name>
>>>
>> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/local</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.map.tasks</name>
>>> <value>1</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.reduce.tasks</name>
>>> <value>4</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.submit.replication</name>
>>> <value>2</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.system.dir</name>
>>>
>>
>> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/system</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.tasktracker.indexcache.mb</name>
>>> <value>10</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.tasktracker.map.tasks.maximum</name>
>>> <value>1</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.tasktracker.reduce.tasks.maximum</name>
>>> <value>4</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.temp.dir</name>
>>>
>> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/temp</value>
>>> </property>
>>>
>>> <property>
>>> <name>webinterface.private.actions</name>
>>> <value>true</value>
>>> </property>
>>>
>>> <property>
>>> <name>mapred.reduce.slowstart.completed.maps</name>
>>> <value>0.01</value>
>>> </property>
>>>
>>> -hdfs-site.xml
>>>
>>> <property>
>>> <name>dfs.block.size</name>
>>> <value>268435456</value>
>>> </property>
>>> PS: I extended dfs.block.size, because I won 50% better performance with
>>> this change.
>>>
>>> I am waiting for your comments...
>>>
>>> Regards,
>>>
>>> Baran
>>

Mime
View raw message