hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From baran cakici <barancak...@gmail.com>
Subject Re: Configuration for small Cluster
Date Mon, 02 May 2011 14:48:26 GMT
without job;

CPU Usage = 0%
Memory      = 585 MB (2GB Ram)

Baran
2011/5/2 baran cakici <barancakici@gmail.com>

> CPU Usage = 95-100%
> Memory      = 650-850 MB (2GB Ram)
>
> Baran
>
>
>  2011/5/2 James Seigel <james@tynt.com>
>
>> If you have windows and cygwin you probably don't have a lot if memory
>> left at 2 gig.
>>
>> Pull up system monitor on the data nodes and check for free memory
>> when you have you jobs running. I bet it is quite low.
>>
>> I am not a windows guy so I can't take you much farther.
>>
>> James
>>
>> Sent from my mobile. Please excuse the typos.
>>
>>  On 2011-05-02, at 8:32 AM, baran cakici <barancakici@gmail.com> wrote:
>>
>> > yes, I am running under cygwin on my datanodes too. OS of Datanodes are
>> > Windows as well.
>> >
>> > What can I do exactly for a better Performance. I changed
>> > mapred.child.java.opts to default value.How can I solve this "swapping"
>> > problem?
>> >
>> > PS: I dont have a chance to get Slaves(Celeron 2GHz) with Liniux OS.
>> >
>> > thanks, both of you
>> >
>> > Regards,
>> >
>> > Baran
>> > 2011/5/2 Richard Nadeau <strouthos@gmail.com>
>> >
>> >> Are you running under cygwin on your data nodes as well? That is
>> certain to
>> >> cause performance problems. As James suggested, swapping to disk is
>> going
>> >> to
>> >> be a killer, running on Windows with Celeron processors only compounds
>> the
>> >> problem. The Celeron processor is also sub-optimal for CPU intensive
>> tasks
>> >>
>> >> Rick
>> >>
>> >> On Apr 28, 2011 9:22 AM, "baran cakici" <barancakici@gmail.com> wrote:
>> >>> Hi Everyone,
>> >>>
>> >>> I have a Cluster with one Master(JobTracker and NameNode - Intel
>> Core2Duo
>> >> 2
>> >>> GB Ram) and four Slaves(Datanode and Tasktracker - Celeron 2 GB Ram).
>> My
>> >>> Inputdata are between 2GB-10GB and I read Inputdata in MapReduce line
>> by
>> >>> line. Now, I try to accelerate my System(Benchmark), but I'm not sure,
>> if
>> >> my
>> >>> Configuration is correctly. Can you please just look, if it is ok?
>> >>>
>> >>> -mapred-site.xml
>> >>>
>> >>> <property>
>> >>> <name>mapred.job.tracker</name>
>> >>> <value>apple:9001</value>
>> >>> </property>
>> >>>
>> >>> <property>
>> >>> <name>mapred.child.java.opts</name>
>> >>> <value>-Xmx512m -server</value>
>> >>> </property>
>> >>>
>> >>> <property>
>> >>> <name>mapred.job.tracker.handler.count</name>
>> >>> <value>2</value>
>> >>> </property>
>> >>>
>> >>> <property>
>> >>> <name>mapred.local.dir</name>
>> >>>
>> >>
>> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/local</value>
>> >>> </property>
>> >>>
>> >>> <property>
>> >>> <name>mapred.map.tasks</name>
>> >>> <value>1</value>
>> >>> </property>
>> >>>
>> >>> <property>
>> >>> <name>mapred.reduce.tasks</name>
>> >>> <value>4</value>
>> >>> </property>
>> >>>
>> >>> <property>
>> >>> <name>mapred.submit.replication</name>
>> >>> <value>2</value>
>> >>> </property>
>> >>>
>> >>> <property>
>> >>> <name>mapred.system.dir</name>
>> >>>
>> >>
>> >>
>> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/system</value>
>> >>> </property>
>> >>>
>> >>> <property>
>> >>> <name>mapred.tasktracker.indexcache.mb</name>
>> >>> <value>10</value>
>> >>> </property>
>> >>>
>> >>> <property>
>> >>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> >>> <value>1</value>
>> >>> </property>
>> >>>
>> >>> <property>
>> >>> <name>mapred.tasktracker.reduce.tasks.maximum</name>
>> >>> <value>4</value>
>> >>> </property>
>> >>>
>> >>> <property>
>> >>> <name>mapred.temp.dir</name>
>> >>>
>> >>
>> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/temp</value>
>> >>> </property>
>> >>>
>> >>> <property>
>> >>> <name>webinterface.private.actions</name>
>> >>> <value>true</value>
>> >>> </property>
>> >>>
>> >>> <property>
>> >>> <name>mapred.reduce.slowstart.completed.maps</name>
>> >>> <value>0.01</value>
>> >>> </property>
>> >>>
>> >>> -hdfs-site.xml
>> >>>
>> >>> <property>
>> >>> <name>dfs.block.size</name>
>> >>> <value>268435456</value>
>> >>> </property>
>> >>> PS: I extended dfs.block.size, because I won 50% better performance
>> with
>> >>> this change.
>> >>>
>> >>> I am waiting for your comments...
>> >>>
>> >>> Regards,
>> >>>
>> >>> Baran
>> >>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message