hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From baran cakici <barancak...@gmail.com>
Subject Re: Configuration for small Cluster
Date Mon, 02 May 2011 15:26:25 GMT
Actually it was one, I changed that, and got better Performance by Reduce,
because my Reduce-Algortihm is a little bit complex.

thanks anyway

Regards,

Baran

2011/5/2 Richard Nadeau <strouthos@gmail.com>

> I would change "mapred.tasktracker.reduce.tasks.maximum" to one. With your
> setting
>
> On May 2, 2011 8:48 AM, "baran cakici" <barancakici@gmail.com> wrote:
> > without job;
> >
> > CPU Usage = 0%
> > Memory = 585 MB (2GB Ram)
> >
> > Baran
> > 2011/5/2 baran cakici <barancakici@gmail.com>
> >
> >> CPU Usage = 95-100%
> >> Memory = 650-850 MB (2GB Ram)
> >>
> >> Baran
> >>
> >>
> >> 2011/5/2 James Seigel <james@tynt.com>
> >>
> >>> If you have windows and cygwin you probably don't have a lot if memory
> >>> left at 2 gig.
> >>>
> >>> Pull up system monitor on the data nodes and check for free memory
> >>> when you have you jobs running. I bet it is quite low.
> >>>
> >>> I am not a windows guy so I can't take you much farther.
> >>>
> >>> James
> >>>
> >>> Sent from my mobile. Please excuse the typos.
> >>>
> >>> On 2011-05-02, at 8:32 AM, baran cakici <barancakici@gmail.com> wrote:
> >>>
> >>> > yes, I am running under cygwin on my datanodes too. OS of Datanodes
> are
> >>> > Windows as well.
> >>> >
> >>> > What can I do exactly for a better Performance. I changed
> >>> > mapred.child.java.opts to default value.How can I solve this
> "swapping"
> >>> > problem?
> >>> >
> >>> > PS: I dont have a chance to get Slaves(Celeron 2GHz) with Liniux OS.
> >>> >
> >>> > thanks, both of you
> >>> >
> >>> > Regards,
> >>> >
> >>> > Baran
> >>> > 2011/5/2 Richard Nadeau <strouthos@gmail.com>
> >>> >
> >>> >> Are you running under cygwin on your data nodes as well? That is
> >>> certain to
> >>> >> cause performance problems. As James suggested, swapping to disk
is
> >>> going
> >>> >> to
> >>> >> be a killer, running on Windows with Celeron processors only
> compounds
> >>> the
> >>> >> problem. The Celeron processor is also sub-optimal for CPU intensive
> >>> tasks
> >>> >>
> >>> >> Rick
> >>> >>
> >>> >> On Apr 28, 2011 9:22 AM, "baran cakici" <barancakici@gmail.com>
> wrote:
> >>> >>> Hi Everyone,
> >>> >>>
> >>> >>> I have a Cluster with one Master(JobTracker and NameNode -
Intel
> >>> Core2Duo
> >>> >> 2
> >>> >>> GB Ram) and four Slaves(Datanode and Tasktracker - Celeron
2 GB
> Ram).
> >>> My
> >>> >>> Inputdata are between 2GB-10GB and I read Inputdata in MapReduce
> line
> >>> by
> >>> >>> line. Now, I try to accelerate my System(Benchmark), but I'm
not
> sure,
> >>> if
> >>> >> my
> >>> >>> Configuration is correctly. Can you please just look, if it
is ok?
> >>> >>>
> >>> >>> -mapred-site.xml
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>mapred.job.tracker</name>
> >>> >>> <value>apple:9001</value>
> >>> >>> </property>
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>mapred.child.java.opts</name>
> >>> >>> <value>-Xmx512m -server</value>
> >>> >>> </property>
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>mapred.job.tracker.handler.count</name>
> >>> >>> <value>2</value>
> >>> >>> </property>
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>mapred.local.dir</name>
> >>> >>>
> >>> >>
> >>>
> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/local</value>
> >>> >>> </property>
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>mapred.map.tasks</name>
> >>> >>> <value>1</value>
> >>> >>> </property>
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>mapred.reduce.tasks</name>
> >>> >>> <value>4</value>
> >>> >>> </property>
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>mapred.submit.replication</name>
> >>> >>> <value>2</value>
> >>> >>> </property>
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>mapred.system.dir</name>
> >>> >>>
> >>> >>
> >>> >>
> >>>
>
> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/system</value>
> >>> >>> </property>
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>mapred.tasktracker.indexcache.mb</name>
> >>> >>> <value>10</value>
> >>> >>> </property>
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>mapred.tasktracker.map.tasks.maximum</name>
> >>> >>> <value>1</value>
> >>> >>> </property>
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>mapred.tasktracker.reduce.tasks.maximum</name>
> >>> >>> <value>4</value>
> >>> >>> </property>
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>mapred.temp.dir</name>
> >>> >>>
> >>> >>
> >>>
> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/temp</value>
> >>> >>> </property>
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>webinterface.private.actions</name>
> >>> >>> <value>true</value>
> >>> >>> </property>
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>mapred.reduce.slowstart.completed.maps</name>
> >>> >>> <value>0.01</value>
> >>> >>> </property>
> >>> >>>
> >>> >>> -hdfs-site.xml
> >>> >>>
> >>> >>> <property>
> >>> >>> <name>dfs.block.size</name>
> >>> >>> <value>268435456</value>
> >>> >>> </property>
> >>> >>> PS: I extended dfs.block.size, because I won 50% better performance
> >>> with
> >>> >>> this change.
> >>> >>>
> >>> >>> I am waiting for your comments...
> >>> >>>
> >>> >>> Regards,
> >>> >>>
> >>> >>> Baran
> >>> >>
> >>>
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message