hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ranjan Banerjee <rbanerj...@wisc.edu>
Subject Re: How to change the scheduler
Date Wed, 07 Mar 2012 01:04:04 GMT
Hello Merto,
     I wish to implement SBF(Stretch Based Fairness), hence I guess the place to look
will be dynamic changing of slaves.
Regards,
Ranjan

On 03/06/12, Merto Mertek   wrote:
> If you will change the default scheduler take a look at the thread "Dynamic
> changing of slaves" on the mailing list where I described my understanding
> of the scheduling process. However if you will modify some fairscheduling
> code, take a look at classes fairscheduler.java and
> schedulingalgorithms.java from the fairscheduler package. Here you can make
> your own comparator that can compare jobs/pools depending on varying
> parameters or you can make your own method for job/pool share calculation.
> If I am not wrong there is another option just to extend a class xy (do not
> know which) and set jobs weights (check the docs from Harsh)
> 
> 
> On 7 March 2012 01:08, Ranjan Banerjee <rbanerjee2@wisc.edu> wrote:
> 
> > Hello Harsh,
> >    Thanks for the quick info. I will go through the specifics. I hope you
> > will address any further doubts that I have on the issue.
> >
> > Regards,
> > Ranjan
> >
> > On 03/06/12, Harsh J   wrote:
> > > Ranjan,
> > >
> > > Schedulers do not apply per-job. You need to change it at the JobTracker.
> > >
> > > Follow instructions at
> > > http://hadoop.apache.org/common/docs/r1.0.0/fair_scheduler.html to
> > > switch scheduler to FairScheduler.
> > >
> > > On Wed, Mar 7, 2012 at 4:08 AM, Ranjan Banerjee <rbanerjee2@wisc.edu>
> > wrote:
> > > >
> > > > Hello,
> > > >    I am relatively new to Hadoop. Started playing with it around two
> > weeks ago. I have finished running the canonical word count map reduce
> > example. My class project involves coming with a different scheduler for
> > Hadoop. I know that by default Hadoop uses the FIFO and it also has the
> > fair scheduler to be used. Can someone suggest where do I exactly write my
> > scheduler code and what change do I need to do to the Conf object so that
> > my scheduler is used by Hadoop and not the default scheduler in scheduling
> > map reduce jobs.
> > > >
> > > > Regards,
> > > > Ranjan
> > > >
> > > >
> > > >
> > > >
> > > > On 03/02/12, mohammed elsaeedy   wrote:
> > > >> Dear Mailing list,
> > > >>
> > > >>    I've been trying to simply build the hadoop source code on my
> > Macbook
> > > >> Pro. I tried following the tutorial mentioned
> > > >> here<http://wiki.apache.org/hadoop/EclipseEnvironment>.
> > > >> I grabbed the source code from github, and then I try  to run maven:
> > > >>
> > > >> mvn install -DskipTests
> > > >>
> > > >> mvn clean package -DskipTests -Pdist -Dtar -Dmaven.javadoc.skip=true
> > > >>
> > > >>
> > > >> but I always get the following error:
> > > >>
> > > >> *[INFO]
> > > >>
> > ------------------------------------------------------------------------*
> > > >> *[INFO] Total time: 14:48.283s*
> > > >> *[INFO] Finished at: Fri Mar 02 13:29:20 EET 2012*
> > > >> *[INFO] Final Memory: 68M/123M*
> > > >> *[INFO]
> > > >>
> > ------------------------------------------------------------------------*
> > > >> *[ERROR] Failed to execute goal
> > > >> org.apache.maven.plugins:maven-pdf-plugin:1.1:pdf (pdf) on project
> > > >> hadoop-distcp: Error during document generation: Error parsing
> > > >>
> > /Users/SaSa/Desktop/Hadoop/src/hadoop-common/hadoop-tools/hadoop-distcp/target/pdf/site.tmp/xdoc/index.xml:
> > > >> Error validating the model: Fatal error:*
> > > >> *[ERROR] Public ID: null*
> > > >> *[ERROR] System ID: http://maven.apache.org/xsd/xdoc-2.0.xsd*
> > > >> *[ERROR] Line number: 2699*
> > > >> *[ERROR] Column number: 5*
> > > >> *[ERROR] Message: The element type "xs:element" must be terminated
by
> > the
> > > >> matching end-tag "</xs:element>".*
> > > >> *[ERROR] -> [Help 1]*
> > > >> *[ERROR] *
> > > >> *[ERROR] To see the full stack trace of the errors, re-run Maven with
> > the
> > > >> -e switch.*
> > > >> *[ERROR] Re-run Maven using the -X switch to enable full debug
> > logging.*
> > > >> *[ERROR] *
> > > >> *[ERROR] For more information about the errors and possible solutions,
> > > >> please read the following articles:*
> > > >> *[ERROR] [Help 1]
> > > >>
> > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException*
> > > >> *[ERROR] *
> > > >> *[ERROR] After correcting the problems, you can resume the build with
> > the
> > > >> command*
> > > >> *[ERROR]   mvn <goals> -rf :hadoop-distcp*
> > > >>
> > > >> I googled for a similar problem, all I got was some people cleared
> > the m2
> > > >> and ivy2 cahce, and rerun the command. But unfortunately didn't work
> > for
> > > >> me. I've been stuck here, I really hope anyone can help me with this
> > issue.
> > > >>
> > > >> Thank you all.
> > > >> Regards,
> > > >> SaSa
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Mohammed El Sayed
> > > >> Computer Science Department
> > > >> King Abdullah University of Science and Technology
> > > >> 2351 - 4700 KAUST, Saudi Arabia
> > > >> Home Page <
> > http://cloud.kaust.edu.sa/SiteCollectionDocuments/melsayed.aspx>
> > > >
> > >
> > >
> > >
> > > --
> > > Harsh J
> >
> >


Mime
View raw message