mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brenden Matthews" <bren...@diddyinc.com>
Subject Re: Review Request: Hadoop changes; see below.
Date Mon, 08 Apr 2013 22:20:30 GMT


> On April 8, 2013, 9:55 p.m., Vinod Kone wrote:
> > hadoop/TUTORIAL.sh, line 569
> > <https://reviews.apache.org/r/10346/diff/1/?file=279031#file279031line569>
> >
> >     What compression does J represent? It doesn't work on my MAC (OSX 10.8).
> >     
> >     How about.. s/-cjf/-czf/

This is to use the XZ format instead of gzip.  Support for XZ should be included in modern
GNU systems (I don't know about others).  It's nice but not necessary.  See http://tukaani.org/xz/format.html
for details.


> On April 8, 2013, 9:55 p.m., Vinod Kone wrote:
> > hadoop/TUTORIAL.sh, lines 581-589
> > <https://reviews.apache.org/r/10346/diff/1/?file=279031#file279031line581>
> >
> >     I think you added this because the jobtracker was failing on your Mac? I would
rather we find and fix that bug, than having to skip it. It just sets a better precedent for
our code.
> >     
> >     Can you file a bug (with a stack trace and logs) on your tutorial output?

I added it because a) it was failing and b) it isn't necessary to run it every time you rebuild
the package.  I'll have to check again to see if it's still not working.


> On April 8, 2013, 9:55 p.m., Vinod Kone wrote:
> > hadoop/hadoop-2.0.0-mr1-cdh4.1.2_hadoop-env.sh.patch, line 10
> > <https://reviews.apache.org/r/10346/diff/1/?file=279032#file279032line10>
> >
> >     We definitely want to be able to run this both locally and on the cluster. And
your change, would render local runs in a non-working state.
> >     
> >     I've sent a review out that should work for both local and remote runs. I will
get it committed. You can base your changes off that to patch cdh4.2.0_hadoop-env.sh.

If you run from ./build/hadoop instead of ./ it should work (or at least it does for me).
 I think the latter is more correct than the former, but perhaps you have a better solution.


> On April 8, 2013, 9:55 p.m., Vinod Kone wrote:
> > hadoop/TUTORIAL.sh, line 165
> > <https://reviews.apache.org/r/10346/diff/1/?file=279031#file279031line165>
> >
> >     i wasn't able to confirm whether skipping the compression level while extracting
always works with all versions of tar.
> >     
> >     so, lets stick with specifying it (here it is 'z') unless we are sure. i'm going
to revert my change in the launcher (in a review that i sent last week) to do the same.

It should work on modern GNU systems, but I can't speak for Mac OS.  Presumably (from your
description) it does not.


> On April 8, 2013, 9:55 p.m., Vinod Kone wrote:
> > hadoop/TUTORIAL.sh, line 547
> > <https://reviews.apache.org/r/10346/diff/1/?file=279031#file279031line547>
> >
> >     Is this required by the pre-package script? If yes, please add a comment here.

In my case I use it, but it's not required.  I use the hook scripts for doing things like
including custom configuration parameters (i.e., 'cp /etc/hadoop/config/* config/') before
the package is created, and copying the package to HDFS after it's ready.


> On April 8, 2013, 9:55 p.m., Vinod Kone wrote:
> > hadoop/TUTORIAL.sh, line 549
> > <https://reviews.apache.org/r/10346/diff/1/?file=279031#file279031line549>
> >
> >     Are these scripts only applicable for Cloudera versions? If yes, may be we can
just test against that, like we do everywhere else.

My scripts are version independent, but you could certainly write scripts that depend on a
particular version.


- Brenden


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10346/#review18797
-----------------------------------------------------------


On April 8, 2013, 9:21 p.m., Brenden Matthews wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10346/
> -----------------------------------------------------------
> 
> (Updated April 8, 2013, 9:21 p.m.)
> 
> 
> Review request for mesos.
> 
> 
> Description
> -------
> 
>     - Added CDH4.2.0 support to TUTORIAL.sh
>     - Use .tar.xz for Hadoop dist (LZMA)
>     - Fixed Hadoop dist extraction in launcher
>     - Add pre/post packaging hooks to TUTORIAL.sh
>     - Don't run the jobtracker at the end by default
> 
> 
> Diffs
> -----
> 
>   hadoop/Makefile.am bd8919a 
>   hadoop/TUTORIAL.sh b0f77f3 
>   hadoop/hadoop-2.0.0-mr1-cdh4.1.2_hadoop-env.sh.patch c61ccec 
>   hadoop/hadoop-2.0.0-mr1-cdh4.2.0_hadoop-env.sh.patch PRE-CREATION 
>   hadoop/hadoop-2.0.0-mr1-cdh4.2.0_mesos.patch PRE-CREATION 
>   src/launcher/launcher.cpp eb8d6fb 
> 
> Diff: https://reviews.apache.org/r/10346/diff/
> 
> 
> Testing
> -------
> 
> Works in production on Hadoop at airbnb.
> 
> 
> Thanks,
> 
> Brenden Matthews
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message