Mailing-List: contact mahout-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mahout-dev@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of gsiasf@gmail.com designates
 209.85.160.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=qITGLGYBs8Ba1x0VhJ8wg6pXEFNGf26oWiXRlYXSI18i982Y4CgCLdg9PLD9COvJY2
         2pLnx1UvAGXK9ISbeoha8es5DYkumFbYi2sDOPeH5XyNGKa1zUkuuSNMBnKObCWqV3cO
         EK++I+nULsUA31JqZBR7akQzzDy2Ie6GIkpts=
Sender: Grant Ingersoll <gsiasf@gmail.com>
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1077)
Subject: Re: The new improved command-line: MahoutDriver (get it?)
From: Grant Ingersoll <gsingers@apache.org>
In-Reply-To: <4b124c311003021210t7488198nc7fb6b419f7498e3@mail.gmail.com>
Date: Fri, 19 Mar 2010 08:34:00 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <EE299DB1-6B8C-44BF-813C-BBFB5EEF9E35@apache.org>
References: <4b124c311003021210t7488198nc7fb6b419f7498e3@mail.gmail.com>
To: mahout-dev@lucene.apache.org

RE: MahoutDriver (get it?)

Perhaps we should have called it MahoutMahout?

At any rate, very good stuff.

On Mar 2, 2010, at 3:10 PM, Jake Mannix wrote:

> Hey all,
>=20
>  Just an update on the new-and-improved command-line "UI" we have now.
> After a ton of iterations back and forth with Drew (thanks!), =
MAHOUT-301
> has been committed, and brings with it the easy ability to trim down =
your
> long long command lines for most of our *Driver main() methods, by =
saving
> your default command-line arguments for various drivers in properties =
files
> (which are then overridable via the command line), either locally or =
on
> hadoop.  Feature-set is as follows (usage after that):
>=20
>  Either from the binary distribution or from source (after having done =
"mvn
> install", naturally), this is the setup - there are a bunch of =
properties
> files with a kludgey format (because I didn't want to dig into the xml
> rathole, and while a nice flexible schema is nice, I opted to follow =
the
> YAGNI principle) :
>=20
>  *) there is a new directory "conf" at the top level (of the binary =
dist,
> as well as source), which contains a bunch of *.props files: one =
special one
> called driver.classes.props, which has the mapping between (the keys)
> fully-qualified class name of a class which has a main() method, and =
the
> "short-name" (the values) and brief description.  The current file is =
just
> the following:
>=20
> ###
> org.apache.mahout.utils.vectors.VectorDumper =3D vectordump : Dump =
vectors
> from a sequence file to text
> org.apache.mahout.utils.clustering.ClusterDumper =3D clusterdump : =
Dump
> cluster output to text
> org.apache.mahout.utils.SequenceFileDumper =3D seqdumper : Generic =
Sequence
> File dumper
> org.apache.mahout.clustering.kmeans.KMeansDriver =3D kmeans : K-means
> clustering
> org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansDriver =3D fkmeans =
: Fuzzy
> K-means clustering
> org.apache.mahout.clustering.lda.LDADriver =3D lda : Latent Dirchlet
> Allocation
> org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver =3D fpg : Frequent =
Pattern
> Growth
> org.apache.mahout.clustering.dirichlet.DirichletDriver =3D dirichlet :
> Dirichlet Clustering
> org.apache.mahout.clustering.meanshift.MeanShiftCanopyDriver =3D =
meanshift :
> Mean Shift clustering
> org.apache.mahout.clustering.canopy.CanopyDriver =3D canopy : Canopy
> clustering
> org.apache.mahout.utils.vectors.lucene.Driver =3D lucene.vector : =
Generate
> Vectors from a Lucene index
> org.apache.mahout.text.SequenceFilesFromDirectory =3D seqdirectory : =
Generate
> sequence files (of Text) from a directory
> org.apache.mahout.text.SparseVectorsFromSequenceFiles =3D seq2sparse: =
Sparse
> Vector generation from Text sequence files
> org.apache.mahout.text.WikipediaToSequenceFile =3D seqwiki : Wikipedia =
xml
> dump to sequence file
> org.apache.mahout.classifier.bayes.TestClassifier =3D testclassifier : =
Test
> Bayes Classifier
> org.apache.mahout.classifier.bayes.TrainClassifier =3D trainclassifier =
: Train
> Bayes Classifier
> org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver =3D =
svd :
> Lanczos Singular Value Decomposition
> org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob =3D =
cleansvd :
> Cleanup and verification of SVD output
> ###
>=20
> It's meant to be read into java.util.Properties, where the values on =
the
> right hand side are further split by ":" into the short-name (to be =
used on
> the command-line) and the description (printed to stdout if an invalid =
input
> is made or "-h" is used with no class name to run).  *If there are =
missing
> classes from this list, please add them!*
>=20
>  *) there are also a bunch of files in conf/ which are named
> <shortName>.props, where <shortName> is one of the =
driver.classes.props
> above.  These files are  mostly empty now (well, commented out), but =
for
> example, conf/svd.props is currently:
>=20
> #i|input =3D
> #o|output =3D
> #nr|numRows =3D
> #nc|numCols =3D
> #r|rank =3D
> #t|tempDir =3D
>=20
> the format of these props files is that the key is of the form
> "singleDashCmdLineOpt|doubleDashCmdLineOpt", (if there is no "|" in =
the key,
> the short and long form will be assumed to be the same) and the value =
is
> whatever you would want that option to be (does not currently support
> options with no value, this is a TODO).   So for example if you had a
> command line such as:
>=20
>> $MAHOUT_HOME/bin/mahout svd --input /path/to/input -o /path/to/output =
-nr
> <numRows> --numCols <numCols> -r <rank> -t /tmp/svd
>=20
> You could just uncomment the lines in conf/svd.conf as
>=20
> i|input =3D /path/to/input
> o|output =3D /path/to/output
> nr|numRows =3D <numRows>
> nc|numCols =3D <numCols>
> r|rank =3D <rank>
> t|tempDir =3D /tmp/svd
>=20
> and run as
>=20
>> $MAHOUT_HOME/bin/mahout svd
>=20
> If you wanted to run a second time, but you didn't want to overwrite =
your
> old results, you could then do
>=20
>> $MAHOUT_HOME/bin/mahout svd -o /path/to/newOutput
>=20
> which would override /path/to/output and instead use =
/path/to/newOutput,
> with all the other properties coming from the svd.props.
>=20
>  *) the $MAHOUT_HOME/conf directory is just a template - the mahout =
shell
> script adds $MAHOUT_CONF_DIR to the classpath (or $MAHOUT_HOME/conf if
> $MAHOUT_CONF_DIR is not defined), and MahoutDriver reads the =
properties
> files from the classpath.
>=20
>  *) running on Hadoop:  if your $HADOOP_HOME and $HADOOP_CONF_DIR are =
set,
> the mahout shell script automatically launches your requested main =
method to
> your hadoop cluster, otherwise it's run locally.
>=20
>  *) if your main() isn't defined in driver.classes.properties, that's =
ok,
> it'll still run via:
>=20
>=20
> $MAHOUT_HOME/bin/mahout org.apache.mahout.blah.blah.SomeOtherDriver =
[remaining
> args]
>=20
> and in fact, if you put =
"org.apache.mahout.blah.blah.SomeOtherDriver.props"
> on your classpath, and has the format for the <shortName>.props listed
> above, it will be used for default properties for this class.
>=20
> --------
>=20
> I'll put this up in some nicer form for the wiki in the next couple of =
days.
>=20
>=20
> Try out various driver classes that you use - we all use different =
ones, so
> getting some dev/user manual test coverage would be nice, because it's =
kinda
> tricky to unit test shell scripts and command line args and env =
variables
> (and running on a real cluster, etc...).  We should try to fix any =
bugs
> before release.
>=20
> Feedback welcome.  It's hacky, but it adds some useful functionality, =
and we
> can clean up the props-file syntax (or ditch it for =
xml/yaml/json/whatever)
> as needed later.
>=20
>  -jake

--------------------------
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem using Solr/Lucene: =
http://www.lucidimagination.com/search