Mailing-List: contact user-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mahout.apache.org
Received-SPF: pass (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CAJwFCa0vwAEVnnQg-_79JjNTT5nMMu-TR_Sye1B87XFsz8GS0Q@mail.gmail.com>
References: 
 <CAJBgT0n7Aq4UB=F0nG8mzfM+p5arfd9p1uaPHDgq-0UjtKRF7Q@mail.gmail.com>
	<CAKBEjTsOZOiFAYEpKhoVgCM_+VSF=bP4YL_BacPKfisXuF1hrg@mail.gmail.com>
	<CAJwFCa0mKDB4szUycHebfe0iJyPobD2wA=kaOG6YjM5rg49c7Q@mail.gmail.com>
	<CAJwFCa3SFqGAHT=jgp=jLHCXLJ=Byu_-M+wFOQumwsfrWW+GgQ@mail.gmail.com>
	<52F0B38E.6060905@apache.org>
	<CAJwFCa0vwAEVnnQg-_79JjNTT5nMMu-TR_Sye1B87XFsz8GS0Q@mail.gmail.com>
Date: Tue, 4 Feb 2014 19:39:27 +0100
Message-ID: 
 <CAJBgT0=6MfT87BsWwROeDT=CNBcy+JzXqzVvR-JuqNqev2KkRQ@mail.gmail.com>
Subject: Re: SGD classifier demo app
From: Frank Scholten <frank@frankscholten.nl>
To: user@mahout.apache.org
Cc: Sebastian Schelter <ssc@apache.org>
Content-Type: multipart/alternative; boundary=20cf300fa9f9c8c7d304f198f5f4

--20cf300fa9f9c8c7d304f198f5f4
Content-Type: text/plain; charset=ISO-8859-1

Thanks Ted!

Would indeed be a nice example to add.


On Tue, Feb 4, 2014 at 10:40 AM, Ted Dunning <ted.dunning@gmail.com> wrote:

> Yes.
>
>
> On Tue, Feb 4, 2014 at 1:31 AM, Sebastian Schelter <ssc@apache.org> wrote:
>
> > Would be great to add this as an example to Mahout's codebase.
> >
> >
> > On 02/04/2014 10:27 AM, Ted Dunning wrote:
> >
> >> Frank,
> >>
> >> I just munched on your code and sent a pull request.
> >>
> >> In doing this, I made a bunch of changes.  Hope you liked them.
> >>
> >> These include massive simplification of the reading and vectorization.
> >>   This wasn't strictly necessary, but it seemed like a good idea.
> >>
> >> More important was the way that I changed the vectorization.  For the
> >> continuous values, I added log transforms.  For the categorical values,
> I
> >> encoded as they are.  I also increased the feature vector size to 100 to
> >> avoid excessive collisions.
> >>
> >> In the learning code itself, I got rid of the use of index arrays in
> favor
> >> of shuffling the training data itself.  I also tuned the learning
> >> parameters a lot.
> >>
> >> The result is that the AUC that results is just a tiny bit less than 0.9
> >> which is pretty close to what I got in R.
> >>
> >> For everybody else, see
> >> https://github.com/tdunning/mahout-sgd-bank-marketing for my version
> and
> >> https://github.com/tdunning/mahout-sgd-bank-marketing/
> >> compare/frankscholten:master...masterfor
> >> my pull request.
> >>
> >>
> >>
> >> On Mon, Feb 3, 2014 at 3:57 PM, Ted Dunning <ted.dunning@gmail.com>
> >> wrote:
> >>
> >>
> >>> Johannes,
> >>>
> >>> Very good comments.
> >>>
> >>> Frank,
> >>>
> >>> As a benchmark, I just spent a few minutes building a logistic
> regression
> >>> model using R.  For this model AUC on 10% held-out data is about 0.9.
> >>>
> >>> Here is a gist summarizing the results:
> >>>
> >>> https://gist.github.com/tdunning/8794734
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Feb 3, 2014 at 2:41 PM, Johannes Schulte <
> >>> johannes.schulte@gmail.com> wrote:
> >>>
> >>>  Hi Frank,
> >>>>
> >>>> you are using the feature vector encoders which hash a combination of
> >>>> feature name and feature value to 2 (default) locations in the vector.
> >>>> The
> >>>> vector size you configured is 11 and this is imo very small to the
> >>>> possible
> >>>> combination of values you have for your data (education, marital,
> >>>> campaign). You can do no harm by using a much bigger cardinality (try
> >>>> 1000).
> >>>>
> >>>> Second, you are using a continuous value encoder with passing in the
> >>>> weight
> >>>> your are using as string (e.g. variable "pDays"). I am not quite sure
> >>>> about
> >>>> the reasons in th mahout code right now but the way it is implemented
> >>>> now,
> >>>> every unique value should end up in a different location because the
> >>>> continuous value is part of the hashing. Try adding the weight
> directly
> >>>> using a static word value encoder, addToVector("pDays",v,pDays)
> >>>>
> >>>> Last, you are also putting in the variable "campaign" as a continous
> >>>> variable which should be probably a categorical variable, so just
> added
> >>>> with a StaticWorldValueEncoder.
> >>>>
> >>>> And finally and probably most important after looking at your target
> >>>> variable: you are using a Dictionary for mapping either y or no to 0
> or
> >>>> 1.
> >>>> This is bad. Depending on what comes first in the data set, either a
> >>>> positive or negative example might be 0 or 1, totally random. Make a
> >>>> hard
> >>>> mapping from the possible values (y/n?) to zero and one, having yes
> the
> >>>> 1
> >>>> and no the zero.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Feb 3, 2014 at 9:33 PM, Frank Scholten <
> frank@frankscholten.nl
> >>>>
> >>>>> wrote:
> >>>>>
> >>>>
> >>>>  Hi all,
> >>>>>
> >>>>> I am exploring Mahout's SGD classifier and like some feedback
> because I
> >>>>> think I didn't properly configure things.
> >>>>>
> >>>>> I created an example app that trains an SGD classifier on the 'bank
> >>>>> marketing' dataset from UCI:
> >>>>> http://archive.ics.uci.edu/ml/datasets/Bank+Marketing
> >>>>>
> >>>>> My app is at:
> >>>>>
> >>>> https://github.com/frankscholten/mahout-sgd-bank-marketing
> >>>>
> >>>>>
> >>>>> The app reads a CSV file of telephone calls, encodes the features
> into
> >>>>> a
> >>>>> vector and tries to predict whether a customer answers yes to a
> >>>>> business
> >>>>> proposal.
> >>>>>
> >>>>> I do a few runs and measure accuracy but I'm I don't trust the
> results.
> >>>>> When I only use an intercept term as a feature I get around 88%
> >>>>> accuracy
> >>>>> and when I add all features it drops to around 85%. Is this perhaps
> >>>>>
> >>>> because
> >>>>
> >>>>> the dataset highly unbalanced? Most customers answer no. Or is the
> >>>>> classifier biased to predict 0 as the target code when it doesn't
> have
> >>>>>
> >>>> any
> >>>>
> >>>>> data to go with?
> >>>>>
> >>>>> Any other comments about my code or improvements I can make in the
> app
> >>>>>
> >>>> are
> >>>>
> >>>>> welcome! :)
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Frank
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>
> >
>

--20cf300fa9f9c8c7d304f198f5f4--