Mailing-List: contact dev-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mahout.apache.org
Received-SPF: pass (nike.apache.org: domain of dlieu.7@gmail.com designates
 209.85.220.170 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CACYXym-e1v0vRznT=W1VR4-D_FT3Srj-XAhyTMD=LJW15CpqjA@mail.gmail.com>
References: 
 <CAJwFCa2i_7N7DG4RHWz4URja5U4WNKrKjfDeHq3sVLgNrdxpdQ@mail.gmail.com>
	<CACYXym-e1v0vRznT=W1VR4-D_FT3Srj-XAhyTMD=LJW15CpqjA@mail.gmail.com>
Date: Wed, 24 Aug 2011 15:35:14 -0700
Message-ID: 
 <CAPud8TqCLtWTtepUq1f8KO=T2emzkmA2cZRbAdPgNn_VSTm+Pw@mail.gmail.com>
Subject: Re: discussion of input conversions
From: Dmitriy Lyubimov <dlieu.7@gmail.com>
To: dev@mahout.apache.org
Cc: praneet mhatre <praneetmhatre@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

somewhat -1 too. Just because :)

as far as i understand, arff just contains a way to name attributes
and present types others than double, which is why it is not DRM and
DRM is not ARFF.

I'd rather re-engineer ARFF parser if needs be.


On Wed, Aug 24, 2011 at 3:16 PM, Jake Mannix <jake.mannix@gmail.com> wrote:
> My initial inclination is -1 on adding a GPL dependency.
>
> Can you spell out exactly what is meant by needing a "general input forma=
t"
> and "general transfer format". =A0We currently take in raw text, and then
> vectorize it. =A0 Are Vectors (with either hashed encoding, or with a
> dictionary
> file) not suitable as a format for some reason?
>
> =A0-jake
>
> On Wed, Aug 24, 2011 at 3:09 PM, Ted Dunning <ted.dunning@gmail.com> wrot=
e:
>
>> Praneet and I were just talking about a project he is working on to do w=
ith
>> higher-order learning methods such as boosting and feature sharding. =A0=
This
>> is all pretty much in the context of classification and possibly
>> clustering.
>>
>> The problems are:
>>
>> a) mahout doesn't have a general input format for classifiable data (thi=
s
>> has been discussed recently)
>>
>> b) hashed vector representations are not suitable for feature sharding
>> since
>> individual features may be redundantly represented in many locations.
>>
>> c) mahout doesn't have a reasonable data structure for general data
>> transfer
>> (related to -a-)
>>
>> One possible thought is that Mahout could introduce Weka as a dependency=
.
>>
>> The virtues would be:
>>
>> 1) Weka has ARFF as a data format and Instance as an object to satisfy (=
a)
>> and (c)
>>
>> 2) Weka provides a bunch of simple classifier algorithms which are not
>> individually scalable, but might be made to be so by model averaging or
>> feature sharding.
>>
>> 3) Praneet could finish his project very quickly.
>>
>> Any thoughts about this?
>>
>> The problems that I see with this include:
>>
>> A) Weka is GPL which might slow adoption of Mahout and would certainly
>> inhibit direct incorporation of any piece of Weka
>>
>> B) Weka appears to have not caught the maven bug which makes it harder t=
o
>> add as a dependency without actually distributing the weka jar.
>>
>> One possible work-around might be to reverse engineer something like
>> Instance and an ARFF reader/writer.
>>
>