Mailing-List: contact mahout-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mahout-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of jake.mannix@gmail.com
 designates 209.85.211.183 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=DsL8S11IiHd0rR+dQTzTp2jpj7GeFqvRP3SSqrqLxtuA1giKR8UG+96Gi8Ul8EDBo2
         oETkuMWvD093eMQ8Vvqr/YJXn5iCvnW1X6ODdaZyPr52V7Zb9Tz11OzS8paujL2FYUaT
         gq7jcc1dTM+sTxrjp6eKJkWNsojj1Z4oXJPnY=
MIME-Version: 1.0
In-Reply-To: <C06106D7-EC48-46F5-AEEE-74C299A476A0@transpac.com>
References: <5E00122C-1C77-4623-922E-867502CD09AA@apache.org>
	 <C06106D7-EC48-46F5-AEEE-74C299A476A0@transpac.com>
Date: Tue, 3 Nov 2009 10:14:11 -0800
Message-ID: <4b124c310911031014n6a352705qc937cfdc158c71cc@mail.gmail.com>
Subject: Re: http://bixolabs.com/datasets/public-terabyte-dataset-project/
From: Jake Mannix <jake.mannix@gmail.com>
To: mahout-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=0016361e7b8000c12f04777b7670

--0016361e7b8000c12f04777b7670
Content-Type: text/plain; charset=ISO-8859-1

Well the minimum size, for the IntDoubleVector which isn't yet in trunk
(it's on Ted's patch which hasn't worked its way in yet) would entail one
int and one double per unique term in the document, so that's 12 bytes
each.  Typical documents have lots of repeat terms, but most terms are
smaller than 12 bytes as well... so the fraction is probably more than 10%
and less than 50% is my guess.  But I'm sure others around here have more
experience producing large vector sets out of the text in Mahout.

  -jake

On Tue, Nov 3, 2009 at 7:49 AM, Ken Krugler <kkrugler_lists@transpac.com>wrote:

>
> On Nov 3, 2009, at 5:43am, Grant Ingersoll wrote:
>
>  Might be of interest to all you Mahouts out there...
>> http://bixolabs.com/datasets/public-terabyte-dataset-project/
>>
>> Would be cool to get this converted over to our vector format so that we
>> can cluster, etc.
>>
>
>
> How much additional space would be required for the vectors, in some
> optimal compressed format? Say as a percentage of raw text size.
>
> I'm asking because I have some flexibility in the processing and associated
> metadata I can store as part of the dataset.
>
> -- Ken
>
> --------------------------------------------
> Ken Krugler
> +1 530-210-6378
> http://bixolabs.com
> e l a s t i c   w e b   m i n i n g
>
>
>
>
>

--0016361e7b8000c12f04777b7670--