mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robin Anil (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAHOUT-391) Make vector more space efficient with variable-length encoding, et al
Date Wed, 05 May 2010 12:07:02 GMT

    [ https://issues.apache.org/jira/browse/MAHOUT-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12864300#action_12864300
] 

Robin Anil commented on MAHOUT-391:
-----------------------------------

No Idea why. I am doing a clean install. Anyways builds fine without the test. Here is the
comparison on unigram reuters vector dataset


$ ls -l reuters-vectors/tfidf/vectors/
-rwxrwxrwx  1   18237354 May  5 17:33 part-00000
 
$ ls -l reuters-vector-small/tfidf/vectors/
-rwxrwxrwx  1   18152858 May  5 17:24 part-00000


> Make vector more space efficient with variable-length encoding, et al
> ---------------------------------------------------------------------
>
>                 Key: MAHOUT-391
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-391
>             Project: Mahout
>          Issue Type: Improvement
>    Affects Versions: 0.3
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>            Priority: Minor
>             Fix For: 0.4
>
>         Attachments: MAHOUT-391.patch
>
>
> There are a few things we can do to make Vector representations smaller on disk:
> - Use variable-length encoding for integer values like size and element indices in sparse
representations
> - Further, delta-encode indices in sequential representations
> - Let caller specify that precision isn't crucial in values, allowing it to store values
as floats
> Since indices are usually small-ish, I'd guess this saves 2 bytes or so on average, out
of 12 bytes per element now.
> Using floats where applicable saves another 4. Not bad.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message