mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Eastman (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAHOUT-846) Improve Scalability of Gaussian Cluster For Wide Vectors
Date Wed, 19 Oct 2011 17:13:10 GMT

     [ https://issues.apache.org/jira/browse/MAHOUT-846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jeff Eastman updated MAHOUT-846:
--------------------------------

    Description: 
The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product
of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and
returns 0. Here's the code:
{noformat}
  public double pdf(VectorWritable vw) {
    Vector x = vw.get();
    // return the product of the component pdfs
    // TODO: is this reasonable? correct? It seems to work in some cases.
    double pdf = 1;
    for (int i = 0; i < x.size(); i++) {
      // small prior on stdDev to avoid numeric instability when stdDev==0
      pdf *= UncommonDistributions.dNorm(x.getQuick(i),
          getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
    }
    return pdf;
{noformat}
  }


  was:
The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product
of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and
returns 0. Here's the code:

  public double pdf(VectorWritable vw) {
    Vector x = vw.get();
    // return the product of the component pdfs
    // TODO: is this reasonable? correct? It seems to work in some cases.
    double pdf = 1;
    for (int i = 0; i < x.size(); i++) {
      // small prior on stdDev to avoid numeric instability when stdDev==0
      pdf *= UncommonDistributions.dNorm(x.getQuick(i),
          getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
    }
    return pdf;
  }


    
> Improve Scalability of Gaussian Cluster For Wide Vectors
> --------------------------------------------------------
>
>                 Key: MAHOUT-846
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-846
>             Project: Mahout
>          Issue Type: Improvement
>    Affects Versions: 0.5
>            Reporter: Jeff Eastman
>            Assignee: Jeff Eastman
>             Fix For: 0.6
>
>
> The pdf() implementation in GaussianCluster is pretty lame. It is computing a running
product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows
and returns 0. Here's the code:
> {noformat}
>   public double pdf(VectorWritable vw) {
>     Vector x = vw.get();
>     // return the product of the component pdfs
>     // TODO: is this reasonable? correct? It seems to work in some cases.
>     double pdf = 1;
>     for (int i = 0; i < x.size(); i++) {
>       // small prior on stdDev to avoid numeric instability when stdDev==0
>       pdf *= UncommonDistributions.dNorm(x.getQuick(i),
>           getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
>     }
>     return pdf;
> {noformat}
>   }

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message