[ https://issues.apache.org/jira/browse/MATH644?page=com.atlassian.jira.plugin.system.issuetabpanels:commenttabpanel&focusedCommentId=13159987#comment13159987
]
Thomas Neidhart edited comment on MATH644 at 12/15/11 8:53 AM:

I digged a bit into the problem. The HypergeometricDistribution calculates the probability
for a given x using the following formula:
{code:java}
private double probability(int n, int m, int k, int x) {
return FastMath.exp(ArithmeticUtils.binomialCoefficientLog(m, x) +
ArithmeticUtils.binomialCoefficientLog(n  m, k  x) 
ArithmeticUtils.binomialCoefficientLog(n, k));
}
{code}
Thus it transforms the binomial coefficients to a logarithmic scale in order to cope with
the possibly large results (and to be able to compute the bincoeff at all). But, imho, the
reverse transformation is broken, as it does not take any scaling into account. As the coefficients
get larger (e.g. due to a large n), the differences between the terms will become smaller
in log scale, and thus incorrectly transformed back to linear scale. Taking scaling into account,
the exp function will most likely fail for large n.
I have created a simple test to illustrate the problem, the t{x} correspond to the binomial
coeff terms from the formula, diff is the input the the exp function. This loop simulates
an increasing n, and the expectation is that the result should get smaller with increasing
n:
{code}
t1=0.0, t2=4547.288942497606, t3=4770.9627189150215, diff=223.67377641741587, result=7.23957639711833E98
t1=0.0, t2=12183.221706275828, t3=12186.968419291079, diff=3.7467130152508616, result=0.023595175513309037
t1=0.0, t2=13444.672093808651, t3=13446.561352727935, diff=1.8892589192837477, result=0.15118380673528464
t1=0.0, t2=14186.229425843805, t3=14187.492505971342, diff=1.2630801275372505, result=0.2827816800804864
t1=0.0, t2=14713.395226772875, t3=14714.343882929534, diff=0.9486561566591263, result=0.3872610921706871
t1=0.0, t2=15122.726785860956, t3=15123.486358374357, diff=0.7595725134015083, result=0.46786639087791215
t1=0.0, t2=15457.396298892796, t3=15458.029636271298, diff=0.6333373785018921, result=0.5308173033122051
t1=0.0, t2=15740.484181590378, t3=15741.027263000607, diff=0.5430814102292061, result=0.5809553297318205
t1=0.0, t2=15985.787659011781, t3=15986.263000234962, diff=0.47534122318029404, result=0.6216728910682101
t1=0.0, t2=16202.21559868753, t3=16202.638224512339, diff=0.4226258248090744, result=0.6553237931479635
t1=0.0, t2=16395.855738580227, t3=16396.236174091697, diff=0.380435511469841, result=0.6835636445695273
{code}
hmm, after some second thoughts, I am not sure if the analysis is correct, and the problem
is hidden somewhere else.
was (Author: tn):
I digged a bit into the problem. The HypergeometricDistribution calculates the probability
for a given x using the following formula:
{code:java}
private double probability(int n, int m, int k, int x) {
return FastMath.exp(ArithmeticUtils.binomialCoefficientLog(m, x) +
ArithmeticUtils.binomialCoefficientLog(n  m, k  x) 
ArithmeticUtils.binomialCoefficientLog(n, k));
}
{code}
Thus it transforms the binomial coefficients to a logarithmic scale in order to cope with
the possibly large results (and to be able to compute the bincoeff at all). But, imho, the
reverse transformation is broken, as it does not take any scaling into account. As the coefficients
get larger (e.g. due to a large n), the differences between the terms will become smaller
in log scale, and thus incorrectly transformed back to linear scale. Taking scaling into account,
the exp function will most likely fail for large n.
I have created a simple test to illustrate the problem, the t{x} correspond to the binomial
coeff terms from the formula, diff is the input the the exp function. This loop simulates
an increasing n, and the expectation is that the result should get smaller with increasing
n:
{code}
t1=0.0, t2=4547.288942497606, t3=4770.9627189150215, diff=223.67377641741587, result=7.23957639711833E98
t1=0.0, t2=12183.221706275828, t3=12186.968419291079, diff=3.7467130152508616, result=0.023595175513309037
t1=0.0, t2=13444.672093808651, t3=13446.561352727935, diff=1.8892589192837477, result=0.15118380673528464
t1=0.0, t2=14186.229425843805, t3=14187.492505971342, diff=1.2630801275372505, result=0.2827816800804864
t1=0.0, t2=14713.395226772875, t3=14714.343882929534, diff=0.9486561566591263, result=0.3872610921706871
t1=0.0, t2=15122.726785860956, t3=15123.486358374357, diff=0.7595725134015083, result=0.46786639087791215
t1=0.0, t2=15457.396298892796, t3=15458.029636271298, diff=0.6333373785018921, result=0.5308173033122051
t1=0.0, t2=15740.484181590378, t3=15741.027263000607, diff=0.5430814102292061, result=0.5809553297318205
t1=0.0, t2=15985.787659011781, t3=15986.263000234962, diff=0.47534122318029404, result=0.6216728910682101
t1=0.0, t2=16202.21559868753, t3=16202.638224512339, diff=0.4226258248090744, result=0.6553237931479635
t1=0.0, t2=16395.855738580227, t3=16396.236174091697, diff=0.380435511469841, result=0.6835636445695273
{code}
> for the class of hypergeometric distribution, for some number the method "upperCumulativeProbability"
return a probability greater than 1 which is impossible.
> 
>
> Key: MATH644
> URL: https://issues.apache.org/jira/browse/MATH644
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2
> Reporter: marzieh
> Priority: Minor
> Labels: hypergeometric, probability
> Fix For: 2.2
>
>
> In windows 7, I used common.Math library. I used class "HypergeometricDistributionImpl"
and method "upperCumulativeProbability" of zero for distribution and the return value is larget
than 1. the following code is working error.
> HypergeometricDistributionImpl u = new HypergeometricDistributionImpl(14761461, 1035
,1841 );
> System.out.println(u.upperCumulativeProbability(0))
> Thanks

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
