hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Bug of InputSampler
Date Thu, 05 May 2011 03:40:50 GMT
Jeff,

Is this similar to what
https://issues.apache.org/jira/browse/MAPREDUCE-1987 had pointed out?

On Fri, Apr 8, 2011 at 2:03 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> Hi all,
>
> I found a probable bug of InputSampler in method writePartitionFile
>
>    for(int i = 1; i < numPartitions; ++i) {
>      int k = Math.round(stepSize * i);
>      while (last >= k && comparator.compare(samples[last], samples[k])
==
> 0) {
>        ++k;
>      }
>      writer.append(samples[k], nullValue);
>      last = k;
>    }
>
>
> In the line of writer.append(samples[k], nullValue), the k may be already
> out of array index. maybe we should add one line to test whether it has been
> out of array index as following
>  for(int i = 1; i < numPartitions; ++i) {
>      int k = Math.round(stepSize * i);
>      while (last >= k && comparator.compare(samples[last], samples[k])
==
> 0) {
>        ++k;
>      }
>      if (k>=samples.length)
>         break;
>
>      writer.append(samples[k], nullValue);
>      last = k;
>    }
>
>
> --
> Best Regards
>
> Jeff Zhang
>



-- 
Harsh J

Mime
View raw message