hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3442) QuickSort may get into unbounded recursion
Date Wed, 11 Jun 2008 00:27:45 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Douglas updated HADOOP-3442:

    Attachment: 3442-4.patch

bq. I guess it is safe to choose a larger k. For example, with k = 8, the stack depth is bounded
by 250 in practice, but a lot fewer cases of input data patterns will need to invoke the heapsort.
bq. I think the 2*log n heuristic not only bail out of worst cases but also good cases since
it is overly strict.

The IntroSort [paper|http://citeseer.ist.psu.edu/musser97introspective.html] suggested that
2 * floor(log\(n)) produced good results. Given that the most degenerate cases cascade, deferring
the change to heapsort may effect longer running times. Still, it does seem low. I ran some
informal benchmarks with a random set of elements in nearly sorted, sawtooth, and pipe organ
patterns and using k = 2 switched to heapsort pretty aggressively, so I increased k to 4.
The average running times were all within a few hundred milliseconds, though; we probably
don't want to get too carried away optimizing and tweaking an operation typically accounting
for no more than a few seconds out of every spill.

> QuickSort may get into unbounded recursion
> ------------------------------------------
>                 Key: HADOOP-3442
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3442
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.17.0
>            Reporter: Runping Qi
>            Assignee: Chris Douglas
>            Priority: Blocker
>             Fix For: 0.17.1, 0.18.0
>         Attachments: 3442-0.patch, 3442-0v17.patch, 3442-1.patch, 3442-2.patch, 3442-3.patch,
3442-4.patch, CheckSortBuffer.java, HADOOP-3442.patch, overflow.zip, spillbuffers.patch

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message