spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From FRosner <>
Subject [GitHub] spark pull request: SPARK-11258 Remove quadratic runtime complexit...
Date Fri, 23 Oct 2015 07:09:22 GMT
Github user FRosner commented on the pull request:
    @felixcheung the ticket is mentioning two numbers (
in the description. For a small data frame it is already a speedup of 4. But anyway - quadratic
complexity is the problem as you run into issues with increasing number of columns quickly.
We were not even able to load one of our data frames (> 1 Million rows, several hundred
of columns) with the old method but it ran through with the new one.
    I can provide some more benchmarks later today.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message