spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwendell <>
Subject [GitHub] spark pull request: [SPARK-3477] Clean up code in Yarn Client / Cl...
Date Thu, 11 Sep 2014 18:42:28 GMT
Github user pwendell commented on the pull request:
    Hey all - regarding backwards compatibility. I agree we definitely need to preserve all
of the publicly documented interfaces, including environment variables etc. And I think that
even includes the original yarn command line `Client` which was both undocumented and deprecated
in Spark 1.0 and is subsumed by spark submit.
    To me though, preserving programmatic linking against the `Client` class doesn't make
sense - it was never intended or documented to be used this way. Really it's an oversight
that this was even public in the first place, but I'm guessing it was just because it was
assumed to be self-evident that this exists for parsing command line options, etc and not
for linking against in a program. The classes are not in our public java doc, etc. Finally,
AFAIK we've already changed a bunch of things in e.g. `ClientArguments` in both Spark 1.0
and 1.1... including renaming that could break access to it from earlier versions of Spark.
So IMO this already isn't a public API and belongs as `private[spark]` (even before Spark
1.0). I am indeed curious if people are actually trying to use this.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message