spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephen Haberman (JIRA)" <>
Subject [jira] [Commented] (SPARK-4877) userClassPathFirst doesn't handle user classes inheriting from parent
Date Wed, 04 Feb 2015 22:22:34 GMT


Stephen Haberman commented on SPARK-4877:

Hi Matt,

I know about the caching/LinkageError issue, because I saw it running a job; my patch has
a test that reproduced it and fixes it.

I'm still willing to believe loadClass might be preferred, but AFAICT it's not required, and
the current findClass approach is working fine (with this patch) in our production jobs.

(Note that I'm very open to refactoring this approach further in the future, especially to
introduce the Jetty/Hadoop concept of system classes, but our jobs have not ran into any issues.)

> userClassPathFirst doesn't handle user classes inheriting from parent
> ---------------------------------------------------------------------
>                 Key: SPARK-4877
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.2.0
>            Reporter: Stephen Haberman
> We're trying out userClassPathFirst.
> To do so, we make an uberjar that does not contain Spark or Scala classes (because we
want those to load from the parent classloader, otherwise we'll get errors like scala.Function0
!= scala.Function0 since they'd load from different class loaders).
> (Tangentially, some isolation classloaders like Jetty whitelist certain packages, like
spark/* and scala/*, to only come from the parent classloader, so that technically if the
user still messes up and leaks the Scala/Spark jars into their uberjar, it won't blow up;
this would be a good enhancement, I think.)
> Anyway, we have a custom Kryo registrar, which ships in our uberjar, but since it "extends
spark.KryoRegistrator", which is not in our uberjar, we get a ClassNotFoundException.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message