spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tal Barda (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24192) Invalid Spark URL in local spark session since upgrading from org.apache.spark:spark-sql_2.11:2.2.1 to org.apache.spark:spark-sql_2.11:2.3.0
Date Wed, 09 May 2018 11:10:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16468708#comment-16468708
] 

Tal Barda commented on SPARK-24192:
-----------------------------------

Regarding the machine details:

*_lsb_release -a_* output:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty


_*uname -a*_ output is:
Linux railsonfire_be98de61-c2bc-4afa-af27-3fe1058e603d_6cc13d6a6b9f 4.4.0-121-generic #145~14.04.1-Ubuntu
SMP Mon Apr 16 18:40:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

> Invalid Spark URL in local spark session since upgrading from org.apache.spark:spark-sql_2.11:2.2.1
to org.apache.spark:spark-sql_2.11:2.3.0
> --------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-24192
>                 URL: https://issues.apache.org/jira/browse/SPARK-24192
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core, SQL
>    Affects Versions: 2.3.0
>            Reporter: Tal Barda
>            Priority: Major
>              Labels: HeartbeatReceiver, config, session, spark, spark-conf, spark-session
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> since updating to Spark 2.3.0, tests which are run in my CI (Codeship) fail due to a
allegedly invalid spark url when creating the (local) spark context. Here's a log from my _*mvn
clean install*_ command execution:
> {quote}{{2018-05-03 13:18:47.668 ERROR 5533 --- [ main] org.apache.spark.SparkContext
: Error initializing SparkContext. org.apache.spark.SparkException: Invalid Spark URL: spark://HeartbeatReceiver@railsonfire_61eb1c99-232b-49d0-abb5-a1eb9693516b_52bcc09bb48b:44284
at org.apache.spark.rpc.RpcEndpointAddress$.apply(RpcEndpointAddress.scala:66) ~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rpc.netty.NettyRpcEnv.asyncSetupEndpointRefByURI(NettyRpcEnv.scala:134)
~[spark-core_2.11-2.3.0.jar:2.3.0] at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
~[spark-core_2.11-2.3.0.jar:2.3.0] at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
~[spark-core_2.11-2.3.0.jar:2.3.0] at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:32)
~[spark-core_2.11-2.3.0.jar:2.3.0] at org.apache.spark.executor.Executor.<init>(Executor.scala:155)
~[spark-core_2.11-2.3.0.jar:2.3.0] at org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalSchedulerBackend.scala:59)
~[spark-core_2.11-2.3.0.jar:2.3.0] at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:126)
~[spark-core_2.11-2.3.0.jar:2.3.0] at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
~[spark-core_2.11-2.3.0.jar:2.3.0] at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
~[spark-core_2.11-2.3.0.jar:2.3.0] at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
[spark-core_2.11-2.3.0.jar:2.3.0] at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
[spark-sql_2.11-2.3.0.jar:2.3.0] at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
[spark-sql_2.11-2.3.0.jar:2.3.0] at scala.Option.getOrElse(Option.scala:121) [scala-library-2.11.8.jar:na]
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921) [spark-sql_2.11-2.3.0.jar:2.3.0]
at com.planck.spark_data_features_extractors.utils.SparkConfig.sparkSession(SparkConfig.java:43)
[classes/:na] at com.planck.spark_data_features_extractors.utils.SparkConfig$$EnhancerBySpringCGLIB$$66dd1f72.CGLIB$sparkSession$0(<generated>)
[classes/:na] at com.planck.spark_data_features_extractors.utils.SparkConfig$$EnhancerBySpringCGLIB$$66dd1f72$$FastClassBySpringCGLIB$$a213b647.invoke(<generated>)
[classes/:na] at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
[spring-core-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361)
[spring-context-5.0.2.RELEASE.jar:5.0.2.RELEASE] at com.planck.spark_data_features_extractors.utils.SparkConfig$$EnhancerBySpringCGLIB$$66dd1f72.sparkSession(<generated>)
[classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171] at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:579)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:583)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:91)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:372)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1344)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:583)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:91)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:372)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1344)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:758)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:868)
~[spring-context-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
~[spring-context-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:751)
~[spring-boot-2.0.0.M7.jar:2.0.0.M7] at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:387)
~[spring-boot-2.0.0.M7.jar:2.0.0.M7] at org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
~[spring-boot-2.0.0.M7.jar:2.0.0.M7] at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:138)
~[spring-boot-test-2.0.0.M7.jar:2.0.0.M7] at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:117)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:107)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.test.context.web.ServletTestExecutionListener.setUpRequestContextIfNecessary(ServletTestExecutionListener.java:190)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.test.context.web.ServletTestExecutionListener.prepareTestInstance(ServletTestExecutionListener.java:132)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:242)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:227)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:289)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
~[junit-4.12.jar:4.12] at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:291)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:246)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
~[junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
~[junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
~[junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
~[junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
~[junit-4.12.jar:4.12] at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
~[junit-4.12.jar:4.12] at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
~[surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
~[surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
~[surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
~[surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
~[surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
~[surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
~[surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
~[surefire-booter-2.20.1.jar:2.20.1]}}
> {quote}
>  
> my SparkConf object is initialized this way:
> {quote}new SparkConf().setAppName("SparkForFeaturesExtraction")
>  .setMaster("local)
>  .set("spark.driver.memory", "512m")
>  .set("spark.executor.memory", "512m")
>  .set("spark.cores.max", 1)
>  .set("spark.default.parallelism", 2)
>  .set("spark.sql.shuffle.partitions", 2)
> {quote}
>  
> also see the issue in https://stackoverflow.com/questions/49143271/invalid-spark-url-in-local-spark-session



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message