spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shivaram Venkataraman (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-6840) SparkR: private package functions unavailable when using lapplyPartition in package
Date Fri, 10 Apr 2015 01:18:12 GMT
Shivaram Venkataraman created SPARK-6840:
--------------------------------------------

             Summary: SparkR: private package functions unavailable when using lapplyPartition
in package
                 Key: SPARK-6840
                 URL: https://issues.apache.org/jira/browse/SPARK-6840
             Project: Spark
          Issue Type: Bug
          Components: SparkR
    Affects Versions: 1.4.0
            Reporter: Shivaram Venkataraman


Developing package that imports SparkR. There is a function in that package that calls lapplyPartition
with a function argument that uses in its body some functions private to the package. When
run, the computation fails because R can not find the private function (details below). If
I fully qualify them with otherpackage:::private.function, the error moves down to the next
private function. This used to work some time ago, I've been working on other stuff for a
little while. This should also work by regular R scope rules. I apologize I don't have a minimal
test case ready, but this was discovered developing plyrmr and the list of dependencies is
long enough that  it's a little bit of a burden to make you install it. I think I can put
together a toy package to demonstrate the problem, if that helps.


Error in FUN(part) : could not find function "keys.spark"
Calls: source ... eval -> eval -> computeFunc -> <Anonymous> -> FUN ->
FUN
Execution halted
15/03/19 12:29:16 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
org.apache.spark.SparkException: R computation failed with
 Error in FUN(part) : could not find function "keys.spark"
Calls: source ... eval -> eval -> computeFunc -> <Anonymous> -> FUN ->
FUN
Execution halted
	at edu.berkeley.cs.amplab.sparkr.BaseRRDD.compute(RRDD.scala:80)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
	at org.apache.spark.scheduler.Task.run(Task.scala:54)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message