spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chutium <teng....@gmail.com>
Subject Re: share/reuse off-heap persisted (tachyon) RDD in SparkContext or saveAsParquetFile on tachyon in SQLContext
Date Tue, 12 Aug 2014 10:23:09 GMT
more interesting is if spark-shell started on master node (test01)

then

parquetFile.saveAsParquetFile("tachyon://test01.zala:19998/parquet_tablex")

14/08/12 11:42:06 INFO : initialize(tachyon://...
...
...
14/08/12 11:42:06 INFO : File does not exist:
tachyon://test01.zala:19998/parquet_tablex/_metadata
14/08/12 11:42:06 INFO : getWorkingDirectory: /
14/08/12 11:42:06 INFO :
create(tachyon://test01.zala:19998/parquet_tablex/_metadata, rw-r--r--,
true, 65536, 1, 33554432, null)
14/08/12 11:42:06 WARN : tachyon.home is not set. Using
/mnt/tachyon_default_home as the default value.
14/08/12 11:42:06 INFO : Trying to get local worker host : test01.zala
14/08/12 11:42:06 ERROR : No local worker on test01.zala
NoWorkerException(message:No local worker on test01.zala)
        at
tachyon.thrift.MasterService$user_getWorker_result$user_getWorker_resultStandardScheme.read(MasterService.java:25675)
        at
tachyon.thrift.MasterService$user_getWorker_result$user_getWorker_resultStandardScheme.read(MasterService.java:25652)
        at
tachyon.thrift.MasterService$user_getWorker_result.read(MasterService.java:25591)
        at
tachyon.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
        at
tachyon.thrift.MasterService$Client.recv_user_getWorker(MasterService.java:832)
        at
tachyon.thrift.MasterService$Client.user_getWorker(MasterService.java:818)
        at tachyon.master.MasterClient.user_getWorker(MasterClient.java:648)
        at tachyon.worker.WorkerClient.connect(WorkerClient.java:199)
        at tachyon.worker.WorkerClient.mustConnect(WorkerClient.java:360)
        at
tachyon.worker.WorkerClient.getUserUfsTempFolder(WorkerClient.java:298)
        at
tachyon.client.TachyonFS.createAndGetUserUfsTempFolder(TachyonFS.java:270)
        at tachyon.client.FileOutStream.<init>(FileOutStream.java:72)
        at tachyon.client.TachyonFile.getOutStream(TachyonFile.java:207)
        at tachyon.hadoop.AbstractTFS.create(AbstractTFS.java:102)
        at tachyon.hadoop.TFS.create(TFS.java:24)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:773)
        at
parquet.hadoop.ParquetFileWriter.writeMetadataFile(ParquetFileWriter.java:344)
        at
org.apache.spark.sql.parquet.ParquetTypesConverter$.writeMetaData(ParquetTypes.scala:345)
        at
org.apache.spark.sql.parquet.ParquetRelation$.createEmpty(ParquetRelation.scala:142)
        at
org.apache.spark.sql.parquet.ParquetRelation$.create(ParquetRelation.scala:120)
        at
org.apache.spark.sql.execution.SparkStrategies$ParquetOperations$.apply(SparkStrategies.scala:197)
        at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
        at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
        at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
        at
org.apache.spark.sql.catalyst.planning.QueryPlanner.apply(QueryPlanner.scala:59)
        at
org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:399)
        at
org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:397)
        at
org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:403)
        at
org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:403)
        at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:406)
        at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:406)
        at
org.apache.spark.sql.SchemaRDDLike$class.saveAsParquetFile(SchemaRDDLike.scala:77)
        at
org.apache.spark.sql.SchemaRDD.saveAsParquetFile(SchemaRDD.scala:103)
        at $line12.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:17)
        at $line12.$read$$iwC$$iwC$$iwC.<init>(<console>:22)
        at $line12.$read$$iwC$$iwC.<init>(<console>:24)
        at $line12.$read$$iwC.<init>(<console>:26)
        at $line12.$read.<init>(<console>:28)
        at $line12.$read$.<init>(<console>:32)
        at $line12.$read$.<clinit>(<console>)
        at $line12.$eval$.<init>(<console>:7)
        at $line12.$eval$.<clinit>(<console>)
        at $line12.$eval.$print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
        at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
        at
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)
        at
org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:814)
        at
org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:859)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:771)
        at
org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:616)
        at
org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:624)
        at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:629)
        at
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:954)
        at
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:902)
        at
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:902)
        at
scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:997)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at
org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:314)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:73)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
14/08/12 11:42:06 INFO : Trying to connect master @
test01.zala/10.10.114.120:19998
14/08/12 11:42:06 INFO : User registered at the master
test01.zala/10.10.114.120:19998 got UserId 32233
14/08/12 11:42:06 ERROR : No local worker on test01.zala
NoWorkerException(message:No local worker on test01.zala)
        at
tachyon.thrift.MasterService$user_getWorker_result$user_getWorker_resultStandardScheme.read(MasterService.java:25675)
        at
tachyon.thrift.MasterService$user_getWorker_result$user_getWorker_resultStandardScheme.read(MasterService.java:25652)
        at
tachyon.thrift.MasterService$user_getWorker_result.read(MasterService.java:25591)
        at
tachyon.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
        at
tachyon.thrift.MasterService$Client.recv_user_getWorker(MasterService.java:832)
        at
tachyon.thrift.MasterService$Client.user_getWorker(MasterService.java:818)
        at tachyon.master.MasterClient.user_getWorker(MasterClient.java:648)
        at tachyon.worker.WorkerClient.connect(WorkerClient.java:199)
        at tachyon.worker.WorkerClient.mustConnect(WorkerClient.java:360)
        at
tachyon.worker.WorkerClient.getUserUfsTempFolder(WorkerClient.java:298)
        at
tachyon.client.TachyonFS.createAndGetUserUfsTempFolder(TachyonFS.java:270)
        at tachyon.client.FileOutStream.<init>(FileOutStream.java:72)
        at tachyon.client.TachyonFile.getOutStream(TachyonFile.java:207)
        at tachyon.hadoop.AbstractTFS.create(AbstractTFS.java:102)
        at tachyon.hadoop.TFS.create(TFS.java:24)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:773)
        at
parquet.hadoop.ParquetFileWriter.writeMetadataFile(ParquetFileWriter.java:344)
        at
org.apache.spark.sql.parquet.ParquetTypesConverter$.writeMetaData(ParquetTypes.scala:345)
        at
org.apache.spark.sql.parquet.ParquetRelation$.createEmpty(ParquetRelation.scala:142)
        at
org.apache.spark.sql.parquet.ParquetRelation$.create(ParquetRelation.scala:120)
        at
org.apache.spark.sql.execution.SparkStrategies$ParquetOperations$.apply(SparkStrategies.scala:197)
        at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
        at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
        at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
        at
org.apache.spark.sql.catalyst.planning.QueryPlanner.apply(QueryPlanner.scala:59)
        at
org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:399)
        at
org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:397)
        at
org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:403)
        at
org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:403)
        at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:406)
        at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:406)
        at
org.apache.spark.sql.SchemaRDDLike$class.saveAsParquetFile(SchemaRDDLike.scala:77)
        at
org.apache.spark.sql.SchemaRDD.saveAsParquetFile(SchemaRDD.scala:103)
        at $line12.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:17)
        at $line12.$read$$iwC$$iwC$$iwC.<init>(<console>:22)
        at $line12.$read$$iwC$$iwC.<init>(<console>:24)
        at $line12.$read$$iwC.<init>(<console>:26)
        at $line12.$read.<init>(<console>:28)
        at $line12.$read$.<init>(<console>:32)
        at $line12.$read$.<clinit>(<console>)
        at $line12.$eval$.<init>(<console>:7)
        at $line12.$eval$.<clinit>(<console>)
        at $line12.$eval.$print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
        at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
        at
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)
        at
org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:814)
        at
org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:859)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:771)
        at
org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:616)
        at
org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:624)
        at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:629)
        at
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:954)
        at
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:902)
        at
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:902)
        at
scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:997)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at
org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:314)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:73)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)



infinite error message like
14/08/12 11:42:06 INFO : Trying to connect master @
test01.zala/10.10.114.120:19998
14/08/12 11:42:06 INFO : User registered at the master
test01.zala/10.10.114.120:19998 got UserId 32233
14/08/12 11:42:06 ERROR : No local worker on test01.zala
NoWorkerException(message:No local worker on test01.zala)

always "got UserId XXX" and then "No local worker on test01.zala"

save text files works fine with spark-shell on both master node and worker
node




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/share-reuse-off-heap-persisted-tachyon-RDD-in-SparkContext-or-saveAsParquetFile-on-tachyon-in-SQLCont-tp11897p11978.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message