spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-19101) Spark Beeline catchs an exeception when run command " load data inpath '/data/test/test.csv' overwrite into table db.test partition(area='021')"
Date Wed, 11 Jan 2017 09:57:58 GMT

     [ https://issues.apache.org/jira/browse/SPARK-19101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen resolved SPARK-19101.
-------------------------------
    Resolution: Not A Problem

> Spark Beeline catchs an exeception when run command " load data inpath '/data/test/test.csv'
 overwrite into table db.test partition(area='021')"
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-19101
>                 URL: https://issues.apache.org/jira/browse/SPARK-19101
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.1
>         Environment: spark2.0.1
>            Reporter: Xiaochen Ouyang
>
> firtstly,two commands as follow:
> 1:load data  inpath '/data/test/lte_cm_projdata_520000.csv' overwrite into table  db.lte_cm_projdata
partition(p_provincecode=520000);
> 2:load data local  inpath '/home/mr/lte_cm_projdata_520000.csv' overwrite into table
 db.lte_cm_projdata partition(p_provincecode=520000);
> the first command run failed,but the second command run success.
> beeline execption:
> 0: jdbc:hive2://10.43.156.221:18000> load data  inpath '/data/test/lte_cm_projdata_520000.csv'
overwrite into table  db.lte_cm_projdata partition(p_provincecode=520000);
> Error: java.lang.reflect.InvocationTargetException (state=,code=0)
> ThriftServer2 logs :
> 2017-01-06 15:16:16,518 INFO HiveMetaStore: 58: get_partition_with_auth : db=zxvmax tbl=lte_cm_projdata[520000]
> 2017-01-06 15:16:16,518 INFO audit: ugi=root    ip=unknown-ip-addr      cmd=get_partition_with_auth
: db=zxvmax tbl=lte_cm_projdata[520000]
> 2017-01-06 15:16:16,611 ERROR SparkExecuteStatementOperation: Error executing query,
currentState RUNNING, 
> java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.spark.sql.hive.client.Shim_v0_14.loadPartition(HiveShim.scala:622)
>         at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadPartition$1.apply$mcV$sp(HiveClientImpl.scala:635)
>         at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadPartition$1.apply(HiveClientImpl.scala:635)
>         at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadPartition$1.apply(HiveClientImpl.scala:635)
>         at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:280)
>         at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:227)
>         at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:226)
>         at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:269)
>         at org.apache.spark.sql.hive.client.HiveClientImpl.loadPartition(HiveClientImpl.scala:634)
>         at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadPartition$1.apply$mcV$sp(HiveExternalCatalog.scala:279)
>         at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadPartition$1.apply(HiveExternalCatalog.scala:271)
>         at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadPartition$1.apply(HiveExternalCatalog.scala:271)
>         at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:72)
>         at org.apache.spark.sql.hive.HiveExternalCatalog.loadPartition(HiveExternalCatalog.scala:271)
>         at org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadPartition(SessionCatalog.scala:317)
>         at org.apache.spark.sql.execution.command.LoadDataCommand.run(tables.scala:325)
>         at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:60)
>         at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:58)
>         at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
>         at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>         at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
>         at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
>         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>         at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
>         at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
>         at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:86)
>         at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:86)
>         at org.apache.spark.sql.Dataset.<init>(Dataset.scala:186)
>         at org.apache.spark.sql.Dataset.<init>(Dataset.scala:167)
>         at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:65)
>         at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:605)
>         at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:685)
>         at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:222)
>         at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:166)
>         at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:163)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
>         at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1.run(SparkExecuteStatementOperation.scala:176)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source hdfs://nameservice/data/test/lte_cm_projdata_520000.csv
to destination hdfs://nameservice/db/telecom/lte/cm/lte_cm_projdata/p_provincecode=520000/lte_cm_projdata_520000.csv
>         at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2644)
>         at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2911)
>         at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1398)
>         at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1324)
>         ... 48 more
> Caused by: java.io.IOException: Filesystem closed
>         at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:838)
>         at org.apache.hadoop.hdfs.DFSClient.getEZForPath(DFSClient.java:3350)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$43.doCall(DistributedFileSystem.java:2030)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$43.doCall(DistributedFileSystem.java:2026)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.getEZForPath(DistributedFileSystem.java:2026)
>         at org.apache.hadoop.hdfs.client.HdfsAdmin.getEncryptionZoneForPath(HdfsAdmin.java:310)
>         at org.apache.hadoop.hive.shims.Hadoop23Shims$HdfsEncryptionShim.isPathEncrypted(Hadoop23Shims.java:1221)
>         at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2607)
>         ... 51 more
> 2017-01-06 15:16:16,613 ERROR SparkExecuteStatementOperation: Error running hive query:

> org.apache.hive.service.cli.HiveSQLException: java.lang.reflect.InvocationTargetException
>         at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:264)
>         at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:166)
>         at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:163)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
>         at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1.run(SparkExecuteStatementOperation.scala:176)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message