hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-7950) StorageHandler resources aren't added to Tez Session if already Session is already Open
Date Thu, 04 Sep 2014 15:34:51 GMT

     [ https://issues.apache.org/jira/browse/HIVE-7950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Josh Elser updated HIVE-7950:
-----------------------------
    Description: 
Was trying to run some queries using the AccumuloStorageHandler when using the Tez execution
engine. Some things that classes which were added to tmpjars weren't making it into the container.
When a Tez Session is already open, as is the normal case when simply using the `hive` command,
the resources aren't added.


  was:
Was trying to run some queries using the AccumuloStorageHandler when using the Tez execution
engine. Some things that I've noticed already (probably more as I can get past the ones I
already found):

* Jars added to the classpath via tmpjars (which is done by the copied HBase Utils class)
aren't available in the Tez Map task -- need to compare to HBaseStorageHandler and see if
there is something magic happening
* Configuration generated by the AccumuloStorageHandler doesn't make it all the way to the
Configuration passed to the AccumuloOutputFormat (probably AccumuloInputFormat, too)

{noformat}
2014-09-03 01:28:45,357 ERROR [TezChild] org.apache.hadoop.hive.ql.exec.tez.TezProcessor:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row {"row":"a","col":"d"}
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.processRow(MapRecordProcessor.java:195)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:161)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:309)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:180)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:172)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:167)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing
row {"row":"a","col":"d"}
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:550)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.processRow(MapRecordProcessor.java:183)
	... 15 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.IOException: java.lang.IllegalStateException: Instance has not been configured for
AccumuloOutputFormat
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:459)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:540)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
	at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
	at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:540)
	... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.IllegalStateException:
Instance has not been configured for AccumuloOutputFormat
	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:286)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:496)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:448)
	... 23 more
Caused by: java.io.IOException: java.lang.IllegalStateException: Instance has not been configured
for AccumuloOutputFormat
	at org.apache.accumulo.core.client.mapred.AccumuloOutputFormat.getRecordWriter(AccumuloOutputFormat.java:553)
	at org.apache.hadoop.hive.ql.io.HivePassThroughOutputFormat.getHiveRecordWriter(HivePassThroughOutputFormat.java:113)
	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:296)
	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:283)
	... 25 more
Caused by: java.lang.IllegalStateException: Instance has not been configured for AccumuloOutputFormat
	at org.apache.accumulo.core.client.mapreduce.lib.impl.ConfiguratorBase.getInstance(ConfiguratorBase.java:335)
	at org.apache.accumulo.core.client.mapred.AccumuloOutputFormat.getInstance(AccumuloOutputFormat.java:229)
	at org.apache.accumulo.core.client.mapred.AccumuloOutputFormat$AccumuloRecordWriter.<init>(AccumuloOutputFormat.java:402)
	at org.apache.accumulo.core.client.mapred.AccumuloOutputFormat.getRecordWriter(AccumuloOutputFormat.java:551)
	... 28 more
{noformat}


> StorageHandler resources aren't added to Tez Session if already Session is already Open
> ---------------------------------------------------------------------------------------
>
>                 Key: HIVE-7950
>                 URL: https://issues.apache.org/jira/browse/HIVE-7950
>             Project: Hive
>          Issue Type: Bug
>          Components: StorageHandler, Tez
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>             Fix For: 0.14.0
>
>
> Was trying to run some queries using the AccumuloStorageHandler when using the Tez execution
engine. Some things that classes which were added to tmpjars weren't making it into the container.
When a Tez Session is already open, as is the normal case when simply using the `hive` command,
the resources aren't added.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message