hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (HIVE-8704) HivePassThroughOutputFormat cannot proxy more than one kind of OF (in one process)
Date Mon, 03 Nov 2014 01:06:35 GMT


Josh Elser commented on HIVE-8704:

Nice find, [~sushanth]! Thanks for getting to the bottom of this.

> HivePassThroughOutputFormat cannot proxy more than one kind of OF (in one process)
> ----------------------------------------------------------------------------------
>                 Key: HIVE-8704
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: StorageHandler
>    Affects Versions: 0.14.0
>            Reporter: Sushanth Sowmyan
>            Assignee: Sushanth Sowmyan
>         Attachments: HIVE-8704.patch
> HivePassThroughOutputFormat is a wrapper HiveOutputFormat used by hive to allow access
to StorageHandlers that use mapred OutputFormats as their primary implementation point, and
do not implement HiveOutputFormat.
> However, HivePassThroughOutputFormat(henceforth called PTOF) has one major bug - it tracks
the underlying outputformat that it is proxying by means of a static string in HiveFileFormatUtils.
There are a few problems with this.
> a) For starters, it means that a given process can only process one PTOF-based output
format. So, in the case of a HS2 instance, where one thread is attempting to start a job based
on HBase and another on Accumulo will cause a problem, and will overwrite each others' "real"
output format. This leads to bugs where a person trying to use a hbase table gets stack traces
from Accumulo like the following:
> {noformat}
> ERROR exec.Task: Job Submission failed with exception 'java.lang.NullPointerException(Expected
Accumulo table name to be provided in job configuration)'
> java.lang.NullPointerException: Expected Accumulo table name to be provided in job configuration
> 	at
> 	at
> 	at
> 	at
> 	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.checkOutputSpecs(
> 	at
> 	at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(
> 	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
> 	at org.apache.hadoop.mapreduce.Job$
> 	at org.apache.hadoop.mapreduce.Job$
> 	at Method)
> 	at
> 	at
> 	at org.apache.hadoop.mapreduce.Job.submit(
> 	at org.apache.hadoop.mapred.JobClient$
> 	at org.apache.hadoop.mapred.JobClient$
> 	at Method)
> 	at
> 	at
> 	at org.apache.hadoop.mapred.JobClient.submitJobInternal(
> 	at org.apache.hadoop.mapred.JobClient.submitJob(
> 	at
> 	at
> 	at org.apache.hadoop.hive.ql.exec.Task.executeTask(
> 	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(
> 	at org.apache.hadoop.hive.ql.Driver.launchTask(
> 	at org.apache.hadoop.hive.ql.Driver.execute(
> 	at org.apache.hadoop.hive.ql.Driver.runInternal(
> 	at
> 	at
> 	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(
> 	at org.apache.hive.service.cli.operation.SQLOperation.access$100(
> 	at org.apache.hive.service.cli.operation.SQLOperation$1$
> 	at Method)
> 	at
> 	at
> 	at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(
> 	at org.apache.hive.service.cli.operation.SQLOperation$
> 	at java.util.concurrent.Executors$
> 	at
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(
> 	at java.util.concurrent.ThreadPoolExecutor$
> 	at
> {noformat}
> b) There is a bug in HiveFileFormatUtils.getOutputFormatSubstitute, which, after it determines
that PTOF should act as a substitute for a process, winds up registering PTOF in the substitute
map. This seems innocuous, but in result, because it winds up registering it as a substitute,
winds up short-circuiting and avoiding the path where it sets the real OF the next time. This
is a problem, because if the same job were to prepare writing to a HBase table, then followed
by preparing to write to an Accumulo table, followed by preparing to write to a HBase table,
then the second time HBase comes along, the underlying "real" OF is set to accumulo, and the
HBase map look up short circuits and avoids the path that would reset the real OF back to

This message was sent by Atlassian JIRA

View raw message