hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin, Nick" <NiMar...@pssd.com>
Subject Re: ORC 'BETWEEN' Error
Date Fri, 28 Feb 2014 00:37:29 GMT
Cool, figured you guys were on it but couldn't locate the JIRA for some reason. I'll blame
user error :)

Thanks for the fix!

Nick


Sent from my iPhone

On Feb 27, 2014, at 6:09 PM, "Prasanth Jayachandran" <pjayachandran@hortonworks.com<mailto:pjayachandran@hortonworks.com>>
wrote:

Hi Martin

This is an known issue and its fixed in hive trunk. It should be available in 0.13 release.

https://issues.apache.org/jira/browse/HIVE-5601

Thanks
Prasanth Jayachandran

On Feb 26, 2014, at 8:55 AM, Martin, Nick <NiMartin@pssd.com<mailto:NiMartin@pssd.com>>
wrote:

Hi all,

(Running Hive 12.0)

I have two tables and both are stored as ORC. I attempted to insert via select from tbl1 to
tbl2 using ‘BETWEEN’ in my where clause to narrow down some dates. Something like so:

“Insert into tbl1 select col1, col2 from tbl2 where col1 between 2 and 4”

I kept hitting the error pasted below. So, I switched to a different approach to see if it
would work:

“Insert into tbl1 select col1,col2 from tbl2 where col1>=2 and col1<=4”

Hit the same error. When I just use “where col1>=2” in the where clause the insert
will run fine.

Is this expected?



2014-02-26 11:22:53,755 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2014-02-26 11:22:53,782 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2014-02-26 11:22:53,902 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded
properties from hadoop-metrics2.properties
2014-02-26 11:22:53,930 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink
ganglia started
2014-02-26 11:22:53,975 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
snapshot period at 10 second(s).
2014-02-26 11:22:53,975 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask
metrics system started
2014-02-26 11:22:53,985 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2014-02-26 11:22:53,985 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job,
Service: job_1392147432508_1108, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@249c2715)
2014-02-26 11:22:54,057 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before
retrying again. Got null now.
2014-02-26 11:22:54,352 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2014-02-26 11:22:54,363 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2014-02-26 11:22:54,409 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir
for child: /hdfs/01/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/02/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/03/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/04/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/05/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/06/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/07/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/08/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/09/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/10/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/11/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/12/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108
2014-02-26 11:22:54,481 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2014-02-26 11:22:54,486 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2014-02-26 11:22:54,542 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.id
is deprecated. Instead, use mapreduce.task.attempt.id
2014-02-26 11:22:54,542 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.is.map
is deprecated. Instead, use mapreduce.task.ismap
2014-02-26 11:22:54,543 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.local.dir
is deprecated. Instead, use mapreduce.cluster.local.dir
2014-02-26 11:22:54,543 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.cache.localFiles
is deprecated. Instead, use mapreduce.job.cache.local.files
2014-02-26 11:22:54,543 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.job.id
is deprecated. Instead, use mapreduce.job.id
2014-02-26 11:22:54,544 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.tip.id
is deprecated. Instead, use mapreduce.task.id
2014-02-26 11:22:54,544 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.partition
is deprecated. Instead, use mapreduce.task.partition
2014-02-26 11:22:54,545 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: job.local.dir
is deprecated. Instead, use mapreduce.job.local.dir
2014-02-26 11:22:54,546 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.cache.localArchives
is deprecated. Instead, use mapreduce.job.cache.local.archives
2014-02-26 11:22:54,710 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id
is deprecated. Instead, use dfs.metrics.session-id
2014-02-26 11:22:55,007 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree
: [ ]
2014-02-26 11:22:55,316 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Paths:/apps/hive/warehouse/edw.db/penopp_fsm/000000_0:0+116456645,/apps/hive/warehouse/edw.db/penopp_fsm/000001_0:0+116484938,/apps/hive/warehouse/edw.db/penopp_fsm/000002_0:0+116473509InputFormatClass:
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat

2014-02-26 11:22:55,370 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: <PERFLOG
method=deserializePlan>
2014-02-26 11:22:55,370 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: Deserializing
MapWork via kryo
2014-02-26 11:22:55,574 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: </PERFLOG
method=deserializePlan start=1393431775370 end=1393431775574 duration=204>
2014-02-26 11:22:55,708 INFO [main] org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: included
column ids = 0,1,2,3,4,5,6
2014-02-26 11:22:55,708 INFO [main] org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: included
columns names = col1,col2,col3,col4,col5,col6,col7
2014-02-26 11:22:55,728 INFO [main] org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: ORC pushdown
predicate: leaf-0 = (BETWEEN col7 201301 201312)
expr = leaf-0
2014-02-26 11:22:55,897 ERROR [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:myusername (auth:SIMPLE) cause:java.io.IOException: java.lang.reflect.InvocationTargetException
2014-02-26 11:22:55,898 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running
child : java.io.IOException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
        at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:343)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:290)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:404)
        at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:556)
        at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:167)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:408)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:329)
        ... 11 more
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.planReadPartialDataStreams(RecordReaderImpl.java:1962)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readPartialDataStreams(RecordReaderImpl.java:2118)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:1775)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:2148)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:2190)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:130)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rows(ReaderImpl.java:335)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:100)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:205)
        at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
        ... 16 more

2014-02-26 11:22:55,902 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the
task


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.

Mime
View raw message