hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin, Nick" <NiMar...@pssd.com>
Subject ORC 'BETWEEN' Error
Date Wed, 26 Feb 2014 16:55:32 GMT
Hi all,

(Running Hive 12.0)

I have two tables and both are stored as ORC. I attempted to insert via select from tbl1 to
tbl2 using 'BETWEEN' in my where clause to narrow down some dates. Something like so:

"Insert into tbl1 select col1, col2 from tbl2 where col1 between 2 and 4"

I kept hitting the error pasted below. So, I switched to a different approach to see if it
would work:

"Insert into tbl1 select col1,col2 from tbl2 where col1>=2 and col1<=4"

Hit the same error. When I just use "where col1>=2" in the where clause the insert will
run fine.

Is this expected?



2014-02-26 11:22:53,755 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2014-02-26 11:22:53,782 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2014-02-26 11:22:53,902 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded
properties from hadoop-metrics2.properties
2014-02-26 11:22:53,930 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink
ganglia started
2014-02-26 11:22:53,975 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
snapshot period at 10 second(s).
2014-02-26 11:22:53,975 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask
metrics system started
2014-02-26 11:22:53,985 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2014-02-26 11:22:53,985 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job,
Service: job_1392147432508_1108, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@249c2715)
2014-02-26 11:22:54,057 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before
retrying again. Got null now.
2014-02-26 11:22:54,352 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2014-02-26 11:22:54,363 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2014-02-26 11:22:54,409 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir
for child: /hdfs/01/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/02/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/03/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/04/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/05/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/06/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/07/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/08/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/09/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/10/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/11/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/12/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108
2014-02-26 11:22:54,481 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2014-02-26 11:22:54,486 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt
to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2014-02-26 11:22:54,542 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.id
is deprecated. Instead, use mapreduce.task.attempt.id
2014-02-26 11:22:54,542 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.is.map
is deprecated. Instead, use mapreduce.task.ismap
2014-02-26 11:22:54,543 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.local.dir
is deprecated. Instead, use mapreduce.cluster.local.dir
2014-02-26 11:22:54,543 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.cache.localFiles
is deprecated. Instead, use mapreduce.job.cache.local.files
2014-02-26 11:22:54,543 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.job.id
is deprecated. Instead, use mapreduce.job.id
2014-02-26 11:22:54,544 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.tip.id
is deprecated. Instead, use mapreduce.task.id
2014-02-26 11:22:54,544 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.partition
is deprecated. Instead, use mapreduce.task.partition
2014-02-26 11:22:54,545 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: job.local.dir
is deprecated. Instead, use mapreduce.job.local.dir
2014-02-26 11:22:54,546 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.cache.localArchives
is deprecated. Instead, use mapreduce.job.cache.local.archives
2014-02-26 11:22:54,710 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id
is deprecated. Instead, use dfs.metrics.session-id
2014-02-26 11:22:55,007 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree
: [ ]
2014-02-26 11:22:55,316 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Paths:/apps/hive/warehouse/edw.db/penopp_fsm/000000_0:0+116456645,/apps/hive/warehouse/edw.db/penopp_fsm/000001_0:0+116484938,/apps/hive/warehouse/edw.db/penopp_fsm/000002_0:0+116473509InputFormatClass:
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat

2014-02-26 11:22:55,370 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: <PERFLOG
method=deserializePlan>
2014-02-26 11:22:55,370 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: Deserializing
MapWork via kryo
2014-02-26 11:22:55,574 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: </PERFLOG
method=deserializePlan start=1393431775370 end=1393431775574 duration=204>
2014-02-26 11:22:55,708 INFO [main] org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: included
column ids = 0,1,2,3,4,5,6
2014-02-26 11:22:55,708 INFO [main] org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: included
columns names = col1,col2,col3,col4,col5,col6,col7
2014-02-26 11:22:55,728 INFO [main] org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: ORC pushdown
predicate: leaf-0 = (BETWEEN col7 201301 201312)
expr = leaf-0
2014-02-26 11:22:55,897 ERROR [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:myusername (auth:SIMPLE) cause:java.io.IOException: java.lang.reflect.InvocationTargetException
2014-02-26 11:22:55,898 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running
child : java.io.IOException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
        at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:343)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:290)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:404)
        at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:556)
        at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:167)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:408)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:329)
        ... 11 more
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.planReadPartialDataStreams(RecordReaderImpl.java:1962)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readPartialDataStreams(RecordReaderImpl.java:2118)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:1775)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:2148)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:2190)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:130)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rows(ReaderImpl.java:335)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:100)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:205)
        at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
        ... 16 more

2014-02-26 11:22:55,902 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the
task






Mime
View raw message