impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amos Bird <amosb...@gmail.com>
Subject Re: Error encounted when doing 'bin/load_data.py -w all'
Date Tue, 16 Aug 2016 09:26:18 GMT

Hmm, building hadoop-lzo trapped me with more errors.
build-native-non-win:
    [javah] Exception in thread "main" java.lang.NullPointerException
    [javah] 	at com.sun.tools.javah.JavahTask.run(JavahTask.java:503)
    [javah] 	at com.sun.tools.javah.JavahTask.run(JavahTask.java:329)
    [javah] 	at com.sun.tools.javah.Main.main(Main.java:46)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.620 s
[INFO] Finished at: 2016-08-16T01:12:28-08:00
[INFO] Final Memory: 21M/1166M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (build-native-non-win)
on project hadoop-lzo: An Ant BuildException has occured: compilation failed
[ERROR] around Ant part ...<javah classpath="${build.classes}" destdir="${build.native}/src/com/hadoop/compression/lzo"
force="yes" verbose="yes">... @ 12:122 in /home/amos/hadoop-lzo/target/antrun/build-build-native-non-win.xml

> Alternatively, you could also to try to only load data for the non-LZO
> formats.
I think I should go with masking the lzo format. However, I couldn't
find a way to only load non-lzo formats. I removed all the lzo
declarations in the testdata csv files, while loading process still
complained about the missing of hadoop-lzo.

Alex Behm writes:

> Looks like the root cause is that Hadoop's LZO codec could not be found.
>
> You probably need to fetch and build this:
> https://github.com/twitter/hadoop-lzo
>
> bin/impala-config.sh assumes that you've checked out that repo into a
> specific directory $IMPALA_HOME/../hadoop-lzo
>
> Alternatively, you could also to try to only load data for the non-LZO
> formats.
>
> Alex
>
> On Mon, Aug 15, 2016 at 7:05 PM, Amos Bird <amosbird@gmail.com> wrote:
>
>> Hi Alex,
>> Thanks for the reply! I have indeed get them all up and running. Here is
>> the jps list:
>>
>> 13184
>> 12833 LlamaAMMain
>> 14081 HQuorumPeer
>> 40870 Jps
>> 11302 DataNode
>> 15081 RunJar
>> 15722 RunJar
>> 13259
>> 11980 NodeManager
>> 11244 DataNode
>> 14700 HRegionServer
>> 11278 NameNode
>> 14353 HRegionServer
>> 11218 DataNode
>> 11923 NodeManager
>> 11955 ResourceManager
>> 12982 Bootstrap
>> 14166 HMaster
>> 11896 NodeManager
>> 16089 RunJar
>> 14527 HRegionServer
>>
>> I also played around a few SQLs inside impala. It works fine. In fact I
>> was able to create the test data before. I have no clue what happened in
>> the newest impala build.
>>
>>
>> Alex Behm writes:
>>
>> > Hi Amos!
>> >
>> > load-data.py assumes that you have a running cluster. You need to first
>> get
>> > these working:
>> > testdata/bin/run-all.sh
>> > bin/start-impala-cluster.py
>> >
>> > The first command starts all dependent services like HDFS, YARN, Hive
>> > Metastore, Hive HS2, etc.
>> > The second command starts an Impala mini-cluster with 3 nodes. This
>> command
>> > assumes all dependent services are already running/
>> >
>> > Hope it helps!
>> >
>> > Alex
>> >
>> > On Mon, Aug 15, 2016 at 5:20 AM, Amos Bird <amosbird@gmail.com> wrote:
>> >
>> >>
>> >> I was trying to build a new test warehouse. After successfully running
>> >> 'bin/create_testdata.sh', I did 'bin/load_data.py -w all'.
>> Unfortunately it
>> >> ended up with this:
>> >>
>> >> ERROR : Job Submission failed with exception 'java.io.IOException(java.
>> >> util.concurrent.ExecutionException: java.io.IOException: Cannot create
>> an
>> >> instance of InputFormat class org.apache.hadoop.mapred.TextInputFormat
>> as
>> >> specified in mapredWork!)'
>> >> java.io.IOException: java.util.concurrent.ExecutionException:
>> >> java.io.IOException: Cannot create an instance of InputFormat class
>> >> org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork!
>> >>   at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(
>> >> CombineHiveInputFormat.java:544)
>> >>   at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(
>> >> JobSubmitter.java:332)
>> >>   at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(
>> >> JobSubmitter.java:324)
>> >>   at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
>> >> JobSubmitter.java:200)
>> >>   at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
>> >>   at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
>> >>   at java.security.AccessController.doPrivileged(Native Method)
>> >>   at javax.security.auth.Subject.doAs(Subject.java:422)
>> >>   at org.apache.hadoop.security.UserGroupInformation.doAs(
>> >> UserGroupInformation.java:1693)
>> >>   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
>> >>   at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:578)
>> >>   at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:573)
>> >>   at java.security.AccessController.doPrivileged(Native Method)
>> >>   at javax.security.auth.Subject.doAs(Subject.java:422)
>> >>   at org.apache.hadoop.security.UserGroupInformation.doAs(
>> >> UserGroupInformation.java:1693)
>> >>   at org.apache.hadoop.mapred.JobClient.submitJobInternal(
>> >> JobClient.java:573)
>> >>   at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:564)
>> >>   at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(
>> >> ExecDriver.java:430)
>> >>   at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(
>> >> MapRedTask.java:137)
>> >>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
>> >>   at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(
>> >> TaskRunner.java:100)
>> >>   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1782)
>> >>   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1539)
>> >>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1318)
>> >>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1127)
>> >>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1120)
>> >>   at org.apache.hive.service.cli.operation.SQLOperation.
>> >> runQuery(SQLOperation.java:191)
>> >>   at org.apache.hive.service.cli.operation.SQLOperation.access$
>> >> 100(SQLOperation.java:79)
>> >>   at org.apache.hive.service.cli.operation.SQLOperation$2$1.
>> >> run(SQLOperation.java:245)
>> >>   at java.security.AccessController.doPrivileged(Native Method)
>> >>   at javax.security.auth.Subject.doAs(Subject.java:422)
>> >>   at org.apache.hadoop.security.UserGroupInformation.doAs(
>> >> UserGroupInformation.java:1693)
>> >>   at org.apache.hive.service.cli.operation.SQLOperation$2.run(
>> >> SQLOperation.java:258)
>> >>   at java.util.concurrent.Executors$RunnableAdapter.
>> >> call(Executors.java:511)
>> >>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> >>   at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> >> ThreadPoolExecutor.java:1142)
>> >>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> >> ThreadPoolExecutor.java:617)
>> >>   at java.lang.Thread.run(Thread.java:745)
>> >> Caused by: java.util.concurrent.ExecutionException:
>> java.io.IOException:
>> >> Cannot create an instance of InputFormat class org.apache.hadoop.mapred.
>> TextInputFormat
>> >> as specified in mapredWork!
>> >>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> >>   at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> >>   at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(
>> >> CombineHiveInputFormat.java:532)
>> >>   ... 37 more
>> >> Caused by: java.io.IOException: Cannot create an instance of InputFormat
>> >> class org.apache.hadoop.mapred.TextInputFormat as specified in
>> mapredWork!
>> >>   at org.apache.hadoop.hive.ql.io.HiveInputFormat.
>> getInputFormatFromCache(
>> >> HiveInputFormat.java:211)
>> >>   at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$
>> >> CheckNonCombinablePathCallable.call(CombineHiveInputFormat.java:111)
>> >>   at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$
>> >> CheckNonCombinablePathCallable.call(CombineHiveInputFormat.java:88)
>> >>   ... 4 more
>> >> Caused by: java.lang.RuntimeException: Error in configuring object
>> >>   at org.apache.hadoop.util.ReflectionUtils.setJobConf(
>> >> ReflectionUtils.java:109)
>> >>   at org.apache.hadoop.util.ReflectionUtils.setConf(
>> >> ReflectionUtils.java:75)
>> >>   at org.apache.hadoop.util.ReflectionUtils.newInstance(
>> >> ReflectionUtils.java:133)
>> >>   at org.apache.hadoop.hive.ql.io.HiveInputFormat.
>> getInputFormatFromCache(
>> >> HiveInputFormat.java:203)
>> >>   ... 6 more
>> >> Caused by: java.lang.reflect.InvocationTargetException
>> >>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >>   at sun.reflect.NativeMethodAccessorImpl.invoke(
>> >> NativeMethodAccessorImpl.java:62)
>> >>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> >> DelegatingMethodAccessorImpl.java:43)
>> >>   at java.lang.reflect.Method.invoke(Method.java:498)
>> >>   at org.apache.hadoop.util.ReflectionUtils.setJobConf(
>> >> ReflectionUtils.java:106)
>> >>   ... 9 more
>> >> Caused by: java.lang.IllegalArgumentException: Compression codec
>> >> com.hadoop.compression.lzo.LzoCodec not found.
>> >>   at org.apache.hadoop.io.compress.CompressionCodecFactory.
>> >> getCodecClasses(CompressionCodecFactory.java:135)
>> >>   at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>
>> >> (CompressionCodecFactory.java:175)
>> >>   at org.apache.hadoop.mapred.TextInputFormat.configure(
>> >> TextInputFormat.java:45)
>> >>   ... 14 more
>> >> Caused by: java.lang.ClassNotFoundException: Class
>> >> com.hadoop.compression.lzo.LzoCodec not found
>> >>   at org.apache.hadoop.conf.Configuration.getClassByName(
>> >> Configuration.java:2105)
>> >>   at org.apache.hadoop.io.compress.CompressionCodecFactory.
>> >> getCodecClasses(CompressionCodecFactory.java:128)
>> >>   ... 16 more
>> >>
>> >> ERROR : FAILED: Execution Error, return code 1 from
>> >> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>> >> INFO  : Completed executing command(queryId=amos_
>> >> 20160815034646_1d786772-c41e-4804-9d3c-dc768656ca3a); Time taken: 0.475
>> >> seconds
>> >> Error: Error while processing statement: FAILED: Execution Error, return
>> >> code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>> >> (state=08S01,code=1)
>> >> java.sql.SQLException: Error while processing statement: FAILED:
>> Execution
>> >> Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>> >>   at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:279)
>> >>   at org.apache.hive.beeline.Commands.executeInternal(
>> Commands.java:893)
>> >>   at org.apache.hive.beeline.Commands.execute(Commands.java:1079)
>> >>   at org.apache.hive.beeline.Commands.sql(Commands.java:976)
>> >>   at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1089)
>> >>   at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:921)
>> >>   at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:899)
>> >>   at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:841)
>> >>   at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(
>> >> BeeLine.java:482)
>> >>   at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
>> >>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >>   at sun.reflect.NativeMethodAccessorImpl.invoke(
>> >> NativeMethodAccessorImpl.java:62)
>> >>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> >> DelegatingMethodAccessorImpl.java:43)
>> >>   at java.lang.reflect.Method.invoke(Method.java:498)
>> >>   at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>> >>   at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>> >>
>> >> It seems like a mr framework is needed to be running, but
>> >> 'testdata/bin/run-all.sh' doesn't start it.
>> >>
>> >> Any help is much appreciated.
>> >>
>> >> regards,
>> >> Amos.
>> >>
>> >>
>> >>
>>
>>


Mime
View raw message