hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-hudi] eigakow commented on issue #1398: [SUPPORT] DeltaStreamer - NoClassDefFoundError for HiveDriver
Date Thu, 12 Mar 2020 08:21:14 GMT
eigakow commented on issue #1398: [SUPPORT] DeltaStreamer - NoClassDefFoundError for HiveDriver
URL: https://github.com/apache/incubator-hudi/issues/1398#issuecomment-598062351
 
 
   Yes, I have tried adding both hive-jdbc-2.1.1-cdh6.1.0.jar and hive-service-2.1.1-cdh6.1.0.jar
and ended up with another missing class error, therefore I was questioning if that is the
right path. Thank you for clarifying.
   
   With both jars added I am now getting ```Cannot find class 'org.apache.hudi.hadoop.HoodieParquetInputFormat'```
while hudi tries to create a new table 
   
   This class is present
   ```
   $ jar tf packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.5.1-incubating.jar
| grep HoodieParquetInputFormat
   org/apache/hudi/hadoop/HoodieParquetInputFormat.class
   ```
   Stacktrace:
   ```
   20/03/12 07:29:31 INFO hive.HiveSyncTool: Hive table test_hudi is not found. Creating it
   20/03/12 07:29:31 INFO hive.HoodieHiveClient: Creating table with CREATE EXTERNAL TABLE
 IF NOT EXISTS `fr24raw`.`test_hudi`( `_hoodie_commit_time` string, `_hoodie_commit_seqno`
string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name`
string, `flights` ARRAY< STRUCT< `fcode` : string, `finfo` : ARRAY< string>>>,
`full_count` int, `ts` string, `version` int) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION 'adl://XXX.azuredatalakestore.net/test-hudi'
   20/03/12 07:29:31 INFO hive.HoodieHiveClient: Executing SQL CREATE EXTERNAL TABLE  IF NOT
EXISTS `fr24raw`.`test_hudi`( `_hoodie_commit_time` string, `_hoodie_commit_seqno` string,
`_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name` string,
`flights` ARRAY< STRUCT< `fcode` : string, `finfo` : ARRAY< string>>>, `full_count`
int, `ts` string, `version` int) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION 'adl://XXX.azuredatalakestore.net/test-hudi'
   20/03/12 07:29:31 ERROR hive.HiveSyncTool: Got runtime exception when hive syncing
   org.apache.hudi.hive.HoodieHiveSyncException: Failed in executing SQL CREATE EXTERNAL TABLE
 IF NOT EXISTS `fr24raw`.`test_hudi`( `_hoodie_commit_time` string, `_hoodie_commit_seqno`
string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name`
string, `flights` ARRAY< STRUCT< `fcode` : string, `finfo` : ARRAY< string>>>,
`full_count` int, `ts` string, `version` int) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION 'adl://XXX.azuredatalakestore.net/test-hudi'
           at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:488)
           at org.apache.hudi.hive.HoodieHiveClient.createTable(HoodieHiveClient.java:276)
           at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:150)
           at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:118)
           at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:91)
           at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncHive(DeltaSync.java:481)
           at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:423)
           at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:238)
           at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:393)
           at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement:
FAILED: SemanticException Cannot find class 'org.apache.hudi.hadoop.HoodieParquetInputFormat'
           at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:266)
           at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:252)
           at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:318)
           at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:259)
           at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:486)
           ... 12 more
   Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement:
FAILED: SemanticException Cannot find class 'org.apache.hudi.hadoop.HoodieParquetInputFormat'
           at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
           at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:207)
           at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
           at org.apache.hive.service.cli.operation.Operation.run(Operation.java:266)
           at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:504)
           at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:490)
           at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:498)
           at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
           at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
           at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
           at java.security.AccessController.doPrivileged(Native Method)
           at javax.security.auth.Subject.doAs(Subject.java:422)
           at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
           at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
           at com.sun.proxy.$Proxy38.executeStatementAsync(Unknown Source)
           at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:295)
           at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:507)
           at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
           at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
           at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
           at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
           at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:605)
           at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
           ... 3 more
   Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Cannot find class 'org.apache.hudi.hadoop.HoodieParquetInputFormat'
           at org.apache.hadoop.hive.ql.parse.ParseUtils.ensureClassExists(ParseUtils.java:263)
           at org.apache.hadoop.hive.ql.parse.StorageFormat.fillStorageFormat(StorageFormat.java:57)
           at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:11666)
           at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10839)
           at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10949)
           at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10639)
           at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
           at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:600)
           at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1414)
           at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1391)
           at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:205)
           ... 26 more
   Caused by: java.lang.ClassNotFoundException: org.apache.hudi.hadoop.HoodieParquetInputFormat
           at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
           at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
           at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:348)
           at org.apache.hadoop.hive.ql.parse.ParseUtils.ensureClassExists(ParseUtils.java:261)
           ... 36 more
   20/03/12 07:29:31 INFO hive.metastore: Closed a connection to metastore, current connections:
0
   
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message