kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "codingforfun@126.com" <codingfor...@126.com>
Subject Re: Kylin to PostgreSQL Error in Cube build Step 1
Date Tue, 15 Oct 2019 23:50:16 GMT
Hi, Molotov, because postgresql's syntax and metadata have certain specialities, need to do
some development work. PR https://github.com/apache/kylin/pull/747 <https://github.com/apache/kylin/pull/747>
id doing this kind of thing, it is in review now.

> 在 2019年10月15日,20:54,Andrey Molotov <molotovav.bcs@gmail.com> 写道:
> 
> Hello, everyone.
> I’ve set up Kylin to access a PostgreSQL Database using JDBC as described in http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html
.
> 
> I’ve also set kylin.source.default=16 and kylin.source.hive.enable.quote=false in kylin.properties.
> 
> But when I try to build a cube a get an error on #1 Step Name: Sqoop To Flat Hive Table.
> 
> My Kylin Version is 2.6.4.
> 
> Here is log:
> 
>                java.io.IOException: OS command error exit with return code: 1, error
message: Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
> 
> SLF4J: Class path contains multiple SLF4J bindings.
> 
> SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 
> SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 
> SLF4J: Found binding in [jar:file:/opt/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> 
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 
> 2019-10-15 08:40:23,908 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
> 
> 2019-10-15 08:40:23,936 WARN tool.BaseSqoopTool: Setting your password on the command-line
is insecure. Consider using -P instead.
> 
> 2019-10-15 08:40:24,004 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit
driver however appropriate connection manager is not being set (via --connection-manager).
Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify
explicitly which connection manager should be used next time.
> 
> 2019-10-15 08:40:24,017 INFO manager.SqlManager: Using default fetchSize of 1000
> 
> 2019-10-15 08:40:24,017 INFO tool.CodeGenTool: Beginning code generation
> 
> 2019-10-15 08:40:24,164 INFO manager.SqlManager: Executing SQL statement: SELECT "installations"."city"
AS "INSTALLATIONS_CITY", "installations"."device_type" AS "INSTALLATIONS_DEVICE_TYPE", "installations"."install_datetime"
> 
> FROM "data"."installations" AS "installations"
> 
> WHERE 1 = 1 AND ("installations"."install_datetime" >= '2019-01-01' AND "installations"."install_datetime"
< '2019-01-03') AND  (1 = 0)
> 
> 2019-10-15 08:40:24,176 INFO manager.SqlManager: Executing SQL statement: SELECT "installations"."city"
AS "INSTALLATIONS_CITY", "installations"."device_type" AS "INSTALLATIONS_DEVICE_TYPE", "installations"."install_datetime"
> 
> FROM "data"."installations" AS "installations"
> 
> WHERE 1 = 1 AND ("installations"."install_datetime" >= '2019-01-01' AND "installations"."install_datetime"
< '2019-01-03') AND  (1 = 0)
> 
> 2019-10-15 08:40:24,200 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop
> 
> Note: /tmp/sqoop-hadoop/compile/33bbb7f633bb5f8338ed0a8e1e7ce3cc/QueryResult.java uses
or overrides a deprecated API.
> 
> Note: Recompile with -Xlint:deprecation for details.
> 
> 2019-10-15 08:40:25,545 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/33bbb7f633bb5f8338ed0a8e1e7ce3cc/QueryResult.jar
> 
> 2019-10-15 08:40:25,564 INFO mapreduce.ImportJobBase: Beginning query import.
> 
> 2019-10-15 08:40:25,565 INFO Configuration.deprecation: mapred.job.tracker is deprecated.
Instead, use mapreduce.jobtracker.address
> 
> 2019-10-15 08:40:25,650 INFO Configuration.deprecation: mapred.jar is deprecated. Instead,
use mapreduce.job.jar
> 
> 2019-10-15 08:40:26,327 INFO Configuration.deprecation: mapred.map.tasks is deprecated.
Instead, use mapreduce.job.maps
> 
> 2019-10-15 08:40:26,392 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
> 
> 2019-10-15 08:40:26,640 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding
for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1571114338286_0004
> 
> 2019-10-15 08:40:33,776 INFO db.DBInputFormat: Using read commited transaction isolation
> 
> 2019-10-15 08:40:33,777 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN("installations"."install_datetime"),
MAX("installations"."install_datetime")
> 
> FROM "data"."installations" AS "installations" WHERE `INSTALLATIONS`.`INSTALL_DATETIME`
>= '2019-01-01' AND `INSTALLATIONS`.`INSTALL_DATETIME` < '2019-01-03'
> 
> 2019-10-15 08:40:33,785 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/hadoop/.staging/job_1571114338286_0004
> 
> 2019-10-15 08:40:33,789 ERROR tool.ImportTool: Import failed: java.io.IOException: org.postgresql.util.PSQLException:
ERROR: syntax error at or near "."
> 
>  Position: 164
> 
>                at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:207)
> 
>                at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:310)
> 
>                at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:327)
> 
>                at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:200)
> 
>                at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
> 
>                at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
> 
>                at java.security.AccessController.doPrivileged(Native Method)
> 
>                at javax.security.auth.Subject.doAs(Subject.java:422)
> 
>                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> 
>                at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
> 
>                at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588)
> 
>                at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:200)
> 
>                at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:173)
> 
>                at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:270)
> 
>                at org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:748)
> 
>                at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:522)
> 
>                at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
> 
>                at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> 
>                at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> 
>                at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> 
>                at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
> 
>                at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
> 
>                at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
> 
> Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "."
> 
>  Position: 164
> 
>                at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2497)
> 
>                at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2233)
> 
>                at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
> 
>                at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446)
> 
>                at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370)
> 
>                at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311)
> 
>                at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297)
> 
>                at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274)
> 
>                at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:225)
> 
>                at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:178)
> 
>                ... 22 more
> 
> 
> 
> The command is:
> 
> /opt/sqoop/bin/sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true  -Dmapreduce.job.queuename=default
--connect "jdbc:postgresql://172.0.0.1:8080/database" --driver org.postgresql.Driver --username
test --password test --query "SELECT \"installations\".\"city\" AS \"INSTALLATIONS_CITY\",
\"installations\".\"device_type\" AS \"INSTALLATIONS_DEVICE_TYPE\", \"installations\".\"install_datetime\"
> 
> FROM \"data\".\"installations\" AS \"installations\"
> 
> WHERE 1 = 1 AND (\"installations\".\"install_datetime\" >= '2019-01-01' AND \"installations\".\"install_datetime\"
< '2019-01-03') AND \$CONDITIONS" --target-dir hdfs://localhost:9000/kylin/kylin_metadata/kylin-530cd3bb-f258-6ff8-9c38-9514a2258848/kylin_intermediate_7_5c51bb9d_b1ad_1135_a08f_12139ba64070
--split-by \"installations\".\"install_datetime\" --boundary-query "SELECT MIN(\"installations\".\"install_datetime\"),
MAX(\"installations\".\"install_datetime\")
> 
> FROM \"data\".\"installations\" AS \"installations\" WHERE \`INSTALLATIONS\`.\`INSTALL_DATETIME\`
>= '2019-01-01' AND \`INSTALLATIONS\`.\`INSTALL_DATETIME\` < '2019-01-03'" --null-string
'' --fields-terminated-by '|' --num-mappers 4
> 
>                at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:96)
> 
>                at org.apache.kylin.source.jdbc.sqoop.SqoopCmdStep.sqoopFlatHiveTable(SqoopCmdStep.java:59)
> 
>                at org.apache.kylin.source.jdbc.sqoop.SqoopCmdStep.doWork(SqoopCmdStep.java:70)
> 
>                at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
> 
>                at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> 
>                at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
> 
>                at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> 
>                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 
>                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 
>                at java.lang.Thread.run(Thread.java:748)
> 
> 
> 
> 
> 
> Please, help me resolve the problem.
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message