kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 余楚倩 <...@meitu.com>
Subject Kylin should override hiveconf settings when connecting to hive using jdbc
Date Mon, 05 Mar 2018 11:32:56 GMT


Hi, kylin developers. My cube building procedure failing at Step 2 "Redistribute Flat Hive
Table" because Kylin always trying to submit a mr job to the default yarn queue.

I have overrided the mapred.job.queue.name property in both kylin_hive_conf.xml and kylin.properties
but it doesn't work. 

kylin.properties
```
kylin.source.hive.beeline-params=-n hive -p hive --hiveconf mapred.job.queue.name=myQueue
 -u "jdbc:hive2://myZk:2181/;serviceDiscoveryMode=zooKeeper;" 
```

kylin_hive_conf.xml
```
    <property>
        <name>mapred.job.queue.name</name>
        <value>myQueue</value>
    </property>
```

After digging into the source code ,I found that kylin try to get the row count of hive table
before redistributing it. But it dose not override the hive configuration when using jdbc
to connect to hive server. I wonder if it is a bug or there is some solutions?

```
public BeelineHiveClient(String beelineParams) {
        if (StringUtils.isEmpty(beelineParams)) {
            throw new IllegalArgumentException("BeelineParames cannot be empty");
        }
        String[] splits = StringUtils.split(beelineParams);
        String url = null, username = null, password = null;
        for (int i = 0; i < splits.length; i++) {
            if ("-u".equals(splits[i])) {
                url = stripQuotes(splits[i + 1]);
            }
            if ("-n".equals(splits[i])) {
                username = stripQuotes(splits[i + 1]);
            }
            if ("-p".equals(splits[i])) {
                password = stripQuotes(splits[i + 1]);
            }
        }
        this.init(url, username, password);
    }

    private void init(String url, String username, String password) {
        try {
            Class.forName("org.apache.hive.jdbc.HiveDriver");
            cnct = DriverManager.getConnection(url, username, password);   //use DriverManager.getConnection(url,properites)
here to override hive configuration settings?
            stmt = cnct.createStatement();
            metaData = cnct.getMetaData();
        } catch (SQLException | ClassNotFoundException e) {
            throw new RuntimeException(e);
        }
    }
```
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message