kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vsevolod Ostapenko (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KYLIN-3139) Failure in map-reduce job due to undefined hdp.version variable when using HDP stack and remote HBase cluster
Date Wed, 03 Jan 2018 16:43:01 GMT

     [ https://issues.apache.org/jira/browse/KYLIN-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vsevolod Ostapenko updated KYLIN-3139:
--------------------------------------
    Description: 
When running on top of HDP stack and using a setup where Hive and HBase run in different clusters
cube build/refresh fails on the step "Extract Fact Table Distinct Columns" with the error
{quote}java.lang.IllegalArgumentException: Unable to parse '/hdp/apps/$\{hdp.version\}/mapreduce/mapreduce.tar.gz#mr-framework'
as a URI, check the setting for mapreduce.application.framework.path{quote}

Based on existing JIRA discussions in Ambari project, it's responsibility of a service to
set hdp.version Java property. When HBase is not installed as a service in a cluster where
Kylin server is running, hbase launcher (invoked by kylin.sh) does not set this property (presumably
because HBase in that case is just a client and not a service).
The only suitable workaround found so far is to set property as part of the conf/setenv.sh
script.

In order to avoid hard coding of the HDP version info, suggested change to setenv.sh will
attempt to detect HDP version at run-time. It should work for all released HDP version from
2.2.x to 2.6.x
In addition to that, it will also try to locate and set Java native library path, when running
on top of HDP.

  was:
When running on top of HDP stack and using a setup where Hive and HBase run in different clusters
cube build/refresh fails on the step "Extract Fact Table Distinct Columns" with the error
{quote}java.lang.IllegalArgumentException: Unable to parse '/hdp/apps/$\{hdp.version\}/mapreduce/mapreduce.tar.gz#mr-framework'
as a URI, check the setting for mapreduce.application.framework.path{quote}

Based on existing JIRA discussions in Ambari project, it's responsibility of a service to
set hdp.version Java property. When HBase is not installed as a service in a cluster hbase
launcher does not set this property (presumably because HBase in that case is just a client
and not a service).
The only suitable workaround found so far is to set property as part of the conf/setenv.sh
script.

In order to avoid hard coding of the HDP version info, suggested change to setenv.sh will
attempt to detect HDP version at run-time. It should work for all released HDP version from
2.2.x to 2.6.x
In addition to that, it will also try to locate and set Java native library path, when running
on top of HDP.


> Failure in map-reduce job due to undefined hdp.version variable when using HDP stack
and remote HBase cluster
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: KYLIN-3139
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3139
>             Project: Kylin
>          Issue Type: Bug
>          Components: General
>    Affects Versions: v2.2.0
>         Environment: HDP 2.5.6, two cluster setup, Kylin 2.2.0 in a cluster with Hive
only, remote HBase cluster for data storage
>            Reporter: Vsevolod Ostapenko
>            Assignee: Vsevolod Ostapenko
>            Priority: Minor
>              Labels: hdp
>         Attachments: KYLIN-3139.master.001.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When running on top of HDP stack and using a setup where Hive and HBase run in different
clusters cube build/refresh fails on the step "Extract Fact Table Distinct Columns" with the
error
> {quote}java.lang.IllegalArgumentException: Unable to parse '/hdp/apps/$\{hdp.version\}/mapreduce/mapreduce.tar.gz#mr-framework'
as a URI, check the setting for mapreduce.application.framework.path{quote}
> Based on existing JIRA discussions in Ambari project, it's responsibility of a service
to set hdp.version Java property. When HBase is not installed as a service in a cluster where
Kylin server is running, hbase launcher (invoked by kylin.sh) does not set this property (presumably
because HBase in that case is just a client and not a service).
> The only suitable workaround found so far is to set property as part of the conf/setenv.sh
script.
> In order to avoid hard coding of the HDP version info, suggested change to setenv.sh
will attempt to detect HDP version at run-time. It should work for all released HDP version
from 2.2.x to 2.6.x
> In addition to that, it will also try to locate and set Java native library path, when
running on top of HDP.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message