hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xiang....@sky-data.cn
Subject Re: how hawq use HA hdfs
Date Thu, 31 May 2018 04:15:39 GMT
Thank you very much. 



From: "Radar Lei" <rlei@pivotal.io> 
To: "user" <user@hawq.incubator.apache.org> 
Cc: "dev" <dev@hawq.incubator.apache.org> 
Sent: Thursday, May 31, 2018 11:46:09 AM 
Subject: Re: how hawq use HA hdfs 

Seems like you set the 'dfs.nameservices' as skydata, but not 'dx' which you defined in hawq-site.xml.


Regards, 
Radar 

On Thu, May 31, 2018 at 11:22 AM, < xiang.dai@sky-data.cn > wrote: 



I had, here is related configs in hdfs-client.xml: 


<property> 
<name>dfs.nameservices</name> 
<value>skydata</value> 
</property> 

<property> 
<name>dfs.ha.namenodes.skydata</name> 
<value>nn1,nn2</value> 
</property> 

<property> 
<name>dfs.namenode.rpc-address.skydata.nn1</name> 
<value> 192.168.60.24:8020 </value> 
</property> 

<property> 
<name>dfs.namenode.rpc-address.skydata.nn2</name> 
<value> 192.168.60.32:8020 </value> 
</property> 

<property> 
<name>dfs.namenode.http-address.skydata.nn1</name> 
<value> 192.168.60.24:50070 </value> 
</property> 
<property> 
<name>dfs.namenode.http-address.skydata.nn2</name> 
<value> 192.168.60.32:50070 </value> 
</property> 



From: "Radar Lei" < rlei@pivotal.io > 
To: "user" < user@hawq.incubator.apache.org > 
Cc: "dev" < dev@hawq.incubator.apache.org > 
Sent: Thursday, May 31, 2018 10:27:21 AM 

Subject: Re: how hawq use HA hdfs 

Have you made changes to HAWQ configuration file ' hdfs-client.xml'? 

Regards, 
Radar 

On Thu, May 31, 2018 at 10:07 AM, < xiang.dai@sky-data.cn > wrote: 

BQ_BEGIN



Change the following parameter in the $GPHOME/etc/hawq-site.xml file: 
<property> <name> hawq_dfs_url </name> <value> hdpcluster/hawq_default
</value> <description> URL for accessing HDFS. </description> </property>



In the listing above: 

    * Replace hdpcluster with the actual service ID that is configured in HDFS. 
    * Replace /hawq_default with the directory you want to use for storing data on HDFS. Make
sure this directory exists and is writable. 

In my case, i think that hdpcluster is "dx" which defined in core-site.xml. 

So i use "dx/hawq_default" but failed. 

Could anyone help me about this? 

Thanks very much. 



From: "Radar Lei" < rlei@pivotal.io > 
To: "user" < user@hawq.incubator.apache.org >, "dev" < dev@hawq.incubator.apache.org
> 
Sent: Thursday, May 31, 2018 9:50:39 AM 

Subject: Re: how hawq use HA hdfs 

If you are installing a new HAWQ, then file space move is not required. 
I think HAWQ will treat the host string as an url unless you configured HAWQ hdfs HA correctly.
So please verify if you missed any other steps. 

Regards, 
Radar 

On Thu, May 31, 2018 at 9:40 AM, < xiang.dai@sky-data.cn > wrote: 

BQ_BEGIN

I think that move filespace is the action that hawq has installed before HA hdfs, 
but in my case, i installed HA hdfs then installed hawq. 

In this way, i think there is no need to move filespace. 

Besides, my question is that why URI can not work as expected? 
Or which URI i should use? 


From: "Radar Lei" < rlei@pivotal.io > 
To: "user" < user@hawq.incubator.apache.org > 
Sent: Wednesday, May 30, 2018 8:30:01 PM 
Subject: Re: how hawq use HA hdfs 

As the steps in the document you mentioned, you need to finish all the steps to convert your
HAWQ cluster to HA mode. 
E.g. Do filespace move, make changes in hdfs-client.xml etc. 

Regards, 
Radar 

On Wed, May 30, 2018 at 7:44 PM, < xiang.dai@sky-data.cn > wrote: 

BQ_BEGIN

Hi! 

I has setting HA hdfs using ZKFC with below config: 

<configuration> 
<property> 
<name>fs.defaultFS</name> 
<value>hdfs://dx</value> 
</property> 
<property> 
<name>ha.zookeeper.quorum</name> 
<value> 192.168.60.24 : 2181,192.168.60.32 : 2181,192.168.60.37 :2181</value>

</property> 
</configuration> 

Then i install hawq in another node(not any node installed hdfs). 

I follow http://hawq.incubator.apache.org/docs/userguide/2.3.0.0-incubating/admin/HAWQFilespacesandHighAvailabilityEnabledHDFS.html

to set `hawq_dfs_url` as "dx/hawq_default", but init failed when check if hdfs path is available:

[WARNING]:-ERROR: Can not connect to 'hdfs://dx:0' 

Then i change `hawq_dfs_url` to "dx:8020/hawq_default", still failed: 
ERROR Failed to setup RPC connection to "dx:8020" caused by: 
TcpSocket.cpp: 171: HdfsNetworkConnectException: Failed to resolve address "dx:8020" Name
or service not known 

It consider "dx" as hostname, which is not expected. 

I also test "hdfs://dx", " file://dx ", all failed. 

I know that " 192.168.60.24 :8020" can work, but it does not use HA hdfs. 

How can hawq use HA hdfs? 

Thanks 





-- 
戴翔 
南京天数信息科技有限公司 
电话: +86 1 3382776490 
公司官网: www.sky-data.cn 
免费使用天数润科智能计算平台 SkyDiscovery 

BQ_END



-- 
戴翔 
南京天数信息科技有限公司 
电话: +86 1 3382776490 
公司官网: www.sky-data.cn 
免费使用天数润科智能计算平台 SkyDiscovery 

BQ_END



-- 
戴翔 
南京天数信息科技有限公司 
电话: +86 1 3382776490 
公司官网: www.sky-data.cn 
免费使用天数润科智能计算平台 SkyDiscovery 

BQ_END



-- 
戴翔 
南京天数信息科技有限公司 
电话: +86 1 3382776490 
公司官网: www.sky-data.cn 
免费使用天数润科智能计算平台 SkyDiscovery 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message