ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yusaku Sako <yus...@hortonworks.com>
Subject Re: How to add configurations to specific host to create cluster using blueprint ?
Date Fri, 06 Jun 2014 04:30:23 GMT
I can understand that there could be a different number of disks on
different VMs, but isn't there some logical grouping?
Something like...
Group 1. VM001-VM100 with 2 disks: /mnt/disks/0, /mnt/disks/1
Group 2. VM101-VM200 with 4 disks: /mnt/disks/0, /mnt/disks/1,
/mnt/disks/2, /mnt/disks/3
and so on?
In such scenario, you would only need 2 host groups for 200 hosts.
It would be beneficial to look into setting up the VMs so that they are
more or less homogeneous with a small number of variations for
manageability.
In your original post, you said that each host has uniquely named mount
points, but instead, is it possible to use the same mount points (e.g.,
/mnt/disks/0, 1, and so on at the time of mounting the disks)?  If that's
not possible, maybe create symlinks?

Yusaku


On Thu, Jun 5, 2014 at 9:10 PM, Qing Chi 79624 <chiq@vmware.com> wrote:

>  Hi Yusaku,
>
>  There is a different number of VMDK in the different virtual machine. So
> mount points are different in these virtual machines.
>
>  Thanks,
>
>  -qing
>
>
>   From: Yusaku Sako <yusaku@hortonworks.com>
> Reply-To: <user@ambari.apache.org>
> Date: Thu, 5 Jun 2014 20:43:04 -0700
>
> To: <user@ambari.apache.org>
> Cc: Emma Lin <line@vmware.com>, Xiaoding Bian <xiaodingbian@vmware.com>,
> John Speidel <jspeidel@hortonworks.com>
> Subject: Re: How to add configurations to specific host to create cluster
> using blueprint ?
>
>  It would be best to not have 100 different configs for 100 different
> hosts.
> Is there anyway you can use the same mount points for these hosts?
> Typically, you may have several classes of machines (say, different number
> of disks).  But having
>
>
> On Thu, Jun 5, 2014 at 8:25 PM, Siddharth Wagle <swagle@hortonworks.com>
> wrote:
>
>>  Could you elaborate on which config needs to customized on a per host
>> basis?
>>
>>  You would only need hundred host_groups if you have 100 hosts each with
>> unique specs.
>>
>>  -Sid
>>
>>
>> On Thu, Jun 5, 2014 at 7:13 PM, Qing Chi 79624 <chiq@vmware.com> wrote:
>>
>>>  Hi Sid,
>>> Thanks, Sid.
>>> I think it's a solution. But if there are hundreds of host with the same
>>> components. In this solution I must create hundreds of host_group as well.
>>>
>>>  -qing
>>>
>>>   From: Siddharth Wagle <swagle@hortonworks.com>
>>> Reply-To: <user@ambari.apache.org>
>>> Date: Thu, 5 Jun 2014 19:00:18 -0700
>>> To: "user@ambari.apache.org" <user@ambari.apache.org>
>>>
>>> Cc: Emma Lin <line@vmware.com>, Xiaoding Bian <xiaodingbian@vmware.com>,
>>> John Speidel <jspeidel@hortonworks.com>
>>> Subject: Re: How to add configurations to specific host to create
>>> cluster using blueprint ?
>>>
>>>   Hi Qing Chi,
>>>
>>>  You can have 2 host_groups with exactly same components say
>>> "salve_type_1", "slave_type_2" with different configs for each host group.
>>>
>>>  -Sid
>>>
>>>
>>> On Thu, Jun 5, 2014 at 6:52 PM, Qing Chi 79624 <chiq@vmware.com> wrote:
>>>
>>>>  Hi Jeff,
>>>> Thanks a lot for your help.
>>>> I think the host_group is the group which includes the same attributes
>>>> hosts. Maybe there is a host or more than one in a host_group. So if I want
>>>> to set configurations for each host. How to do?
>>>> I think it isn't convenient to set configurations to each host when a
>>>> host_group has many hosts.For example,
>>>>  {
>>>>   "blueprint": "multi-node-hdfs-yarn",
>>>>   "host_groups":[
>>>>     {
>>>>       "name": "master",
>>>>       "hosts": [
>>>>         {
>>>>           "fqdn": "c6401.ambari.apache.org
>>>> <https://urldefense.proofpoint.com/v1/url?u=http://c6401.ambari.apache.org&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=XFPetyuh5lGv3twiMNs7nw%3D%3D%0A&m=zr3ckRNJEVJ%2B7zQShmUFM9vYstWx6%2Fe%2B0VI7WGP7RIo%3D%0A&s=f9b31ec62449af463e2dee594973c5ff1797187ac253ba66f7dac46f8c0594bc>
>>>> "
>>>>         }
>>>>       ]
>>>>     },
>>>>     {
>>>>       "name": "slaves",
>>>>       "hosts": [
>>>>         {
>>>>           "fqdn": "c6402.ambari.apache.org
>>>> <https://urldefense.proofpoint.com/v1/url?u=http://c6402.ambari.apache.org&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=XFPetyuh5lGv3twiMNs7nw%3D%3D%0A&m=zr3ckRNJEVJ%2B7zQShmUFM9vYstWx6%2Fe%2B0VI7WGP7RIo%3D%0A&s=6a60e82c0bc013fceb7671c67c8addfc8528a6e8470498fa68604562431785f3>
>>>> "
>>>>         },
>>>>         {
>>>>           "fqdn": "c6403.ambari.apache.org
>>>> <https://urldefense.proofpoint.com/v1/url?u=http://c6403.ambari.apache.org&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=XFPetyuh5lGv3twiMNs7nw%3D%3D%0A&m=zr3ckRNJEVJ%2B7zQShmUFM9vYstWx6%2Fe%2B0VI7WGP7RIo%3D%0A&s=469c66697be63cdeab1006495edf62f78e72db0072a176e4a96bd9d961ddf788>
>>>> "
>>>>         }
>>>>       ]
>>>>     }
>>>>   ]
>>>> }
>>>>
>>>>  The host_groups slaves has two hosts. In this case how to set
>>>> configurations for each host?
>>>>
>>>>   From: Jeff Sposetti <jeff@hortonworks.com>
>>>> Reply-To: <user@ambari.apache.org>
>>>> Date: Thu, 5 Jun 2014 06:25:07 -0700
>>>> To: <user@ambari.apache.org>
>>>> Cc: Emma Lin <line@vmware.com>, Xiaoding Bian <xiaodingbian@vmware.com>,
>>>> John Speidel <jspeidel@hortonworks.com>
>>>> Subject: Re: How to add configurations to specific host to create
>>>> cluster using blueprint ?
>>>>
>>>>  You can set configurations at the blueprint level (to apply to all
>>>> host_groups) and you can also set configurations in the host_groups element
>>>> to set configurations for that host_group.
>>>>
>>>> Will that help? Or will it just mean you will have to have a host_group
>>>> that you have to apply per host?
>>>>
>>>>
>>>> https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-Configurations
>>>> <https://urldefense.proofpoint.com/v1/url?u=https://cwiki.apache.org/confluence/display/AMBARI/Blueprints%23Blueprints-Configurations&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=XFPetyuh5lGv3twiMNs7nw%3D%3D%0A&m=EfpGl9vWkh%2FuFYOC%2FWIgKw9XBLClivVRI1LFsC2njqs%3D%0A&s=3b6c5134ae1180ec8287000ba557ef4b0030e02634e6e5fccc15feed9f094d67>
>>>>
>>>>
>>>> On Thu, Jun 5, 2014 at 12:58 AM, Qing Chi 79624 <chiq@vmware.com>
>>>> wrote:
>>>>
>>>>>  Hi guys,
>>>>> I want to use specific data dir on each host. For example:
>>>>>
>>>>>  Host 1, following is content of hdfs-site.xml on a host.
>>>>>
>>>>>  <?xml version="1.0"?>
>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>>> <!-- check for all settings at
>>>>> http://hadoop.apache.org/common/docs/stable/hdfs-default.html
>>>>> <https://urldefense.proofpoint.com/v1/url?u=http://hadoop.apache.org/common/docs/stable/hdfs-default.html&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=XFPetyuh5lGv3twiMNs7nw%3D%3D%0A&m=EfpGl9vWkh%2FuFYOC%2FWIgKw9XBLClivVRI1LFsC2njqs%3D%0A&s=b8f8fb2e9d43adf43de08a22b58d97c0a0e5ee3eb1bccb6d669e17df5e59f3df>
>>>>> -->
>>>>> <configuration>
>>>>>  …
>>>>>  <property>
>>>>>   <name>dfs.data.dir</name>
>>>>>
>>>>> <value>/mnt/scsi-36000c297241e2328f0c26ccf28525ada-part1/hadoop/hdfs/data,/mnt/scsi-36000c296928f65c7749de0620b422dde-part1/hadoop/hdfs/data,/mnt/scsi-36000c2970047fad07f4662de91001b8e-part1/hadoop/hdfs/data</value>
>>>>>   <final>true</final>
>>>>> </property>
>>>>>  ….
>>>>> </configuration>
>>>>>
>>>>>  Host 2,  following is content of hdfs-site.xml on another host.
>>>>>
>>>>>   <?xml version="1.0"?>
>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>>> <!-- check for all settings at
>>>>> http://hadoop.apache.org/common/docs/stable/hdfs-default.html
>>>>> <https://urldefense.proofpoint.com/v1/url?u=http://hadoop.apache.org/common/docs/stable/hdfs-default.html&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=XFPetyuh5lGv3twiMNs7nw%3D%3D%0A&m=EfpGl9vWkh%2FuFYOC%2FWIgKw9XBLClivVRI1LFsC2njqs%3D%0A&s=b8f8fb2e9d43adf43de08a22b58d97c0a0e5ee3eb1bccb6d669e17df5e59f3df>
>>>>> -->
>>>>> <configuration>
>>>>>  …
>>>>>   <property>
>>>>>   <name>dfs.data.dir</name>
>>>>>
>>>>> <value>/mnt/scsi-36000c2966300dabe41c0193161330a2c-part1/hadoop/hdfs/data,/mnt/scsi-36000c29ce17ee6c47e1a329791cbbe61-part1/hadoop/hdfs/data,/mnt/scsi-36000c29694c370eb163114067ffed1bb-part1/hadoop/hdfs/data</value>
>>>>>   <final>true</final>
>>>>> </property>
>>>>>  ….
>>>>> </configuration>
>>>>>
>>>>>  Values of data dir on all hosts are different. So on this situation
>>>>> how to add configurations to specific host to create cluster using
>>>>> blueprint? Can somebody give me some help?
>>>>>
>>>>>  Thanks,
>>>>>
>>>>>  -qing
>>>>>
>>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message