ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Mildebrandt <ch...@woodenrhino.com>
Subject Re: Overriding values in component configuration using the Ambari UI
Date Tue, 11 Mar 2014 23:08:13 GMT
In case anyone else comes across this thread, here's the command to use to
make sure the capacity scheduler field is populated in the YARN config for
the Ambari UI:

curl -i -H 'X-Requested-By: company' -X PUT -d '{"Clusters":
{"desired_config": {"tag": "custom_capacity", "type": "capacity-scheduler",
"properties": {"yarn.scheduler.capacity.maximum-applications": "10000",
"yarn.scheduler.capacity.maximum-am-resource-percent": "0.1",
"yarn.scheduler.capacity.resource-calculator":
"org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator",
"yarn.scheduler.capacity.root.queues": "default",
"yarn.scheduler.capacity.root.default.capacity": "100",
"yarn.scheduler.capacity.root.default.user-limit-factor": "1",
"yarn.scheduler.capacity.root.default.maximum-capacity": "100",
"yarn.scheduler.capacity.root.default.state": "RUNNING",
"yarn.scheduler.capacity.root.default.acl_submit_applications": "*",
"yarn.scheduler.capacity.root.default.acl_administer_queue": "*",
"yarn.scheduler.capacity.node-locality-delay": "-1"}}}}' --user admin:admin
http://hadoop1.company.com:8080/api/v1/clusters/cl1<http://hadoop1.robinsystems.com:8080/api/v1/clusters/cl1>

-Chris


On Tue, Feb 18, 2014 at 2:04 PM, Chris Mildebrandt <chris@woodenrhino.com>wrote:

> Hi Jaimin,
>
> Thanks for your patience, I really appreciate it. I still don't entirely
> understand how this field in the UI will get populated when I configure the
> cluster using the Ambari REST APIs. Since I'm not using the wizard, there
> must be a command I'm missing on my side. Can you please provide an example
> how I would set this (or point me in a place I can look at the commands the
> wizard executes during deployment)?
>
> On a side note, my coworker just told me that he put one value in the text
> area (yarn.scheduler.capacity.maximum-applications=10000) and the file on
> the hosts was not touched. The other modified parameters in the YARN config
> were updated, and the yarn-site.xml file was updated correctly. Any idea
> why it wouldn't have updated the capacity-scheduler.xml file?
>
> I'd like to suggest that this field get initially populated with the
> default values when creating a cluster through the REST APIs. The reason
> being, this file exists on the hosts and is being actively used by the YARN
> application....so Ambari should know about it.
>
> I hope everything above makes sense. Please let me know if you need more
> details, and thanks again for your continuing help,
> -Chris
>
>
> On Tue, Feb 18, 2014 at 12:47 PM, Jaimin Jetly <jaimin@hortonworks.com>wrote:
>
>> Hi Chris,
>>
>> Hi Chris,
>>
>> HDP stack RPM deploys the capacity-scheduler.xml by default at
>> /etc/hadoop/conf/capacity-scheduler.xml path. So initially it's the RPM
>> and not the REST APIs that create capacity-scheduler.xml file. This file
>> is unknown to Ambari until the "capacity-scheduler" configuration resource
>> is created with a list of properties by installer wizard via REST APIs.
>> Once this is done by the wizard and further yarn service is started by the
>> wizard, then the properties listed in the configuration completely
>> overrides the RPM deployed file at /etc/hadoop/conf/capacity-scheduler.xml
>> path
>>
>>
>> So I suggest insert the same default values in the test area as below
>> which are suggested by Ambari-web ui during installation.
>>
>> yarn.scheduler.capacity.maximum-am-resource-percent=0.2
>> yarn.scheduler.capacity.maximum-applications=10000
>> yarn.scheduler.capacity.node-locality-delay=40
>> yarn.scheduler.capacity.root.acl_administer_queues=*
>> yarn.scheduler.capacity.root.capacity=100
>> yarn.scheduler.capacity.root.default.acl_administer_jobs=*
>> yarn.scheduler.capacity.root.default.acl_submit_jobs=*
>> yarn.scheduler.capacity.root.default.capacity=100
>> yarn.scheduler.capacity.root.default.maximum-capacity=100
>> yarn.scheduler.capacity.root.default.state=RUNNING
>> yarn.scheduler.capacity.root.default.user-limit-factor=1
>> yarn.scheduler.capacity.root.queues=default
>> yarn.scheduler.capacity.root.unfunded.capacity=50
>>
>>
>>
>>
>>
>> On Tue, Feb 18, 2014 at 11:21 AM, Chris Mildebrandt <
>> chris@woodenrhino.com> wrote:
>>
>>> Hi Jaimin,
>>>
>>> I've never seen this field in any state other than empty. I'll populate
>>> it with the key value pairs which match the current file on my system.
>>>
>>> Is this due to the fact that I create the cluster entirely through the
>>> REST APIs? I don't set a value for this using the REST APIs, so I'd assume
>>> it would have just been populated with the capacity-scheduler.xml values.
>>> Is this a bug that should be opened, or expected behavior?
>>>
>>> Thanks for the format of this field and for the tip about the shell
>>> script to change one property at a time. That helps a lot.
>>>
>>> -Chris
>>>
>>>
>>> On Tue, Feb 18, 2014 at 11:13 AM, Jaimin Jetly <jaimin@hortonworks.com>wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> Normally, The content in the UI "Capacity Scheduler" text area Which
>>>> should be list of key=value pairs delimited by newline is reflected in the
>>>> /etc/hadoop/conf/capacity-scheduler.xml file. Both of these are always
>>>> in sync.
>>>>
>>>> If you are seeing something similar to below inline image, then
>>>> Clicking on undo link will again populate the text area with last saved
>>>> value:
>>>>
>>>> [image: Inline image 1]
>>>>
>>>>
>>>> Otherwise the default value as suggested by Ambari for this field is :
>>>>
>>>> yarn.scheduler.capacity.maximum-am-resource-percent=0.2
>>>> yarn.scheduler.capacity.maximum-applications=10000
>>>> yarn.scheduler.capacity.node-locality-delay=40
>>>> yarn.scheduler.capacity.root.acl_administer_queues=*
>>>> yarn.scheduler.capacity.root.capacity=100
>>>> yarn.scheduler.capacity.root.default.acl_administer_jobs=*
>>>> yarn.scheduler.capacity.root.default.acl_submit_jobs=*
>>>> yarn.scheduler.capacity.root.default.capacity=100
>>>> yarn.scheduler.capacity.root.default.maximum-capacity=100
>>>> yarn.scheduler.capacity.root.default.state=RUNNING
>>>> yarn.scheduler.capacity.root.default.user-limit-factor=1
>>>> yarn.scheduler.capacity.root.queues=default
>>>> yarn.scheduler.capacity.root.unfunded.capacity=50
>>>>
>>>>
>>>> Hope this helps.
>>>>
>>>>
>>>>
>>>> On Tue, Feb 18, 2014 at 10:38 AM, Chris Mildebrandt <
>>>> chris@woodenrhino.com> wrote:
>>>>
>>>>> Hi Jaimin,
>>>>>
>>>>> The graphic didn't come through, but I understand what you're
>>>>> explaining. We do have a missing value for the "Capacity Scheduler".
How
>>>>> should this be normally filled? We have the standard untouched
>>>>> /etc/hadoop/conf/capacity-scheduler.xml file on our machines. Should
the
>>>>> contents of this file normally show up in the UI? Should I copy the
>>>>> contents of the file into this text area?
>>>>>
>>>>> Thanks,
>>>>> -Chris
>>>>>
>>>>>
>>>>> On Tue, Feb 18, 2014 at 10:28 AM, Jaimin Jetly <jaimin@hortonworks.com
>>>>> > wrote:
>>>>>
>>>>>> Hi Chris,
>>>>>>
>>>>>> In context to your first email regarding Ambari UI issue,
>>>>>> Ambari web-ui disables the save button whenever there is a validation
>>>>>> error for any property on a service config page. In that case you
should
>>>>>> also see the invalidated property highlighted with red color as shown
below
>>>>>> in the image. Please expand all the categories and check if any property
is
>>>>>> invalidated.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [image: Inline image 1]
>>>>>>
>>>>>> If Ambari UI is disabling the button with no validation error of
any
>>>>>> property on the page then it's a bug and We will need more information
to
>>>>>> debug the cause. To start with a screenshot of the yarn config page
and
>>>>>> javascript console output in case if any JS  error is logged in the
console
>>>>>> will be helpful.
>>>>>>
>>>>>> In regards to your second email,
>>>>>> The API call that you mentioned should work and it's the correct
way
>>>>>> to change configs at the cluster level. In addition to directly using
API
>>>>>> for configuration changes, you can also use the shell script located
at
>>>>>> /var/lib/ambari-server/resources/scripts/configs.sh path on ambari-server
>>>>>> host to add/override/delete a single property. The shell script in
itself
>>>>>> uses the RESTful API in it's implementation but it's really useful
while
>>>>>> overriding a single config property.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Feb 18, 2014 at 10:09 AM, Chris Mildebrandt <
>>>>>> chris@woodenrhino.com> wrote:
>>>>>>
>>>>>>> Hi Sid,
>>>>>>>
>>>>>>> I'm actually setting the desired config. Here's an example of
the
>>>>>>> REST call:
>>>>>>>
>>>>>>> curl -i -H 'X-Requested-By: xxx' -X PUT -d '{"Clusters":
>>>>>>> {"desired_config": {"tag": "custom1", "type": "yarn-site", "properties":
>>>>>>> {"yarn.scheduler.minimum-allocation-mb" : "7168"}}}}' --user
admin:admin
>>>>>>> http://xxx:8080/api/v1/clusters/cl1
>>>>>>>
>>>>>>> Now, of course, I'd have all my YARN variables in here not just
one.
>>>>>>> This is just an example to focus on the technique I'm using to
change the
>>>>>>> property. This is the property I'm interested in changing. After
running
>>>>>>> the above command, I then restart the required components to
re-read the
>>>>>>> config.
>>>>>>>
>>>>>>> Am I doing this incorrectly? Thanks,
>>>>>>> -Chris
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Feb 18, 2014 at 9:38 AM, Siddharth Wagle <
>>>>>>> swagle@hortonworks.com> wrote:
>>>>>>>
>>>>>>>> Hi Chris,
>>>>>>>>
>>>>>>>> Are you using Config groups to overrides configs as per the
latest
>>>>>>>> API docs ?
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/config-groups.md
>>>>>>>>
>>>>>>>> It is possible to do this using the Web UI too, using the
Manage
>>>>>>>> Config Groups link on the Config page.
>>>>>>>> Could you please provide the name of the Service and property
that
>>>>>>>> you are trying to override ?
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Sid
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Feb 18, 2014 at 9:17 AM, Chris Mildebrandt <
>>>>>>>> chris@woodenrhino.com> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I've searched quite a bit, maybe I'm just not using the
correct
>>>>>>>>> terms. I'm trying to override certain configuration values
for YARN using
>>>>>>>>> the Ambari UI. Can someone please give a step-by-step
method to do this?
>>>>>>>>> When I try to update the values, the save button stays
disabled. I can get
>>>>>>>>> around this by using the REST APIs for now...but it would
be nice to do
>>>>>>>>> this in the UI for quick changes.
>>>>>>>>>
>>>>>>>>> I'm using HDP 2.0.6 with Ambari 1.4.3.38-1.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> -Chris
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>> NOTICE: This message is intended for the use of the individual
or
>>>>>>>> entity to which it is addressed and may contain information
that is
>>>>>>>> confidential, privileged and exempt from disclosure under
applicable law.
>>>>>>>> If the reader of this message is not the intended recipient,
you are hereby
>>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>>> disclosure or forwarding of this communication is strictly
prohibited. If
>>>>>>>> you have received this communication in error, please contact
the sender
>>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Thanks
>>>>>>  Jaimin
>>>>>>
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to which it is addressed and may contain information that
is
>>>>>> confidential, privileged and exempt from disclosure under applicable
law.
>>>>>> If the reader of this message is not the intended recipient, you
are hereby
>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>> disclosure or forwarding of this communication is strictly prohibited.
If
>>>>>> you have received this communication in error, please contact the
sender
>>>>>> immediately and delete it from your system. Thank You.
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> -Thanks
>>>>  Jaimin
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>>
>>>
>>>
>>
>>
>> --
>> -Thanks
>>  Jaimin
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>

Mime
View raw message