apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sasha Parfenov <sa...@datatorrent.com>
Subject Re: Application Memory Reducing
Date Mon, 15 Feb 2016 19:56:12 GMT
To restart the Hadoop services on the sandbox, there are several options:

1.  Use the script /sfw/dtbox/current/bin/dtdemo (or just dtdemo) with
start or stop arguments to start/stop all services, including YARN, HDFS,
Gateway.  There is also a rebuild argument, which will wipe all HDFS files,
and restart all the services.

2.  Restart individual services with "sudo service hadoop-yarn-nodemanager
restart".  Services
include: hadoop-hdfs-namenode, hadoop-hdfs-datanode,
hadoop-yarn-resourcemanager, hadoop-yarn-nodemanager,
and dtgateway.

Also, all the Hadoop configuration files can be found
under /sfw/hadoop/conf/, including hdfs-site.xml, yarn-site.xml, etc.  And
dt-site.xml can be configured under /home/dtadmin/.dt/dt-site.xml with
custom settings.

Thanks,
Sasha

On Mon, Feb 15, 2016 at 8:57 AM, Munagala Ramanath <ram@datatorrent.com>
wrote:

> I believe foo.xml need only contain those you want to override from other
> places lower down on the priority list.
>
> Properties can come from many places and their relative priority is
> discussed at:
>
> http://docs.datatorrent.com/application_packages/#properties-source-precedence
>
> There is additional discussion of memory allocation and management at the
> "Advanced Features" section of the
> TopNWords tutorial at:
> http://docs.datatorrent.com/tutorials/topnwords-c7/
>
> Ram
>
> On Mon, Feb 15, 2016 at 8:47 AM, Jim <jim@facility.supplies> wrote:
>
>> Ram,
>>
>>
>>
>> Do the items in the foo.xml need to contain all the properties in the
>> properties.xml, or just the high level ones from the system configuration
>> that we want to change (i.e. will it still grab custom settings from the
>> properties.xml if I use a config override)?
>>
>>
>>
>> Jim
>>
>>
>>
>> *From:* Munagala Ramanath [mailto:ram@datatorrent.com]
>> *Sent:* Monday, February 15, 2016 10:36 AM
>>
>> *To:* users@apex.incubator.apache.org
>> *Subject:* Re: Application Memory Reducing
>>
>>
>>
>> Jim,
>>
>>
>>
>> On the sandbox, the start/stop scripts are at
>> /sfw/hadoop/hadoop-2.2.0/sbin
>>
>> so you can run stop-yarn.sh followed by start-yarn.sh but there may not
>> be any need to do so.
>>
>>
>>
>> Your values are probably being overridden by values in ~/.dt/dt-site.xml
>>
>>
>>
>> If you create config file called, say, foo.xml under
>> <your-application>/src/site/conf and
>>
>> use that when you launch the application (by selecting "Use saved
>> configuration"
>>
>> in the popup) its values will override those in dt-site.xml
>>
>>
>>
>> Ram
>>
>>
>>
>> On Mon, Feb 15, 2016 at 6:06 AM, Jim <jim@facility.supplies> wrote:
>>
>> Tushar,
>>
>>
>>
>> I found the yarn-site.xml, but how do I restart the yarn service?
>>
>>
>>
>> Does ‘restart the gateway’ under Configuration do that?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Jim
>>
>>
>>
>> *From:* Tushar Gosavi [mailto:tushar@datatorrent.com]
>> *Sent:* Monday, February 15, 2016 7:46 AM
>> *To:* users@apex.incubator.apache.org
>> *Subject:* Re: Application Memory Reducing
>>
>>
>>
>> Hi Jim,
>>
>>
>>
>> Can you check your yarn-site.xml for
>> yarn.scheduler.minimum-allocation-mb, it may be set to 1G (default for 2.7
>> hadoop).
>>
>> Reduce this value to 256MB and restart yarn service and try again running
>> the application.
>>
>>
>>
>>     <property>
>>
>>         <name>yarn.scheduler.minimum-allocation-mb</name>
>>
>>         <value>256</value>
>>
>>     </property>
>>
>>
>>
>> Regards,
>>
>> -Tushar.
>>
>>
>>
>>
>>
>> On Mon, Feb 15, 2016 at 7:06 PM, Jim <jim@facility.supplies> wrote:
>>
>> I have successfully built my first test Apex package on the most recent
>> Sandbox, but only one half of it can start because of memory allocation.
>>
>>
>>
>> From the article:
>>
>>
>>
>>
>> http://docs.datatorrent.com/tutorials/topnwords-c7/#managing-memory-allocation-for-containers
>>
>>
>>
>> I have put the following into my properties.xml:
>>
>>
>>
>> <*property*>
>>   <*name*>dt.application.*.attr.MASTER_MEMORY_MB</*name*>
>>   <*value*>1024</*value*>
>> </*property*>
>> <*property*>
>>   <*name*>dt.application.*.operator.*.attr.MEMORY_MB</*name*>
>>   <*value*>200</*value*>
>> </*property*>
>> <*property*>
>>   <*name*>dt.application.*.operator.*.port.*.attr.BUFFER_MEMORY_MB</
>> *name*>
>>   <*value*>200</*value*>
>> </*property*>
>>
>>
>>
>> Yet the memory allocated to my operators is still 1 GB each for the most
>> part (1 exception has 2 GB), as can be seen below:
>>
>>
>>
>>
>>
>> I would like to get this down to 512MB per operator – what is not
>> allowing the settings to take?
>>
>>
>>
>> I did see a comment where it may be the yarn configuration, but I don’t
>> know where that is within the Datatorrent Apex Sandbox configuration; nor
>> do I know if there is anything else that could be causing this?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Jim
>>
>>
>> Jim
>>
>>
>> jim@facility.supplies (414) 760-7711
>> ------------------------------
>>
>> *The information contained in this communication, including any files or
>> attachments transmitted with it, may contain copyrighted information or
>> information that is confidential and exempt from disclosure under
>> applicable laws and regulations, is intended only for the use of the
>> recipient(s) named above, and may be legally privileged. If the reader of
>> this message is not the intended recipient, you are hereby notified that
>> any dissemination, distribution, or copying of this communication, or any
>> of its contents, files or attachments, is strictly prohibited. If you have
>> received this communication in error, please return it to the sender
>> immediately and delete the original message and any copy of it from your
>> computer system. If you have any questions concerning this message, please
>> contact the sender. *
>>
>>
>>
>>
>>
>>
>> ------------------------------
>>
>> *The information contained in this communication, including any files or
>> attachments transmitted with it, may contain copyrighted information or
>> information that is confidential and exempt from disclosure under
>> applicable laws and regulations, is intended only for the use of the
>> recipient(s) named above, and may be legally privileged. If the reader of
>> this message is not the intended recipient, you are hereby notified that
>> any dissemination, distribution, or copying of this communication, or any
>> of its contents, files or attachments, is strictly prohibited. If you have
>> received this communication in error, please return it to the sender
>> immediately and delete the original message and any copy of it from your
>> computer system. If you have any questions concerning this message, please
>> contact the sender. *
>>
>>
>> Jim
>>
>>
>> jim@facility.supplies (414) 760-7711
>> ------------------------------
>>
>> *The information contained in this communication, including any files or
>> attachments transmitted with it, may contain copyrighted information or
>> information that is confidential and exempt from disclosure under
>> applicable laws and regulations, is intended only for the use of the
>> recipient(s) named above, and may be legally privileged. If the reader of
>> this message is not the intended recipient, you are hereby notified that
>> any dissemination, distribution, or copying of this communication, or any
>> of its contents, files or attachments, is strictly prohibited. If you have
>> received this communication in error, please return it to the sender
>> immediately and delete the original message and any copy of it from your
>> computer system. If you have any questions concerning this message, please
>> contact the sender. *
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------
>>
>> *The information contained in this communication, including any files or
>> attachments transmitted with it, may contain copyrighted information or
>> information that is confidential and exempt from disclosure under
>> applicable laws and regulations, is intended only for the use of the
>> recipient(s) named above, and may be legally privileged. If the reader of
>> this message is not the intended recipient, you are hereby notified that
>> any dissemination, distribution, or copying of this communication, or any
>> of its contents, files or attachments, is strictly prohibited. If you have
>> received this communication in error, please return it to the sender
>> immediately and delete the original message and any copy of it from your
>> computer system. If you have any questions concerning this message, please
>> contact the sender. *
>>
>> Jim
>>
>>
>> jim@facility.supplies (414) 760-7711
>> ------------------------------
>> The information contained in this communication, including any files or
>> attachments transmitted with it, may contain copyrighted information or
>> information that is confidential and exempt from disclosure under
>> applicable laws and regulations, is intended only for the use of the
>> recipient(s) named above, and may be legally privileged. If the reader of
>> this message is not the intended recipient, you are hereby notified that
>> any dissemination, distribution, or copying of this communication, or any
>> of its contents, files or attachments, is strictly prohibited. If you have
>> received this communication in error, please return it to the sender
>> immediately and delete the original message and any copy of it from your
>> computer system. If you have any questions concerning this message, please
>> contact the sender.
>>
>
>


-- 
*Sasha Parfenov*
Software Engineer, DataTorrent Inc.
3200 Patrick Henry Drive, 2nd floor
Santa Clara, CA  95054
sasha@datatorrent.com
https://www.datatorrent.com/

Mime
View raw message