apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaurav Gupta <gau...@datatorrent.com>
Subject Re: HDFS Space Utilization keeps on increasing
Date Thu, 03 Sep 2015 05:56:00 GMT
Shashi,

I looked at the logs again and looks like you upgraded from 2.x to 3.x. Is
that the case?
>From 2.x to 3.x there is big change of how License behaves. In 3.x you
don't need any Licensing Agent. Please follow the following steps. You
should have Yarn Application running with Id dt-20150514010221-88gt3kvr

1. Shutdown Gateway (Make sure that gateway script is pointing to 3.x
binary file)
2. Kill Yarn Application with Id dt-20150514010221-88gt3kvr. You can kill
it using yarn application -kill dt-20150514010221-88gt3kvr
3. Start the Gateway

Thanks
-Gaurav

On Mon, Aug 31, 2015 at 7:07 AM, Shashi Vishwakarma <
shashi.vish123@gmail.com> wrote:

> Hi All,
>
> Thanks for your reply. I believe you guys are right. There is data torrent
> application which keeps on restarting. I observed resource manager UI, I
> always see one application running even no one running app from my team.
>
> Chetan,
>
> yarn.resourcemanager.am.max-attempts property is currently set to 2. I
> checked a log for that application,there are some
> AlreadybeingCreatedException is coming.Attaching log along this mail.Can
> some one help me on this?
>
> Thanks and Regards,
> Shashi
>
>
>
> On Thu, Aug 27, 2015 at 1:01 AM, Chetan Narsude <chetan@iitbombay.org>
> wrote:
>
>> Can you check: yarn.resourcemanager.am.max-attempts setting for YARN
>> (yarn-site.xml or yarn-default.xml whichever you are using)?
>>
>> Also can you look at the application master logs for one of the app
>> instances you did not start to see why it was shutdown?
>>
>>
>> --
>> Chetan
>>
>>
>> On Wed, Aug 26, 2015 at 9:51 AM, Tushar Gosavi <tushargosavi@gmail.com>
>> wrote:
>>
>>> You can also check yarn resource manager ui and logs to verify which
>>> applications are getting restarted continuously.
>>>
>>> On Wed, Aug 26, 2015 at 9:08 AM, David Yan <david@datatorrent.com>
>>> wrote:
>>>
>>>> That's a lot of applications.  I suspect there is something that keeps
>>>> starting the application, which causes the folder to keep increasing in
>>>> size. Can you just run get-app-info on dtcli on just one application and
>>>> see what is being spawned up?
>>>>
>>>> David
>>>>
>>>> On Tue, Aug 25, 2015 at 11:44 PM, Shashi Vishwakarma <
>>>> shashi.vish123@gmail.com> wrote:
>>>>
>>>>> Thanks David for detailed explanation. I checked apps directory in
>>>>> HDFS,there are around 12858 application in that folder each of having
6.2 M
>>>>> size. It will be a time consuming process to find status of each
>>>>> application by running get-app-info in dtcli. So logged in to web
>>>>> interface of datatorrent(port 9090) but there is no application running
at
>>>>> this moment.
>>>>>
>>>>> Still HDFS space utilization  is increasing,any pointers on this?
>>>>>
>>>>> Thanks and Regards,
>>>>> Shashi
>>>>>
>>>>> On Wed, Aug 26, 2015 at 2:16 AM, Amol Kekre <amol@datatorrent.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Adding dev@apex.incubator.apache.org
>>>>>>
>>>>>> Thks,
>>>>>> Amol
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 25, 2015 at 10:34 AM, David Yan <david@datatorrent.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Shashi,
>>>>>>>
>>>>>>> That directory is where Apex stores application information,
like
>>>>>>> application jar files, checkpoints, container information, etc.
>>>>>>> Please run this command to see which directory is taking the
most
>>>>>>> space.
>>>>>>>
>>>>>>> $ hdfs dfs -du /user/dtadmin/datatorrent/apps
>>>>>>>
>>>>>>> Then open dtcli and use the get-app-info command look at the
>>>>>>> information of that application.  For example:
>>>>>>>
>>>>>>> dt> get-app-info application_1439598948299_0557
>>>>>>>
>>>>>>> The field "state" will tell you whether the application is running
>>>>>>> or not.
>>>>>>>
>>>>>>> If you don't care about the application, you can safely kill
it if
>>>>>>> it's running and delete the HDFS directory by doing hdfs dfs
-rm -r
>>>>>>> /user/dtadmin/datatorrent/apps/application_xxx_yyy (replace xxx
and yyy
>>>>>>> with appropriate values).  Note that doing so will wipe all stored
>>>>>>> information about that application.
>>>>>>>
>>>>>>> David
>>>>>>>
>>>>>>> On Tue, Aug 25, 2015 at 6:32 AM, Shashi Vishwakarma <
>>>>>>> shashi.vish123@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have  DataTorrent 3.x installed on my cluster.Even thought
there
>>>>>>>> is no data torrent application is running , still my hdfs
space utilization
>>>>>>>> goes on increasing. Below is hdfs path that has occupied
most of the space.
>>>>>>>>
>>>>>>>> /user/dtadmin/datatorrent/apps
>>>>>>>>
>>>>>>>> Why this is happening? Am I missing something here?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Shashi
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the
Google
>>>>>>>> Groups "apex-dev" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails
from it,
>>>>>>>> send an email to apex-dev+unsubscribe@googlegroups.com.
>>>>>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com
>>>>>>>> <https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "apex-dev" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from
it,
>>>>>>> send an email to apex-dev+unsubscribe@googlegroups.com.
>>>>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com
>>>>>>> <https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "apex-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>> “I'd have blown my top, because I want to beat this damn thing,
>>>  as long as I've gone this far. I can't just leave it after I've found
>>>  out so much about it. I have to keep going to find out ultimately
>>> what is the matter with it in the end."
>>>                 Richard P. Feynman
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "apex-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>> To post to this group, send email to apex-dev@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "apex-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to apex-dev+unsubscribe@googlegroups.com.
> To post to this group, send email to apex-dev@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com
> <https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message