apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaurav Gupta <gau...@datatorrent.com>
Subject Re: HDFS Space Utilization keeps on increasing
Date Mon, 31 Aug 2015 15:53:17 GMT
Shashi,
I see what is happening. For now, please stop gateway, clear
/user/dtadmin/datatorrent/audit/
folder and start gateway again. This should resolve the issue for now.


Thanks
-Gaurav

On Mon, Aug 31, 2015 at 7:07 AM, Shashi Vishwakarma <
shashi.vish123@gmail.com> wrote:

> Hi All,
>
> Thanks for your reply. I believe you guys are right. There is data torrent
> application which keeps on restarting. I observed resource manager UI, I
> always see one application running even no one running app from my team.
>
> Chetan,
>
> yarn.resourcemanager.am.max-attempts property is currently set to 2. I
> checked a log for that application,there are some
> AlreadybeingCreatedException is coming.Attaching log along this mail.Can
> some one help me on this?
>
> Thanks and Regards,
> Shashi
>
>
>
> On Thu, Aug 27, 2015 at 1:01 AM, Chetan Narsude <chetan@iitbombay.org>
> wrote:
>
>> Can you check: yarn.resourcemanager.am.max-attempts setting for YARN
>> (yarn-site.xml or yarn-default.xml whichever you are using)?
>>
>> Also can you look at the application master logs for one of the app
>> instances you did not start to see why it was shutdown?
>>
>>
>> --
>> Chetan
>>
>>
>> On Wed, Aug 26, 2015 at 9:51 AM, Tushar Gosavi <tushargosavi@gmail.com>
>> wrote:
>>
>>> You can also check yarn resource manager ui and logs to verify which
>>> applications are getting restarted continuously.
>>>
>>> On Wed, Aug 26, 2015 at 9:08 AM, David Yan <david@datatorrent.com>
>>> wrote:
>>>
>>>> That's a lot of applications.  I suspect there is something that keeps
>>>> starting the application, which causes the folder to keep increasing in
>>>> size. Can you just run get-app-info on dtcli on just one application and
>>>> see what is being spawned up?
>>>>
>>>> David
>>>>
>>>> On Tue, Aug 25, 2015 at 11:44 PM, Shashi Vishwakarma <
>>>> shashi.vish123@gmail.com> wrote:
>>>>
>>>>> Thanks David for detailed explanation. I checked apps directory in
>>>>> HDFS,there are around 12858 application in that folder each of having
6.2 M
>>>>> size. It will be a time consuming process to find status of each
>>>>> application by running get-app-info in dtcli. So logged in to web
>>>>> interface of datatorrent(port 9090) but there is no application running
at
>>>>> this moment.
>>>>>
>>>>> Still HDFS space utilization  is increasing,any pointers on this?
>>>>>
>>>>> Thanks and Regards,
>>>>> Shashi
>>>>>
>>>>> On Wed, Aug 26, 2015 at 2:16 AM, Amol Kekre <amol@datatorrent.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Adding dev@apex.incubator.apache.org
>>>>>>
>>>>>> Thks,
>>>>>> Amol
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 25, 2015 at 10:34 AM, David Yan <david@datatorrent.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Shashi,
>>>>>>>
>>>>>>> That directory is where Apex stores application information,
like
>>>>>>> application jar files, checkpoints, container information, etc.
>>>>>>> Please run this command to see which directory is taking the
most
>>>>>>> space.
>>>>>>>
>>>>>>> $ hdfs dfs -du /user/dtadmin/datatorrent/apps
>>>>>>>
>>>>>>> Then open dtcli and use the get-app-info command look at the
>>>>>>> information of that application.  For example:
>>>>>>>
>>>>>>> dt> get-app-info application_1439598948299_0557
>>>>>>>
>>>>>>> The field "state" will tell you whether the application is running
>>>>>>> or not.
>>>>>>>
>>>>>>> If you don't care about the application, you can safely kill
it if
>>>>>>> it's running and delete the HDFS directory by doing hdfs dfs
-rm -r
>>>>>>> /user/dtadmin/datatorrent/apps/application_xxx_yyy (replace xxx
and yyy
>>>>>>> with appropriate values).  Note that doing so will wipe all stored
>>>>>>> information about that application.
>>>>>>>
>>>>>>> David
>>>>>>>
>>>>>>> On Tue, Aug 25, 2015 at 6:32 AM, Shashi Vishwakarma <
>>>>>>> shashi.vish123@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have  DataTorrent 3.x installed on my cluster.Even thought
there
>>>>>>>> is no data torrent application is running , still my hdfs
space utilization
>>>>>>>> goes on increasing. Below is hdfs path that has occupied
most of the space.
>>>>>>>>
>>>>>>>> /user/dtadmin/datatorrent/apps
>>>>>>>>
>>>>>>>> Why this is happening? Am I missing something here?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Shashi
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the
Google
>>>>>>>> Groups "apex-dev" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails
from it,
>>>>>>>> send an email to apex-dev+unsubscribe@googlegroups.com.
>>>>>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com
>>>>>>>> <https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "apex-dev" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from
it,
>>>>>>> send an email to apex-dev+unsubscribe@googlegroups.com.
>>>>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com
>>>>>>> <https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "apex-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>> “I'd have blown my top, because I want to beat this damn thing,
>>>  as long as I've gone this far. I can't just leave it after I've found
>>>  out so much about it. I have to keep going to find out ultimately
>>> what is the matter with it in the end."
>>>                 Richard P. Feynman
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "apex-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>> To post to this group, send email to apex-dev@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "apex-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to apex-dev+unsubscribe@googlegroups.com.
> To post to this group, send email to apex-dev@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com
> <https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message