giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From José Luis Larroque <larroques...@gmail.com>
Subject Re: RELATION BETWEEN THE NUMBER OF GIRAPH WORKERS AND THE PROBLEM SIZE
Date Mon, 27 Feb 2017 16:57:07 GMT
Could be a lot of different reasons. Memory problems, algorithm problems,
etc.

I recommend you to focus in reach the logs instead of guessing why the
worker's are dying. Maybe you are looking in the wrong place, maybe you can
access to them though web ui instead of command line.

>From terminal, doing yarn logs -applicationId "id" doing will be enough for
seeing them. If you want to access your phyisical files in your nodes, you
should go to all nodes and check everyone of them, and search for the
different containers of your application in the directory where those are.

Another link with help:
http://stackoverflow.com/questions/32713587/how-to-keep-yarns-log-files.

Maybe you could test the algorithm locally instead of running it on the
cluster, for a better understanding of the relation between yarn and Giraph.

Bye

-- 
*José Luis Larroque*
Analista Programador Universitario - Facultad de Informática - UNLP
Desarrollador Java y .NET  en LIFIA

2017-02-27 12:27 GMT-03:00 Sai Ganesh Muthuraman <saiganeshpsn@gmail.com>:

> Hi,
>
> The first container in the application logs usually contains the gam logs.
> But the first container logs are not available. Hence no gam logs.
> What could be the possible reasons for the dying of some workers?
>
>
> Sai Ganesh
>
>
>
> On Feb 25, 2017, at 9:30 PM, José Luis Larroque <user@giraph.apache.org>
> wrote:
>
> You are probably looking at your giraph application manager (gam) logs.
> You should look for your workers logs, each one have a log (container's
> logs). If you can't find them, you should look at your yarn configuration
> in order to know where are them, see this: http://stackoverflow.com/
> questions/21621755/where-does-hadoop-store-the-logs-of-yarn-applications.
>
> I don't recommend you to enable checkpointing until you now the specific
> error that you are facing. If you are facing out of memory errors for
> example, checkpointing won't be helpful in my experience, the same error
> will happen over and over.
>
> --
> *José Luis Larroque*
> Analista Programador Universitario - Facultad de Informática - UNLP
> Desarrollador Java y .NET  en LIFIA
>
> 2017-02-25 12:38 GMT-03:00 Sai Ganesh Muthuraman <saiganeshpsn@gmail.com>:
>
> Hi Jose,
>
> Which logs do I have to look into exactly, because in the application
> logs, I found the error message that I mentioned and it was also mentioned
> that there was *No good last checkpoint.*
> I am not able to figure out the reason for the failure of a worker for
> bigger files. What do I have to look for in the logs?
> Also, How do I enable Checkpointing?
>
>
> - Sai Ganesh Muthuraman
>
>
>
>
>
>
>

Mime
View raw message