apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandeep Deshmukh <sand...@datatorrent.com>
Subject Re: Retrieving logs
Date Tue, 25 Aug 2015 16:46:44 GMT
Hi Ilya,

Here is something that I use to extract the container logs. You get one
file per container log:

~/bin/extractContainerLogs.awk
------------------------------------------------
BEGIN{
}

/^Container: container_/{found=1;container=$2;
OUT_FILE="containerLogs/"$2;print "Writing logs to " OUT_FILE ;
containerCount++}
{if(found) print $0 >> OUT_FILE;}

END{
printf "Extracted logs for %d  containers.",containerCount;
print ;
}
------------------------------------------------
Covering shell script:

~/bin/extractContainerLogs.sh
------------------------------------------------
# Check if input file is passed as parameter.
if [ $1 -e ]
INPUT_LOGFILE=$1

echo "Extracting container logs from: $INPUT_LOGFILE"
awk -f ~/bin/extractContainerLogs.awk $INPUT_LOGFILE;

echo "Done."
------------------------------------------------
All the logs are saved in a directory - containerLogs/ . Please create one
upfront.

Regards,
Sandeep

On Mon, Aug 24, 2015 at 7:57 PM, Ganelin, Ilya <Ilya.Ganelin@capitalone.com>
wrote:

> Apologies! This was a half-baked e-mail that I didn¹t intend to send.
>
> For others¹ reference, one may also be able to use.
>
> yarn logs -applicationId APP_ID -ContainerId ContainerID to get logs for a
> specific container.
>
> One other thing that we¹re seeing is that the yarn logs are saved by node,
> rather than by container.
> E.g.
> /tmp/logs/yarn_logs/NODE_1
> /tmp/logs/yarn_logs/NODE_2
>
> /tmp/logs/yarn_logs/NODE_3
>
>
> Is this by design? Is there a way to change it so logs are output per
> container? This would greatly facilitate debugging since then we would
> have a unique log per JVM.
>
> Thank you!
>
> On 8/24/15, 7:53 PM, "Chinmay Kolhatkar" <chinmay@datatorrent.com> wrote:
>
> >Hi,
> >
> >If you have applicationId of the application you can use following command
> >to get all the logs of all the containers:
> >
> >yarn logs -applicationId <applicationId>
> >
> >Output of this command will be huge, so you might want to redirect the
> >output to a file for analysis.
> >
> >-Chinmay.
> >
> >
> >On Tue, Aug 25, 2015 at 6:32 AM, Ganelin, Ilya
> ><Ilya.Ganelin@capitalone.com>
> >wrote:
> >
> >> Hi all ­ we¹re seeing failures in individual containers and wanted to
> >> figure out how to retrieve logs.
> >> ________________________________________________________
> >>
> >> The information contained in this e-mail is confidential and/or
> >> proprietary to Capital One and/or its affiliates and may only be used
> >> solely in performance of work or services for Capital One. The
> >>information
> >> transmitted herewith is intended only for use by the individual or
> >>entity
> >> to which it is addressed. If the reader of this message is not the
> >>intended
> >> recipient, you are hereby notified that any review, retransmission,
> >> dissemination, distribution, copying or other use of, or taking of any
> >> action in reliance upon this information is strictly prohibited. If you
> >> have received this communication in error, please contact the sender and
> >> delete the material from your computer.
> >>
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message