apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandeep Deshmukh <sand...@datatorrent.com>
Subject Re: Retrieving logs
Date Tue, 25 Aug 2015 18:06:58 GMT
Use the following shell script, the earlier one had a bug. This works fine
for me.

---------------------------------------
# Check if input file is passed as parameter.
if [ -f $1 ]
then
    INPUT_LOGFILE=$1;

    echo "Extracting container logs from: $INPUT_LOGFILE"
    awk -f ~/bin/extractContainerLogs.awk $INPUT_LOGFILE;
else
    echo "ERROR: input file does not exist.";
    echo "usage: $0 log_file_name";
fi;
-----------------------------------------------

Regards,
Sandeep

On Tue, Aug 25, 2015 at 9:46 AM, Sandeep Deshmukh <sandeep@datatorrent.com>
wrote:

> Hi Ilya,
>
> Here is something that I use to extract the container logs. You get one
> file per container log:
>
> ~/bin/extractContainerLogs.awk
> ------------------------------------------------
> BEGIN{
> }
>
> /^Container: container_/{found=1;container=$2;
> OUT_FILE="containerLogs/"$2;print "Writing logs to " OUT_FILE ;
> containerCount++}
> {if(found) print $0 >> OUT_FILE;}
>
> END{
> printf "Extracted logs for %d  containers.",containerCount;
> print ;
> }
> ------------------------------------------------
> Covering shell script:
>
> ~/bin/extractContainerLogs.sh
> ------------------------------------------------
> # Check if input file is passed as parameter.
> if [ $1 -e ]
> INPUT_LOGFILE=$1
>
> echo "Extracting container logs from: $INPUT_LOGFILE"
> awk -f ~/bin/extractContainerLogs.awk $INPUT_LOGFILE;
>
> echo "Done."
> ------------------------------------------------
> All the logs are saved in a directory - containerLogs/ . Please create one
> upfront.
>
> Regards,
> Sandeep
>
> On Mon, Aug 24, 2015 at 7:57 PM, Ganelin, Ilya <
> Ilya.Ganelin@capitalone.com> wrote:
>
>> Apologies! This was a half-baked e-mail that I didn¹t intend to send.
>>
>> For others¹ reference, one may also be able to use.
>>
>> yarn logs -applicationId APP_ID -ContainerId ContainerID to get logs for a
>> specific container.
>>
>> One other thing that we¹re seeing is that the yarn logs are saved by node,
>> rather than by container.
>> E.g.
>> /tmp/logs/yarn_logs/NODE_1
>> /tmp/logs/yarn_logs/NODE_2
>>
>> /tmp/logs/yarn_logs/NODE_3
>>
>>
>> Is this by design? Is there a way to change it so logs are output per
>> container? This would greatly facilitate debugging since then we would
>> have a unique log per JVM.
>>
>> Thank you!
>>
>> On 8/24/15, 7:53 PM, "Chinmay Kolhatkar" <chinmay@datatorrent.com> wrote:
>>
>> >Hi,
>> >
>> >If you have applicationId of the application you can use following
>> command
>> >to get all the logs of all the containers:
>> >
>> >yarn logs -applicationId <applicationId>
>> >
>> >Output of this command will be huge, so you might want to redirect the
>> >output to a file for analysis.
>> >
>> >-Chinmay.
>> >
>> >
>> >On Tue, Aug 25, 2015 at 6:32 AM, Ganelin, Ilya
>> ><Ilya.Ganelin@capitalone.com>
>> >wrote:
>> >
>> >> Hi all ­ we¹re seeing failures in individual containers and wanted to
>> >> figure out how to retrieve logs.
>> >> ________________________________________________________
>> >>
>> >> The information contained in this e-mail is confidential and/or
>> >> proprietary to Capital One and/or its affiliates and may only be used
>> >> solely in performance of work or services for Capital One. The
>> >>information
>> >> transmitted herewith is intended only for use by the individual or
>> >>entity
>> >> to which it is addressed. If the reader of this message is not the
>> >>intended
>> >> recipient, you are hereby notified that any review, retransmission,
>> >> dissemination, distribution, copying or other use of, or taking of any
>> >> action in reliance upon this information is strictly prohibited. If you
>> >> have received this communication in error, please contact the sender
>> and
>> >> delete the material from your computer.
>> >>
>>
>> ________________________________________________________
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message