hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: MapReduce logs
Date Wed, 21 Nov 2012 19:55:54 GMT
Thanks for the info.

I have quickly draft this bash script in case it can help someone...
You just neeed to make sure the IP inside is replaced.
To call it, you need to give the job task page.

./showLogs.sh "http://192.168.23.7:50030/jobtasks.jsp?jobid=job_201211211408_0001&type=map&pagenum=1"

Then you can redirect the output, or do what ever you want.

I was wondering if there was a "nicer" solution...

:~/test$ cat showLogs.sh
#!/bin/bash
rm -f tasks.html
wget --quiet --output-document tasks.html $1
for i in `cat tasks.html | grep taskdetails | cut -d"\"" -f2 | grep
taskdetails`; do
	rm -f tasksdetails.html
	wget --quiet --output-document tasksdetails.html http://192.168.23.7:50030/$i
	for j in `cat tasksdetails.html | grep "all=true" | cut -d"\"" -f6`; do
		printf "*"%.0s {1..80}
		echo
		echo $j
		printf "*"%.0s {1..80}
		echo
		rm -f logs.txt
		wget --quiet --output-document logs.txt $j
		tail -n +31 logs.txt | head -n -2
	done
done
rm -f tasks.html
rm -f tasksdetails.html
rm -f logs.txt


2012/11/21, Dino Kečo <dino.keco@gmail.com>:
> Hi,
>
> We had similar requirement and we built small Java application which gets
> information about task nodes from Job Tracker and download logs into one
> file using URLs of each task tracker.
>
> For huge logs this becomes slow and time consuming.
>
> Hope this helps.
>
> Regards,
> Dino Kečo
> msn: xdinno@hotmail.com
> mail: dino.keco@gmail.com
> skype: dino.keco
> phone: +387 61 507 851
>
>
> On Wed, Nov 21, 2012 at 7:55 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> Hi,
>>
>> When we run a MapReduce job, the logs are stored on all the tasktracker
>> nodes.
>>
>> Is there an easy way to agregate all those logs together and see them
>> in a single place instead of going to the tasks one by one and open
>> the file?
>>
>> Thanks,
>>
>> JM
>>
>

Mime
View raw message