hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michel Tourn (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-489) Seperating user logs from system logs in map reduce
Date Tue, 29 Aug 2006 22:02:30 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-489?page=comments#action_12431388 ] 
            
Michel Tourn commented on HADOOP-489:
-------------------------------------

>>About (c) and providing a http url to the log file which contains 
>>the log for this task. on the client screen 
>>Would that solve your second requirement Michel?
Yes and no. 
With access to such a URL I would not ask the uer to go browse it.
But I could have HadoopStreaming retrieve the content then stream it to the user like "tail
-f". 

>polling for the last N lines would not be easy
Use HTTP Keep-Alive to do this. I.e. just keep the GET socket open and wait for this "slow
webserver" to send you more lines as they get available.


Now the scalability story:
===================

when you have access to all this real-time log information, 
a "smart" client will not blindly echo everything from every Task.
It will just show you what is wrong "by example"

Assuming we have a way to do some classification of the Tasks into 
categories:
"failed", "still going" and "successfully finished":

o pick the first one in each category
o by default, ignore log URLs for the other logs
o do a merged "tail -f" with a banned on the client. "tail -f " is implemented as HTTP Keep-Alive
as described. 3 HTTP sockets at most. 

Yes, all this assumes some smart-client support.

But we can easily package the code as reusable logic to be used from any user-written JobSubmitter
main class. 
One of these clients is HadoopStreaming.

If a client application does not use this support code:
it is just ignoring the log URLs made available, but otherwise works as usual.

> Seperating user logs from system logs in map reduce
> ---------------------------------------------------
>
>                 Key: HADOOP-489
>                 URL: http://issues.apache.org/jira/browse/HADOOP-489
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Mahadev konar
>         Assigned To: Mahadev konar
>            Priority: Minor
>
> Currently the user logs are a part of system logs in mapreduce. Anything logged by the
user is logged into the tasktracker log files. This create two issues-
> 1) The system log files get cluttered with user output. If the user outputs a large amount
of logs, the system logs need to be cleaned up pretty often.
> 2) For the user, it is difficult to get to each of the machines and look for the logs
his/her job might have generated.
> I am proposing three solutions to the problem. All of them have issues with it -
> Solution 1.
> Output the user logs on the user screen as part of the job submission process. 
> Merits- 
> This will prevent users from printing large amount of logs and the user can get runtime
feedback on what is wrong with his/her job.
> Issues - 
> This proposal will use the framework bandwidth while running jobs for the user. The user
logs will need to pass from the tasks to the tasktrackers, from the tasktrackers to the jobtrackers
and then from the jobtrackers to the jobclient using a lot of framework bandwidth if the user
is printing out too much data.
> Solution 2.
> Output the user logs onto a dfs directory and then concatenate these files. Each task
can create a file for the output in the log direcotyr for a given user and jobid.
> Issues -
> This will create a huge amount of small files in DFS which later can be concatenated
into a single file. Also there is this issue that who would concatenate these files into a
single file? This could be done by the framework (jobtracker) as part of the cleanup for the
jobs - might stress the jobtracker.
>  
> Solution 3.
> Put the user logs into a seperate user log file in the log directory on each tasktrackers.
We can provide some tools to query these local log files. We could have commands like for
jobid j and for taskid t get me the user log output. These tools could run as a seperate map
reduce program with each map grepping the user log files and a single recude aggregating these
logs in to a single dfs file.
> Issues-
> This does sound like more work for the user. Also, the output might not be complete since
a tasktracker might have went down after it ran the job. 
> Any thoughts?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message