hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabriele Kahlout <gabri...@mysimpatico.com>
Subject Re: How to debug why I don't get hadoop logs?
Date Wed, 01 Jun 2011 12:58:09 GMT
Apparently the issue is more complicated that I first thought, and it is not
a Nutch issue.

Submitting a MapReduce Job to the JobTracker (through JobClient interface)
the task is executed on another node, with different stdout and stderr from
the ones the job is submitted. Hence, no matter what log settings in the
job, those won't surface to my console.
"The standard output (stdout) and error (stderr) streams of the task are
read by the TaskTracker and logged to ${HADOOP_LOG_DIR}/userlogs" instead,
possibly through:
org.apache.hadoop.mapred.TaskLog.captureOutAndError(..).


Postmortem one could use debugging scripts to read those
logs.<http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Debugging>

In my imagination it would be possible to aggregate the stdout outputs of
each task (as they are logged to ${HADOOP_LOG_DIR}/userlogs) and output them
to the console of the JobTracker, but it seemly horribly complicated.

Perhaps the easiest thing would be for me to have a script that watches
every addittion to ${HADOOP_LOG_DIR}/userlogs and outputs the content of
each new syslog.

The reason I'm posting is to share my findings, and possibly receive expert
feedback /validate my thesis and workaround proposal.

On Thu, May 26, 2011 at 11:58 PM, Gabriele Kahlout <gabriele@mysimpatico.com
> wrote:

> Hello,
>
> I'm running nutch on a hadoop cluster but unfortunately I don't find under
> hadoop_home/logs datanote logs but only a jobtracker log. I've not modified
> nutch log4j.properties nor hadoops.
> To the console I get printed mapred.JobClient stuff and also nutch stuff
> the nutch class logs directly before running as a job.
>
>
> --
> Regards,
> K. Gabriele
>
> --- unchanged since 20/9/10 ---
> P.S. If the subject contains "[LON]" or the addressee acknowledges the
> receipt within 48 hours then I don't resend the email.
> subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧
> time(x) < Now + 48h) ⇒ ¬resend(I, this).
>
> If an email is sent by a sender that is not a trusted contact or the email
> does not contain a valid code then the email is not received. A valid code
> starts with a hyphen and ends with "X".
> ∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧
y ∈
> L(-[a-z]+[0-9]X)).
>
>


-- 
Regards,
K. Gabriele

--- unchanged since 20/9/10 ---
P.S. If the subject contains "[LON]" or the addressee acknowledges the
receipt within 48 hours then I don't resend the email.
subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧ time(x)
< Now + 48h) ⇒ ¬resend(I, this).

If an email is sent by a sender that is not a trusted contact or the email
does not contain a valid code then the email is not received. A valid code
starts with a hyphen and ends with "X".
∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈
L(-[a-z]+[0-9]X)).

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message