hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Sivachenko <trtrmi...@gmail.com>
Subject Log Aggregation
Date Fri, 13 Feb 2015 14:54:14 GMT

I am using hadoop-2.4.1 in distributed mode.  After a job completes, logs are aggregated to
hdfs and are available via history server.

Sometimes logs appear very fast after the job completes (or fails), but sometimes it takes
long (10-20-30 minutes).

During that period history server reports:
Logs not available for attempt_1422914757889_1881_m_000000_0. Aggregation may not be complete,
Check back later or try the nodemanager at <host>

It seems that it does not depend on log size, it is not so big to take 20 minutes to copy
to hdfs.

Why this can happen?  How can I debug the issue to understand what is happening during that
period before logs appear at history server?

View raw message