Retaining Job and Task Logs in Hadoop 1

Hadoop’s JobTracker (JT) generates logs such as job configuration XML and job statistics. A TaskTracker (TT) generates logs such as Standard Error for a specific task attempt. These logs serve various purposes such as debugging job/task failures that occur because of hardware or platform issues. The logs are stored in different locations and retained for a specific amount of time as required. There are different configurations set to control logs, which are done in mapred-site.xml by default.

Retaining Job Tracker Logs

Job Tracker logs for in-memory and retired jobs are as explained below:

  • In-memory jobs - An in-memory job means that JT responds to API Calls about job status and so on through a JT API. It also means that JT contains detailed information about these jobs when compared to the Retired Jobs such as task-level counters. Jobs maintained in memory is controlled by:
    • mapred.jobtracker.retirejob.interval - The default interval is 24 hours. Jobs that are older than 24 hours are considered as retired.
    • mapred.jobtracker.completeuserjobs.maximum - The default value is 100 that is, when the number of jobs exceeds 100, the excess jobs are considered as retired.
  • Retired Jobs - Once a job is retired, its information is retrievable from the file system through history files. The following parameter is set to retrieve a retired job’s details.
    • mapred.job.tracker.retiredjobs.cache.size

The default parameter value is 1000, which means that you can retrieve a maximum of 1000 retired jobs in the QDS UI from the JT URLs.

Note

History files are only maintained for a month and the home page of the JT shows only 100 retired jobs and does not list all 1000 retired jobs if any.

Retaining Task Tracker Logs

A TT operates on its own and is independent of a JT. It cleans logs of task attempts based on the following parameters:

  • mapred.userlog.retain.hours - The default value is 24 hours.
  • mapred.userlog.files.limit - The default value is 20000.

So, in QDS, you can get the JT UI to render a certain job’s information. However, you may possibly not find the task logs because they would have been purged because of the above parameters.

If task logs are being backed up to an Amazon S3 location, then relying on TT for logs is not required.