Configuring Fair Scheduler on Hadoop

To configure Fair Scheduler in Qubole on Hadoop 0.20.1, perform the following steps:

1. Create a XML file for the Fair Scheduler configuration. A sample Fair Scheduler XML is as provided below.

<allocations>
   <pool name="default">
     <minMaps>72</minMaps>
     <canBePreempted>true</canBePreempted>
     <fifo>false</fifo>
   </pool>
   <pool name="sqoop-conf">
     <maxMaps>32</maxMaps>
     <maxReduces>8</maxReduces>
     <fifo>true</fifo>
   </pool>
   <pool name="sqoop-conf2">
     <maxMaps>2</maxMaps>
     <fifo>true</fifo>
   </pool>
   <pool name="batch">
     <minMaps>216</minMaps>
     <fifo>true</fifo>
   </pool>
   <pool name="fast-paced">
     <minMaps>50</minMaps>
     <fifo>false</fifo>
     <minSharePreemptionTimeout>300</minSharePreemptionTimeout>
   </pool>
   <fairSharePreemptionTimeout>60</fairSharePreemptionTimeout>
</allocations>

Within a <pool> tag, <canBePreempted>true</canBePreempted> is an important XML tag. By default, the value defaults to true and all pools are preemptible. You can set an individual pool to false, to prevent tasks from that pool from being preempted.

Use <canBePreempted>true</canBePreempted> with caution because this can allow tasks from this (non-preemptible) pool to hog the cluster. It makes sense to use the tag for small jobs that do important activity.

The last element, <fairSharePreemptionTimeout>60</fairSharePreemptionTimeout> means by default tasks are preempted after 60s (if they do not get their fair share of the cluster). This is something you frequently want to change to see how aggressive preemption is.

2. The Fair Scheduler configuration is at the cluster level that is available in the Edit Cluster tab of the Clusters page.

3. The Fair Scheduler configuration can include certain custom XML tags that are specific to Qubole.

   Example: Qubole specifies use of FIFO scheduling within a pool differently than open source Hadoop.

   Use <fifo>true</fifo> instead of <schedulingMode>FIFO</schedulingMode> (applicable to later versions of Hadoop Fair Scheduler).

   This property is specified within a Fair Scheduler pool. It denotes whether the resources within a pool are allocated to jobs in a FIFO way. If it is set to false, resources within a pool are allocated fairly.

4. If you restart the cluster or Push the configuration (Push is available as one of the actions in the actions (reperesented by the ellipse) of the Clusters page of the QDS UI), the new Fair Scheduler settings become effective. See Cluster Operations for more information.

5. When a job is being submitted, you can specify which Fair Scheduler pool to use. The specified Fair Scheduler pool overrides the default Fair Scheduler pool specified (if any). When you run database import/export commands, you can select a Fair Scheduler pool from the QDS UI. If you are running sqoop from a command line interface (through a Shell command), you can use the -Dmapred.fairscheduler.pool switch.

6. Small jobs limit

   In Fair Scheduler, Qubole allows small jobs to go through even if the system is burdened with many big jobs.

   A mapreduce job is a small job if it has a maximum of 10 tasks or less than (mapred.fairscheduler.smalljob.taskperc * totalTaskLimit) tasks.

   Where:

   mapred.fairscheduler.smalljob.taskperc is a Hadoop configuration (float) and must be configured in the value range of (0 - 1).

   totalTaskLimit is mapred.fairscheduler.total.task.limit - It denotes the maximum number of jobs permitted by the JobTracker (default value is 800000).

7. Auto task limit

   Qubole uses a smarter way to calculate the total task limit based on the machine specifications on which the JobTracker (JT) runs. It is enabled by setting this parameter, mapred.fairscheduler.auto.task.limit.enable (default parameter value is true).

   Total task limit in this case is: (20000 * heap_memory of the JT)

   You can change the value, 20000 by using the parameter, mapred.fairscheduler.auto.task.limit.pergb.