Understanding Hive Versions

QDS supports these versions of Hive:

  • Hive 1.2.0 on Hadoop 2 clusters
  • Hive 2.1.1 on Hadoop 2 clusters
  • Hive 2.3 (beta) on Hadoop 2 clusters

Using Different Versions of Hive on QDS

Qubole supports Hive versions 1.2.0 and 2.1.1 only on Hadoop 2 clusters. Hadoop 2 clusters are also known as Hadoop 2 (Hive) clusters.

Note

You can configure the Pig version on an Hadoop 2 (Hive) cluster. Pig 0.11 is the default version. Pig 0.15 and Pig 0.17 (beta) are the other supported versions. You can also choose between MapReduce and Tez as the execution engine when you set the Pig 0.17 (beta) version. Pig 0.17 (beta) is only supported with Hive 1.2.0.

Qubole supports Hive versions 1.2.0 and 2.1.1 on different ways of running Hive that are described in Understanding Different Ways to Run Hive.

Use different versions of Hive on QDS as described here:

  • Hive 1.2.0 - This version is supported only on Hadoop 2 clusters. You can configure it as follows:

    • Select it from the Hive Version drop-down when you configure a Hadoop 2 (Hive) cluster in QDS.
    • You can run it with Hive on Master. Create a ticket with Qubole Support for an account-wide configuration, so that all clusters in the account use Hive on Master. For more information, see Enabling Hive on the Cluster Master.
    • To use Hive Server 2, enable Hive Server 2, under the Advanced Configuration tab of the Hadoop 2 (Hive) cluster, For more information, see Configuring a HiveServer2 Cluster (AWS).

  • Hive 2.1.1 - This is a stable version supported only on Hadoop 2 clusters. You can configure it by setting 2.1.1 as the Hive version when you configure a Hadoop 2 (Hive) cluster. To use Hive Server 2, choose Enable Hive Server 2 under the Advanced Configuration tab. In this case, QDS will always use Hive-on-Master for this cluster.

    Note

    LLAP from the Hive open source is not verified in Qubole’s Hive 2.1.1.

    For more information, see Configuring a HiveServer2 Cluster (AWS). You can set the versions through the API as described in engine_config for an HiveServer2 Cluster. After setting Hive 2.1.1 as the Hive version:

    • Hive 2.1.1 queries run through QDS servers by default.

      Note

      The feature to run Hive 2.x queries on QDS servers by default is not available on all QDS accounts. Create a ticket with Qubole Support to get it enabled on the QDS account.

    • To run Hive 2.1.1 queries on Hive-on-master, refer to How to Enable Hive-on-Master. If Hive 2.x queries do not run on QDS servers by default, then they run on the master by default.

    • If you enable Hive Server 2, Hive 2.1.1 queries run through HiveServer2.

  • Hive 2.3 (beta) - This is a beta version. The /media/ephemeral0/hive1.2/metastore.properties file is deleted. Remove the dependency on the metastore.properties file if you use Hive 2.3 (beta). When Hive 2.3 (beta) is running on QDS servers, it uses Java 8. It is compatible with Java 7 as well.

    You can configure it by setting 2.3 (beta) as the Hive version when you configure a Hadoop 2 (Hive) cluster. To use Hive Server 2, choose Enable Hive Server 2 under the Advanced Configuration tab. In this case, QDS will always use Hive-on-Master for this cluster.

    For more information, see Configuring a HiveServer2 Cluster (AWS). You can set the versions through the API as described in engine_config for an HiveServer2 Cluster. After setting Hive 2.3 (beta) as the Hive version:

    • Hive 2.3 (beta) queries run through QDS servers by default.
    • To run Hive 2.3 (beta) queries on Hive-on-master, refer to How to Enable Hive-on-Master. If Hive 2.x queries do not run on QDS servers by default, then they run on the master by default.
    • If you enable Hive Server 2, Hive 2.3 (beta) queries run through HiveServer2.