QDS Components: Supported Versions and Cloud Platforms¶

Supported Versions
Deprecated Versions

Supported Versions¶

The following table shows the currently supported open source versions of QDS Components, and the Cloud platforms on which they run.

QDS Component	Currently Supported Versions	Supported on (Cloud Platforms)
Airflow	1.7.0, 1.8.2, 1.10.0 (beta)	AWS, Azure, Oracle OCI, GCP
Cascading	All	AWS, Azure, Oracle OCI
Hadoop 1	0.20.1 (deprecated)	AWS
Hadoop 2	2.6.0	AWS, Azure, Oracle OCI, GCP
Hive	1.2, 2.1.1, 2.3 (beta)	AWS
	1.2, 2.1.1	Azure, Oracle OCI
	2.1.1	GCP
Java	Hadoop 1 supports Java 1.7	AWS
	Hadoop 2 and Spark support Java 1.7 as the default version, but 1.8 can be enabled through the node bootstrap.	AWS, Azure, Oracle OCI
	Presto supports only 1.8	AWS, Azure
	GCP supports only 1.8	GCP
MapReduce	0.20.1	AWS
MapReduce	2.6.0	AWS, Azure, Oracle OCI, GCP
Pig	0.11, 0.15	AWS
Pig	0.11	Azure, Oracle OCI
Presto	0.157, 0.180, 0.193, and 0.208 (beta)	AWS, Azure
Python	2.6, 2.7, and 3.5 Airflow supports only 2.7 and later	AWS See Can I use Python 2.7 for Hadoop tasks? for more information
	2.7 and 3.5	Azure, Oracle OCI
	3.6.x and 3.7	GCP
R	3.3.3	AWS, Oracle OCI
	3.3.2	Azure
	3.5.2	GCP
RubiX	0.2.11	AWS, Azure
Scala	2.10 for Spark versions older than 2.0.0	AWS
Scala	2.11 for Spark 2.0.0 and later	AWS, Azure, Oracle OCI, GCP
Spark: see Spark Version Support	1.6.2, 2.0.2, 2.1.1, 2.2.0, 2.2.1, 2.3.1, 2.4.0	AWS
	2.0.2, 2.2.1, 2.3.1	Azure, Oracle OCI
	2.3.2, 2.4	GCP
	Note In the Spark Version drop-down list on the Clusters page of the QDS UI, `latest` means the latest open-source maintenance version. If you choose `latest`, Qubole Spark is automatically upgraded when a new maintenance version is released. For example, if `2.x latest` currently points to Spark 2.x.y, then when 2.x.(y+1) is released, QDS clusters running `2.x latest` automatically start using 2.x.(y+1) on a cluster restart.
Sqoop	1.4.7	AWS
Sqoop	1.4.6	AWS, Azure, Oracle OCI
Tez	0.7	AWS, Azure, Oracle OCI
Tez	0.8.4	GCP
Zeppelin (notebooks)	0.6.2	AWS, Azure, Oracle OCI, GCP

Deprecated Versions¶

The following table shows the deprecated versions of the engines and the corresponding timelines.

Note

Deprecation Timeline: Deprecated version is marked as (deprecated) on the UI. Existing clusters with this version function normally, however, new features and bug fixes are not available. It is not recommended for production use, therefore, you should upgrade the clusters to the later versions of engines. For assistance, contact Qubole Support.
Removal Timeline: Version in the removal timeline is completely removed from the cluster image. Existing clusters with this version function when they are in running state. After the next restart, these clusters come up with the default version on the available version list.

Engines on QDS	Deprecated Versions	Deprecation Timeline	Removal Timeline
Spark	1.3.1, 1.4.0, 1.4.1, 1.5.0	2017	Yet to be removed. Estimated release: R56
Spark	1.5.1, 1.6.0, 1.6.1, 2.0.0, 2.1.0	R54 (November 2018)	Yet to be removed. Estimated release: R56
Hive	0.13	Not supported for new customers. For legacy customers, support ended in R55.	Removed in R55
Presto	0.142	R52 (January 2018)	Yet to be removed. Estimated release: R56
Presto	0.157	R56	Yet to be removed. Estimated release: R57

Spark Version Support¶

QDS supports two major versions of Spark, 1.x and 2.x. As of April 2017, Qubole began phasing out support for older versions. Those versions are marked deprecated in the drop-down list of available versions on the Clusters page of the QDS UI. You can still launch clusters running a version marked deprecated, but:

No new features or bug fixes will be applied to this version.

The version will no longer be eligible for Qubole Customer Support; tickets will not be addressed.

Hadoop 1 is Deprecated¶

Qubole has deprecated Hadoop 1 as-a-service. Qubole will support Hadoop 1 on the existing clusters until 31 December 2018. Creating a new Hadoop 1 cluster or cloning an existing Hadoop 1 cluster is not supported through the API and the UI.

If you are using Hadoop 1 clusters, then you must plan to migrate the workloads to Hadoop 2 clusters. Creating a new Hadoop 2 (Hive) cluster is one of the easiest ways to migrate the workload. As an alternative option, you can migrate the workloads to a Presto or a Spark cluster. For more information, contact the Qubole Support or the Customer Success Manager associated with the account.