Compatible with IntelliJ IDEA Ultimate, DataGrip and 2 more

Included In

Submit Spark/PySpark jobs via using run configurations in your IDE.

Monitor Spark jobs from the IDE.

The Spark plugin will automatically install the Metastore Core and Remote File Systems plugins. The Remote File Systems plugin lets you connect to remote storages, and Metastore Core lets you connect to:

  • AWS Glue
  • AWS EMR
  • Dataproc
  • Hive Metastore
  • Hadoop YARN

This plugin is a part of the Big Data Tools plugin bundle.

What’s New

Changes in 2023.3

Features

  • Added ability to submit Spark applications on a DataProc cluster
  • Added ability custom Spark cluster connection for submitting applications
  • Added automatic run configuration suggestion for PySpark projects
  • New Pyspark project wizard available
  • Spark project wizard reworked
  • Added automatic run configuration suggestion for Spark projects using Scala 3
  • Added debugging support for JVM Spark applications that were run with Run configuration
  • Added static analysis for Spark and Pyspark DataFrame columns
  • Added autocompletion for Spark and Pyspark DataFrame columns
  • Added Spark jobs monitoring (EMR, Dataproc, Custom cluster) into Services toolwindow
  • Spark monitoring Executor logs now provide two options - stdout and stderr
  • View settings for Spark monitoring Stages in the Jobs tab and in the Stages tab are independent now
  • Added Spark monitoring and SFTP templates for EMR clusters in the Big Data Tools toolwindow
  • Added button to copy Spark application logs from Console
  • Added ability to filter Spark applications `by me` (submitted by the current user)

Fixed Bugs

  • Fixed limitations for the Spark applications list to avoid lack of necessary applications
  • Spark and Hadoop monitoring view settings are saved correctly now
  • `Delete Job` button is now disabled when a Job is executing
  • Supported sso-session property for AWS connections
  • AWS Glue region is saved correctly now

The full release notes are available here.

Feb 06, 2025
Version 251.20015.29

Rating & Reviews

4.5
1 Rating (1,031,686 Downloads)
5
4
3
2
1

Additional Information

Vendor:
Plugin ID:
com.intellij.bigdatatools.spark