Skip to content Skip to sidebar Skip to footer
Home / Apache Spark / Pyspark / Python

SparkException: Python Worker Failed To Connect Back When Execute Spark Action

Post a Comment
When I try to execute this command line at pyspark arquivo = sc.textFile('dataset_analise_sentimento.csv') I got the following error message: Py4JJavaError: An error occurred whil

Solution 1:

I just configure the following variables environment and now it's working normally:

  • HADOOP_HOME = C:\Hadoop
  • JAVA_HOME = C:\Java\jdk-11.0.6
  • PYSPARK_DRIVER_PYTHON = jupyter
  • PYSPARK_DRIVER_PYTHON_OPTS = notebook
  • PYSPARK_PYTHON = python

Actually I´m using the following versions:

Python 3.7.3, Java JDK 11.0.6, Windows 10, Apache Spark 2.4.3 and using Jupyter Notebook with pyspark.


Solution 2:

I got the same error. Got it resolved by Java classpath to JDK 11.

enter image description here


Post a Comment for "SparkException: Python Worker Failed To Connect Back When Execute Spark Action"