0

I keep running into this issue when running PySpark.

I was able to connect to my database and retrieve data, but whenever I try do operations like .show() or .count(), or when I try to save a Spark DataFrame to a CSV, it keeps crashing with the following error traceback.

(Note: I am using SparkSession.builder)

Error 1:

py4j.protocol.Py4JJavaError: An error occurred while calling o121.count.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 13 in stage 0.0 failed 1 times, most recent failure: Lost task 13.0 in stage 0.0 (TID) ( driver): org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)
        at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:624)
        at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:599)

Error 2:

25/09/29 15:21:22 WARN TaskSetManager: Lost task 1.0 in stage 2.0 (TID) ( executor driver): TaskKilled (Stage cancelled: Job aborted due to stage failure: Task 4 in stage 2.0 failed 1 times, most recent failure: Lost task 4.0 in stage 2.0 (TID 4) (executor driver): org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)
        at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:624)
        at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:599)
Improve this question
3
  • 1
    You can probably find more details about the error in stdout log produced by python runner itself than in stderr printed by Spark driver. Commented Sep 29 at 14:39
  • Per mazaneicha, please look in the driver logs to see what actually happened and use this to create a question that people can answer. As it stands there is no real information for people here to be able to help. Commented Sep 30 at 10:22
  • the key detail in the error is Python worker exited unexpectedly (crashed). this could be due to multiple reasons - one of which is incompatible python or java version; another is incorrect configs. try looking into those. Commented Sep 30 at 10:33
Related questions
0
Error from PySpark code to showdataFrame : py4j.protocol.Py4JJavaError
0
fail DF show pyspark
18
Apache Spark: pyspark crash for large dataset
Load 4 more related questions Show fewer related questions

0

Reset to default

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.