I found that adding the spark location through findspark and java8 with os at the beginning of the script the easiest solution:
import findsparkimport osspark_location='/opt/spark-2.4.3/' # Set your ownjava8_location= '/usr/lib/jvm/java-8-openjdk-amd64' # Set your ownos.environ['JAVA_HOME'] = java8_locationfindspark.init(spark_home=spark_location)