Python Dummy

In another similar question, they hint 'install older spark 2.4.5.' EDIT: the solution from… Read more No Module Named 'pyspark.streaming.kafka' Even With Older Spark Version

I have a Spark DataFrame (using PySpark 1.5.1) and would like to add a new column. I've tried t… Read more How Do I Add A New Column To A Spark Dataframe (using Pyspark)?

I am having 100,000+ of records in dataframe. I want to create a file dynamically and push 1000 rec… Read more How To Get 1000 Records From Dataframe And Write Into A File Using Pyspark?

I'm trying to configure apache-spark on MacOS. All the online guides ask to either download the… Read more Spark Installation And Configuration On Macos Importerror: No Module Named Pyspark

I have Hortonworks HDP 2.6.3 running Spark2 (v2.2). My test case is very simple: Create a Hive tab… Read more Spark 2.2 Thrift Server Error On Dataframe Numberformatexception When Query Hive Table

Trying to cast StringType to ArrayType of JSON for a dataframe generated form CSV. Using pyspark on… Read more How To Cast String To Arraytype Of Dictionary (json) In Pyspark

After reading through the documentation I do not understand how does Spark running on YARN account … Read more How Does Spark Running On Yarn Account For Python Memory Usage?

How can I change spark.driver.maxResultSize in pyspark interactive shell? I have used the following… Read more How To Change Sparkcontext Properties In Interactive Pyspark Session