Try to search your question here, if you can't find : Ask Any Question Now ?

How to convert pivot table into df using PySpark

HomeCategory: stackoverflowHow to convert pivot table into df using PySpark
julie asked 2 weeks ago

I have the following problem: I am using PySpark to read a table from Snowflake as a dataframe. Then I perform pivot in the dataframe. After that I want to convert the pivot object back to a dataframe so to write it again in snowflake.

 df = spark.read.format(snowflakename) 
        .options(**snowflOpt) 
        .option("query",  "select DATE, PRODUCT, 
                           AMOUNT from DELIVERY_TBL") 
        .load()

df = df.withColumn("AMOUNT", df["AMOUNT"].cast(IntegerType()))

fs =df
        .groupby(df.PRODUCT)
        .pivot("DATE")
        .sum("AMOUNT")
        .show()
fs.show()

This is my script. When executing it, (after printing the pivoted table) I get the following error

AttributeError: 'NoneType' object has no attribute 'show'

My question is how can I convert it to df again so I can write it in Snowflake? Thank you.

1 Answers
Best Answer
Matthias answered 2 weeks ago
Your Answer

8 + 8 =

Popular Tags

WP Facebook Auto Publish Powered By : XYZScripts.com