pyspark column transformation

You can use the isin function.

East = ["Bengal", "Bihar", "Assam"]
West = ["Bombay", "Gujarat", "Goa"]

from pyspark.sql.functions import when, col

df.withColumn("state", when(col("City").isin(East), "East")\
    .when(col("City").isin(West), "West").otherwise(None)).show()

|Num|  City|state|
|  1|Bengal| East|
|  2|   Goa| West|
|  3|Bombay| West|
|  4| Bihar| East|

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top