You can use the isin function.
East = ["Bengal", "Bihar", "Assam"]
West = ["Bombay", "Gujarat", "Goa"]
from pyspark.sql.functions import when, col
df.withColumn("state", when(col("City").isin(East), "East")\
.when(col("City").isin(West), "West").otherwise(None)).show()
+---+------+-----+
|Num| City|state|
+---+------+-----+
| 1|Bengal| East|
| 2| Goa| West|
| 3|Bombay| West|
| 4| Bihar| East|
+---+------+-----+
CLICK HERE to find out more related problems solutions.