How to preserve default order in Spark dataframe after first order by inside window

As guided by @Jxc in comments, I was able to to it with help of adding monotonically_increasing_id() before window operation.

Here is the example

df= df.withColumn("idx", monotonically_increasing_id())
w= Window.partitionBy("user-id").orderBy('C1','idx')

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top