scala tasks are not serializable when using closure

Instead of defining a function, try using a val fun

val difference = (x:Int) => {x-mean}

When you use def to define a function, Spark would try to serialize your object that has this function. This usually results into TaskNotSerializable as there might be something (val or var) in that Object which is not serializable.

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top