Handyspark Dataframe works on driver or executor

HandySpark isn’t a “bridge” – it’s a wrapper round a Spark DataFrame which gives it a pandas-like API. Therefore it executes on the executors; there would be little point in the project if it executed on the driver as you could always just to toPandas on your DataFrame to pull it back to the driver (don’t do this!)

That all said, HandySpark seems to be abandoned, the last commit being eighteen months ago in May 2019. I’d suggest looking at Koalas instead which does the same thing but is actively developed.

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top