how to write dataframes to s3 efficiently?

It turns out you have to manually specify the committer (otherwise the default one will be used, which isn’t optimized for S3):

result_df \
    .write \
    .partitionBy('my_column') \
    .option('', 'partitioned') \
    .option('fs.s3a.committer.staging.conflict-mode', 'replace') \
    .option("", "bytebuffer") \ # Buffer in memory instead of disk, potentially faster but more memory intensive
    .mode('overwrite') \
    .csv(path='s3a://mybucket/output', sep=',')

Relevant documentation can be found here:

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top