AWS Glue crawler able to parse the struct definition but Athena fails to read correctly

Glue has a tendency of creating tables that don’t work with Athena, and this is just yet another example.

The OpenCSVSerde does not have support for complex types, so there’s no chance of this working. Why Glue does this is anyone’s guess.

LazySimpleSerDe has some support for complex types, but they need to be encoded differently than in your data.

My suggestion is to stop using Glue crawler altogether and create the table manually using string as the type for the publisher_parameters column and then use JSON functions to parse it at query time. There is to my knowledge no way to get any Athena CSV serde to parse that data as a struct.

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top