is there any way to set the glue crawler recrawl policy in my cf template?

As per my understanding, Incremental policy is a relatively new feature in Glue and not supported in Cloud Formation yet.

A workaround I can suggest to overcome this limitation is creating a crawler using cloudformation and then use AWS CLI to update its RecrawlPolicy property.

When you create a crawler using cloudformation and try to retrieve its properties using CLI, RecrawlPolicy” has “RecrawlBehavior” set to “CRAWL_EVERYTHING”. You can use the below command to change it to incremental crawls (Crawl new folders only).

aws glue update-crawler 
    --name <crawlername> 
    --recrawl-policy '{"RecrawlBehavior": "CRAWL_NEW_FOLDERS_ONLY"}' 
    --schema-change-policy '{"UpdateBehavior":"LOG","DeleteBehavior":"LOG"}'

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top