how can i repair a delta location file in adls gen 2?

Sometimes changes in table partitions/columns will not be picked by hive megastore, refresh the table is always a good practice before you trying to do some queries. This exception can occur if the metadata picked up from the current job is altered from any other job while this job still running.

Refresh Table: Invalidates the cached entries, which include data and metadata of the given table or view. The invalidated cache is populated in a lazy manner when the cached table or the query associated with it is executed again.

%sql
REFRESH [TABLE] table_identifier

OR

Here are some recommendations to resolve this issue:

  • Add the configuration either on cluster label (spark.databricks.io.cache.enabled false) or in first command of master notebook using spark.conf.set("spark.databricks.io.cache.enabled", "false")
  • Add the “sqlContext.clearCache()” after the delete operation.
  • Add the “FSCK REPAIR TABLE [db_name.]table_name” after delete operation.

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top