drop rows in a dataframe where column x is a datetime value

Use to_datetime with errors='coerce' for convert non datetimelike to NaNs, so filter by Series.isna in boolean indexing:

df = df[pd.to_datetime(df['Column X'], errors='coerce').isna()]

But sometimes pandas recognise some integers like 2000 for datetimes, so if possible more accurate is specify format of datetimes, e.g. here YYYY-MM-DD:

df = df[pd.to_datetime(df['Column X'], errors='coerce', format='%Y-%m-%d').isna()]