Geom_text(): Error in is.factor(x) : object ‘Year’ not found

@Tung’s note about inherit.aes = FALSE, along with more precise x/y values, will give you the plot you want:

X2015_2019_R_Use %>% 
  mutate(Year = as.factor(Year)) %>%
  ggplot(aes(x = Year, y = `Relative Abundance`, fill = Year)) + 
  geom_boxplot() + 
  facet_grid(Site ~ Bacteria) + 
  geom_text(
    data = dat_text,
    mapping = aes(label = paste0("p = ", label)),
    x = "2019", y = .25, 
    inherit.aes = FALSE
  ) + 
  labs(x="Year", fill = "Year")

*Note that x needs to be specified as a string and not a number, given its factor data type.