Performing operation among levels of grouped variable in R/dplyr

You can subset the Response values for "A" and "B" and then take the difference.

library(dplyr)

test %>%
  group_by(Year) %>%
  summarise(diff = Response[Letter == 'A'] - Response[Letter == 'B'])

#    Year  diff
#   <int> <int>
# 1  2011    45
# 2  2012   -11
# 3  2013   -71
# 4  2014   -51
# 5  2015   -17
# 6  2016   -35
# 7  2017    66
# 8  2018   -45
# 9  2019   -71
#10  2020   -35

In this example, we can also take advantage of the fact that if we arrange the data "A" would come before "B" so we can use diff :

test %>%
  arrange(Year, desc(Letter)) %>%
  group_by(Year) %>%
  summarise(diff = diff(Response))