if a non-compatibility group is created

You can use

string <- "text1 text2(stuff)(blahh)  text3"
sub(".*text2\\([^()]*\\)(?:\\(([^()]*)\\))?.*", "\\1", string, perl=TRUE)
# => blah

See the regex demo and the R demo.

Details

• .* – any 0 or more chars (other than line break chars) as many as possible
• text2\\( – literal text2( text
• [^()]* – zero or more chars other than ( and )
• \) – a ) char
• (?:\(([^()]*)\))? – an optional non-capturing group:
  • \( – a ( char
  • ([^()]*) – Group 1 (\1 in the replacement pattern refers to the text captured with this group): zero or more chars other than ( and )
  • \) – a ) char
• .* – any 0 or more chars (other than line break chars) as many as possible