passing an extremely long filter in a dotnet test?

It is a OS limitation. not much you can do about it.

See: Exec and (very) long commands on Windows

cmd.exe has a maximum command-line length limit of 8191 characters
Win32’s CreateProcess has a maximum lpCommandLine length of 32,768 characters.

Workaround

See: Command prompt (Cmd. exe) command-line string limitation

Modify programs that require long command lines so that they use a file that contains the parameter information, and then include the name of the file in the command line.