TinyMCE doesn’t support rendering custom UI directly in the dialogs due to primarily CSS cascading issues. For these more advanced use-cases you can use the URL dialog type which renders the dialog body content inside a iframe
CLICK HERE to find out more related problems solutions.