how can i use udf and other functions when running a big query query using the dataflow engine?

I do not think you will be able to. Dataflow SQL uses a variant of ZetaSQL and even with that only supports a subset. Below is the supported functions:

https://cloud.google.com/dataflow/docs/reference/sql

ZetaSQL itself does have an ARRAY_AGG function, however it does not seem to be supported yet in Dataflow SQL. https://github.com/google/zetasql

Separately what is the use case for the Dataflow engine here, typically you would use it to gain access to querying a pubsub subscription directly for streaming analysis.