what are best ways to configure a data pipeline in the google cloud platform?

One of the options is to use Cloud Functions configured to be triggered by uploading a file. Another option is to subscribe to Pub/Sub notifications for Cloud Storage. One more option is to use Apache Beam‘s Google Cloud Dataflow Runner.

Under heavy load sometimes triggering events can be duplicated, so you need to make sure that single file won’t be processed multiple times by your cloud function.

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top