Ask Experienced Devs hahattpro • 4 months ago • 93%

Apache Beam - How to control/limit the concurrency process of ParDo ?

The problem is I am using beam.ParDo to call api, to enrich my data. ``` pipeline | "read_csv" >> beam.io.ReadFromCsv("...") | "get_info_api" >> beam.ParDo(GetApiDoFN()) | "write_output" >> beam.io.WriteToBigQuery(...) ``` The problem is that api have limit. I need a way to control so it do not overload api or get 429 error. I would like to set api call should not exceed 4 call / second. How should I do it ?