Introduction

Either for specific business needs or company policies or any other reasons, an enricher might need to restrict its call rate to an external API. 

A frequent example is that, to bring company public information, you might need to call a data provider that restricts 500 calls per minute while you want to refresh your full MDM.

As of today, this is not natively possible, but a few workarounds exist.

Main workaround

One of the main workarounds consists of:

  • Creating a function responsible to wait for x seconds between the processing of each record

  • Calling this function in an enricher

This can be achieved by following steps:

  1. Create a PL/SQL function whose purpose is to sleep for x seconds. 
    Here is an example for PostgreSQL database:

create or replace function temporize_call(vNbSec int)

returns void as $$

begin

perform pg_sleep(vNbSec);

end;

$$ language plpgsql;

  1. Declare it in the model’s database functions:

  1. Use it in an enricher: for that, you can choose between those 2 distinct options

    1. Either via concatenating the function to an input field:

    2. Or by creating a REST API enricher solely dedicated to call the sleep function, by:

  • Enriching a dummy field. This enricher should be right before the REST API that needs to be temporized.

  • Activating the PARAM_AGGREGATE_JOB_PLUGIN_ENRICHERS in the integration used:


Please note that this option:
  • Is an untested hypothesis, that remains interesting to share as it might help some users, depending on the business context
  • Requires the “sleep” enricher to be a REST API one, as only successive enrichers of the same type can be aggregated (see documentation for further details).

Additional workarounds

If the main workaround is not applicable in your context, here are some other options that can be implemented to temporize API calls:

  • Increasing the number of max retries: if the execution of the REST client or plug-in fails, it is repeated this number of times. But this might not answer every use cases because if restrictions are applied at some point, Semarchy will retry as many times as mentioned, but not really temporize the calls

  • Using a middleware that would take care of temporizing the calls

  • Creating an in-house REST API that would take care of temporizing and call said third-party REST API at the same time