Avoid creating accidental duplicates when using the Marketo REST API
In a recent client project, we discovered duplicate records being created by a third-party system integrated via Marketo’s REST API. These duplicate records had the same email address.
While Marketo does de-duplicate leads (based on email addresses) from web forms and list imports, it does not necessarily de-dupe leads coming in via the REST API. It depends which REST operation is used.
Sending a POST request to the /rest/v1/leads.json endpoint, there are three different actions that can be sent in the syncLeadRequest parameter to create lead records: createOnly, createDuplicate or createOrUpdate.
Most developers will probably want to use createOrUpdate in their integrations. This is a so-called “upsert” operation, as it can either update or insert a record. It first checks Marketo to see whether a lead exists. If it does exist, the query updates the existing record. If it does not exist, the query inserts a new record to the Marketo database.
The other two operations — createOnly and createDuplicate — will insert a new record each time they are called, with the exception that createOnly will fail if a matching record is found in the defined lookupField, which defaults to email if not set.
Here is some documentation on this functionality:
The client’s implementation correctly used the createOrUpdate operation, and the specified lookupField was correctly set to email. Despite the check-for-duplicates nature of this operation, however, duplicates were being created nonetheless, which left the developer scratching his head.
Here’s the gotcha: it is actually possible in some cases for createOrUpdate to create a duplicate record!
This note below from the Marketo docs explains (emphasis added):
When performing lead upsert requests concurrently or in quick succession, duplicate records may result when making multiple requests with the same key value if a subsequent call with the same value is made before the first returns. This can be avoided either by using the createOnly, or updateOnly as appropriate, or by queuing calls and waiting for your call to return before making subsequent upsert calls with the same key.
So, basically, hold your horses. Slow your app down. Throttle the API calls in your app so that they are not sent so quickly that Marketo cannot respond before subsequent requests come in. Happy coding!