feature: handle 429 error code #37

Activity

1. pitchmuc

   added

   enhancementNew feature or requestNew feature or request

   questionFurther information is requestedFurther information is requested

   on Aug 14, 2025
2. pitchmuc commented on Aug 14, 2025

   

   pitchmuc

   on Aug 14, 2025

   Contributor

   More actions

   Hello @benedikt-buchert,
   What do you mean by the rate limit of the API ?
   I have used the AEP API quite intensively and I have never reached the limit. Have you faced any specific limit for some operations ?
3. benedikt-buchert commented on Aug 15, 2025

   

   benedikt-buchert

   on Aug 15, 2025

   Last edited by benedikt-buchert

   ContributorAuthor

   More actions

   This is the script I executed>

   def send_deletion_requests(chunks):
       for idx, chunk in enumerate(chunks):
           # Format identities for deletion request
           identities = [
               {"namespace": {"code": row["namespaceCode"]}, "id": row["value"]}
               for _, row in chunk.iterrows()
           ]
   
           # Create deletion request
           job_name = f"deletion_job_chunk_{idx + 1}"
           try:
               response = myHygieneSandbox.createRecordDeleteRequest(
                   name=job_name, identities=identities
               )
               log.info(f"Successfully sent deletion request for chunk {idx + 1}")
               log.debug(response)
           except Exception as e:
               log.error(
                   f"{e} Failed to send deletion request for chunk {idx + 1}",
                   exc_info=True,
               )
   
   def split_and_send_deletions(file_path, chunk_size=10000):
       try:
           # Read the original CSV file into a DataFrame
           df = pd.read_csv(file_path)
           if "value" in df.columns:
               df["value"] = df["value"].astype(str)
           log.info(f"Read CSV file: {file_path}")
       except Exception as e:
           log.error(f"{e} Failed to read CSV file: {file_path}", exc_info=True)
           return
   
       # Get the number of chunks
       num_chunks = len(df) // chunk_size + (1 if len(df) % chunk_size != 0 else 0)
       chunks = []
   
       # Split the DataFrame into chunks
       for i in range(num_chunks):
           # Get the start and end index for the current chunk
           start_idx = i * chunk_size
           end_idx = start_idx + chunk_size
   
           # Get the chunk
           chunk = df.iloc[start_idx:end_idx]
           chunks.append(chunk)
   
           log.info(f"Prepared chunk {i + 1} of {num_chunks}")
   
       # Send deletion requests for each chunk
       send_deletion_requests(chunks)

   I think the main issue is that I'm sending a bunch of requests quickly after each other.

   https://developer.adobe.com/experience-platform-apis/references/data-hygiene/#operation/createWorkOrder
   Seems to be related to this: https://experienceleague.adobe.com/en/docs/experience-platform/data-lifecycle/best-practices#manage-429-errors

   Should have set the chunk size to 100.000... But still might have ran into the issue...
4. benedikt-buchert

   changed the title [-]feature: Respect Rate limit[/-] [+]feature: handle 429 error code[/+] on Aug 15, 2025
5. pitchmuc commented on Sep 2, 2025

   

   pitchmuc

   on Sep 2, 2025

   Contributor

   More actions

   Yeah, I think handling them in 100K chunk would have been better as it is the number of records supported for each Payload.

   Overall, it is a complex topics as I am not aware if the issue is going to be the same for all endpoints, so it could be block.
   I am actually thinking to move to httpx to provide capability for async call...
   This makes it even more complex

   Honestly not sure if I want/can tackle that natively, I will let it open if someone see it and can add more