I previously wrote a python lambda function that copies AWS RDS snapshots to a different region. This has been working for months but recently started throwing this obfuscated error:
An error occurred (DBSnapshotAlreadyExists) when calling the CopyDBSnapshot operation: Cannot create the snapshot because a snapshot with the identifier copy-snap-of-TIMESTAMP-DB_NAME already exists.
Thinking this might be due to some timestamp shenanigans, I looked at the Cloudwatch Events trigger for the lambda and saw that there were two triggers instead of the original one that I setup. Both were scheduled for the same time. I deleted the new one and waited until the next day to see if the error re-occurred, which it did.
Looking through the Cloudwatch errors, even though the second trigger was gone, the lambda was still trying to execute twice. I’ve filed a support ticket with AWS, but in the meantime, needed to silence the false positives to keep people from getting paged.
The error handling had been done as:
except botocore.exceptions.ClientError as e: raise Exception("Could not issue copy command: %s" % e)
Initially, I tried:
except botocore.exceptions.ClientError as e: if 'DBSnapshotAlreadyExists' in e: pass else: raise Exception("Could not issue copy command: %s" % e)
Instead, I had to do:
except botocore.exceptions.ClientError as e: if e.response['Error']['Code'] == 'DBSnapshotAlreadyExists': pass else: raise Exception("Could not issue copy command: %s" % e)