Last week when I check on my AWS S3 usage and found out that there are a lot of old and small log file which store in the date format folder name, I try to delete the file using the AWS S3 console and due to the number of unique file, I may need to spend a lot of time to delete it. So I try to look into the CLI way to try to remove it, but found out that the S3 CLI command only allow to delete the bucket or only a single object in the bucket which not what I looking for.
Manage to Google it for few minutes and the shell script looping come into my mind and help me up to clean in my old log file which about 10++GB in just few minutes. The following is the sample code that I use to delete my file.
for m in 01 02 03 04 05 06 07 08 09 10 11 12 do for i in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 27 28 29 30 31; do aws s3 rm s3://[bucketname]/2017$m$i --recursive; done done
After the deletion done, I plan to setup the lifecycle for my S3 bucket so that I can automate the house keep process without any manual work needed in future.
This may not be the best and optimise solution but it’s the solution that suddenly come into my mind. If you have any better suggestion or solution on this, please share with me so that I can make use of a better and smarter solution if I come into this use case again.