{"id":569,"date":"2013-09-09T07:20:21","date_gmt":"2013-09-09T07:20:21","guid":{"rendered":"https:\/\/virtualchaos.co.uk\/blog\/?p=569"},"modified":"2013-09-09T07:20:21","modified_gmt":"2013-09-09T07:20:21","slug":"copy-files-from-one-s3-bucket-to-another","status":"publish","type":"post","link":"https:\/\/virtualchaos.co.uk\/blog\/2013\/09\/09\/copy-files-from-one-s3-bucket-to-another\/","title":{"rendered":"Copy files from one S3 bucket to another"},"content":{"rendered":"<p>I have an S3 bucket that contains several hundred files in a folder. I needed to copy those files into a different folder in another bucket. Sounds simple enough? but was unable to find a simple way to do this through the AWS Console. I found a number of stack overflow articles that talked about using Sync, or downloading the files and re-uploading them. None of which sounded particularly appealing.<\/p>\n<p>In the end I just wrote this bash one liner (which I can probably optimise further by not repeating the sourcebucket \/ sourcefolder three times):<\/p>\n<p><script src=\"https:\/\/gist.github.com\/kiyanwang\/6492329.js\"><\/script><\/p>\n<p>This just uses s3cmd to list all the files in the bucket\/folder I wish to copy from. The output of that is piped to awk which I use to extract the s3 url of each file. I then use tail to remove the first line which I don&#8217;t need. I then use sed to build up a &#8216;s3cmd cp&#8217; command which copies the file from its original location to my new location.<\/p>\n<p>If anyone can suggest a better way that doesnt require me having to download the source files &#8230; I&#8217;d love to hear it.<br \/>\nIf you can&#8217;t see the embedded Gist above then you can view it <a href=\"https:\/\/gist.github.com\/kiyanwang\/6492329\" title=\"Gist Copying all files from one s3 bucket to another\">here<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I have an S3 bucket that contains several hundred files in a folder. I needed to copy those files into a different folder in another bucket. Sounds simple enough? but was unable to find a simple way to do this &hellip; <a href=\"https:\/\/virtualchaos.co.uk\/blog\/2013\/09\/09\/copy-files-from-one-s3-bucket-to-another\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[24],"tags":[380,272,381],"class_list":["post-569","post","type-post","status-publish","format-standard","hentry","category-development","tag-aws","tag-bash-one-liner","tag-s3"],"_links":{"self":[{"href":"https:\/\/virtualchaos.co.uk\/blog\/wp-json\/wp\/v2\/posts\/569","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/virtualchaos.co.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/virtualchaos.co.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/virtualchaos.co.uk\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/virtualchaos.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=569"}],"version-history":[{"count":0,"href":"https:\/\/virtualchaos.co.uk\/blog\/wp-json\/wp\/v2\/posts\/569\/revisions"}],"wp:attachment":[{"href":"https:\/\/virtualchaos.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=569"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/virtualchaos.co.uk\/blog\/wp-json\/wp\/v2\/categories?post=569"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/virtualchaos.co.uk\/blog\/wp-json\/wp\/v2\/tags?post=569"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}