How to Setup Any User Put to S3 and Private Get…
It is not possible to have an anonymous Put to S3 and a private Get. Turns out the anonymous user owns the Object. Therefor any anonymous user (which is everyone) can read the file. That’s bad. If you figure out a way for anonymous puts with private gets, please leave a comment.
I was able to figure out a slightly different mechanism that works almost as well as a PUT/GET. Oh – I don’t want to do an authenticated Put as I don’t want to have to hit the application server for every file piece (I am uploading in multiple chunks as I believe the browser will timeout the http connection – need to figure out the timeout limits though) and I don’t want to have to include the S3 secret keys in the client code (both options don’t appeal as a solution for this use case).
For background, I am creating a file upload utility for a client. They want to provide the ability for their clients to upload 700MB – 1.4GB data files with on eye towards even larger files. I want to make sure the client has the best experience possible, I realize large files for some clients can be a burden given the current state of upload bandwidth for most people. However, upload speeds are rapidly changing. At 2Mbps a 1.4GB file takes….1 hour 40 minutes. Now on my personal puny link of 256Kbps it takes 13 hours for the upload (here is a handy calculator). So that the receiving bandwidth is not an issue and can support multiple clients concurrently, the upload is being outsourced to S3 for now.
Anyway, I have to create a mechanism to get the files to S3. There are lots of moving parts for the application, one of the trickiest was the actual upload connection. I tried several techniques. Turns out amazon has a Form Post protocol that can be utilized. The important part is to create a “policy” in S3 parlance that is signed. The policy contains some specifics about the upload and an expiration.
Once I realized what to do it took a little playing around to create the correct html form post (I’m actually mimic it; but it looks the same on the receiving end). There is also a great utility to help create the policy – actually it was a big time saver for the prototype. Check it out on the amazon site: Policy Creator. It’s not pretty; but works really well.
The docs for the Post Protocol are in the Amazon Documentation section. Check them out and read carefully.
Chris….
