Binh Codes

Storing and Retrieving Images in S3

Binh-Nguyen Ngo's photo
Binh-Nguyen Ngo
·

4 min read

Black Sony point-and-shoot by C D-X via Unsplash

Intro

As some of you may or may not know, before making this pivot into the web development world, I was a chef at some of the best restaurants in Hawaii. Naturally, I wanted to make a website where chefs far and wide could showcase their signature dishes in the hopes of getting their name out there and gaining clients along the way. But for now, my project’s MVP is a personally curated gallery of dishes from some of Hawaii’s culinary heavyweights. And for that, I need to handle one thing well and efficiently. Image storage.

Here’s my GitHub repo and deployed website if you want to check it out!

Tech Stack

Tech flow

If you tuned into the Stack Overflow Clone series, this tech stack and API flow is similar. S3 and Cloudfront was necessary in that project as well, but this project differs because I need to find an efficient way to store and retrieve images. Having an S3 bucket fronted by Cloudfront is beneficial because it caches files from the bucket in a server near the user, making each subsequent request faster, thus reducing latency.

I looked up possible solutions a few days ago and tried a few methods:

  • Convert the image into a base64 string and store that in DynamoDB then revert the change when you retrieve it. This method was extremely convoluted don’t even bother trying this one.

  • Direct upload to S3. I kept encountering “Access Denied” errors when I tried this method or I was able to send the file to S3, but it was an empty file (0B).

  • I also read articles where they had suggested altering your VTL resolvers and it just seemed way too much work for something as simple as storing images.

After spending hours trying these methods, it made it that much more rewarding when solving this issue with a fraction of the code. Here’s how you do it.

Permissions

First off, you’re going to need to create an IAM Role that has the permissions to read/write from the bucket as well as giving these permissions to authenticated Cognito users. Make sure to record the Access Key and Secret Access Key for your .env file when you initiate the S3 instance. Then, you need to give all users access to read from the bucket.

After you’ve changed these IAM and Bucket policies, you have to enable CORS in the Permissions tab of your S3 bucket. There are a lot of examples online, but you want to enable GET and PUT from all origins just for testing’s sake. You can always restrict it to your domain later on.

Presigned URL

Finally, we got to the meat of the project. The solution that I worked out thankfully was the most efficient way to handle this problem. When we create the Chef entity:

  • We allow the user to enter their information through a form that has an encType = “multipart/form-data”. This is important because the form consists of multiple text fields, but it also requires us to send a file for the user’s profile image.

  • Once the form is validated, our Lambda takes the information and generates a presigned URL. This URL allows the user that has it to either GET or POST an object in the S3 bucket.

presigned url function

  • Therefore, after creating the Chef entity successfully, we generate a unique presigned URL (based on the user's name or unique id) which is then stored in DynamoDB, and immediately retrieved and utilized in a fetch request to POST the image from the form to the S3 bucket.

example fetch request

   const removeParams = (url:string) => {
      return url.split('?')[0];
    }

Well there you have it! You can now store and retrieve user uploaded images in your S3 bucket and store the image URL in your DynamoDB table for easy fetching. Thanks for reading everybody and I hope it helps! :)

DynamoDBlambdaS3GraphQLAppSync