API with NestJS #10. Uploading public files to Amazon S3

August 3, 2020

This entry is part 10 of 121 in the API with NestJS

While storing files directly in the database is doable, it might not be the best approach. Files can take a lot of space, and it might impact the performance of the application. Also, it increases the size of the database and, therefore, makes backups bigger and slower. A good alternative is storing files separately using an external provider, such as Google Cloud, Azure, or Amazon AWS.

In this article, we look into uploading files to Amazon Simple Storage Service, also referred to as S3. You can find all the code from this series in this repository.

Connecting to Amazon S3

Amazon S3 provides storage that we can use with any type of file. We organize files into buckets and manage them in our API through an SDK.

Once we create the AWS account, we can log in as a root user. Even though we might authorize as root to use the S3 through our API, this is not the best approach.

Setting up a user

Let’s create a new user that has a restricted set of permissions. To do so, we need to open the Identity and Access Management (IAM) panel and create a user:

Amazon S3 interface

Since we want this user to be able to manage everything connected to S3, let’s set up proper access.

Amazon S3 interface

After doing that, we are presented with Access key ID and Secret access key. We need those to connect to AWS through our API. We also need to choose one of the available regions.

Let’s add them to our .env file:

.env

# ...

AWS_REGION=eu-central-1

AWS_ACCESS_KEY_ID=*******

AWS_SECRET_ACCESS_KEY=*******

Also, let’s add it to our environment variables validation schema in AppModule:

ConfigModule.forRoot({

validationSchema: Joi.object({

POSTGRES_HOST: Joi.string().required(),

POSTGRES_PORT: Joi.number().required(),

POSTGRES_USER: Joi.string().required(),

POSTGRES_PASSWORD: Joi.string().required(),

POSTGRES_DB: Joi.string().required(),

JWT_SECRET: Joi.string().required(),

JWT_EXPIRATION_TIME: Joi.string().required(),

AWS_REGION: Joi.string().required(),

AWS_ACCESS_KEY_ID: Joi.string().required(),

AWS_SECRET_ACCESS_KEY: Joi.string().required(),

PORT: Joi.number(),

})

}),

Connecting to AWS through SDK

Once we have the necessary variables, we can connect to AWS using the official SDK for Node. Let’s install it first.

1	npm install aws-sdk @types/aws-sdk

Since we’ve got everything that we need to configure the SDK, let’s use it. One of the ways to do so is to use aws-sdk straight in our main.ts file.

main.ts

import { NestFactory } from '@nestjs/core';

import { AppModule } from './app.module';

import * as cookieParser from 'cookie-parser';

import { ValidationPipe } from '@nestjs/common';

import { ExcludeNullInterceptor } from './utils/excludeNull.interceptor';

import { ConfigService } from '@nestjs/config';

import { config } from 'aws-sdk';

async function bootstrap() {

const app = await NestFactory.create(AppModule);

app.useGlobalPipes(new ValidationPipe());

app.useGlobalInterceptors(new ExcludeNullInterceptor());

app.use(cookieParser());

const configService = app.get(ConfigService);

config.update({

accessKeyId: configService.get('AWS_ACCESS_KEY_ID'),

secretAccessKey: configService.get('AWS_SECRET_ACCESS_KEY'),

region: configService.get('AWS_REGION'),

});

await app.listen(3000);

}

bootstrap();

Creating our first bucket

In Amazon S3 data is organized in buckets. We can have multiple buckets with different settings.

Let’s open the Amazon S3 panel and create a bucket. Please note that the name of the bucket must be unique.

Amazon S3 interface

We can set up our bucket to contain public files. All files that we upload to this bucket will be publicly available. We might use it to manage files such as avatars.

Amazon S3 interface

The last step here is to add the name of the bucket to our environment variables.

.env

1 2	# ... AWS_PUBLIC_BUCKET_NAME=nestjs-series-public-bucket

src/app.module.ts

ConfigModule.forRoot({

validationSchema: Joi.object({

// ...

AWS_PUBLIC_BUCKET_NAME: Joi.string().required(),

})

}),

Uploading images through our API

Since we’ve got the AWS connection set up, we can proceed with uploading our files. For starters, let’s create a PublicFile entity.

src/files/publicFile.entity.ts

import { Column, Entity, PrimaryGeneratedColumn } from 'typeorm';

@Entity()

class PublicFile {

@PrimaryGeneratedColumn()

public id: number;

@Column()

public url: string;

@Column()

public key: string;

}

export default PublicFile;

By saving the URL directly in the database, we can access the public file very quickly. The key property uniquely identifies the file in the bucket. We need it to access the file, for example, if we want to delete it.

The next step is creating a service that uploads files to the bucket and saves the data about the file to our Postgres database. Since we want keys to be unique, we use the uuid library:

1	npm install uuid @types/uuid

src/files/files.service.ts

import { Injectable } from '@nestjs/common';

import { InjectRepository } from '@nestjs/typeorm';

import { Repository } from 'typeorm';

import PublicFile from './publicFile.entity';

import { S3 } from 'aws-sdk';

import { ConfigService } from '@nestjs/config';

import { v4 as uuid } from 'uuid';

@Injectable()

export class FilesService {

constructor(

@InjectRepository(PublicFile)

private publicFilesRepository: Repository<PublicFile>,

private readonly configService: ConfigService

) {}

async uploadPublicFile(dataBuffer: Buffer, filename: string) {

const s3 = new S3();

const uploadResult = await s3.upload({

Bucket: this.configService.get('AWS_PUBLIC_BUCKET_NAME'),

Body: dataBuffer,

Key: `${uuid()}-${filename}`

})

.promise();

const newFile = this.publicFilesRepository.create({

key: uploadResult.Key,

url: uploadResult.Location

});

await this.publicFilesRepository.save(newFile);

return newFile;

}

The uploadPublicFile method expects a buffer. It is a chunk of memory that keeps a binary representation of our file. If you want to know more about it, check out Node.js TypeScript #3. Explaining the Buffer.

Creating an endpoint for uploading files

Now we need to create an endpoint for the user to upload the avatar. To link the files with users, we need to modify the UserEntity by adding the avatar column.

src/users/user.entity.ts

import { Entity, JoinColumn, OneToOne } from 'typeorm';

import PublicFile from '../files/publicFile.entity';

@Entity()

class User {

// ...

@JoinColumn()

@OneToOne(

() => PublicFile,

{

eager: true,

nullable: true

}

)

public avatar?: PublicFile;

}

export default User;

If you wan to know more about relationships with Postgres and TypeORM, check out API with NestJS #7. Creating relationships with Postgres and TypeORM

Let’s add a method to the UsersService that uploads files and links them to the user.

src/users/users.service.ts

import { Injectable } from '@nestjs/common';

import { InjectRepository } from '@nestjs/typeorm';

import { Repository } from 'typeorm';

import User from './user.entity';

import { FilesService } from '../files/files.service';

@Injectable()

export class UsersService {

constructor(

@InjectRepository(User)

private usersRepository: Repository<User>,

private readonly filesService: FilesService

) {}

// ...

async getById(id: number) {

const user = await this.usersRepository.findOne({ id });

if (user) {

return user;

}

throw new HttpException('User with this id does not exist', HttpStatus.NOT_FOUND);

}

async addAvatar(userId: number, imageBuffer: Buffer, filename: string) {

const avatar = await this.filesService.uploadPublicFile(imageBuffer, filename);

const user = await this.getById(userId);

await this.usersRepository.update(userId, {

...user,

avatar

});

return avatar;

}

This might be a fitting place to include some functionalities like checking the size of the image or compressing it.

The last piece is adding an endpoint the users can send avatars to. To do that, we follow the NestJS documentation and use the FileInterceptor that utilizes multer under the hood.

src/users/users.controller.ts

import { UsersService } from './users.service';

import { Controller, Post, Req, UploadedFile, UseGuards, UseInterceptors } from '@nestjs/common';

import JwtAuthenticationGuard from '../authentication/jwt-authentication.guard';

import RequestWithUser from '../authentication/requestWithUser.interface';

import { FileInterceptor } from '@nestjs/platform-express';

import { Express } from 'express';

@Controller('users')

export class UsersController {

constructor(

private readonly usersService: UsersService,

) {}

@Post('avatar')

@UseGuards(JwtAuthenticationGuard)

@UseInterceptors(FileInterceptor('file'))

async addAvatar(@Req() request: RequestWithUser, @UploadedFile() file: Express.Multer.File) {

return this.usersService.addAvatar(request.user.id, file.buffer, file.originalname);

}

The file above has quite a few useful properties such as the mimetype. You can use it if you want some additional validation and disallow certain types of files.

Postman interface

Deleting existing files

Aside from uploading files, we also need a way to remove them. To keep our database consistent with the Amazon S3 storage, we remove the files from both places. First, let’s add the method to the FilesService.

src/files/files.service.ts

import { Injectable } from '@nestjs/common';

import { InjectRepository } from '@nestjs/typeorm';

import { Repository } from 'typeorm';

import PublicFile from './publicFile.entity';

import { S3 } from 'aws-sdk';

import { ConfigService } from '@nestjs/config';

@Injectable()

export class FilesService {

constructor(

@InjectRepository(PublicFile)

private publicFilesRepository: Repository<PublicFile>,

private readonly configService: ConfigService

) {}

// ...

async deletePublicFile(fileId: number) {

const file = await this.publicFilesRepository.findOne({ id: fileId });

const s3 = new S3();

await s3.deleteObject({

Bucket: this.configService.get('AWS_PUBLIC_BUCKET_NAME'),

Key: file.key,

}).promise();

await this.publicFilesRepository.delete(fileId);

}

Now, we need to use it in our UsersService. Important addition is that when a user uploads an avatar while already having one, we delete the old one.

src/users/users.service.ts

import { Injectable } from '@nestjs/common';

import { InjectRepository } from '@nestjs/typeorm';

import { Repository } from 'typeorm';

import User from './user.entity';

import { FilesService } from '../files/files.service';

@Injectable()

export class UsersService {

constructor(

@InjectRepository(User)

private usersRepository: Repository<User>,

private readonly filesService: FilesService

) {}

// ...

async addAvatar(userId: number, imageBuffer: Buffer, filename: string) {

const user = await this.getById(userId);

if (user.avatar) {

await this.usersRepository.update(userId, {

...user,

avatar: null

});

await this.filesService.deletePublicFile(user.avatar.id);

}

const avatar = await this.filesService.uploadPublicFile(imageBuffer, filename);

await this.usersRepository.update(userId, {

...user,

avatar

});

return avatar;

}

async deleteAvatar(userId: number) {

const user = await this.getById(userId);

const fileId = user.avatar?.id;

if (fileId) {

await this.usersRepository.update(userId, {

...user,

avatar: null

});

await this.filesService.deletePublicFile(fileId)

}

Summary

In this article, we’ve learned the basics of how Amazon S3 works and how to use it in our API. To do that, we’ve provided the necessary credentials to AWS SDK. Thanks to that, we were able to upload and delete files to AWS. We’ve also kept our database in sync with Amazon S3, to track our files. To upload files through API we’ve used the FileInterceptor, which uses Multer under the hood.

Since there is more to Amazon S3 than handling public files, there is still quite a bit to cover here, and you might expect it in this series.

Series Navigation<< API with NestJS #9. Testing services and controllers with integration testsAPI with NestJS #11. Managing private files with Amazon S3 >>

9 Comments

Oldest

Newest Most Voted

Inline Feedbacks

View all comments

psz

3 years ago

does this solution upload files directly to s3 from frontend? or multer saves the file on disk first and then it’s transfered to s3?

wsdydeni

Reply to psz

upload files directly to s3 from frontend

-3

SpaceCowboy74

I made a slight adjustment for code reuse.

I refactored deleteAvatar to be a wrapper to deleteUserAvatar. The second function takes a user object. deleteAvatar now looks up a user and passes it to deleteUserAvatar and addAvatar now passes a user object to deleteUserAvatar. It wasn’t much, but was some code duplication in the existing functionality.

await this.deleteUserAvatar(user);

await this.userRepository.update(userId, {

async deleteUserAvatar(user: User) {

await this.userRepository.update( user.id, {

await this.filesService.deletePublicFile(fileId);

Anton

cant update user have this errors:
No entity column “posts” was found
It happens because

1	const user = await this.usersRepository.findOne({ id });

return user with this properties:
{
…
posts: undefined
}
what is wrong ?

Cybertech

Reply to Anton

info is not complete to help u sort, where is id coming from? are u sure it’s not meant to be await this.usersRepository.findOne({ id: userId });

I figure out the main reason of it, it is because I had typescript config esnext but in tutorial it is es2019, so how can I avoid this error? should I delete all undefined properties before update ?

David

Don’t forget to specify the suitable access control list (ACL), also the folder if you want to organize the files.

const extension = filename.split('.').reverse()[ 0 ];

ACL: 'public-read',

Bucket: bucket,

Key: <code>${folder}/${Gen_v4()}.${extension}</code>,

ContentType: contentType,

Last edited 3 years ago by David

Bhargav Gohil

2 years ago

There is an UPDATE in S3. After creating a bucket,

Go to Permissions > Object Ownership > select ACL’s enabled > and select Bucket owner preferred

Otherwise you will get an error for ACL related.

Last edited 2 years ago by Bhargav Gohil

helios

1 year ago

I’m getting this error when trying to run the application:

1	UnhandledPromiseRejectionWarning: AccessDenied: Access Denied