API with NestJS #69. Database migrations with TypeORM

NestJS SQL

This entry is part 69 of 121 in the API with NestJS

When working with relational databases, we define the structure of the data rather strictly. For example, we need to specify the format of every table along with fields, relations, indexes, and other structures. By doing that, we also tell the database how to validate the incoming data.

It is crucial to think about the structure of our database carefully. Even if we do that, the requirements that our application has to meet change. Because of the above, we rarely can avoid having to modify the structure of our database. When doing that, we need to be careful not to lose any existing data.

With database migrations, we can define a set of controlled changes that aim to modify the structure of the data. They can include adding or removing tables, changing columns, or changing the data types, for example. While we could manually run SQL queries that make the necessary adjustments, this is not the optimal approach. Instead, we want our migrations to be easy to repeat across different application environments.

Also, we need to acknowledge that modifying the structure of the database is a delicate process where things can go wrong and damage the existing data. Fortunately, writing database migrations includes committing them to the repository. Therefore, they can undergo a rigorous review before merging to the master branch. In this article, we go through the idea of migrations and learn how to perform them with TypeORM.

Working with migrations using TypeORM

When configuring TypeORM, we can set the property to . This causes TypeORM to synchronize the database with our entities automatically. However, using it in production is highly discouraged because it might lead to unexpected data loss.

Instead, TypeORM has a tool that helps us create and run migrations. Unfortunately, its migration documentation is outdated and does not match the latest version.

Configuring the TypeORM CLI

To start working with migrations using TypeORM, we need to properly configure its command line interface (CLI). To do that, we need to create a designated configuration file.

typeOrm.config.ts

Above, we use dotenv to make sure the ConfigService loaded the environment variables before using it.

We also need to add some entries to the in our .

package.json

Unfortunately, the feature is not supported by yarn.

Generating our first migration

Let’s define a straightforward entity of a post.

post.entity.ts

If you want to know more about identity columns, check out Serial type versus identity columns in PostgreSQL and TypeORM

There is a significant caveat regarding the directory in our configuration. Let’s take a look at our NestJS database configuration.

database.module.ts

The @nestjs/typeorm library implements the that analyzes our NestJS application and identifies all of our entities. Unfortunately, the basic TypeORM configuration can’t do that.

We still need to add the entities we don’t use through to the array.

Because of the above, we need to manually add the to our array in our CLI configuration.

typeOrm.config.ts

We might be able to figure out a better approach if the CLI would support asynchronous DataSource creation. Once the pull request with the improvement is merged, we could create a NestJS application in our file and take advantage of the property.

Once we have all of the above set up, we can use the command to let TypeORM generate the migration file.

Running the migration command creates a file with the code that can bring our database from one state to another and back. Its filename consists of the current timestamp followed by the name provided when using the command.

1658694616973-CreatePost.ts

I used prettier on the generated file.

Above, there are two methods:

  • – performs the migration,
  • – reverts it.

Running the migrations

To run a migration, we must add it to the array in our file. Unfortunately, using strings with the array is deprecated and will stop working in TypeORM 0.4. Because of that, we should import the migration classes manually.

typeOrm.config.ts

Once we have the migration added to the array, we can run the command to execute it.

The above command yields the following logs:

Running the migration command does a few things. First, it identifies that the array contains a migration that wasn’t executed yet. It runs the method and creates the posts table.

Besides that, it also adds an entry to the table in the database. It indicates that the migration was executed.

Reverting migrations

To revert a migration, we need to use the command.

The above command produces the following logs:

Running the command executes the method in the latest performed migration and removes the respective row from the array. Therefore, if we need to revert more than one migration, we must use the command multiple times.

Creating migrations manually

Besides relying on TypeORM to generate the migrations for us, we can write their logic manually. Let’s start by making a slight change to the .

post.entity.ts

If you want to know more about managing dates with PostgreSQL, check out Managing date and time with PostgreSQL and TypeORM

Now, let’s run a command that tells TypeORM to create the basics of the migration for us.

By doing the above, we end up with the following file:

1658701645714-PostCreationDate.ts

I used prettier on the generated file.

When generating migrations, TypeORM uses and provides a raw SQL query. While that’s a viable approach, we can also use the migration API.

1658701645714-PostCreationDate.ts

To run our migration, we also need to add it to the array.

typeOrm.config.ts

Now, we can execute the migration by running the appropriate command.

Running the command gives us the following logs:

We can notice that using the migrations API seems to produce bigger queries when executing the migrations.

If we run our new migration, the table gets a new row:

Summary

In this article, we’ve learned what migrations are and how to manage them with TypeORM. To do that, we’ve set up the TypeORM CLI and used commands for creating, generating, running, and reverting migrations. We’ve also used the migrations API and noticed that it produces rather big SQL queries. All of the above gave us a thorough understanding of how to deal with migrations using TypeOrm.

Series Navigation<< API with NestJS #68. Interacting with the application through REPLAPI with NestJS #70. Defining dynamic modules >>
Subscribe
Notify of
guest
19 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Dave Lowe
Dave Lowe
2 years ago

this is super helpful – thanks so much for taking the time to put it together, i ran into a slight issue running on windows – the $npm_config_ process doesnt seem to work (at all) so i resolved by adding cross-env and then updating the npm scripts as follows

4
Reply
Tri
Tri
2 years ago
Reply to  Dave Lowe

Thanks you, but why cross-env solve that kind of problem, can you provide some info ?

0
Reply
mahsa
mahsa
2 years ago
Reply to  Dave Lowe

    “typeorm:generate-migration”: “npm run typeorm — -d ./typeOrm.config.ts migration:generate ./src/migrations/%npm_config_name%”,

Last edited 2 years ago by mahsa
9
Reply
Aidar
Aidar
1 year ago
Reply to  mahsa

thanks a lot, really appreciate it

0
Reply
TinPham
TinPham
2 years ago

Can i change table name when migration:generate, i pass the name to @Entity decorator but get ‘No changes in database was found….’
Edit: i fixed it, it is because some how the table was created before

Last edited 2 years ago by TinPham
0
Reply
Sote
Sote
2 years ago

how to define command to refresh schema, example: migration:refresh
pls help me! Thanks

0
Reply
Kevin
Kevin
2 years ago

Very good article. Thank you, really helpful.
Instead using real entities class, I use dist entities like below.

const dataSource = new DataSource({
 …,
 entities: [‘dist/**/*.entity{.ts,.js}’],
 migrations: [‘dist/migrations/*{.ts,.js}’],
});

4
Reply
Angel
Angel
1 year ago
Reply to  Kevin

In case dev.
entities: [‘src/**/*.entity{.ts,.js}’], migrations: [‘src/migrations/*{.ts,.js}’],

0
Reply
Alejandro
Alejandro
2 years ago

Awesome post!. I have one question, do you know how to load another services that “Config Service” use?. On my code I have something very similar to you, but with the difference that the Config Service uses a nest-aws service to fetch the database values from secret manager, so on my migrations this dependency is not “available” (or I dont know how to use it). Do you know how to do this?

1
Reply
Danilo Marques
Danilo Marques
2 years ago

Hi there,
First of all, thanks for sharing such a big article(s) about Nestjs.
Just want to say that the pull request you mentioned about “support asynchronous DataSource creation.” is merged.
It would be good to see this on the article.

1
Reply
pontus
pontus
2 years ago

Where in the folder structure should I put typeOrm.config.ts in order to make this work?

0
Reply
Eduardo
Eduardo
1 year ago
Reply to  pontus

Following the article you can put outside of src folder

Last edited 1 year ago by Eduardo
0
Reply
Pdh
Pdh
1 year ago

My env variables come as undefined in the data source. Why is it?

0
Reply
Jim Bone
Jim Bone
1 year ago

Thanks for the tutorial. However I don’t want the typeOrm.config.ts to be part of the dist/ directory created by the compilation. I’m unable to tell the compiler to ignore this file for running NestJs. Any ideas?

0
Reply
David
David
1 year ago

Thanks a lot!

I have finally solved (npm and mac, keeps absolute path, setup entities and migrations) as follows:

add to the scripts in package.json :
“typeorm”: “ts-node -r tsconfig-paths/register ./node_modules/typeorm/cli -d orm.config.ts”,
“migrations-generate”: “npm run typeorm migration:generate src/database/migrations/$npm_config_name”,
“migrations:run”: “npm run typeorm migration:run”,
“migrations:show”: “npm run typeorm migration:show”,

//* Load enviroment variables
config({ path: envPath, debug: true });
const configService = new ConfigService();

const cnn = new DataSource({
type: ‘postgres’,
host: configService.get(‘POSTGRES_HOST’),
port: configService.get(‘POSTGRES_PORT’),
username: configService.get(‘POSTGRES_USER’),
password: configService.get(‘POSTGRES_PASSWORD’),
database: configService.get(‘POSTGRES_DB’),
logging: configService.get(‘POSTGRES_DEBUG_QUERIES’) === 1 ? true : false,
entities: [ </span><span style="color: rgb(86, 156, 214);">${</span><span style="color: rgb(156, 220, 254);">__dirname</span><span style="color: rgb(86, 156, 214);">}</span><span style="color: rgb(206, 145, 120);">/src/entities/**/*{entity.js,entity.ts} ],
migrations: [ </span><span style="color: rgb(86, 156, 214);">${</span><span style="color: rgb(156, 220, 254);">__dirname</span><span style="color: rgb(86, 156, 214);">}</span><span style="color: rgb(206, 145, 120);">/src/database/migrations/**/*{.js,.ts} ],
migrationsTableName: “_migrations”
});

cnn.initialize().then(() => {
console.log(‘cnn initialized’);
}).catch((err) => {
console.log(‘cnn error’, err);
});

export default cnn;

PD :

  • npm install tsconfig-paths –save-dev
  • “entity.ts and entity.js” my entities are named like that.
  • -r tsconfig-paths/register : allows using absolute path
0
Reply
David
David
1 year ago

I have finally solved (npm and mac, keeps absolute path, setup entities and migrations) as follows:

add to the scripts in package.json :

Create the orm.config.ts file (in the root in these case) and include something like this:

PD :

  • npm install tsconfig-paths –save-dev
  • “entity.ts and entity.js” my entities are named like that.
  • -r tsconfig-paths/register : allows using absolute path
0
Reply
Orhan
Orhan
1 year ago

Hello,

I keep getting the following:

‘No changes in database schema were found – cannot generate a migration. To create a new empty migration use “typeorm migration:create” command’

What could be the issue?

2
Reply
Dhanunjaya
Dhanunjaya
1 year ago
Reply to  Orhan

Your entities are not loaded proeprly

0
Reply
Mark
Mark
1 year ago

I am facing such error in the end:

Why so?

0
Reply