Rust: How to consume REST API and persist results in Postgres
As enterprise developers, we often find ourselves solving similar problems from one project to another. Undoubtedly, working with databases is one of those. In this tutorial I will show you a way to consume a paginated REST API reqwest running in tokio runtume and persist it into Postgres. Despite Rust being a relatively new language there are already plenty of crates for database handling to choose from. Here we will work with sqlx.
Prerequisites
I’m using Rust version 1.66.0. This tutorial assumes prior knowledge of Rust and concentrates more on practical usage of the language. Do check the Background section of my previous tutorial to better understand the API we are working with. But in short it is ImmutableX API with the help of which we can fetch all the mint events that ever happened on the blockchain.
As always, project sources are available at GitHub, find the link at the end of the page.
There are 3 parts in this tutorial: Postgres and environment setup, API reading, and Working with Postgres.
Part 1. Postgres and environment setup
This section is not required for the Rust coding part of this tutorial. Nevertheless, for the sake of completeness I wanted to also show you how to set up your local (and potentially production) environment. But if you are only interested in the language — feel free to skip to the next chapter.
In order to have a hassle-free and reproducible local environment we will be using docker-compose:
On top of that we will have two bash scripts to simplify the process further:
Now you can run the start-local-environment.sh script and it will fetch the container and start it:
To finalize the environment setup we have an important decision to make — how to migrate the database? There are plenty of options available for this task and the ultimate choice is yours. If you prefer to stick to the tools written in Rust I can suggest refinery, but here I’m going with flyway due to it’s out-of-the-box docker-compose support:
Note! As you can see all the DB credentials are intentionally hardcoded. Since it is your local development environment (or maybe CI pipeline), it is considered okay to have it this way. Obviously, for production infrastructure code you should be using a safe place to store those credentials.
The last bit here is to create the migrations folder where all the future migration scripts will reside:
Local environment is now complete!
Part 2. API reading
As mentioned previously, in order to read the mints API the reqwest crate will be used. What differs in this tutorial from the previous one is that in this case we will be reading it in a paginated way. First of all we should create a model:
We will only process the fields that are later used, so not all the fields that API exposes. Next, let’s create a module responsible for fetching the data. In order to make use of response pagination the cursor field will be used. This field is returned in the response and its value points to the previous page if there is one, otherwise it is an empty string.
In some other languages there is a do..while kind of loop which is idiomatically equivalent to
loop {
call_function();
if condition {
break;
}
}which was implemented with the help of the Option type.
Once again I’d like to emphasize the importance of proper error handling. You should limit the usage of the unwrap function to either test/prototyping or when not having the expected result is the “valid” reason to crash the application — although here I would still recommend using expect to specify an exact error. By following this your app will run reliably and most likely panic-free.
If you execute the app now it will expectedly produce a bunch of logs. It is time to move to the persistence layer of the application.
Part 3. Working with Postgres
We already know the model to persist — mint. Thus we can create a migration script for it:
Restart the environment with the bash scripts created previously, and you should see two tables created:
Now let’s create a module responsible for dealing with the database — db_handler.rs. I’ve chosen sqlx because I prefer to have full control over SQL scripts rather than offloading it to an ORM. From the previous section we know that API is consumed in a loop, thus we should open_connection to the database before the loop begins, save_mints and close_connection once all the mints are saved. With all of that we can already define those three functions to finish the mints_reader.rs:
Next, we will implement the missing functions — starting with connection management. In order to connect to the database the expected credentials should be provided and there are various ways to do it. In this tutorial we will use the dotenvy crate which relies on the .env file being present and filled with the expected values. Following sqlx official documentation the implementation will be as following:
Lastly, let’s have the code to save the mints data to the DB. Each API response returns 200 values and we could insert them one by one. But luckily, we don’t have to since sqlx has a very handy QueryBuilder::push_values method that generates a bulk insert query:
For simplicity’s sake all potential errors on insert are handled the same way. In case you need to distinguish errors from one another the returned error codes should be matched accordingly. To learn further — check sqlx official documentation about DatabaseErrors.
Afterwards
Knowing how to work with databases is an essential skill each developer should have in their toolset. With an abundance of crates, Rust offers a great variety of ways to achieve this. The choice is yours.
All of the code can be found at this GitHub repository.
For more Rust tutorials be sure to check Rust: How to consume REST API and Rust: How to create Telegram bot.
Special thanks to my dearest friend and comrade Andrei Kochemirovskii for code review and valuable inputs.
Support
If you like the content you read and want to support the author — thank you very much!
Here is my Ethereum wallet for tips:
0xB34C2BcE674104a7ca1ECEbF76d21fE1099132F0
