Tarantool is an in-memory database, that happens to have a general purpose programming language onboard. While some databases like MySQL or Postgres allow you to write stored procedures, they are nowhere near as powerful as, say, Python. And even if they have support for external languages, they always feel like second-class citizens because usually they were bolted on after these products have matured. In tarantool, Lua has been added very early, and now covers the entirety of the database API.
It was 2018 and we started building many small and large apps in Tarantool. They ranged from caches that pulled their data from relational databases with custom expiry strategies, to larger systems that combined stream processing, graphql, user-extensible logic, data access policies, etc.
Things went well in the beginning, but I quickly noticed that every team had a different way in which their app was laid out in source control. They had different sets of scripts to launch the app, test harnesses that started the database and loaded code, dependency management for both development and production, and deployment scripts. Well, you can easily find the same examples in the Python world if you browse GitHub repos with web apps.
What I figured out was the root cause of this drift, is that there was no standard way to pass configuration parameters that would be portable and universal across different types of deployment: for Docker, local development and plain old system packages. As every person reinvented their own way to do it, the chances of one team’s tools working with other team’s project diminished.
To solve this problem I borrowed ideas from the Heroku 12-factor app methodology. Namely:
- Every Tarantool app is a regular unix process, that is self-sufficient
- No specific application startup order is needed, even for replicated instances
- Instance configuration can be passed through environment variables starting with TARANTOOL_, or through a simple plaintext configuration file. Both environment variables and config file have feature parity.
- All dependencies are managed in one standard file committed to the root of the app repository
- Do not rely on the presence of system-wide packages
- In fact, don’t rely on the specifics of host system as much as possible
- Don’t require any external processes in order to function properly
In addition to problems with application startup and configuration, I really wanted to solve one problem that bugged me a lot when working with other platforms like Python and Ruby: packaging the app for your Linux distro of choice requires getting really familiar with the ins and outs of specific packaging systems, even if your app doesn’t deviate much from the “typical” app for the language.
The exact programmatic solution for those problems is 2-fold: firstly, there is a tool called cartridge-cli that allows you to create a project from template, that contains everything set up just right. It has dependency management, application initialization, unit and integration test stubs, etc. And it allows you to create a distributable rpm, deb or Docker container with your app – all with a single command. Secondly, there is a framework called Cartridge that enforces how applications are started and configured. It hides some of the complexity that every production-level Tarantool app accumulates, and makes sure that every app can interoperate with our Ops tools.
The important distinction between Cartridge and application frameworks like Rails, Django and Flask, is that it doesn’t enforce any restrictions on how you write business logic. This was a deliberate choice, because standardization of business logic structure has significantly less impact on our delivery, than standardized management and operations.
The exact moment I realized we achieved the goal is when it became possible to create an end-to-end getting started guide that covered everything from project bootstrap to launching it and setting up a sharded cluster.