Blog: Classier Twitter threads - Tag Performance

Twice as fast deployments from GitLab pipelines

All CI pipelines would finish in a few minutes in an ideal world, but sadly we don’t live in a perfect world. Sometimes your pipelines take 15 minutes or more, and reducing that time would be pretty hard. That alone can be rather unpleasant, but the real problem begins if you’re trying to ensure that you verify all your commits correctly. First, you have to wait for the CI to pass on a rebased merge request, then you merge the merge request, then you wait for the CI to pass on the merge commit in main, and only then a conditional job that runs only on main would deploy the application.

If you sum up all that waiting, you’ll realize that a pipeline that takes 15 minutes means a full CI cycle of a bugfix will take you at least 30 minutes to deploy - potentially 30 minutes of downtime on production.

In this article, I’ll show you one of the possible optimizations you can do to cut that time in half without losing any safety. A significant benefit of this approach is that you can implement it without re-architecting your workflow and CI from the ground up because it’s only an incremental improvement.

Continue reading ...

Optimizing PostgreSQL queries with Multicolumn and Partial Indexes

I have an application that does asynchronous data processing, and at the core of the application are simulated queues in a PostgreSQL table. Each row in that queue represents a task and also contains the result of that task. You can imagine this table as a sort of multi-tenant where the rows belong to a data_source and queue. There are multiple DataSources, and each can have multiple queues. Some of the combinations contain very few rows, and some of them contain several million.

This uneven distribution of rows caused that while some of the queues can be queried rather quickly, the largest queue has slowly grown in size to the point where the job iterating over it took around 9 hours.

Continue reading ...