We’re developing a software product whose production environments run entirely on AWS. Keeping local development as close as possible to those environments is important to us: It reduces surprises when deploying, shortens the feedback loop when debugging infrastructure-related issues, and means developers can work confidently without needing access to a shared cloud account. To achieve…
Read MoreThere’s a particular kind of irony in watching your observability stack become the thing you most need to observe. That’s more or less our day job, and it’s why we tend to be unusually deliberate about tooling decisions: The small ones have a habit of compounding over time. So when it came to choosing a…
Read MoreFor a long time, Sealed Secrets was our go-to solution for managing confidential data like database credentials, tokens and private keys within Kubernetes. However, as our operations grew, the operational overhead of this approach led us to migrate to External Secrets. This transition has significantly streamlined our workflow, enhanced our security posture during credential rotations,…
Read MoreHi everyone 😃 Today I’d like to walk you through some experiments we ran about load-balancing requests to LLMs in Kubernetes. Let’s dive deep into it! Why Deploy LLMs in Kubernetes? Well, Kubernetes has now established itself as a leading technology when it comes to workloads orchestration. And with time the increasing support for accelerators…
Read MoreOne of your servers is reporting moderate CPU usage, enough available memory, and storage that’s busy but not saturated. The dashboard looks healthy, the application does not. Requests to it are slowing down, background jobs are taking longer, and latency is rising even though no resource appears fully exhausted. The problem here is that most…
Read MoreHow we adapted our monitoring approach when the default choice didn’t fitand what we learned along the way. You’ve probably been there. You walk into a new observability project, playbook in hand, only to realize that no two projects are ever really the same. Every team has its own tools, its own platform quirks, and…
Read MoreA Simple Reverse Proxy Might Turn out to Be Dangerous Sometimes we inadvertently make assumptions that undermine our infrastructure security. In today’s article I want to share with you one of the most common mistakes that are made when setting up a reverse proxy. As always, real world use cases are the best ones to…
Read MoreThe year has just come to an end, and after a long stretch of work I could really use some rest. But before closing everything out completely, there’s one last thing I wan to share, and it needs a bit of context about the project. We’re currently building a distributed application consisting of several components…
Read MoreKubernetes Operators are one of those ideas that feel magical when they work: you declare an intent/goal in YAML, and software continuously makes the cluster match it – handling upgrades, failures, drift, and lifecycle cleanup: like a purpose-built SRE on autopilot. Although in theory it looks like sci-fi fiction, in practice Operators are just code…
Read MoreWhen I got back from my last trip to Japan, I quickly bumped into a small but annoying problem: I had finally reached my free storage limit on Google Photos. Instead of just buying more space for a few euros a year, I saw it as the perfect excuse to try something I’ve been considering…
Read MoreA modern web app isn’t one single big monolith: it’s made of quite a lot of pieces! For instance, we relied on a setup such as this one for a recent one we are developing: That’s a lot of moving parts. You could glue them together with scripts, sticky notes, and caffeine… but then most…
Read MoreAs our company’s NetEye cloud solution NetEye.cloud expands, we’re deploying compute nodes not only in our own data centers but, on customer premises across the globe – connected through satellite links. This hybrid, geo-distributed model creates a very tough challenge: How can we manage configuration across hundreds of remote machines reliably, and at scale? Why…
Read MoreHi 😀 Today I’d like to explore with you a migration that we performed to a service that’s used internally to monitor the performance of various DBs, gathering data that’s especially useful for troubleshooting. This tool is the Percona Monitoring and Management (PMM) platform, which combines agents or direct access to various supported DBMS (MySQL,…
Read MoreRecently, we needed to upload build artifacts to allow developers to visualize Playwright test recordings. Initially, we used a simple PVC and an NGINX server with basic authentication, but this approach has a major drawback: it doesn’t allow uploads from different namespaces. As a result, we had to choose whether to deploy this service and…
Read MoreOne of the most annoying aspects of using Sealed Secrets is their encryption design. They are tightly coupled to the namespace and name of the target Secret, meaning any changes to either require re-encrypting the Sealed Secret. This rigidity complicates workflows, especially when you need to duplicate or promote Secrets across environments. It breaks the…
Read More