When the product is already hacked: IR through a DevOps's eyes, not a security department's
kosar

This is not a guide where you press the right button in a SIEM, and it is not a forty-slide retelling of NIST.
Three incidents stacked up for me in three weeks. Two of the products were not mine at all: they called me when it was already on fire. The third sat in a neighboring environment I was formally barely attached to, but I still had to dig through it. Names, IPs, brands, and CMS vendors are cut out below. I talk about PIM and CMS in general terms. If the architecture looks familiar, that is a coincidence.
I am not a certified security specialist, but sometimes people write me something like "dude, we're in deep shit." Out of these three investigations I ended up with a pretty boring picture: how to tell that we are already being broken into, what to do first, why "we cleaned it and brought it back the way it was" often does not fix anything, and two stories after which "well, we have Cloudflare, we're fine" sounds a lot less confident.
What counts as a "good" incident
From the attacker's point of view, a good incident is one that goes unnoticed for a long time. If the goal is to take the site down, people see that immediately. If the goal is to collect credentials, databases, and access to neighboring machines, then the longer everyone assumes nothing is happening, the better it is for that side.
For the product it works the other way around. While it is quiet, they have time to persist, read whatever is sitting nearby, and widen the surface. So if the product does not have at least basic monitoring, you are already starting from a loss. You just do not know it yet.
Security courses like to talk about IDS and IPS. In production I have seen a system like that once, at a very large customer, with a separate security department and a budget a startup would not dream of. A lot of the products I work with have no separate security department, and waiting for one to suddenly appear by Monday is pointless. The minimum I expect instead is different: metrics of how the product behaves on an ordinary day (OpenTelemetry, and on top of that Grafana, Datadog, or Victoria; the specific dashboard is secondary here), alerts on odd CPU, memory, disk, and outbound traffic, and at least one person who sometimes opens the dashboard and notices that the picture does not look like yesterday's.
An incident almost never walks into an empty room. DDoS shows up on the network. Password guessing and decryption eat the CPU. On one of the servers I ran into software that was reading RAM looking for secrets, and that was not quiet either. After that comes the usual set, except you have to look at it not as "admin commands" but as traces: the process list, outbound traffic to places it should not go, and everything that can start on a schedule. Cron, Task Scheduler, a systemd timer, an ordinary service. The less inventive ones leave a crontab line that says "every five minutes, download a script and hand it to bash." The more inventive ones hide in systemd and in startup items nobody has opened for months.
In this story, monitoring is only the entrance. The moment you realize this is not "just load," the argument is no longer with the graphs. It is with the business.
The product has to work. But not at any cost
When production is on fire, everyone wants the same thing: for it to open again. The business pays the salary, not a pretty incident report, and that is fine. The trap is that the fastest way to "make it open" often drops you back into the same hole.
You can clean the server and bring the same code back up on the same hardware. You can wait a day, see that nothing popped up, and decide it is clean. I have watched people step on that rake more than once, and it was always an illusion. The break-in could have come through the infrastructure, through a stolen token, or through a hole in the product itself. If even one of those is true, and you simply moved the same build to a neighboring address, congratulations: you might as well buy a welcome mat, strap it to your asshole, say "I'm open for business". You did write down a recovery plan after the first breach, right?

The frame that actually helps in moments like this is boring to the point of horror: preparation, detection and analysis, containment, eradication, recovery, post-incident. Containment and removing the cause come before "put it back the way it was, as fast as possible." Otherwise recovery turns into a rehearsal of the next visit.
Which brings up the question that surfaces every time: fix the current server, or stand up new infrastructure. I usually go for new. It takes longer and it hurts more. On one of the investigations the pain was very concrete: a European "sovereign" provider with no proper API, no Terraform, servers created through a support ticket. The upside is fewer thoughts along the lines of "what if something we never found is still sitting somewhere."
If you are alone on the incident, I am sorry. At that moment you are simultaneously a DevOps, a developer, QA, a manager, a developer, and the person who explains the calendar to the customer. The customer usually wants two different estimates at the same time. An hourly one and a calendar one. "Thirty hours of work" does not mean "done in two days": those thirty hours can stretch across a month, and you have to say that up front, not when you are already in the middle of it.
If there are several of you, it is better to hand out roles the same day, not once everyone is already fixing something in parallel. Who cuts the network and collects whatever can still be preserved. Who looks for the hole in the code. Who writes the plan and does the talking.
And there are two documents people constantly mix up. A postmortem: what happened, what we did, what we got burned on, and what to do differently next time. An incident report. I usually write two versions, a technical one and one for the business. Sometimes that is the best thing you have when the customer calls in an external audit. Sometimes it is the last nail, if the text has holes and sugarcoating. I try not to hide the uncomfortable parts.
Next, two cases. Not about how to repeat the attack, but about what I saw from the other side, and which doors turned out to be open simply because that is how it had worked out historically.
Case 1. Supply chain through "convenient" CI
The stack was ordinary: Ubuntu, a PIM, almost classic nginx + php + mysql, and a self-hosted GitHub Actions runner on the same production box, because that made deploys more convenient. The call was coming from inside the house: the runner lived on the production server.
At some point the server simply fell over from load. Someone finally looked at the monitoring, I got onto the host, and I saw roughly what you expect to see at a moment like that: extra processes, a reverse shell, stray jobs in cron. That got turned off. The more unpleasant part was something else. Someone else's code was already sitting in the deploy artifact.
For us, continuous delivery is usually one-way. There should be no road from the server back into the repository. If the dirt is already inside the artifact, the issue is not that someone logged onto the machine and edited files in place. They are breaking the chain that code takes to get onto production at all.
At first glance, main looked normal: commits from known contractors, nothing loud. The customer was on a free GitHub organization, and there was no proper branch protection: you could push straight to main, and you could force-push. The picture had to be assembled from the organization's audit logs and from dangling commits. Objects on GitHub keep living for a while, on the order of thirty days, even when no branch points at them anymore.
Later the contractor wrote in the shared chat that their GitHub token had been stolen. We never officially found out how, but everyone understands how that usually goes. In my experience most of these stories are human factor, not some especially clever research.
The workflow was green. The secrets were gone. Several things were done in that account's name, and they line up into one thread rather than a pile of random commits. First, under labels like Security, Audit, and CI Check, workflows pulled secrets and sent them out. A security pipeline is especially convenient for that: you set it up once, and after that almost nobody looks at it.
Remember the self-hosted runner, conveniently placed in a group with passwordless sudo? From there they were already pulling .env files, SSH keys, and tokens, and cron had a beacon every five minutes: download a script and feed it to bash, so the whole thing would come back if someone simply deleted the files.
They also put a "font" with a .woff2 extension into the repository, which was in fact a Node script, and a VS Code task that runs when the folder is opened. On PHP and nginx itself that does not blow up. The hit is aimed at a developer who clones the repo and opens it in an IDE. Works on attacker's machine. Unfortunately, it also ran on yours.
For the attacker the timeline was almost ideal. The pipelines had started leaking data long before the server fell over and someone finally looked at it. For us this is exactly the case where the incident stayed "good" for the other side for too long.
After that the work is boring: remove the persistence, sinkhole and DROP the C2, stop the runner, clean the disk and _work, stand up new infrastructure, and before rolling out onto the clean environment, make a list of secrets and rotate them. Neighboring machines on the contractor's private network had to be looked at too. One compromised host with access over the local network is already not "one server."
Branch protection would have helped here. This is not bureaucracy for the sake of bureaucracy. If you cannot push straight to main, only through a PR and review, an approach like this usually runs into a living person before it ever reaches production. If you cannot force-push, you do not later have to reassemble history from pieces that have already been thrown out of the branch. On a free plan that often simply is not there. That is also a decision about risk, even if nobody explicitly made it.
Case 2. "We have Cloudflare" and one database for two brands
Another project. Windows, IIS, old .NET, a CMS. They called me in again to sort it out after the fact.
Day zero looked like this. CPU at a hundred percent, the cursor over RDP moving like a slideshow, GodPotato in the process list, and that is a way to raise privileges. The application pool ran as Network Service and could also write the site's files, so the path from there was short: SYSTEM, a reverse shell under names like "almost IIS logging" and "Winlog Helper," with normal File Descriptions, not a dumb renamed file, then mimikatz, dumps, and Scheduled Tasks.
The IIS logs for the day of entry had already been wiped. We put the picture together from what was left: the SQL ERRORLOG, Event 7045 for a service install (especially telling when PathName is suddenly out of Temp), tasks, files on disk, and the Cloudflare logs. After that the customer suddenly agreed to pay for separate storage that the web logs stream into. What used to look like an unnecessary expense became one of the few sources that had not been erased on the machine itself.
The way in was almost insulting. An old CMS, a file upload form with no authorization, on a long URL nobody had opened for years. The WAF and the CDN looked at it calmly, because the page was "ours." They uploaded an .aspx there, reset the CMS admin, and from inside the CMS they pulled privilege escalation.
Cloudflare guarded the front door. The attacker used the side entrance nobody remembered.
What worked next was not cleverness. It was architecture that, for some reason, the project considered normal. Servers on the provider's LAN could see each other more generously than anyone wanted. Two brands shared one SQL database, so from Alpha, through that database, they created an admin on Beta and repeated the upload and the shell. On Beta the OS was newer, and antivirus cut off part of the chain, but they still managed to take the database. Orders went through a legitimate FTP setup: it was enough to change FTPHostname to someone else's address, and the attacker does not need their own client. Your zoo is enough. And I still do not understand why FTP was not banned worldwide in 2007. Separately, the same server held one key to a shared S3 with all of the customer's backups. I did not see a suspicious download in the S3 logs, but the surface was enough to burn keys like that and then live with the thought.
And my mistake on this case, which is worth saying out loud. I dug around for too long before I cut the network. The channel should have been killed first, and only then should I have figured out what was actually running there. Maybe the IIS logs would still have been there. Detection without containment is a nice tour of someone else's party while the party is still going.
A few things worth saying separately
Once it is clear this is an incident, first you save whatever can still be saved, and you shut the channel going out. Who exactly did it can be figured out later. While the channel is alive, you keep feeding the other side your data.
Do not go visit the attacker, even if passwords to their own services are suddenly sticking out of the scripts. A blocklist and a sinkhole, yes. "Let's knock, delete it on their side, check what is sitting there," no. At that moment you are getting distracted from the only job the business has: get our own thing back up. And there is another unpleasant possibility. If it seems like everyone around you is an idiot, there is a non-zero chance they are making the idiot out of you, and the password in the script was left there precisely so you would widen the surface yourself. I am not a certified red teamer. From my own experience I will just say: on defense, you should not do that.
A self-hosted runner on production inherits the blast radius of the machine it sits on. If it was given passwordless sudo and access to the disk, that is no longer "just CI." A separate host, narrow permissions, and egress you can actually describe. "We only run our own workflows" is not a threat model.
A CDN does not replace control of the origin. It is worth walking every name that points at the same address. You put Cloudflare in front, then close the origin's web ports to everything except the CDN. Otherwise the CDN stays decoration, and an unauthenticated form, like in the second case, still opens from the street.
A shared SQL database and a shared backup key across brands are a bridge of trust.
RTO and RPO are better discussed with the business before the fire, not in the middle of the night. RTO is how much downtime is still tolerable. RPO is how much data you are willing to lose if you roll back to a backup you trust. Pulling a live database off an already compromised host is faster in hours and more expensive in compliance: users' data suddenly ends up with people who will "just help compare."
Most of the incidents I have seen did not look like elite research. There was human factor, a forgotten form, a shared database, a contractor's token, and FTP that is still alive "because that is how it historically worked out." Boring, expensive, and very repeatable.
An incident report after that is not the ending. It is the moment when you still have to decide what you are rebuilding, which secrets you have already burned, and how long to keep watching the graphs until "quiet" is quiet for a real reason, and not because the beacon simply stopped knocking on the same port.