Ask the Internet, Not the Model

An AI assistant can read every rule in your firewall config and still be wrong about what's exposed. So can you. The only way to know which ports answer is to knock on them from outside.

Use AI on your servers by all means. Just give its work a check that can't share its mistakes.

The Agent That Was Sure

In July 2025, SaaStr founder Jason Lemkin was about a week into a public experiment building an app with Replit's AI agent when, by his account, the agent deleted his production database during a code freeze. The useful part is what happened next. Asked about recovery, the agent said rollback didn't cover databases and that every database version was gone. Whatever you call that, it was confident, specific and wrong: the rollback worked.

Unless something makes an agent observe a system, its answers about that system are inferences.

Inferred Versus Observed

There are two ways to answer "what's exposed on this server?"

The first is inference. You, or your AI, read the firewall rules, the container settings and the cloud provider's security groups, and reason about what they add up to. It's fast and usually right.

The second is observation. A machine on another network sends a packet to each port and records what comes back. For a TCP port, the scanner sends the opening packet of a connection, called a SYN. An open port replies with a SYN-ACK, a closed port replies with a reset, and a port behind a firewall usually says nothing, which scanners report as "filtered". Very little interpretation is involved, so the method is close to deterministic: scan an unchanged server twice and you get the same answer, give or take dropped packets and rate limits, which good scanners retry around.

Language models are less repeatable than they look. Researchers at Thinking Machines sent one prompt 1,000 times with temperature, the randomness setting, at zero and got 80 different completions. They traced it to the way model servers batch requests, and showed how to fix it. A perfectly repeatable model would still be reading the config rather than testing the port and could be wrong the same way every time.

Inference fails quietly when a config doesn't mean what it appears to mean, and Docker has a well-known case of it.

Docker's documentation warns that when you publish a container's port, the traffic is diverted before it reaches the rules of ufw, the firewall front end that ships with Ubuntu. Published ports also listen on every network interface by default. So a server can carry a tidy, deny-by-default ufw policy and still serve its database to the world.

We reproduced this on a test machine running Docker 29.9.2 and ufw 0.36.2. ufw status verbose showed incoming traffic denied by default, with 22, 80 and 443 allowed. A test service running directly on the host at port 6379 (Redis's usual port) came back filtered from outside, as you'd expect. A container started with -p 5432:5432 (PostgreSQL's port) did not: a scan from a second network found 5432 open. Most people reading that ufw output, human or AI, would call the server locked down.

One Mistake, Three Engines

Asking the AI to re-check its own work helps less than you'd expect.

On 5 May 1983, Eastern Air Lines Flight 855, a three-engine Lockheed L-1011, left Miami for Nassau. The night before, mechanics had replaced a part called the master chip detector on each engine, and all three replacements went in without their O-ring oil seals. There was even a check: each engine ran for 10 seconds and showed no leak. Investigators later found that a leak without O-rings takes at least 30 seconds to appear.

In the air, the oil quantity gauges for all three engines read zero. The crew's first thought was faulty instruments, because three engines failing at once seemed too unlikely to be real. The gauges were right. The crew restarted one engine and landed back in Miami on it, and none of the 172 people aboard was hurt.

Redundancy only works if failures are independent. That night a single maintenance error reached all three engines, which reliability engineers call a common-mode failure.

Asking an AI to review a config it wrote is like asking the crew that fitted the parts to inspect them. It brings the assumptions that produced the error, so it tends to approve the error. Some checks can't see the fault at all, either: reading ufw status on a Docker host is the 10-second leak test. People reviewing their own work have the same problem. Richard Feynman told Caltech's graduating class of 1974 that you must not fool yourself, "and you are the easiest person to fool."

That's why skill doesn't fix it. An experienced engineer who knows Docker well can still miss the one compose file (Docker's per-project settings file), in one project, that publishes a port. Solo operators have it hardest, because the AI may be the only reviewer they have. Teams aren't immune: reviewers read the same diff and share the same picture of the system.

A port scanner is far less clever than a language model, and that's what makes it useful here. It has no opinion about what should be open.

The Internet Checks Anyway

Someone will scan your server whether you do or not. In 2013, researchers showed that one ordinary machine could scan the entire public IPv4 address space for a single port in under 45 minutes, and that was more than a decade ago.

What you control is whether you see the results first. Checking yourself costs a little time with free tools, and a service that checks daily costs a small monthly fee. On the other side of the ledger is a database port you believed was closed.

How to Use This

1. Write Down What Should Be Open

For each server, list the ports you mean to expose and why. That list is the expected answer. Without it, a scan result is just a list of numbers.

2. After Every Change, Yours or an Agent's, Scan From Outside

Use a machine on a different network, and only scan servers you're authorized to test:

sudo nmap -Pn -p- --open 203.0.113.10

-p- covers all 65,535 TCP ports (nmap's default is the 1,000 most common), -Pn skips the ping check and treats the host as up, and --open shows only open ports. With sudo, nmap uses the SYN scan described above; without it, nmap completes a full connection on each port instead. Against a well-firewalled host a full sweep can take a while. Anything not on your list is a finding.

3. Ask Agents What They Ran and What Came Back

When an assistant says a server is locked down, ask for the command and its output. Reading a config is inference. A probe from outside is an observation.

4. Look for the Known Bypasses

On a Docker host, list published ports:

docker ps --format 'table {{.Names}}\t{{.Ports}}'

An entry starting 0.0.0.0: or [::]: is published on every interface. If a service only needs to be reached from the machine itself, bind it to loopback, for example "127.0.0.1:5432:5432" under ports: in your compose file. In our test that change took port 5432 from open to filtered, and local connections kept working.

5. Make It Recurring

One scan is a snapshot, and servers drift. Put the scan on a schedule, or hand it to a service.

A few limits. These steps cover TCP; UDP scanning (nmap -sU) is much slower and less conclusive. If the server has an IPv6 address, scan that separately with nmap -6. And a scan only shows what's reachable from where the scanner sits, so if a rule admits only certain source addresses, scan from outside that list.

Where AegisWhite Fits

Step 5 is what AegisWhite's endpoint monitoring does. Once a day it scans all 65,535 TCP ports on each server from outside. Between those full sweeps, lighter passes every 30, 10 or 5 minutes, depending on plan, check every port already known to be open plus a rotating tenth of the rest, and changes are confirmed with a deeper scan.

Results are compared with an Allowed Ports list where nothing is allowed by default and every rule needs a written reason. When something unexpected answers, the alert page in the portal explains it in plain language and carries fix commands for UFW, iptables, AWS, GCP and Azure. Nothing is installed on the server for scanning.

The Firewall Map is included on every plan and shows the Docker case above, open ports that no firewall rule explains. It reads your rules either from rules you paste in or from an optional collector script that runs on the server, reports every 6 hours by default and opens no ports.

Plans start at $19 a month for up to 10 endpoints, each an IP address or hostname.

Let the model help write the config. Then ask the internet what's open.

If you'd like to see how that looks in practice, the AegisWhite demo runs with no sign-up.

Sources

• The Register: Vibe coding service Replit deleted user's production database, faked data, told fibs galore (21 Jul 2025)
• Thinking Machines Lab: Defeating Nondeterminism in LLM Inference (Horace He et al., 10 Sep 2025)
• Docker docs: Packet filtering and firewalls (the ufw bypass)
• Docker docs: Publishing and exposing ports (default bind on all interfaces; 127.0.0.1 form, including Compose)
• Wikipedia: Eastern Air Lines Flight 855
• NTSB Aircraft Accident Report AAR-84/04 (archived)
• Richard Feynman: Cargo Cult Science (Caltech, Engineering and Science, June 1974)
• Durumeric, Wustrow, Halderman: ZMap, USENIX Security 2013
• Nmap reference guide: Port specification