What it was

A publicly-accessible Linux VPS hosting a Flask application, on Oracle Cloud’s ARM free tier. Oracle Linux, nginx terminating TLS, Gunicorn behind a Unix socket, five scoped users, two independent firewalls, custom Python monitoring with push notifications to my phone, and fail2ban catching the noise the whole time.

I built it, ran it, and wrote it up as the Oracle Free Tier policy changed and the instance was scheduled for termination. The infrastructure is gone. The writeups are what remain, and they are the actual artifact of this project: a public, deployed system, run long enough to see what actually breaks, then documented in enough detail that someone could reproduce every decision.

Why bother, when the answer could have been “just use Docker”

I could have containerized the whole application, used Fly.io or Railway to handle TLS and deployment, wired up UptimeRobot for monitoring, and had the site live in a Saturday afternoon. That is the right choice for a business. It is the wrong choice for a learning project.

Every tool I would have used automates something I did not yet understand. Containers hide the permission model. Managed platforms hide the deployment pipeline. Commercial monitoring hides the question of what is even worth watching. Using any of them would have given me a working site and none of the knowledge that a working site is supposed to teach.

So I deployed the way sysadmins did before those tools existed: scoped Unix users, sudoers.d rules with full binary paths, a bare Git repo with a post-receive hook, Unix sockets for interprocess communication, and a Python script that runs from cron and alerts me when something breaks. Every layer was manual, every layer broke at least once, and every layer taught something.

The layered architecture

Working from the outside in, five layers of independent controls, each of which would have to fail for the layer inside it to be reachable.

The cloud firewall. Oracle Cloud Security Lists sit above the VM entirely. They drop unauthorized traffic before it ever touches the OS. Three ports open: a non-default SSH port, 80, and 443. Configuration lives in the OCI console, requires separate credentials from the server itself, and cannot be modified from a shell inside the VM even if that shell is root. That last property is the whole point.

The host firewall. firewalld on the OS, zone-based rules, service-name references instead of raw port numbers so an auditor reading firewall-cmd --list-all six months later can understand what is allowed and why. The cockpit web console (port 9090, shipped enabled by default) was removed on day one. An open port for a service I do not use is textbook unnecessary attack surface.

SSH. Root login disabled. Password authentication disabled. AllowUsers whitelist so that even if some other vulnerability lets an attacker create a user, that user cannot SSH in. Non-default port for log-noise reduction, with no illusions that this is “security through obscurity, therefore secure.” Idle session timeout. And every change validated with sshd -t before restart, with a second terminal session held open as a rollback path. Locking yourself out of a cloud VM is a rite of passage I chose to skip.

User roles. Three system users (five for the deployment pipeline), each scoped to one job, none capable of doing another’s. The admin has sudo but must type a password (Oracle’s default was NOPASSWD: ALL, removed on day one). The deployer owns web files and can restart nginx, nothing else. The service user runs the application with a /sbin/nologin shell so it cannot be logged into at all. Each sudo rule is stored in its own file under /etc/sudoers.d/, uses full binary paths to prevent PATH manipulation, and grants exactly one command with no wildcards.

The application layer. nginx terminates TLS with Let’s Encrypt certificates auto-renewed by a systemd timer, redirects all HTTP to HTTPS with a permanent 301, and proxies dynamic requests to Gunicorn over a Unix socket. Static files nginx serves directly. The Unix socket instead of a TCP loopback is a small decision that has a real effect: access is controlled by file permissions rather than a firewall rule, and there is no open port for anything to even attempt to connect to.

Monitoring. Fail2ban watches SSH authentication attempts and bans offending IPs at the firewall layer, so banned sources cannot even complete a TCP handshake. A custom Python script running from cron every five minutes checks service health, disk and memory, certificate expiry, and failed-login counts, and sends push notifications to my phone with per-issue cooldown so I get one alert per problem instead of one every five minutes. The script also attempts an automatic restart on down services and reports whether the restart succeeded.

The design principles, named

Three things this project taught me that I would carry into any production infrastructure role.

Defense in depth is not a slogan. Every layer above assumes the layers below it might fail. The cloud firewall assumes firewalld might be misconfigured. firewalld assumes an application might bind a port it should not. SSH hardening assumes the firewall might have a hole. User scoping assumes SSH might be compromised. The value of the layering is not that each layer is bulletproof. It is that compromise requires simultaneous failure of independent controls, which is much rarer than any single failure.

Least privilege is worth the friction. Scoping five users with non-overlapping permissions made the deployment pipeline substantially harder to build. Every cross-user action required a scoped sudo rule with exact path matching. Every mistake produced a cryptic “command not allowed” error that took real time to debug. The alternative was one user with sudo access to everything, which would have worked on the first try and would have made compromise of that account a full compromise of the server. The friction is the security.

Manual now buys understanding later. When I eventually containerize a project or deploy through a managed platform, I will know exactly what those tools abstract. The bare-Git-repo deploy flow, the sudoers pattern, the Unix socket, the fail2ban jail configuration, the certbot renewal timer, the SELinux context relabeling: none of these are obsolete knowledge. Every one of them is what the tool I am not using would be doing on my behalf.

Detailed writeups

The individual layers, each in a standalone post:

What I would do differently

Put the application under /opt from day one. SELinux is more permissive there, and it is the conventional location for application-specific software. Half the SELinux debugging in the deployment writeup would have been avoided by starting there.

External monitoring. My custom monitoring script cannot detect its own death. If the server goes down, the alerts stop, and I do not know. A cheap external ping from a separate machine (or a free UptimeRobot check) would have closed this blind spot for zero effort. I did not do it. That is on the list for the next deployment.

Test the pipeline with a dummy application first. Half the deployment debugging was pipeline debugging, not application debugging. Getting the pipeline working against a Flask “hello world” before deploying real code would have saved substantial time and produced a cleaner separation between “the pipeline is broken” and “my application is broken.”

Structured logs from the start. I collected plain-text logs throughout, then wished I had JSON when it was time to actually query them. Structuring logs at the source is trivial upfront and expensive to retrofit.

What remains

The infrastructure is scheduled for termination, so there is no live URL to point at. The value now is in the notes: seven detailed writeups linked above, each covering one layer of the system in enough depth that the decisions are defensible and reproducible. The next iteration will run on different hardware, probably in a different provider, with a different domain. The principles transfer. The commands mostly do not.