blip zip blog

so I stand a chance of remembering things

Not a lot to do today. Just added this VPS to the Prometheus monitoring suite.

A few things to do next: – Put together a rootless Docker container for writefreely; there doesn't seem to be one from the publisher. Might even be bold and compile from source. – Sort out the DNS approach for WireGuard peers. – Lob static config files into a public git repo so they can be pulled easily.

Since the point of this blog is self-reflection, I'll start that now.

iptables

Before

Prior to this, I've always survived with ufw when I've needed a firewall. It's been simple, and I've never needed to do more than limit port access. I've been baffled by the various tables and rules that get mashed in by e.g., Docker; definitely a case of seeing something new and assuming it's far too complicated to understand.

Now

I can write rules on the fly for the sorts of use cases I have, namely allowing traffic based on: – IP – Port – Protocol – Connection state

Next

So far I've just bunged everything into the default tables and I'm going to explore those more. Particularly with WireGuard interfaces, where everything gets spun up and torn down on a whim, I think it would make more sense to extend INPUT/OUTPUT/FORWARD via dedicated tables.

Docker-compose

Before

I only had a very basic grasp of how Docker worked, and putting an abstraction on top of it with networking and state was too much to manage. I've since used Docker a load more, starting from base images and building up hardened images from scratch.

Now

I've got enough confidence to use it, but still a fair way to go before I'm good writing it from memory without a basic structure.

Next

Carry on where I am unless I have another reason to go and learn more.

Systemd timers over crontab

Before

It's been ages since I last set up a cron job, and I couldn't remember the arg I've always used in the past (it was crontab -e). I googled it to see, and was shown a bunch of articles explaining the metrics of systemd timers. Sidenote: out of interest, I asked ChatGPT for a modern way of doing scheduling tasks on Ubuntu 24.04 LTS (which comes with systemd) and it still recommends crontab.

Now

I know they exist!

There's plenty of other stuff that I've left out, mostly networking. Next recap will cover those.

IONOS

A quid a month is pretty hard to argue with. Other pros: I can customise inbound firewalls so I can restrict SSH to only operate over the WireGuard management. If the interface goes down for whatever reason, I can just allow SSH in from my current WAN IP address and fix things.

The whole setup for the VPS is script so if I need to burn/replace it should be trivial.

One thing to I always seem to forget is the name of the default NIC. IONOS' Ubuntu image uses ens6 which probably makes sense to someone somewhere. The first time I ran the build script, my iptables templates had the wrong name and I locked myself out. Nice to know how the reimagine process works, I guess.

Writefreely

Enjoying using this so far. Really really simple to set up.

Only weird behaviour (to me; might be obvious to others) that I've found is that the bind config parameter is only a single interface on the hosting machine. Had to bust out tcpdump and test out where it was falling over. The default value is localhost which is fine, but I need it to listen on the IP of the WireGuard interface. I've modified as below:

bind                 = 192.168.115.2

It's worked fine since I did this. Kind of annoying but not a big issue.

Had a look at OAuth setup using Entra, but the config items appear to assume that the userinfo endpoint is on the same domain as the auth/token ones. MS loves to do things their own way, so they've exposed it via Graph instead. I'll spend some more time reviewing later.

NUCs

Neither had a VPN connection prior to setting up this blog solution so firewalls felt unnecessary. I'll sort out the iptables rules later on. For now, I'm just enjoying having a public thing that I can see.

You can see this because the VPS is up and running!

VPS is the smallest and cheapest I could find (from a trustworthy provider). The basic setup is nginx + certbot on the VPS for this web interface, with a WireGuard tunnel set up direct to a machine in the “prod” network of my homelab. The machine's running the writefreely service. It'd be rude not to include some code in the first post, so here we go with the systemd unit file:

cat <<EOF | sudo tee /etc/systemd/system/writefreely.service > /dev/null
[Unit]
Description=Write Freely Instance
After=syslog.target network.target

[Service]
User=writefreely
Group=writefreely
Type=simple
StandardOutput=syslog
StandardError=syslog
WorkingDirectory=/etc/writefreely
ExecStart=/usr/local/bin/writefreely
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now writefreely.service

What's the point in running a homelab and all the associated VPNs to get traffic off/onto the internet unless you actually use them?!

To that end, GitHub pages and Jekyll is out, and some simple microblog is in.

I've done some reading around, and writefreely looks like a solid bet. Authn is basic in the OOTB config, but it does support OAuth so I can use that if needs be. In the mean time, a really long password should do.

Plan is to get a cheap new VPS, and use that for various different blip.zip subdomains (initially just blog and blog-test). It'll act as the gateway for inbound traffic. nginx will be a reverse proxy to point each subdomain to a distinct service. It's public-facing, so certbot can take care of the TLS cert configuration.

Two WireGuard tunnels will take traffic from the VPS gateway into the homelab; one for lab machine, and another for prod.

Ionos looks remarkably cheap, and given that I need the bare minimum bandwidth, it's probably the best bet.

TODO list for tomorrow: – Order the VPS – Setup DNS (A record for blip.zip, then aliases for blog and blog-test) – Customise existing VPS setup scripts for this gateway: – install nginx and certbot – configure nginx to proxy blog and blog-test into nuc2 (lab) and nuc3 (prod) – Customise existing nuc scripts for these machines: – download, add, setup writefreely – use iptables

  • Scripted out most of the config for prometheus/alertmanager/blackbox_exporter.
  • AlertManager had disappeared (can't remember why) but config was still there so that kept things simple.
  • Going to set up NGINX as a reverse proxy for prometheus.blip.zip, alertmanager.blip.zip. Is it worth putting it in docker? Probably not.
  • I'm going to script up the key renewal and distribution from (probably) the monitoring box.
    • Setup:
      • Create tls_certs (?) user on automation host
      • Create SSH key
      • Create tls_certs user on each machine that needs it.
      • Add pub to authorized_keys.
      • Add all process user (i.e., nginx) that need it to a group tls_certs.
      • Create new directory /etc/tlscerts/<domainname>
      • Make tls_certs owner/group. Set 640 -rw-r——– perms.
    • Script:
      • (Maybe) prompt for porkbun API secret
      • Call API and write key/cert to /tmp/keys
      • Iterate over all stored hosts and SCP new key/cert onto /etc/tlscerts<domainname> (replacing previous ones)
      • Notify via webhook if there are any failures
  • Stupid and annoying learning point: if you add a new group, and add yourself as a member of that group, it won't take effect in that session.
  • TODO – set up a new jump box for remote access?

Scripted out the rest of the seedbox build config in what little bit of time I had.

Had a quick look at GitHub pages for actually publishing this microblog. Need to read more about Jeykll, but it'll probably do to repurpose my existing GitHub pages thing.

  • Worked out what was wrong! The docker network that was created by the NGINX/Deluge compose was just using a default bridge setup. It was using 192.168.0.0/20 – i.e., all up to 192.168.15.255. Inbound packets were getting routed on to the docker network, rather than SSHD. That explains why NUC3 was working – VLAN_40 uses 192.168.40.0/24.
  • Did some research into the networks used by docker-compose here.
  • Network config is now the below. It's basically arbitrary, and I should make sure that it's not conflicting with any WG network (especially third-party ones). networks: default: ipam: driver: default config: - subnet: 10.100.0.0/24
  • USB HDD connector arrived, so I need to dig into my trusty box of old power supplies to find a 12v one
  • Plan for later: read the hard drives from my old gaming PC, copy anything super-important to OneDrive, and put some labels on the drives so I know for next time.

  • I've updated my docker-compose file for Deluge. It now includes nginx listening on 443, which is proxying to 8112 for the Deluge web interface.
  • Took ages to get the NGINX config set up properly, mostly because I was being lazy and trying to copy things rather than starting from scratch and understanding what was going on.
  • Weird – when NGINX/Deluge are running together, I can't SSH from the LAN network i.e., laptop on VLAN101 –> seedbox on VLAN60. iptables firewalls and pfsense are all fine. tcpdump shows the packet being received, but there's no reply. Same goes for pings. WG110 works fine.
  • Added in a rule to allow SSH from VLAN_40 and could SSH fine from NUC3 to seedbox, so it's not a problem on the interface.
  • Added in some permissive firewall rules in pfsense to debug. Don't forget to remove those later.