guest@crt.os ~ $ whoami

// IDENT 0xA17E Joran Beasley

technical lead · software & architecture · on-site

01 // brief

Full-stack engineer on a small team at a small company, building an IoT platform for large-scale agriculture — firmware to fleet, dashboards to data pipelines. Also write the developer & test tools that keep the rest of it honest.

02 // frontend & runtime

web

JavaScript TypeScript HTML CSS React TanStack Zustand Tailwind MUI Bootstrap

runtime & realtime

Node.js WebSockets REST

03 // backend & data

python

Python FastAPI Flask Django NumPy Pandas

datastores

PostgreSQL SQLite TimescaleDB Timestream MongoDB Redis

04 // cloud & infra

aws

EC2 S3 SES SQS Timestream IAM CloudWatch

containers & orchestration

Docker Compose Swarm Kubernetes

01 / 04

// stack

Python10y
TypeScript/Javascript7y
WebGL / WGSL2y
C/C++2y
Postgres8y

// shipping

github writings cv.pdf mail

#01 · 2025-03-18 · 6 min · mqttrealtimeiot

MQTT for people who have used WebSockets

Both push bytes over a long-lived connection. They are not the same animal.

If you came up doing web work, the obvious frame for MQTT is "WebSockets, but for IoT." It's a tempting shortcut and it will hurt you for about six months until you learn the differences the painful way. Here are the ones that mattered for our fleet.

publishers, broker, subscribers — nobody talks to anyone directly.

one connection, many topics

A WebSocket is a pipe. You decide what goes in it. MQTT puts a topic tree in front of the pipe and lets a central broker route each message to whoever has subscribed to a matching pattern. farm/north/+/temperature fans out to every device-id under farm/north, and you didn't have to know about any of them ahead of time. That's not just an addressing convenience — it's the entire shape of the system.

QoS is not a knob you set once

WebSockets give you delivery semantics roughly equivalent to "if the socket is open, the bytes show up — otherwise they don't." MQTT has three QoS levels, and the right one depends on what you're sending:

QoS 0 — fire and forget. Telemetry that's fine to lose.
QoS 1 — at-least-once. Ack required, dedupe on your end.
QoS 2 — exactly-once via a four-step handshake. Slow. Reserve for state changes you cannot replay.

We started with QoS 1 everywhere. Three months in, half our problems were duplicate inserts in Timescale. The fix was a (device_id, ts) unique key and an INSERT … ON CONFLICT DO NOTHING. The fact that QoS 1 implies duplicates is in the spec. Nobody read the spec.

retained messages and the last-will-and-testament

These are the two MQTT features I miss every time I go back to a plain WebSocket. Retained messages stick on a topic so a new subscriber gets the latest known value the instant they connect — perfect for state ("device 0a1f is currently irrigating"). LWT lets a client tell the broker, "if I disappear without a clean disconnect, publish this message on my behalf." You get presence for free.

so, when WebSockets?

Browsers, real-time UI updates, and anything where the producers and consumers all live in your own infrastructure. MQTT when you have a fleet, when the network is flaky, and when who is on the other end is the question you're trying not to answer.

6 min

#02 · 2025-02-04 · 5 min · websocketsrealtimeops

WebSockets: the quiet parts

Heartbeats, backpressure, and what your reverse proxy is silently doing to you.

The WebSocket protocol is a paragraph of RFC 6455 and a hundred pages of operational lore. Most teams I've worked on knew the paragraph and were slowly, painfully accumulating the lore. A short collection of the lore that mattered.

handshake → frames → ping/pong → close. The middle bit is where you live.

your proxy will kill the connection

Nginx defaults to a 60-second idle timeout on proxied upgrades. ALB defaults to 60s as well. If your app's protocol has a quiet period longer than that — and most do — the connection is severed without ceremony. The fix is two things, not one:

Raise the idle timeout to something generous (we use 300s).
Send your own application-level heartbeat on a shorter interval. PING/PONG control frames work, but a {type: "hb"} payload is easier to debug and easier to reason about across reconnects.

backpressure is your problem

A WebSocket gives you a send() method. It does not give you a "I am overwhelmed, slow down" signal — at least not one most libraries surface. If your client is on a poor link and your server is firing 50 messages a second, the kernel buffer grows, then bufferedAmount grows, and at some point you OOM the tab.

Practical rule: before every send(), check bufferedAmount. If it's above some threshold (we use 1 MB), drop or coalesce. Telemetry that's a few seconds stale and present beats fresh telemetry that crashed the page.

reconnect with jitter, always

A datacenter blip drops 10,000 of your connections. Every client retries in 1 second. You get a 10,000-RPS thundering-herd reconnect. Use exponential backoff with full jitter — the AWS Architecture blog has the canonical write-up. The first retry should be a random delay between 0 and 2s, not 2s.

message framing for things humans will read

Resist the urge to send raw protobuf to a browser. The hour you save in serialization you spend forever in the network tab squinting at hex. JSON until proven slow; MessagePack when you've measured it.

5 min

#03 · 2025-01-12 · 8 min · agricultureiotphilosophy

Sensing beats modelling

On a farm, the ground truth is the ground. Reach for a sensor before reaching for a model.

This is a piece of advice I'd give my younger self a thousand times if I could. When you're building software for the physical world — farms, factories, energy grids, anything where the thing being software'd is made of dirt and water and metal — your first instinct will be to model. Models are clean. Models compose. Models are how you got hired. They are also, almost always, the wrong first move.

a real season, drawn from real data. the model started fine. by harvest, it was guessing.

the seductive thing about a model

A model is a bet that the future will look like the past. Sometimes that bet pays off — the model gets cheaper than the sensor, faster than the sensor, prettier in a deck than the sensor. So you ship the model, take the sensor out of the loop, and for a while the bet looks free.

Then a thing happens that didn't happen last year. A new pest. A weather pattern the training set didn't see. A dirty boom on a sprayer. The model has no idea. The model cannot have any idea. By the time you notice, the season is gone, and you cannot get a season back.

what we did instead

We built our platform on a hard rule: measurements are sources of truth, models are auxiliary. Concretely:

Every irrigation decision is anchored to a soil-moisture probe in the block, not a calculated ET budget. The ET model is shown next to it for context. When they disagree, the probe wins by default.
Every yield prediction has a what would I expect if the model is wrong by 20% readout next to it. We make growers do the math.
Every alert has a "go look" affordance — a one-tap action that opens a Google Maps pin to the sensor, with photos from the most recent visit.

The phrase we ended up using internally was "observe, don't infer." It's a mantra. It's also a code-review checklist item.

the dollar argument

The pushback I get on this, every time, is: sensors are expensive, you can't put one on every plant, the math has to fill the gap. That's true. But the dollar argument runs in the other direction too. A mid-size orchard that loses a single block to a missed frost has paid for fifty soil/air sensors. The sensors do not have to be everywhere. They have to be where the decisions are made.

what models are good for

I'm not anti-model. I'm anti-model-as-source-of-truth. Models are great at:

Filling in the spaces between sensors for visualization. (Heatmaps, not decisions.)
Anomaly detection. A model that's been wrong by the same amount for ten years and is suddenly wrong by twice that is telling you something — about the field, the sensor, or itself. All three are useful.
Explaining the past. Post-season, a model is a great way to understand why a year went the way it did. That's different from using it to drive next week's irrigation.

the cultural part

This is mostly a software-engineering blog so I'll keep this brief, but the cultural shift is the hard part. Every PM I've worked with has at some point asked, "can we replace this sensor with a model?" and the honest answer is almost always "yes, and you'll regret it."

Build the platform so that the people making decisions can see, at all times, how much of what they're looking at is measured and how much is inferred. Color-code it. Caption it. Put a tiny m next to inferred values. The growers we work with picked it up in about a week. It's the single piece of feedback that gets repeated back to me unprompted.

The land is older than your model. The instrumentation is how you stay honest about that.

8 min

#04 · 2024-11-22 · 6 min · postgrestimescaledbdata

A small Timescale cookbook

Hypertables, continuous aggregates, and the three queries that pay rent.

I've spent four years inside TimescaleDB schemas at varying scale. This is not a tutorial — the docs are good. It is the small set of patterns I reach for, and the ones I'd warn a younger me away from.

chunks roll forward in time; the old ones get compressed and queried less often.

chunk size matters more than you think

The default chunk interval is 7 days. For most workloads that's fine. For very high cardinality (millions of distinct devices) or very low (a single sensor at 1Hz), it isn't. The rule of thumb that's served me well: target chunks of ~25% of available RAM, not a calendar period. A pg_buffercache query and a Sunday afternoon will tell you if you got it wrong.

continuous aggregates are not optional

A raw SELECT avg(value) FROM readings WHERE ts > now() - interval '90 days' across a billion rows is the kind of query that makes a dashboard look broken. Continuous aggregates materialize a rollup (typically time_bucket('1h', ts)) and refresh on a policy. Done well, your 90-day chart hits a 100k-row view, not a 1B-row table.

CREATE MATERIALIZED VIEW readings_1h
WITH (timescaledb.continuous) AS
SELECT
  time_bucket('1 hour', ts) AS bucket,
  device_id,
  avg(value)  AS avg_v,
  min(value)  AS min_v,
  max(value)  AS max_v,
  count(*)    AS n
FROM readings
GROUP BY 1, 2;

SELECT add_continuous_aggregate_policy('readings_1h',
  start_offset => INTERVAL '7 days',
  end_offset   => INTERVAL '1 hour',
  schedule_interval => INTERVAL '15 minutes');

Two gotchas worth surfacing:

The view is eventually consistent. Charts at "now" will lag by your end_offset. Pick a value the product can tolerate.
You cannot index a continuous aggregate the way you index a hypertable. Plan your WHERE clauses against the bucketed columns.

compression isn't free, but it's cheap

Once a chunk is older than your hot window, compress it. We use 14 days hot, everything older compressed. Disk drops by ~10x; query time on compressed chunks is worse (single-digit-ms turns into double-digit-ms) but for the older data nobody cares.

ALTER TABLE readings SET (
  timescaledb.compress,
  timescaledb.compress_segmentby = 'device_id',
  timescaledb.compress_orderby   = 'ts DESC');

SELECT add_compression_policy('readings', INTERVAL '14 days');

the three queries that pay rent

After a few years, ~80% of our dashboard hits one of three shapes:

Latest value per device — SELECT DISTINCT ON (device_id) … ORDER BY device_id, ts DESC. There is no faster way to phrase this in SQL.
Bucketed series for one device — time_bucket against the continuous aggregate, with device_id = $1.
Bucketed series for many devices, summed — same shape, GROUP BY bucket, SUM(avg_v * weight) if you're rolling up.

Build for those three. Everything else is rare enough to be slow on purpose.

6 min

#05 · 2024-09-08 · 5 min · edgeembeddediot

Edge gateways on a bad LTE connection

What you do at the device, so the cloud doesn't have to.

The cloud half of an IoT platform gets all the architecture diagrams. The half that lives in a metal box on a fence post, on a stalled LTE link, in a thunderstorm, gets a Slack message every two months that says "the gateway in block 14 went weird again." The gateway software is where most of the operational budget actually goes. A few principles that have made ours boring.

the gateway is a buffer with opinions. the link to the cloud is the unreliable part.

assume the link is down

Not "the link might be down." Build as if it is, and let connectivity be the optimization. Every payload goes into a local SQLite queue first, with a (sent_at IS NULL) index. The transmit thread reads batches off that queue. If the link is up, batches go out. If not, they don't, and the queue grows.

This sounds obvious. The trap is the team that puts a HTTPS POST in the same code path as the sensor read. That code path will hang for 60 seconds, miss the next sample, and you'll find out three weeks later when somebody asks why a graph has stripes.

deduplicate at the gateway, not the cloud

Every payload has a (device_id, ts, kind) triple. The gateway keeps a small bloom filter of recently-sent triples. On retry — and there will be retries — duplicates are dropped before they hit the radio. This saves you bytes (cellular data has a price) and saves the cloud ingest from doing the same dedup work for 12,000 devices in parallel.

batch ruthlessly, but with a deadline

A single MQTT publish per sample is a lot of overhead. We batch up to 256 readings or 30 seconds, whichever comes first. The deadline matters: a quiet sensor would otherwise sit in the queue forever waiting for a partner to fill the batch. Pick a deadline shorter than your worst-acceptable dashboard staleness.

sign at the edge

Every batch is signed with a per-device key (we use Ed25519). The cloud verifies before it inserts. This costs a few microseconds at the device and saves an entire class of "is this telemetry from who it says it's from" questions. It also makes a replayed payload detectable, because the signature is over (payload || nonce) and the nonce is monotonic.

log levels that survive a power cycle

A flash-friendly ring buffer of the last ~2k log lines, fsynced every N seconds, is the single highest-ROI debugging feature we've built. When a gateway misbehaves, cat /var/log/ringbuf over SSH is the first thing the on-call does. Without it, the answer to "what was it doing right before it rebooted?" is forever "we don't know."

the human at the gate

There is, on average, a person physically driving up to the gateway once a quarter. Build for them: a status LED with a documented language (slow blink == ok, fast blink == queue backed up, solid == no link), a button to force a sync attempt, and a QR code on the case that opens a maintenance page. None of this is hard. All of it is easy to forget.

5 min

#06 · 2024-06-30 · 5 min · observabilityopsiot

Fleet observability for 12,000 devices

What's broken right now, what's about to break, and what just stopped breaking.

Most of what's written about observability assumes the things being observed are services. A service is well-behaved: it exposes metrics, it logs to stdout, it dies on the same continent as you. Devices don't. A few patterns that have made our 12k-device fleet bearable.

the cluster of failures is the headline. the rest of the grid is reassurance.

the inverse health check

The natural way to monitor a service is am I getting a response? The correct way to monitor a fleet is am I getting silence where I shouldn't? Every device has an expected reporting cadence. The cloud keeps a last_seen and a cadence per device. A scheduled job emits an alert when now - last_seen > cadence * 1.5.

This sounds simple. The trick is the per-device cadence. Soil probes report every 10 minutes. Weather stations every 60. Cameras when motion fires. There is no global timeout that's right for all of them, and a single global timeout will drown you in false alerts.

shape, not series

A dashboard that plots 12,000 lines on top of each other tells you nothing. Aggregate to histograms — at this hour, how many devices reported, what's the distribution of their values, how many are missing. The headline metric of fleet health is the shape of the histogram, not any single device.

We use a "small multiples" view: one tile per region, each tile a sparkline of number of reporting devices over the last 24h. A region that should be flat suddenly dipping is the headline. The 50 individual device pages are drill-downs you almost never need.

incidents are spatial

Service incidents are temporal — between 14:02 and 14:11 things were broken. Fleet incidents are usually spatial — every device in the northeast pasture is offline. Build the incident view to reflect that. A map. A grouping by physical location. The same alert plotted as a time series will tell you "it's bad," but the map tells you "the cell tower is down."

a SLO that means something

We don't track uptime per device. We track freshness of the customer's view of their farm. SLO: 95% of dashboards loaded by a customer in a given week show data less than 15 minutes stale. That metric ties directly to a thing the customer can feel, and it survives a single device dying without firing.

A device-level uptime SLO would have us paging on every chipmunk-chewed cable. The freshness SLO lets us batch and prioritize repairs the way we actually want to.

5 min