OneShot Apps

How it fits together

OneShot is two systems. A pipeline on my own machine polls job boards into a 39 GB PostgreSQL database. Once an hour it publishes only the rows that changed to a small hosted Postgres on Neon, which is all the website reads. A heavy backfill on the pipeline side can never slow down a student's search.

1. Find boards. Mostly from Common Crawl's URL index, scanned with DuckDB: 219K candidate ATS boards, of which 112K validated and are polled.
2. Poll. A scheduler fetches each board and stores the raw response untouched.
3. Normalize. Each ATS's payload becomes one posting schema, including a structured location.
4. Enrich. Job type, skills, degree fields, and duplicate groups.
5. Publish. An hourly diff to the serving database.

Polling 112K boards without getting blocked

Every board is a row in a feed table, and that table is the queue. A worker claims the next due board with FOR UPDATE SKIP LOCKED and a lease, so two workers never poll the same board, and a crashed worker's board is picked up again when its lease expires.

Workers run in lanes: one per ATS, and one per Workday data center. I started with rounds that polled every vendor together, but a round can't end until its slowest poll does. One 10-minute Avature board left Lever idle 96% of six hours. With lanes, a slow vendor only slows itself.

Each lane's speed comes from measuring that vendor, not guessing:

• Workday's Cloudflare limit counts per data center, not per company, so each data center gets its own rate budget.
• Workable ran clean at 0.42 requests per second and started refusing at 0.45, so it's paced just under that.
• When a host returns a 429, every request to that host waits out one shared cooldown. Independent retries had been hitting the same wall and wasting 34–41% of attempts on large Workday runs.

Failing boards back off exponentially, up to a day. Three 404s in a row retire a board.

Not re-downloading what hasn't changed

Most polls find nothing new, so the work is in finding that out cheaply. Last week, 44% of 802K polls ended as "unchanged."

• Where the ATS supports ETags, an unchanged board costs one empty 304 response. Lever answers 304 about 91% of the time.
• Workday has no ETags, so the poller reads the job count on the first page and stops if it matches, with a full pass every 24 hours.
• Responses are hashed after stripping fields that change on every request (relative "posted 3 days ago" text, security nonces, server clocks), so a reshuffled page doesn't look like new data.
• Detail pages are fetched only when a listing row changed, and for Workday only when the title could be a student role. That cut Workday detail requests by 98.4%.

Closing jobs only on evidence

A job that vanishes from a listing isn't closed right away. Absence only counts if the poll saw the complete listing; a partial or failed poll proves nothing. Missing jobs go to a checker that opens the job's own link and closes it on a 404, a redirect, a "no longer accepting applications" page, or the ATS's own closed flag.

One schema from 36 ATSs

Every ATS describes jobs differently. I keep every raw response, so when a parser improves I re-parse history instead of re-fetching it. 27 JSON-based ATSs run through one config-driven parser; the rest, such as Avature, Taleo, and Workable, have their own adapters. Most of the work was in edge cases measured on real data:

• Job type. Ordered title rules decide intern, co-op, new grad, student job, or research, with word boundaries everywhere: "International" isn't an intern, "Cooper St" isn't a co-op, and in Workday URLs "Grad" is Croatian for city.
• Dates. Each posted date carries a precision (exact, day, or unknown) instead of turning "Posted today" into a made-up timestamp.
• Remote. Stored as yes, hybrid, or unknown, never "no", because many ATSs report "onsite" by default.

Location normalization

The site shows US roles, so every location has to answer one question: could this job be in the US? Across 315K distinct location strings, the answers arrive as "New York, NY", "2 Locations", "US-CA-San Francisco", facility codes, and everything in between.

The rule I held to: an unresolved location is fine, but wrongly hiding a US job is not. When the evidence is incomplete, the job stays in.

The normalizer splits the text into offices and parts, looks up every word span in GeoNames, and keeps only cities that agree with a nearby state or country. Short codes like "CA" count only when written in capitals. A dynamic program picks the reading that explains the most words, and if any word is left unexplained it drops the city rather than guess. Later fixes may add a location but never remove one. All 315K strings normalize in 31 seconds.

Raw text	Result	Why it's hard
Toronto, CA	Toronto, Canada	CA alone means California.
Franklin, LA	Franklin, Louisiana	LA is also Los Angeles.
Tbilisi, Georgia	Tbilisi, Georgia (country)	Atlanta, Georgia resolves to the state.
Remote, North Carolina, USA	North Carolina · remote	North and Carolina are both towns.
Amsterdam, Noord-Holland, Nederland	Amsterdam, Netherlands	Nederland is also a town in Texas.
Nederland	Nederland, Texas	No Dutch evidence, so the job stays in.

Two cases took the longest:

• Nederland, Texas. "Nederland" is Dutch for the Netherlands and also a town in Texas, so Dutch jobs were landing in the US feed. Dutch provinces now count only as supporting evidence within one address, never across offices. The fix moved about 15,500 Dutch postings out of the US feed without dropping any posting that named the US, and a 198-case test grid pins it down.
• Workday's "2 Locations". Workday's list view hides multi-site jobs behind a count. The real sites come from the detail page, so a job listed as "3 Locations" is still found by its US office.

Serving the site

The hourly publish compares an md5 fingerprint per row and sends only the differences, in one transaction, with IDs that stay stable across both databases. The web app is FastAPI on Vercel with plain JavaScript and no build step; its US filter is a JSON containment query on an indexed column. Resume PDFs compile in-process with Typst, and tailoring is rule-based: it shows which terms a job asks for that the resume doesn't.

What's next

• Poll intervals are fixed per board today. Busy boards should poll more often and quiet ones less, based on how often each actually changes.
• Duplicate grouping runs as a full rebuild, which competes with the pollers for the database. It should run incrementally.
• A bare city name still goes to its most populous namesake, so "Paris" alone resolves to France. Those should stay unresolved, along with loose alias matches like "Al Sila Tower" reading as Selah, Washington.