I use the OpenAI GPT-4o model for all my LLM-related work. I completely missed the release of a cheaper version of the GPT-4o model, GPT-4o-2024-08-06, which is 50% cheaper than the default GPT-4o variant (GPT-4o-2024-05-13). I strive to stay on top of pricing changes, but I still managed to miss this one. To help with any future surprises, I decided to write a Puppeteer script to notify me of such changes. Along the way, we encountered some interesting challenges, particularly with Cloudflare’s bot detection, and had to employ a range of tools to overcome them.
The Challenge
Our primary goal was straightforward: capture screenshots of the OpenAI pricing page and detect any changes over time. However, we quickly discovered that Cloudflare’s robust bot detection mechanisms were standing in our way. Cloudflare is known for its sophisticated security features designed to protect websites from malicious bots, and our initial attempts to automate screenshot capture were frequently thwarted by these defenses.
Continue reading “Bypassing Cloudflare Bot Detection: How We Automated Change Detection on OpenAI’s Pricing Page”