Bypassing Cloudflare Bot Detection: How We Automated Change Detection on OpenAI’s Pricing Page

I use the OpenAI GPT-4o model for all my LLM-related work. I completely missed the release of a cheaper version of the GPT-4o model, GPT-4o-2024-08-06, which is 50% cheaper than the default GPT-4o variant (GPT-4o-2024-05-13). I strive to stay on top of pricing changes, but I still managed to miss this one. To help with any future surprises, I decided to write a Puppeteer script to notify me of such changes. Along the way, we encountered some interesting challenges, particularly with Cloudflare’s bot detection, and had to employ a range of tools to overcome them.

The Challenge

Our primary goal was straightforward: capture screenshots of the OpenAI pricing page and detect any changes over time. However, we quickly discovered that Cloudflare’s robust bot detection mechanisms were standing in our way. Cloudflare is known for its sophisticated security features designed to protect websites from malicious bots, and our initial attempts to automate screenshot capture were frequently thwarted by these defenses.

Embracing Puppeteer and Its Plugins

To tackle this problem, we turned to Puppeteer, a popular Node.js library for controlling headless Chrome or Chromium. Puppeteer provides a rich API for interacting with web pages programmatically, which is ideal for tasks like automated screenshot capture. However, Cloudflare’s defenses meant we needed to use Puppeteer’s plugins to help us navigate these challenges.

We began by incorporating the puppeteer-extra-plugin-stealth plugin. This plugin helps Puppeteer scripts evade detection by making them less recognizable as automated bots. It does this by modifying the user agent, adding browser-like features, and more, to mimic a real user’s browsing behavior more closely.

import puppeteer from 'puppeteer-extra';
import StealthPlugin from 'puppeteer-extra-plugin-stealth';
import path from 'path';
import fs from 'fs';
import { PNG } from 'pngjs';
import pixelmatch from 'pixelmatch';
import { connect } from 'puppeteer-real-browser';

// Use Stealth plugin
puppeteer.use(StealthPlugin());

Going Real with puppeteer-real-browser

Despite the improvements from the Stealth plugin, we found that Cloudflare’s protection was still quite stringent. To further enhance our script’s capability, we decided to use the puppeteer-real-browser package. This package is designed to bypass puppeteer’s bot-detecting captchas such as Cloudflare. It acts like a real browser and can be managed with puppeteer.

const realBrowserOption = {
    args: ["--start-maximized"],
    turnstile: true,
    headless: false,
    customConfig: {},
    connectOption: {
        defaultViewport: null
    },
};
const { browser, page } = await connect(realBrowserOption);

await page.goto(pageData.url, { waitUntil: 'networkidle2' });

const lastImage = getLastImage();

await page.screenshot({ path: todayImage, fullPage: true });

const isImageSame = compareImages(todayImage, lastImage);

if (!isImageSame) {
    console.log("Page has changed");
} else {
    console.log("Page has not changed.");
}

Image Comparison for Change Detection

With the technical hurdles addressed, we moved on to implementing the core functionality: detecting changes. We captured screenshots of the OpenAI pricing page and compared them over time. For this, we used the pixelmatch library to compare images and identify differences.

const compareImages = (imgPath1, imgPath2) => {
    const img1 = PNG.sync.read(fs.readFileSync(imgPath1));
    const img2 = PNG.sync.read(fs.readFileSync(imgPath2));

    const { width, height } = img1;
    const diff = new PNG({ width, height });

    try {
        const numDiffPixels = pixelmatch(img1.data, img2.data, diff.data, width, height, { threshold: 0.1 });
        return numDiffPixels === 0;
    } catch (error) {
        return false;
    }
};

Summarizing changes with LLM

I also tried comparing the two images using OpenAI gpt-4o model but it didn’t work as expected. So, I parked it for future.

Running in a scheduled manner with GitHub Actions

Finally, we wrapped it in a GitHub Action workflow to run daily. Since we are using Puppeteer in non-headless manner we had to install xvfb. xvfb is a virual display server for emulating a graphical environment.

name: Check Page Change

on:
  push:
  schedule:
    - cron: '30 5 * * *'
jobs:
  check-page-change:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20.x'

      - name: Install dependencies
        run: npm install

      - name: Install xvfb
        run: sudo apt-get install -y xvfb


      - name: Run Puppeteer with xvfb
        env:
          DOWNLOAD_DIR: './images'
        run: |
          xvfb-run -a node screenshot.js