Overview

The AI A/B Testing Engine is a Next.js + Prisma service that plugs into this site via a single <script> tag. It lets an admin select any paragraph or heading on a lesson page, generate an AI variant (text rewrite, Mermaid diagram, or interactive GameEngine level), and launch a 50/50 split test — all without touching the page source.

Engine repo: magic005/ai_ab_test_engine
Deployed at: https://ai-ab-test-engine.vercel.app

How It Works

┌─────────────────────────────────────────────────────────┐
│                   OCS Jekyll Site                       │
│                                                         │
│  <script src=".../sdk.js"                              │
│    data-project-id="..."                               │
│    data-site-baseurl="/">                              │
│                                                         │
│  SDK runs on every page load:                           │
│    1. Assigns user to A or B group (deterministic hash) │
│    2. Fetches active tests for this page                │
│    3. Applies variant (text / diagram / game) to DOM    │
│    4. Tracks view + click conversions                   │
└─────────────────────────┬───────────────────────────────┘
                          │ fetch /api/tests + /api/events
                          ▼
┌─────────────────────────────────────────────────────────┐
│           AI A/B Engine (Vercel / Next.js)              │
│                                                         │
│  /api/tests       → returns active tests for project    │
│  /api/generate    → LLM generates variant content       │
│  /api/admin/tests → admin creates / reads tests         │
│  /api/events      → records view/click events           │
│                                                         │
│  DB: Neon PostgreSQL (Prisma ORM)                       │
└─────────────────────────────────────────────────────────┘

Variant Types

1. Text Variant

The most basic variant. The LLM (llama-3.1-8b-instant) rewrites the selected element with a punchier or simplified version.
Stored as plain text in the database.

2. Mermaid Diagram Variant

The selected paragraph is converted into a Mermaid flowchart by llama-3.3-70b-versatile. The SDK swaps the paragraph for a rendered <div class="mermaid"> block.
Stored with MERMAID: prefix.

The LLM extracts 2–4 key concepts from the selected text and outputs structured JSON. The server assembles a valid GameEnginev1.1 ES module — with NPC characters that each teach one concept via dialogue — and stores it with a GAME: prefix.

When a user lands in the B group, the SDK:

  1. Hides the original element
  2. Imports GameExecutor from /assets/js/pages/runners/index.js (this site)
  3. Injects a Play/Pause/Stop/Fullscreen runner UI
  4. Passes the generated level code to GameExecutor.run()

Players walk around with WASD and press E near NPCs to learn the concepts.

Generated Game Level Structure

The API generates a complete GameEnginev1.1 module. Here is a simplified example from a data structures lesson:

import GameControl from '/assets/js/GameEnginev1.1/essentials/GameControl.js';
import GameEnvBackground from '/assets/js/GameEnginev1.1/essentials/GameEnvBackground.js';
import Player from '/assets/js/GameEnginev1.1/essentials/Player.js';
import Npc from '/assets/js/GameEnginev1.1/essentials/Npc.js';

class DataStructureExplorer {
  constructor(gameEnv) {
    const { path, innerWidth: width, innerHeight: height } = gameEnv;
    this.classes = [
      { class: GameEnvBackground, data: { src: path + '/images/gamebuilder/bg/clouds.jpg' } },
      { class: Player, data: { src: path + '/images/gamify/chillguy.png', /* ... */ } },
      { class: Npc, data: {
          id: 'ArrayList',
          src: path + '/images/gamify/r2_idle.png',
          INIT_POSITION: { x: width * 0.25, y: height * 0.5 },
          dialogues: [
            'Dynamic resizing is supported.',
            'Use ArrayList for indexed access.',
          ],
          reaction: function() { if (this.dialogueSystem) this.showReactionDialogue(); },
      }},
      // ... more NPCs for each concept
    ];
  }
}

export const gameLevelClasses = [DataStructureExplorer];
export { GameControl };

Key design decision: the LLM only fills in content (NPC names, dialogue lines, positions). The structural boilerplate is assembled server-side from a verified template, so the output is always a valid ES module.

Admin Overlay

Append ?ab_admin=true to any page URL to activate the admin panel.

Steps:

  1. Click Select Element → hover over any paragraph/heading
  2. Click the element to lock it in
  3. Click Generate AI Variants, Generate as Diagram, or Generate as Game
  4. Review the generated content in the panel
  5. Click Launch 50/50 Test — the test is live immediately

The overlay is served from the engine at /overlay.js and injected by the SDK dynamically — no changes to individual pages required.

Jekyll Integration

The only change needed in this repo is one line in _includes/head-custom.html:

<script src="https://ai-ab-test-engine.vercel.app/sdk.js"
  data-site-baseurl=""
  data-project-id="cmnfour780000f63gu69kg2v8"></script>

The data-site-baseurl attribute (rendered by Liquid at build time) lets the SDK resolve asset paths correctly on GitHub Pages — critical for loading GameEngine sprites like /images/gamify/chillguy.png.

%%html
<!-- Live Demo: visit any lesson page with ?ab_admin=true to try the overlay -->
<div style="background:#0f172a;border:1px solid #1e293b;border-radius:10px;padding:20px;font-family:monospace;color:#94a3b8;">
  <div style="color:#38bdf8;font-size:13px;margin-bottom:12px;">▶ Try it on any lesson page</div>
  <div style="display:flex;gap:10px;flex-wrap:wrap;">
    <a href="/LessonMethodology?ab_admin=true"
       style="background:#1e40af;color:#fff;padding:8px 16px;border-radius:6px;text-decoration:none;font-size:13px;">
      Lesson Methodology
    </a>
    <a href="/csp?ab_admin=true"
       style="background:#1e40af;color:#fff;padding:8px 16px;border-radius:6px;text-decoration:none;font-size:13px;">
      CSP Lessons
    </a>
  </div>
  <div style="margin-top:16px;font-size:12px;color:#475569;">
    Select any paragraph → Generate as Game → Launch 50/50 Test
  </div>
</div>