Applied AI Engineering for Google Ads with Python

Paid search is a good place to start with Large Language Models (LLMs). Campaign management runs on text and data, which is what these models handle well. The bottleneck for PPC specialists is the number of decisions a large multi-account structure holds.

Most people start by wiring a coding agent like Claude Code, Codex, or Cursor directly to Google Ads (e.g. with MCP). Good for ad-hoc questions, fragile at scale: the same prompt yields different output across runs, tokens stack with every turn, no safety gates on bulk edits, and nothing runs unless you are typing. A Python pipeline runs the same steps the same way every time, at a fraction of the token cost.

What works at scale: AI systems in Python, running on a schedule, with every decision logged in BigQuery. One pipeline run works through a month of search terms in batches, routes the outputs to the right campaigns, and queues a Monday digest for review. SQL and models do the math, the LLM does the judgment, and the schedule runs it without you.

This article builds one of those systems end to end: search term and keyword management. Everything around it (the data layer, the pipeline shape, the evals, the review gate) is the same whether the decision is a search term, an ad headline, a product title, or a landing page. The other use cases sit at the end, including the math-driven ones (tROAS bidding, budget allocation, pLTV modeling) that swap the LLM for a statistical engine.

#1 Where to start using AI

Prioritize the tasks where volume is too high for humans to keep up. Imagine you hired a team of specialists who never tire: what repetitive judgment work would you hand them?

Not every decision needs an LLM. Threshold-based pausing, anomaly detection, bid adjustments, budget shifts, pLTV scoring, geo-test readouts: SQL and statistical models handle those cleanly, every time, with no prompt to drift. Reserve the LLM for the calls those can’t make: reading intent in a search term, deciding whether two products belong in the same theme, writing ad copy in a brand voice.

This is also where you keep control of the surface Google is wrapping in black boxes (Performance Max, Broad Match, AI Max for Search).

#2 Give the model your business context

Without context, an LLM behaves like a new PPC analyst on their first day. They know the platforms and miss the details that make their output strategically right: the success criteria, the acceptable trade-offs, the hard limits.

Context comes in two layers, and you need both.

Shared context lives in one central place: account strategy, KPIs, products, business rules, brand voice. Your coding agents read it during builds. Your pipelines pull it into prompts at runtime:

• Goals and KPIs. What each campaign is optimized against: tROAS targets, CPA limits, new-customer share, lifetime value tiers.
• Account strategy. Why certain campaigns exist, which themes are deliberate, what each campaign should and shouldn’t capture.
• Products and offers. Categories, pricing tiers, seasonal items, loss leaders, protected SKUs, what’s allowed in non-brand vs. brand.
• Business rules and guardrails. Budget splits, competitor strategy, exclusion lists, hard caps, the things an experienced operator does without thinking.
• Voice and brand. Tone for ad copy, claims you can and can’t make, terminology that has to be exact.

Task-specific context lives in the prompts that drive specific output. Each one is a Markdown file owned by its pipeline, layered on top of the shared docs at runtime. The classification prompt further down is one.

Documenting context properly is the single biggest lever on output quality. Markdown files in the repo are usually enough. For context that lives elsewhere, pull it in via the Confluence or Notion API. For context too large to fit inline, stand up a vector store and retrieve the relevant chunks at runtime. If your warehouse runs on dbt, the project doubles as agent context for free: column descriptions, lineage, and tests are already in schema.yml.

Then treat the docs like code: review, update, delete what’s stale. A stale promotional calendar, inventory list or margin target is worse than a missing one: the system will confidently execute last month’s strategy. A last_reviewed tag in each file header keeps ownership explicit, and a CI check that flags docs untouched in 90 days removes the reliance on remembering.

#3 The data foundation: BigQuery and Data Transfer

Next, LLMs need access to your performance data. The Google Ads Data Transfer lands a documented set of reports in BigQuery as daily snapshots: campaigns, ad groups, keywords, search terms, conversions, performance stats, tROAS targets, budget settings, audiences. Check the transfer’s report list before you design around it, and write a custom query for anything it does not carry.

Each report shows up as two objects. A p_ads_* table holds the history, partitioned by the day the rows landed. An ads_* view sits over the same rows and adds _DATA_DATE for the snapshot day and _LATEST_DATE for the newest one. Look up a campaign name or a keyword through the view with _DATA_DATE = _LATEST_DATE, or every entity comes back once per snapshot day and any join multiplies the stats. Aggregate performance over a date range from the p_ads_* table, filtered on segments_date.

Your coding agent queries it interactively while building, and your production pipelines read the same tables on a schedule.

It also accumulates history as it runs. Keyword statuses, paused ads, label changes and bid adjustments are all captured over time, so you can ask what a campaign’s tROAS was six months ago and how performance responded. Anything else sitting in your warehouse (revenue, margins, pLTV, forecasts, product info) feeds the same pipelines.

#4 The pattern: BigQuery → decision engine → BigQuery

A workflow built on this foundation typically takes the same shape:

Only the decision logic differs between use cases. Reach for the LLM last: every step you can keep in SQL, rules, or a model running in BigQuery is one less source of prompt variance.

# LLM-driven (search terms, ad copy, classification)data    = bq_client.fetch("get_search_terms.sql")    # pull dataprompt  = open("prompts/classify-main.md").read()    # load instructionsresults = llm_client.classify(prompt, data)          # LLM recommendsbq_client.write(results)                             # write back # Statistical or rule-based (anomaly detection, budget models)data    = bq_client.fetch("daily_spend.sql")results = detect_anomalies(data)                     # z-scores, regression, custom logicslack.send(results)                                  # notify, or write to BQ, or call an API

Sign up with Anthropic, OpenAI, or Google, grab an API key, and call the model in a few lines of Python. The work sits in the prompt, the data shape, the schema check, and what you do with the output.

A coding agent builds the staging tables, the MERGE statements and the chained pipelines from a spec, behind guards it cannot widen.

#5 Wrapping it all in a Python pipeline

A “pipeline” is a Python script that wraps a small set of components and runs them in order. Each one is small, replaceable, and testable in isolation.

Most of the substance lives outside Python: the SQL, the prompts, the schema, the API clients and the BigQuery tables. Python holds them together.

The whole thing (Python, SQL, prompts, configs) sits in a git repo. For a handful of pipelines on a simple schedule, GitHub Actions handles the daily cron, stores secrets, and keeps run logs without a separate orchestrator. Trigger any pipeline manually from the CLI for ad-hoc runs.

Move to a real orchestrator (Airflow, Prefect, Dagster) once pipelines start chaining, backfills become routine, or you need per-task observability. dbt is the default for the SQL layer, and the orchestrator just calls dbt run as one task in a larger DAG.

#6 Building it: search term and keyword management

Search term classification is a good place to start. It’s the densest decision surface in paid search: thousands of terms a month, each one a small business decision (relevant? off-target? bid on it? block it? in which campaign/ad group?) that compounds at scale.

The worked example throughout is Nike running: non-brand campaigns targeting people searching for running shoes, apparel, and accessories, with brand searches routed to dedicated brand campaigns.

How the system fits together

The classification pipeline follows the pattern from earlier:

1. BigQuery in. Pull unclassified search terms from the Data Transfer, filtered and ranked in SQL by whichever performance signal matters (spend without conversions, conversion volume, recency, account-specific thresholds). Skip data that hasn’t matured yet: a term with zero conversions yesterday can convert by the end of the week.
2. Business logic as a prompt. A Markdown file defines what to do with each term: intent scoring, audience detection, category relevance, branded vs. competitor handling, the action (ADD_KEYWORD, ADD_NEGATIVE, TO_REVIEW), match type, target campaign, target ad group.
3. Call the LLM. Send a batch with the prompt. Validate the structured output against a schema. Uncertain terms get enriched (web search, internal docs) and reclassified.
4. Decisions back to BigQuery. Write to staging, MERGE into production. Every recommendation is queryable: input, prompt hash, output, run ID, timestamp. A pinned prompt hash lets you replay any historical run and bisect exactly when a regression appeared.
5. Human review. Approved recommendations push to Google Ads as ADD_KEYWORD or ADD_NEGATIVE via the API. Rejections, with a reviewer comment, flow into the eval pipeline.

Pulling search terms from BigQuery

Step 1 in code, everything except exact match, ordered by spend so the highest-cost terms get processed first:

-- Fetch unclassified non-exact terms, ordered by spend descending.-- p_ads_SearchQueryStats: the partitioned search terms table from Data Transfer.-- search_term_classification: written by this pipeline, starts empty.SELECT  sq.search_term_view_search_term AS search_term,  SUM(sq.metrics_cost_micros) AS total_cost_microsFROM  `your-project.your_dataset.p_ads_SearchQueryStats` AS sq  LEFT JOIN `your-project.your_dataset.search_term_classification` AS cls    ON sq.search_term_view_search_term = cls.search_termWHERE  cls.search_term IS NULL                              -- not yet classified  AND sq.segments_search_term_match_type != 'EXACT'   -- exclude live exact keywords  AND sq.segments_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 90 DAY)  AND sq.segments_date <= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)   -- let conversions landGROUP BY  sq.search_term_view_search_termHAVING  total_cost_micros > 5000000                          -- >$5 spend thresholdORDER BY  total_cost_micros DESCLIMIT  100;

The LEFT JOIN against your own classification table makes the pipeline idempotent: terms already classified don’t come back. The seven-day upper bound is the maturity guard: a term that first showed up this week has not had time to convert yet. Because a classified term never comes back, an ADD_NEGATIVE written before the conversions landed is never revisited. The spend threshold keeps cost in check (Google Ads stores costs in millionths of the currency unit, so 5000000 means $5). Adjust all three to your account size.

Prompt engineering

The prompt does most of the work.

Sequential, decomposed steps. Walk each term through a numbered process, starting with a confidence check that returns UNKNOWN when the model is unsure. Hard rubrics inside each step keep it consistent across thousands of runs.

A trimmed version of the Nike running classification prompt. Production prompts run longer, with detailed category definitions and worked examples.

The STEPS 0-13 logic is reusable as-is across accounts. What you customize is the REFERENCE MATERIAL block at the bottom: products, audiences, competitors, branded terms. The one rubric worth setting per account is the ADD_KEYWORD match type in STEP 12. BROAD is a reasonable default for Smart Bidding accounts with room to grow, and tighter structures warrant PHRASE or EXACT for proven terms.

### ROLE & GOALYou receive a list of 100 search terms from the Nike Running Google Ads account.Classify each one for Nike non-brand running campaigns. For eachterm, follow the strict sequential process below using your ownknowledge in combination with the REFERENCE MATERIAL. --- ### STEP 0: Confidence Check (no exceptions) Make a binary decision: do you KNOW this term with certainty, or NOT?Flag as UNKNOWN (competitor: 99) if ANY is true:- Multiple plausible meanings and you're not 100% sure which is intended- You don't explicitly recognize the exact term as a known brand or product- Your reasoning includes "probably", "likely", "seems", "might be"- You are inferring meaning from parts rather than knowing the whole term UNKNOWN output. Stop, return only:{"search_term": "...", "competitor": 99} KNOWN terms proceed to Steps 1-10. --- ### STEPS 1-10: Classification (KNOWN terms only) 1. commercial_intent_score (1-10)   1-2: irrelevant, off-topic, no running context   3-5: informational, how-to, training tips (no buying intent)   6-8: evaluation, comparison, product research   9-10: transactional (specific product, buy, discount, near me) 2. audience: one of men, women, kids, unknown 3. competitor (0/1): does the term mention a competing brand?   [See REFERENCE MATERIAL for full competitor list] 4. score_shoes (1-10): relevance to running footwear5. score_apparel (1-10): relevance to running clothing6. score_accessories (1-10): relevance to running accessories7. score_training (1-10): relevance to training, recovery, activity 8. justification: one sentence explaining scores 1-79. language: language code (en, de, fr, nl, ...)10. branded (0/1): does the term mention Nike?    [See REFERENCE MATERIAL for full branded term list] --- ### STEP 11: Recommended Action (priority order) Exception first: comparison queries ("brand A vs brand B") → TO_REVIEWregardless of branded/competitor flags. Human judgment required. 1. Branded (branded=1)       → ADD_NEGATIVE (always: protect non-brand)2. Competitor (competitor=1) → TO_REVIEW (team decides on competitor strategy)3. Standard terms:   - commercial_intent_score >= 6 → ADD_KEYWORD   - commercial_intent_score <= 5 → ADD_NEGATIVE   - Borderline or ambiguous      → TO_REVIEW --- ### STEP 12: Match Type ADD_KEYWORD → BROADADD_NEGATIVE:  - Branded terms                               → EXACT  - How-to / instructional queries              → EXACT (avoid overblocking)  - Refined phrase keeps running-adjacent words → EXACT  - Refined phrase has no running context       → PHRASETO_REVIEW → EXACT (human decides) --- ### STEP 13: Keyword Text ADD_KEYWORD: original search term, unmodified.ADD_NEGATIVE:  - Branded: full original search term  - Non-branded: refine to the core irrelevant concept    (strip prefixes like "how to", isolate competitor brand name)  Risk check: if refined phrase contains running-adjacent words,  use the full original search term + EXACT instead. --- ### OUTPUT FORMAT Single JSON array. UNKNOWN: {"search_term": "...", "competitor": 99}KNOWN, all fields in this order: {  "search_term": "best running shoes for women 2026",  "commercial_intent_score": 8,  "audience": "women",  "competitor": 0,  "score_shoes": 10,  "score_apparel": 2,  "score_accessories": 1,  "score_training": 3,  "justification": "High-intent category search, women's audience. No brand signal.",  "language": "en",  "branded": 0,  "recommended_action": "ADD_KEYWORD",  "recommended_match_type": "BROAD",  "recommended_to_add": "best running shoes for women 2026",  "recommended_reason": "Score 8, non-brand, core product search. Broad to capture variations."} --- ### REFERENCE MATERIAL Company: Nike (non-brand running campaigns)Products: shoes (road, trail, racing flats), apparel (shorts, tights,  jackets, vests), accessories (socks, bags, hats)Audiences: men, women, kidsCompetitors: [list your direct running brand competitors]Branded terms: nike, pegasus, vaporfly, dri-fit, [your product lines]

Use the context window. A prompt carries system instructions and business logic you would otherwise put in code. Prompts grow longer than you’d expect once you’ve encoded every rubric, exception, and worked example a domain expert carries in their head. Use as much context as the task needs, but watch for attention degradation on very long prompts: critical rubrics buried in the middle of a massive prompt get less reliable than ones near the top or bottom.

Cost is rarely the constraint. The gains from tighter routing and negative coverage outpace API costs before you’ve run any optimization. When it matters: enable prompt caching from the start if your system prompt includes a large REFERENCE MATERIAL block. Anthropic’s prompt caching keeps the cached prefix token cost at roughly 10% of the normal input rate, which is material at the volumes this system implies. The Batch API cuts costs further for non-real-time workloads. Smaller models for confident cases once you have signal.

Capture as much as you can per term. Every score and flag becomes input for a downstream pipeline. Audience and category scores feed routing, commercial intent decides ADD_KEYWORD against ADD_NEGATIVE, branded and competitor flags feed negative scoping. Output the full reasoning once and query it from whichever pipeline needs it. The marginal cost of extra fields is a few hundred tokens; the downstream reuse is worth a lot more.

Batch to fit the context window. Send terms in batches of around 100 per call. The prompt is the same every batch; only the input array changes. Cuts cost and latency, keeps quality stable across runs.

Web search for the UNKNOWN cases. When the model flags a term as UNKNOWN, send it to a web search API like Tavily or Exa to enrich the context. The web result gives the model a working definition of what the term actually is, and then it reclassifies with that. Catches new brands, regional terms, and competitor mentions the model wouldn’t otherwise know.

Example output

What the classification pipeline returns after a batch of search terms for a non-branded Nike running campaign:

This row-per-term output is the audit log and the input to every downstream pipeline. Audience and category scores route each ADD_KEYWORD to the right campaign and ad group. Match type and keyword text are ready to push to the API as-is. Justifications surface why each term was classified that way (the first place to look when something’s off), and the same rows feed a Monday Slack digest, a dashboard tracking rejection rates, and a QA loop pulling the lowest-confidence justifications.

Negative scoping is the next pipeline in the chain, deciding where each ADD_NEGATIVE applies: ad group, campaign, or account. The system grows as a chain of small pipelines, each one doing a single job.

The spend > $5 filter deciding what reaches the classifier is a blunt cutoff. Cosine similarity between a search term and its matched keyword is a smarter pre-filter, and it flags loose broad-match decisions cheaply along the way. BigQuery has ML.GENERATE_EMBEDDING and VECTOR_SEARCH built in. Treat it as a later layer on top of the core pipeline.

These pipelines don’t become accurate by themselves. Evals are how you measure and improve them over time.

#7 Evals: continuous system optimization

LLMs can hallucinate and make wrong decisions. Even with good prompts, good context, and good data, the model will misclassify or ignore a rule you thought you wrote clearly.

An eval is just a measurement: take a small set of inputs where you already know the correct answer (a “gold set”), run your prompt against it, count how often the output matches. That number is your accuracy. Evals repay more time than anything else here. Hamel Husain’s writing on AI evals is the clearest material I’ve found on the practice.

Start with gold sets for regression and human review for ground truth. Add LLM-as-judge once volume outpaces manual review, pulling rejections from the approval queue as real-world signal. Calibrate the judge against human labels periodically; it drifts. The same term labeled differently across runs is the first sign it needs tightening.

1. Build a gold set. Start with 25-50 terms with known correct answers, covering your real categories: a clear keyword, a clear negative, a competitor, a branded term, an informational query, an ambiguous case. One type repeated dozens of times tells you nothing. The set grows continuously from here. Expect it to grow into the hundreds before you stop finding new failure modes.
2. Stabilize the prompt on the gold set. Run it against the same terms ten or more times before going anywhere near production. LLM outputs vary across runs, so a one-off pass at 95% can hide a flaky prompt swinging between 80% and 100%. Iterate the prompt until accuracy is high and consistent across runs. Only then move to real data.
3. Run real data. Once the prompt is stable, classify a few hundred terms from the live account, push to your review interface, approve or reject with specific notes. “Wrong” tells you nothing. “Classified as shoes but this is an apparel query” is signal.
4. Open coding. One person, the domain expert, reads every rejection and writes a short note describing what went wrong. No predefined categories. Don’t run this by committee.
5. Axial coding. Group the notes into failure modes (COMPETITOR_ERROR, CATEGORY_ERROR, MATCH_TYPE_ERROR). Count them. The most frequent one is where you start. An LLM can do the grouping.
6. Targeted experiment. Add the failing terms to a dev gold set. Copy the production prompt, never edit it directly. Change only the section that controls the failure mode. Run it.
7. Two regression gates. Dev gold set: did the change fix the failures? Prod gold set: did it break anything that was passing (including the original 25-50 terms you started with)? Both pass: merge the prompt change to production and promote the dev terms into the prod gold set. The prod gold set grows with every successful fix, permanently guarding against the failure modes you’ve already solved. Either fails: keep iterating.
8. Repeat. Early cycles catch big errors fast. Later cycles catch edge cases. At some point you stop finding new failure modes. That’s theoretical saturation, and it’s the signal the prompt is production-ready for the current data.

You do the open coding. Claude Code handles everything else: triage, the prompt diff, both regressions, the deltas, the gold set graduation, the commit. A typical loop:

You:    "Run triage on the last 40 rejections and show me the top failure mode" Claude: → reads eval_feedback (status: RECEIVED)        → classifies each rejection into a failure mode        → reports: COMPETITOR_ERROR: 18 (45%) ← top failure mode You:    "Improve competitor detection. Identify why this happens." Claude: → reads prompts/classify-main.md        → identifies Step 0 (confidence check) as the root cause        → proposes a diff: stricter UNKNOWN criteria + new examples        → eval --regression --gold-set dev  → 71% → 96%  ✓        → eval --regression --gold-set prod → 96% → 96%  ✓ no regression        "Both gates pass. Approve to merge." You:    approve Claude: → updates the prompt        → graduates dev gold set terms into prod        → commits: "tighten competitor confidence check, 71→96%"

The part you keep is deciding which failure modes matter. Evals also tell you when to stop. When new batches stop producing new failure modes, the prompt is done for now. Move on to the next decision surface.

Each term has six-plus output fields: intent, audience, category, flags, action, match type, so one aggregate accuracy score hides which of them moved. Track per-field precision and recall in your gold set. A prompt change that fixes COMPETITOR_ERROR can quietly regress MATCH_TYPE on the same terms.

#8 From recommendations to live in your account

Nothing in the warehouse changes the account until someone pushes it to the ad platform, and how you do that sets how much risk lives in the system. There are two modes worth running.

Manual review, manual application. The system writes recommendations to a review interface (Google Sheets, a custom dashboard, a Slack thread). A reviewer reads each one, approves or rejects with a comment, and applies approved changes in the Google Ads UI or via Editor. Slow, but safe, and the right starting point. Run here until you trust the system’s accuracy on real data.

Manual review, one-click push. Same review interface, same human-in-the-loop, but the approve button calls the Google Ads API and pushes the whole approved batch in one action. This is where most teams should land once accuracy is proven.

Keep a human at the gate. Even for boring operations (negatives that match a strict rule, pausing zero-conversion keywords with high spend over an extended window) a 30-second sanity check costs nothing and catches the kind of mistakes that quietly compound. The cost of one bad batch is much higher than the cost of a daily approval queue.

Connecting to the API. Use the official Google Ads Python client. Authenticate via OAuth refresh tokens. A git-ignored .env is fine for local development, production credentials belong in a secrets manager, and your coding agent’s deny list stops it reading that file. Test mutations against a test account first. Scope credentials to the minimum customer IDs each pipeline needs. Plan for partial batch failures (some mutations succeed, others fail on policy or dependencies) and respect object ordering: you can’t add a keyword to an ad group that was removed two seconds earlier.

Anomaly detection. Whichever tier you run, a separate watchdog has to monitor the account: spend over budget, CPA blowouts, bid or budget shifts that look off, mutations the system shouldn’t have made. Mix statistical detection (z-scores) with fixed rules (auto-pause if spend exceeds 2x daily budget). It runs alongside human review and catches what review misses.

#9 Building the review app

The custom dashboard is the option worth building, and it is a small job. A screen a handful of reviewers use never has to scale, so plain HTML, CSS and JavaScript with a small server behind it is enough: no framework, no build step, and the browser never holds a credential or queries the warehouse.

Give the agent the behavior rather than the fields:

A page that reads the current batch from the classifier’s export view, one row per search term with its recommended action, match type, target campaign and justification. A reviewer accepts a row or rejects it with a reason. Decisions go back to BigQuery through the server. With no backend it falls back to a built-in sample and the header says so.

Four things decide whether it keeps working when the pipeline changes.

One contract, owned by the backend. The app never imports or runs the pipeline. What crosses between them is a versioned file listing the columns the export promises, vendored into the app. A test on each side reads it: the backend checks its export still produces those columns, the app checks its fixtures against the same file. Rename a column upstream and one of the two tests fails on the next run, instead of the screen quietly rendering a blank.

Writes are MERGE statements. Approvals go to the table you implement from, rejections to the table the eval loop reads. Key the MERGE on everything that makes a row distinct. The same term can be approved for two campaigns, and one accepted keyword can ship as exact and broad, which are two different keywords in Google Ads. Leave either out of the key and the second row updates the first, so half of those decisions never reach the account.

Then have the export query exclude decided terms, so a reviewed row leaves the queue on the next pull without anything deleting it.

A refresh is SQL. The button and the daily timer re-run the export query. Neither triggers the classifier, which is a long and expensive job. Say that in the button’s own label, because a reviewer who thinks Refresh reruns the model will wait for the wrong thing.

Sample data by default. Ship a fictional batch inside the page and render it whenever there is no backend. The screen becomes shareable with anyone, and real account data stays out of every screenshot and design file.

The working method behind it, and what changes when a coding agent builds a screen rather than a pipeline, is in Agentic Coding for PPC.

#10 Engineering for production

Every pipeline in this system needs the same engineering rigor underneath it. The patterns below separate a script that runs once from a system that runs every day on a real account. Build them into the first pipeline.

Tests and CI. The first thing you wire in. Unit tests on pure functions, integration tests against a sandbox dataset, end-to-end tests against a frozen sample, all in CI. Gold sets run on every prompt change; build fails if accuracy drops. Tests verify the code is correct before it ships; a dry run verifies a specific batch is safe when you run it.

Thin API clients. Wrap every external service (LLM, web search, ad platform) in a thin client with no business logic. Swapping providers (one model for another, one search API for another) stays a one-file change instead of a refactor.

Structured output validation. Use the model’s native structured output mode (Anthropic tool schemas, OpenAI structured outputs, or a library like instructor on top of Pydantic) so the model returns valid JSON in the first place. Validate every response against the schema before it touches the warehouse. Invalid objects get logged and dropped or retried, not silently propagated.

Fail soft between clients. An enrichment failure (web search down, a 429 from a side API) shouldn’t abort the pipeline. Return empty, log it, let downstream steps continue with partial context. Failed items rerun next cycle.

Mutation safety. Every UPDATE, DELETE, or external API write goes through a protocol: scope lock (an explicit list of IDs or a fixed predicate, never an open-ended WHERE clause), pre-snapshot, pre-count, execute, post-count, validate. Warehouse writes can roll back; external API writes can’t, so store the previous state and reverse with a compensating mutation if validation fails.

Dry runs and --limit. Every pipeline ships with --dry-run (runs everything except the final write) and --limit N (caps input volume). New code, new prompt, new SQL? Dry-run first. Cheap insurance against schema mismatches, off-by-one filters, runaway API costs.

Batching, concurrency, and backoff. LLM and ad platform APIs have request-size limits, per-call latency, and throttling. Batch inputs into chunks, run batches concurrently with asyncio or a thread pool where rate limits allow, aggregate before writing. Set explicit timeouts so a hung connection never blocks the pipeline, use exponential backoff with jitter on retries, and cap total retries so failures fail loudly instead of silently looping.

Idempotency. Every pipeline must be safe to re-run. If a job crashes halfway, the next run picks up where it stopped, no duplicates. Staging tables with run IDs, MERGE statements with dedup keys, append-only audit logs.

Logging with execution IDs. Every run gets a UUID. Every log line, every BigQuery write, every API mutation includes that ID. When something breaks, you grep one ID and reconstruct the entire run.

Cost and token observability. Log token usage and dollar cost per run alongside every decision in BigQuery. Build a small dashboard or daily Slack digest, and alert when daily spend deviates meaningfully from the moving average. A prompt change or an infinite retry loop can quietly 10x the API bill, and the daily alert catches it.

Encode them once in the project rules your coding agent reads, and it writes them into every new pipeline.

#11 Where else this pattern goes

The pipeline above is one decision surface. Paid search is full of them, and it stops being just PPC quickly, since every tool you touch ships an API. A few directions:

Shopping feed optimization. Join feed data (from a feed management tool like Channable or Productsup, or straight from your PIM) against shopping performance to surface products losing impression share or missing margin targets. The LLM only enriches that subset: titles, categories, custom labels.

Ad copy automation. SQL ranks winners and losers in BigQuery (CTR, conversion rate, share by ad slot). The LLM extracts the themes behind them (hooks, CTAs, claims) and generates new headlines and descriptions against product attributes and voice rules. Candidates land in BigQuery for human review.

Landing page alignment. Rank pages by paid spend and conversions first, so the pipeline only scrapes what matters. The LLM classifies each page against the keywords and ads routing traffic to it, and flags mismatches: transactional queries on glossary pages, ads promising a discount the page doesn’t show.

Competitor monitoring. Pull competitor ads and their landing pages, overlay your own impression share and CPC trend so shifts surface where they actually hurt you. The LLM classifies positioning and messaging changes, SQL ranks which shifts matter. Manual review caps out at a handful of competitors; a pipeline scales to dozens.

Analyst agents and reporting. Scheduled insight runs that pull from BigQuery, summarize, and post to Slack or Notion. Same plumbing, except a human consumes the output instead of the account.

Analytical and ML work. The same pipelines wrap statistical and ML models: pLTV-aware bidding via Offline Conversion Import, lead scoring and audience building, budget and tROAS automation, demand forecasting, geo tests and holdouts. The math changes, the pipeline around it does not.

Each of these needs its own SQL and, where an LLM makes the call, its own prompt and gold set. Everything else in this article carries over.

#12 Alternatives: n8n, Zapier, and no-code automation

You can build a lot of what we’ve described in n8n, Zapier, Make, or similar no-code automation tools. They ship nodes for HTTP, BigQuery, OpenAI and Anthropic, scheduling, branching, retries.

The wall comes when you try to scale. Gold sets, regression gates, prompt versioning, data engineering, SQL writing and validation: each one is friction in a no-code editor, and some are flat-out impossible without dropping into a code node.

The reason no-code was attractive in the first place (the “I don’t want to write code” barrier) is mostly gone. A coding agent writes the SQL, the Python, the schema and the CI, and debugs them when they fail.

I’ve tried both. n8n is a good place to start if you’ve never wired one of these things up: seeing what a pipeline is, what a trigger does, what a structured output looks like, all useful first steps. Once you understand the shape, a real system is easier to build with code plus an agent, or with an engineer who knows what they’re doing.

#13 The bigger picture

Once you start thinking in systems, the impact compounds: each pipeline feeds the next, and each eval makes the next prompt sharper. Paid search at scale is going to be run by the people who can design, build, test, and improve these systems.

Today. You are the orchestrator: you write specs, design prompts, run evals, and decide when each piece is good enough to ship. Your coding agent builds the system, the pipelines run in production, and you validate and steer.

The boundary. It is moving. For every task you still do by hand, ask whether you can structure it clearly enough that coding agents can own the build and pipelines can own the execution: the spec, the success criteria, the eval that proves it worked.