Every time I open LinkedIn, Instagram or TikTok, there's another post telling people to fire their SEO agency. Claude did this. ChatGPT did that. Usually it's a sped-up screen recording of an AI tool “doing SEO” in the background.
I went back on Edward Sturm's show to test that claim properly. I took one real page, gave the exact same prompt to ChatGPT, Claude and Gemini, and went through every recommendation they gave back.
This page is my write-up of that episode. I'll walk through what each tool said, where they agreed, where they were wrong, and how I actually use AI in my own SEO work.
The short version: AI can speed up an SEO who knows what they're doing. It can't replace one. The recommendations aren't always wrong. They're just in the wrong order, and if you're new to SEO, you have no way to tell.
The test: one page, one prompt, three LLMs
I used Nike's men's running shoes page on the Canadian site. It's a big brand with a strong technical foundation, which makes it a fair test. Any tool can find problems on a broken site.
The prompt was deliberately simple, the kind of thing a business owner would type:
Give me SEO and AEO/GEO recommendations for this page. Prioritize them based on impact.
I turned memory off and used temporary chats, so none of the tools knew I work in SEO. Each one saw the same page and the same question.
Here's what I was looking for:
- Overlap: which recommendations showed up across all three.
- Priority: whether the “high impact” items would actually move rankings.
- Usability: whether a beginner could act on the advice, or would just get lost.
ChatGPT: a long list, in the wrong order
ChatGPT opened with a fair summary. It said the page has a solid technical foundation, and the opportunity is to become a better shopping resource, not just a product listing. I agree with that.
Then it gave me a long list of “very high” and “high” priorities. A few were good. The order was the problem.
What it got right
A shoe comparison and selection guide. ChatGPT picked up that Nike leans on brand language like “soft landings” and “responsive strides”. That copy sounds nice, but it doesn't help a shopper choose between shoes. A “find the right running shoe” section would help. From a UX point of view, I like this. Whether it moves SEO or AI visibility much, nobody really knows.
Intent-based category links. It suggested a visible “shop by running goal” section that links to related categories. I agree with this one. When we compared Nike, Adidas and New Balance in an earlier episode, New Balance was outranking both. It wasn't the only reason, but New Balance linked internally really well from its category pages.
Where it went wrong
It gave me homework instead of answers. One “very high” item was to validate crawlability, rendering and indexation on the live site. It had the HTML. If something was wrong, it should have said so.
Edward made a good point here: if everything is already in the HTML, why send someone to Search Console to check? If there's a JavaScript problem, the tool should just say “too much JavaScript”. As you'll see, Claude did exactly that.
FAQs were ranked as a high priority. I'd add FAQs to this page for users. Most people don't know the difference between a Pegasus and a Vomero, and I didn't either. But as an SEO priority? No. There are plenty of things on Nike's site that should come first. More on why every LLM pushes FAQs below.
The title tag was sixth, and the H1 wasn't mentioned at all. The title tag is the first thing a searcher sees. If you're new to SEO, the title tag, H1 and H2s are where you should start, because they're quick to fix and they matter.
Localization was seventh. It should have been near the top. This is the Canadian site, but the page sends UK signals. The HTML says en-CA, the content-language meta tag says en-gb, and there's no hreflang. The title tag is “Men's Running Shoes & Trainers”. Canadians don't say trainers. That's a UK term. In Canada and the US, people say sneakers. If you sell into more than one market, your pages need to use the words people in that market actually search with.
This matters more for AI search too. ChatGPT already gets localization wrong a lot. People see pages from the wrong country in their answers, partly because hreflang isn't being respected, or the wrong site is pulled in. Clean international signals help.
Good ideas were buried in “medium”. It suggested adding contextual links from Nike's long block of category copy at the bottom of the page. New Balance does this well. I'd move this up. Meanwhile, Open Graph and Twitter images made the list, which isn't really an SEO item.
It never asked for more of the site. I gave it one page. It never asked for more pages or how healthy the rest of the site is. You could implement every tip perfectly, and if the rest of your site is in bad shape, this page would still struggle.
It finished with “if you could only do three things, do these three”. They weren't the three that would make the biggest difference.
Claude: the best of the three, but still not ready to ship
Claude went straight into analysis, and its top priority was the one I'd have picked: the local signals and the missing hreflang cluster. If you have a page built for Canada that sends UK signals, you're targeting the wrong keywords. Fix that first.
Then it found something ChatGPT missed. Over half the products aren't crawlable from the page. The header says 45 products, but only 24 are in the HTML. The rest load with JavaScript. That matters for Google, and it matters even more for AI crawlers, which often don't render JavaScript at all. Same prompt, and Claude found a bigger issue.
Where Claude fell short:
- The lazy-loaded LCP image was ranked high. It's worth a look, but it isn't going to change much.
- Product structured data was ranked high. On a product listing page, you do want it, because it can improve your click-through rate. But it won't make you rank better. Get the technical foundations and relevance right first. Once you're on page one and a competitor above you has rich results, then add it.
- It was overconfident about FAQ markup. It said that even where Google doesn't show FAQ rich results, the format still gets pulled into AI answers. Nobody has shown that clearly. It's repeating what the internet says.
- A big issue was ninth. Footer links on the Canadian site point to the UK site. If you're shopping in Canada, you don't want to end up there. That's a UX problem and an SEO problem, and it should have been near the top.
- More homework. “Check AI crawler access in your robots.txt” is fine advice for an SEO. A business owner now has to go read an SEO 101 guide to understand what that means.
So Claude gave the best answer, but you'd still need someone who knows SEO to pick out what matters.
Gemini: an SEO checklist from another decade
Gemini's first recommendation, a category content block, overlapped with ChatGPT's. After that it fell apart.
Its number two high-impact recommendation was to optimize the meta description for keywords. Its number three was to remove meta keywords. Neither of those is going to move your rankings. Its AEO and GEO section suggested breadcrumb schema and FAQ content with schema, which won't have a big impact either.
Edward and I agreed on this one. Google has more data than anyone, so Gemini should be the best at this. Its output was the weakest by far.
Why every LLM tells you to add FAQ schema
All three tools pushed FAQs or FAQ schema. That isn't a coincidence.
LLMs are trained on years of content from across the internet. For a long time, SEOs said FAQs were important. Google supported FAQ schema, showed FAQ rich results, and everyone added them to their pages. Google stopped showing FAQ rich results for most sites in 2023, but all that older content is still out there.
Then AI search arrived, and FAQs and FAQ schema became the go-to AEO tip. You'll see it on big SEO blogs, AEO vendor sites and in viral posts everywhere. Edward called it one of the most repeated bits of bad advice from SEO influencers. So the models learned it, and now they repeat it back confidently. (It sits right alongside the other AEO and GEO myths we need to leave behind.)
It's spreading to executives too. I had a client message me on Slack this week asking if adding FAQ schema would help them show up in Google's AI Overviews. People outside SEO are seeing this content, believing it, and asking for it.
A few years ago at MozCon, Lily Ray gave a talk about the problem with SEO checklists. That's what these LLM answers felt like: a checklist, with no sense of what matters for this page, this business or this market.
The real cost is the rabbit holes
The biggest risk isn't that the advice is wrong. It's that it eats your time.
Edward shared a story from the night before we recorded. He wanted to add one field to a contact form in Zapier and an AI step to sort replies. He thought it would take 30 minutes. It took three and a half hours, and he was up until 2 a.m.
That's what “validate your crawlability” or “check your robots.txt” does to a beginner. It looks like a small check. If you're inexperienced, you'll spend four, six, eight hours learning what everything means and wondering if each thing is a problem. By the end, you're too tired to do the things that actually make a difference.
The whole point of using AI is to work faster. If it sends you down rabbit holes, it's doing the opposite.
What to measure instead
ChatGPT also told me how to measure success: impressions and clicks for queries like “men's running shoes Canada”. With more zero-click answers, I'd expect impressions to go up. Whether clicks do is less certain.
Edward's view is simpler, and I agree with it. The KPI that matters most is your rankings for the keywords you're actively targeting. If you're targeting a keyword, it's because it's valuable to you. Group those keywords in a way that's useful (by subfolder or intent), check search volume, and use clicks as a sanity check. You don't need much more than that.
How I actually use AI for SEO
I use LLMs every day to speed up my work. The difference is that I know which recommendations to keep and which to ignore. If you don't have that experience yet, here's how to get much better answers.
1. Tell it to follow the 80/20 rule
SEO follows the Pareto principle like everything else: 20% of the work drives 80% of the results. LLMs give you everything at once, which is overwhelming, and that's when beginners give up.
Edward's fix: tell the LLM to keep it short and leave out everything else. Here's a prompt you can paste in:
Act like an experienced SEO, not a generic checklist. Give me no more than five recommendations for this page. Only include the 20% of changes that will drive 80% of the results, and leave everything else out. Put what searchers see first at the top: title tag, H1 and localization. Don't tell me to "check" or "validate" something. If you can see the problem, tell me what it is and how to fix it. Ask me for more pages if you need to judge the rest of the site.
2. Give it real audit data, not one page
This is the tip I'd start with. Run your site through Screaming Frog. The free version crawls up to 500 URLs, which is enough for most small businesses. (If you've never done it, here's how to do an SEO audit for free.)
Then:
- Go to the Issues tab and export the issues list.
- Upload that file to Claude (or your LLM of choice) with the prompt above.
- Ask it to prioritize recommendations based on the issues Screaming Frog found.
Now the LLM is working with real audit data from your whole site, not guessing from one page. The recommendations get a lot better. If you want a ready-made version, use my Screaming Frog issues prioritizer prompt. This is the kind of thing I share in my short videos on fixing your SEO with Claude.
3. Fix what searchers see first
If you do nothing else, start with what a searcher sees: your title tag, your H1 and, if you sell into more than one country, your localization. Use the words people in each market actually search with. Then fix crawlability. Schema, FAQs and meta descriptions come later.
4. Use Claude, and check its work
In this test, Claude gave the most useful answer and was the only one to catch the JavaScript problem. ChatGPT was middle of the pack. Gemini was the weakest. But even Claude's list needed an experienced SEO to reorder it. There are more prompts built this way in my AI prompt library.
Ignore the “fire your SEO agency” posts
Anyone telling you AI will replace SEOs, or that you can fire your agency and do it all in Claude, ChatGPT or Gemini, is doing it for the views. As this test shows, there's a lot more to SEO than a list of recommendations.
Edward pointed out that a lot of these posts are gamed. He runs an automation that tracks viral SEO posts on X. Every few days, another “fire your SEO agency” thread shows up, boosted by friends with big accounts, and often written by AI.
Enterprise teams can see through it. Small business owners often can't. When a post has hundreds of reposts, it's easy to assume the advice is true. It won't help you grow. Get your SEO advice from people with real experience.
If you want an experienced SEO to sort through your AI recommendations, or to tell you what actually matters on your site, let's talk.
Thanks to Edward Sturm for having me back on the show.