How to Compare Screaming Frog Crawls and Track SEO Changes

Comparing two Screaming Frog crawls is useful when you need to see what changed after a release, migration, content update, or technical fix. The comparison becomes more useful when you reduce the output to meaningful URL-level differences, assign owners, and keep the final implementation history after the crawl files are archived.

TL;DR

• Run the two crawls with the same scope and configuration wherever possible.
• Save both crawl files or export the same tabs and columns from each crawl.
• Use Screaming Frog’s comparison workflow for the fastest in-app review.
• Use Excel or Google Sheets when the dataset is manageable and the team needs a shareable comparison.
• Use Python when the site is large, the process repeats, or you need custom normalization and filtering.
• Compare status codes, redirects, indexability, titles, descriptions, headings, canonicals, schema, internal links, and selected content.
• Separate expected changes from regressions before creating tasks.
• Remember that two crawl snapshots cannot show a change that appeared and disappeared between crawl dates.
• Use SEO Logbook when crawl findings, intentional work, monitoring detections, tasks, and later outcomes need to stay connected to each URL.

Make the two crawls comparable

The quality of the comparison depends on how similar the crawl conditions are.

Before running the second crawl, check:

• Crawl mode
• Starting URL or URL list
• Subdomain and subfolder rules
• Include and exclude patterns
• User-agent
• Rendering mode
• Robots.txt settings
• Canonical handling
• URL-parameter rules
• Authentication
• Custom extraction
• API integrations
• Crawl limits
• Saved configuration

If one crawl uses HTML-only rendering and the other uses JavaScript rendering, the comparison may show thousands of differences caused by the setup rather than the website.

The same problem appears when:

• One crawl includes subdomains and the other does not
• One crawl respects robots.txt and the other ignores it
• Query parameters are handled differently
• One crawl starts from the homepage and the other from a sitemap
• The site served a staging, security, cookie, or geolocation response during one crawl
• GSC or analytics integrations used different date ranges

Decide which comparison you are running

Not every crawl comparison has the same purpose.

Define the purpose before exporting data.

A migration comparison may prioritize redirects, final status, canonicals, indexability, and internal links. A content refresh may prioritize titles, H1s, word counts, internal links, sources, and schema.

Save both crawls instead of relying only on CSV exports

Screaming Frog SEO Spider can preserve crawl data for later review and comparison. Saving the crawl is preferable when you may need to revisit tabs or filters that were not included in the original export.

Use consistent names:

client-domain_2026-07-01_pre-release
client-domain_2026-07-08_post-release

For agencies, also store:

• Client
• Project
• Environment
• Crawl owner
• Related release or ticket
• Known limitations
• Configuration file
• Export folder
• Review status

A simple folder structure:

/crawls
  /client-a
    /2026-07-01-pre-release
      crawl-file
      configuration
      notes.md
      exports/
    /2026-07-08-post-release
      crawl-file
      configuration
      notes.md
      exports/

Do not overwrite the earlier crawl after running the new one.

Compare the crawls inside Screaming Frog first

The in-app comparison is usually the fastest starting point.

A practical process:

1. Open Screaming Frog.
2. Switch to the crawl comparison workflow.
3. Load the previous crawl.
4. Load the current crawl.
5. Confirm that the selected crawls use comparable scope and settings.
6. Run the comparison.
7. Review changed, new, and missing URLs.
8. Filter the result by the element relevant to the project.
9. Export only the differences that need review.

Focus on questions such as:

• Which URLs are new?
• Which URLs disappeared?
• Which status codes changed?
• Which redirects changed destination?
• Which pages became non-indexable?
• Which canonicals changed?
• Which titles, descriptions, or H1s changed?
• Which schema types or validation states changed?
• Which internal-link counts changed materially?
• Which pages changed outside the approved scope?

The exact tabs and labels can vary by Screaming Frog version and the data available in the crawl. Review the current interface rather than assuming an older tutorial matches every label.

Review new, missing, and changed URLs separately

These groups require different decisions.

New URLs

A URL can appear only in the current crawl because:

• It was intentionally published
• New internal links exposed it
• A sitemap changed
• Parameters or faceted URLs became crawlable
• A redirect created a new destination
• The crawler configuration changed
• The site generated duplicate or unwanted URLs

Check:

• Is the page intentional?
• Should it be indexable?
• Is the canonical correct?
• Is it linked from the right places?
• Should it appear in the sitemap?
• Is it part of the approved release?

Missing URLs

A URL can exist only in the previous crawl because:

• It was removed
• It now redirects
• Internal links no longer expose it
• Robots rules changed
• The crawl scope changed
• The server failed during the crawl
• A sitemap stopped listing it

Check the URL directly before reporting it as removed.

Changed URLs

A URL exists in both crawls, but one or more fields differ.

Classify the difference as:

• Expected
• Unexpected
• Partially expected
• Harmless
• Needs investigation
• Crawl configuration difference
• Temporary fetch problem

Compare the signals that can create real SEO risk

Use a focused field list.

Do not treat every length or word-count difference as an SEO problem. A footer, recommendation widget, cookie notice, or dynamic block can change the numbers.

Export the same columns from both crawls

For an external comparison, export the same tab, filter, and columns from each crawl.

A practical internal HTML comparison may include:

Address
Content Type
Status Code
Status
Indexability
Indexability Status
Title 1
Title 1 Length
Meta Description 1
Meta Description 1 Length
H1-1
H1-1 Length
Canonical Link Element 1
Meta Robots 1
X-Robots-Tag 1
Structured Data Types
Word Count
Crawl Depth
Unique Inlinks

Column names can vary by version, configuration, and export.

Before comparing:

1. Confirm the URL column exists in both files.
2. Remove summary or blank rows.
3. Normalize URLs where appropriate.
4. Confirm numeric columns were not imported as text.
5. Check encoding.
6. Keep a copy of the raw exports.
7. Use a separate file for the comparison output.

Compare crawl exports in Excel with XLOOKUP

Excel works well when the exports fit comfortably in a workbook and the comparison does not need complex normalization.

Assume:

• Previous contains the earlier crawl
• Current contains the later crawl
• Column A is Address
• Column B is Status Code
• Column C is Title 1
• Column D is Meta Description 1
• Column E is H1-1
• Column F is Canonical Link Element 1
• Column G is Indexability

In the Current sheet, return the previous title:

=XLOOKUP(A2,Previous!$A:$A,Previous!$C:$C,"URL not in previous crawl")

Return the previous canonical:

=XLOOKUP(A2,Previous!$A:$A,Previous!$F:$F,"URL not in previous crawl")

Flag a title change:

=IF(C2=XLOOKUP(A2,Previous!$A:$A,Previous!$C:$C,""),"Same","Changed")

Flag any major change:

=IF(
  OR(
    B2<>XLOOKUP(A2,Previous!$A:$A,Previous!$B:$B,""),
    C2<>XLOOKUP(A2,Previous!$A:$A,Previous!$C:$C,""),
    E2<>XLOOKUP(A2,Previous!$A:$A,Previous!$E:$E,""),
    F2<>XLOOKUP(A2,Previous!$A:$A,Previous!$F:$F,""),
    G2<>XLOOKUP(A2,Previous!$A:$A,Previous!$G:$G,"")
  ),
  "Review",
  "Same"
)

Find URLs missing from the current crawl from inside Previous:

=IF(COUNTIF(Current!$A:$A,A2)=0,"Missing from current crawl","Present")

Use filters and conditional formatting on the review columns.

Compare crawl exports in Google Sheets

Google Sheets works for smaller exports and collaborative review.

Return the current status beside a previous URL:

=XLOOKUP(A2,Current!A:A,Current!B:B,"Missing")

For accounts without XLOOKUP, use:

=IFERROR(VLOOKUP(A2,Current!A:G,2,FALSE),"Missing")

Create a compact review message:

=TEXTJOIN(
  " | ",
  TRUE,
  IF(B2<>XLOOKUP(A2,Previous!A:A,Previous!B:B,""),"Status changed",""),
  IF(C2<>XLOOKUP(A2,Previous!A:A,Previous!C:C,""),"Title changed",""),
  IF(E2<>XLOOKUP(A2,Previous!A:A,Previous!E:E,""),"H1 changed",""),
  IF(F2<>XLOOKUP(A2,Previous!A:A,Previous!F:F,""),"Canonical changed",""),
  IF(G2<>XLOOKUP(A2,Previous!A:A,Previous!G:G,""),"Indexability changed","")
)

Use one review sheet with:

• URL
• Previous values
• Current values
• Changed fields
• Expected?
• Severity
• Owner
• Task
• Resolution

Avoid sharing the raw crawl tabs as the final review file. Most reviewers need the differences, not every column from both exports.

Use Python for repeatable and larger comparisons

Python is useful when:

• The process repeats
• The exports are large
• The team needs custom rules
• URL normalization is required
• Several fields need comparison
• The output should feed another system
• Crawl comparisons are part of a release process

The following script:

• Loads two Screaming Frog CSV exports
• Detects common column-name variations
• Normalizes selected values
• Performs an outer merge
• Identifies new, missing, and changed URLs
• Creates a readable changed-fields column
• Exports the result

from __future__ import annotations

import re
from pathlib import Path
from typing import Iterable
from urllib.parse import urlsplit, urlunsplit

import pandas as pd

PREVIOUS_FILE = Path("previous-crawl.csv")
CURRENT_FILE = Path("current-crawl.csv")
OUTPUT_FILE = Path("crawl-comparison.csv")

COLUMN_CANDIDATES = {
    "url": ["Address", "URL"],
    "status_code": ["Status Code"],
    "indexability": ["Indexability"],
    "indexability_status": ["Indexability Status"],
    "title": ["Title 1", "Title"],
    "meta_description": ["Meta Description 1", "Meta Description"],
    "h1": ["H1-1", "H1 1", "H1"],
    "canonical": [
        "Canonical Link Element 1",
        "Canonical Link Element",
        "Canonical",
    ],
    "meta_robots": ["Meta Robots 1", "Meta Robots"],
    "x_robots_tag": ["X-Robots-Tag 1", "X-Robots-Tag"],
    "schema_types": ["Structured Data Types", "Schema Types"],
    "word_count": ["Word Count"],
    "crawl_depth": ["Crawl Depth"],
    "unique_inlinks": ["Unique Inlinks"],
}


def find_column(
    columns: Iterable[str],
    candidates: list[str],
) -> str | None:
    available = {str(column).strip(): str(column) for column in columns}

    for candidate in candidates:
        if candidate in available:
            return available[candidate]

    lowered = {
        str(column).strip().lower(): str(column)
        for column in columns
    }

    for candidate in candidates:
        match = lowered.get(candidate.lower())
        if match:
            return match

    return None


def normalize_url(value: object) -> str:
    text = str(value or "").strip()

    if not text:
        return ""

    parts = urlsplit(text)

    scheme = parts.scheme.lower()
    hostname = (parts.hostname or "").lower()

    netloc = hostname
    if parts.port:
        netloc = f"{hostname}:{parts.port}"

    path = re.sub(r"/{2,}", "/", parts.path or "/")

    return urlunsplit(
        (
            scheme,
            netloc,
            path,
            parts.query,
            "",
        )
    )


def normalize_text(value: object) -> str:
    if pd.isna(value):
        return ""

    return " ".join(str(value).split())


def prepare_export(path: Path) -> pd.DataFrame:
    frame = pd.read_csv(
        path,
        dtype=str,
        keep_default_na=False,
        low_memory=False,
    )

    selected: dict[str, pd.Series] = {}

    for standard_name, candidates in COLUMN_CANDIDATES.items():
        source_column = find_column(frame.columns, candidates)

        if source_column is None:
            continue

        selected[standard_name] = frame[source_column]

    if "url" not in selected:
        raise ValueError(
            f"No URL column found in {path}. "
            f"Available columns: {list(frame.columns)}"
        )

    output = pd.DataFrame(selected)
    output["url"] = output["url"].map(normalize_url)

    for column in output.columns:
        if column != "url":
            output[column] = output[column].map(normalize_text)

    output = output[output["url"] != ""]
    output = output.drop_duplicates(subset=["url"], keep="first")

    return output


def build_comparison(
    previous: pd.DataFrame,
    current: pd.DataFrame,
) -> pd.DataFrame:
    comparison = previous.merge(
        current,
        on="url",
        how="outer",
        suffixes=("_before", "_after"),
        indicator=True,
    )

    comparison["url_state"] = comparison["_merge"].map(
        {
            "left_only": "Missing from current crawl",
            "right_only": "New in current crawl",
            "both": "Present in both",
        }
    )

    shared_fields = sorted(
        (set(previous.columns) & set(current.columns)) - {"url"}
    )

    changed_fields: list[str] = []

    for _, row in comparison.iterrows():
        if row["_merge"] != "both":
            changed_fields.append("")
            continue

        row_changes = []

        for field in shared_fields:
            before = normalize_text(row.get(f"{field}_before", ""))
            after = normalize_text(row.get(f"{field}_after", ""))

            if before != after:
                row_changes.append(field)

        changed_fields.append(", ".join(row_changes))

    comparison["changed_fields"] = changed_fields
    comparison["has_changes"] = (
        (comparison["_merge"] != "both")
        | (comparison["changed_fields"] != "")
    )

    return comparison.drop(columns=["_merge"])


def main() -> None:
    previous = prepare_export(PREVIOUS_FILE)
    current = prepare_export(CURRENT_FILE)

    comparison = build_comparison(previous, current)

    comparison = comparison[comparison["has_changes"]].copy()
    comparison = comparison.sort_values(
        by=["url_state", "url"],
        kind="stable",
    )

    comparison.to_csv(OUTPUT_FILE, index=False)

    print(f"Previous URLs: {len(previous):,}")
    print(f"Current URLs: {len(current):,}")
    print(f"Rows requiring review: {len(comparison):,}")
    print(f"Saved: {OUTPUT_FILE.resolve()}")


if __name__ == "__main__":
    main()

Install pandas:

python -m pip install pandas

Run:

python compare_crawls.py

The output keeps the before and after columns and adds:

• url_state
• changed_fields
• has_changes

Review the normalization rules before using the script on a site where fragments, query parameters, uppercase paths, ports, or duplicate URL forms matter.

Create one row per changed field when you need assignments

A wide comparison is useful for analysis. A long format is easier for review queues, databases, and task creation.

Add this function:

def to_long_format(comparison: pd.DataFrame) -> pd.DataFrame:
    records: list[dict[str, str]] = []

    for _, row in comparison.iterrows():
        url = row["url"]
        state = row["url_state"]

        if state != "Present in both":
            records.append(
                {
                    "url": url,
                    "field": "url_state",
                    "before": "",
                    "after": "",
                    "change": state,
                }
            )
            continue

        fields = [
            field.strip()
            for field in str(row["changed_fields"]).split(",")
            if field.strip()
        ]

        for field in fields:
            records.append(
                {
                    "url": url,
                    "field": field,
                    "before": row.get(f"{field}_before", ""),
                    "after": row.get(f"{field}_after", ""),
                    "change": "Changed",
                }
            )

    return pd.DataFrame(records)

Then export:

long_output = to_long_format(comparison)
long_output.to_csv("crawl-comparison-long.csv", index=False)

Example output:

The long format can be imported into a database, shared review sheet, or task-creation workflow more easily than hundreds of before-and-after columns.

Add rules for severity and expected changes

Not every change needs the same response.

Example severity rules:

CRITICAL_FIELDS = {
    "status_code",
    "indexability",
    "indexability_status",
    "canonical",
    "meta_robots",
    "x_robots_tag",
}

HIGH_FIELDS = {
    "title",
    "h1",
    "schema_types",
}

MEDIUM_FIELDS = {
    "meta_description",
    "unique_inlinks",
    "crawl_depth",
    "word_count",
}


def classify_severity(field: str, before: str, after: str) -> str:
    if field in CRITICAL_FIELDS:
        return "Critical"

    if field in HIGH_FIELDS:
        return "High"

    if field in MEDIUM_FIELDS:
        return "Medium"

    return "Low"

This is only a starting point.

A title change on the homepage may be critical. A canonical change may be expected during consolidation. A word-count change may represent a major deleted section or an unimportant dynamic widget.

Add context fields:

• Expected?
• Related release
• Related task
• Page priority
• Template
• Client
• Owner
• Review decision
• Resolution

Compare redirect migrations with a dedicated mapping

For migrations, do not rely only on a crawl-to-crawl merge.

Start with the approved redirect map:

Then test:

• Actual status
• First redirect destination
• Final destination
• Number of hops
• Final status
• Final indexability
• Final canonical
• Match with expected destination

A Python structure:

import pandas as pd
import requests

mapping = pd.read_csv("redirect-map.csv")

results = []

for row in mapping.itertuples(index=False):
    old_url = row.old_url
    expected = row.expected_destination

    try:
        response = requests.get(
            old_url,
            timeout=20,
            allow_redirects=True,
        )

        results.append(
            {
                "old_url": old_url,
                "expected_destination": expected,
                "final_url": response.url,
                "final_status": response.status_code,
                "redirect_hops": len(response.history),
                "matches_expected": response.url.rstrip("/")
                == expected.rstrip("/"),
            }
        )
    except requests.RequestException as exc:
        results.append(
            {
                "old_url": old_url,
                "expected_destination": expected,
                "error": str(exc),
            }
        )

pd.DataFrame(results).to_csv(
    "redirect-validation.csv",
    index=False,
)

The redirect map represents intent. The crawl shows what the site currently exposes. Both are needed.

Compare selected content and AI visibility signals

Traditional crawl columns may not cover the content blocks the team needs to protect.

Use custom extraction for values such as:

• Author name
• Author credentials
• Publication and update dates
• Product description
• Pricing
• Lead statistic
• Source list
• Expert quote
• Comparison table
• Organization details
• sameAs links
• AI-related bot directives
• Selected structured-data properties

Example extraction plan:

Compare extracted values the same way as titles or canonicals.

For AI visibility work, the crawl comparison can confirm that the page changed. It cannot confirm that ChatGPT, Gemini, Perplexity, Copilot, or AI Overviews changed their answer.

Use separate AI visibility checks for:

• Brand mentioned
• Brand cited
• Official URL cited
• Competitors mentioned
• Answer accuracy
• Prompt group
• Platform
• Review date

The page comparison and AI-answer comparison should connect to the same work record where relevant.

Track approved changes before running the comparison

A comparison is much easier to review when the team already knows what was supposed to change.

Create an approved-change list:

Then classify crawl differences:

Expected and correct
Expected but incorrect
Unexpected
Unconfirmed
No change detected

This prevents approved work from becoming an alert and makes failed implementation easier to identify.

Turn meaningful differences into a review queue

Do not send the complete comparison output directly to developers or clients.

Create a shorter queue:

A useful review process:

1. Filter critical and high-priority changes.
2. Match them to approved tasks.
3. Confirm the live page directly.
4. Assign unexpected problems.
5. Record expected changes.
6. Re-run or re-crawl after fixes.
7. Close only after verification.

Keep the crawl evidence connected to the URL history

A crawl export is a snapshot. A task is an assignment. A URL history explains the sequence.

SEO Logbook can connect:

• Intentional work
• Affected URL
• Related task
• Crawl or monitor detection
• Previous and current values
• Reviewer
• Follow-up
• Resolution
• Later impact note

Example:

The July 1 and July 6 crawls show the release result. Repeated monitoring preserves the regression that happened afterward.

This is the main limitation of comparing two crawl files: the comparison shows two states, not every event between them.

Know when to use crawl comparison and when to use monitoring

The methods are complementary.

Automate recurring comparisons carefully

A recurring process may include:

Scheduled crawl
→ Export selected columns
→ Compare with previous export
→ Apply severity rules
→ Send review queue
→ Approve or assign changes
→ Store reviewed history

Before automating, handle:

• Failed crawls
• Partial crawls
• Scope changes
• Configuration changes
• Authentication
• Crawl timestamps
• Dynamic content
• Duplicate URLs
• URL normalization
• Export locations
• File retention
• Notification noise
• Ownership
• Re-runs

Do not automatically create hundreds of Jira or ClickUp tasks from raw crawl differences.

Use a review step before task creation.

Use a release-specific comparison checklist

Before the release

• Save the current crawl.
• Save the configuration.
• Record crawl scope.
• Export priority URLs.
• Record approved changes.
• Select representative template URLs.
• Confirm the release owner and SEO reviewer.

After the release

• Run the crawl with the same configuration.
• Check whether the crawl completed successfully.
• Compare new, missing, and changed URLs.
• Review status, redirects, indexability, canonicals, and templates first.
• Match differences to approved work.
• Confirm critical URLs manually.
• Assign unexpected regressions.
• Re-crawl after fixes.
• Store the final evidence and work history.

After the initial comparison

• Increase checking frequency for high-risk URLs.
• Review GSC and analytics after enough data exists.
• Recheck AI visibility prompts where relevant.
• Lower frequency after the release stabilizes.
• Archive the crawl files according to the team’s retention policy.

Avoid the common comparison mistakes

Different configurations

The output reflects setup differences instead of site changes.

Comparing every available column

The review becomes too noisy to use.

Treating a missing URL as deleted

The crawler may no longer discover it, while the URL still exists.

Ignoring redirects

A missing old URL may be correctly redirecting.

Normalizing URLs too aggressively

Two technically different URLs may be merged incorrectly.

Treating every text change as important

Dynamic blocks and formatting can create harmless differences.

Creating tasks before checking expected work

Approved changes become duplicate tickets.

Comparing only status codes

Titles, canonicals, directives, headings, schema, links, and content can change while the page still returns 200.

Using crawl comparison as continuous monitoring

Temporary changes between crawls remain invisible.

Reviewing crawl output without owners

Findings remain in an export instead of becoming action or accepted risk.

Use this practical comparison output

The final review file should contain:

Keep the raw crawl exports. Use the review file for action.

The purpose of comparing Screaming Frog crawls is not to create another large spreadsheet. It is to find the changes that matter, confirm whether planned work reached production, catch regressions, and preserve the result against the affected URLs.

FAQs

Can Screaming Frog compare two crawls?

Yes. Screaming Frog provides a crawl comparison workflow for reviewing differences between saved crawl states. The available comparison data depends on the crawl setup, stored data, and software version.

Should I compare crawl files or CSV exports?

Use saved crawl files for the fastest in-app workflow and broader access to crawl data. Use CSV exports when you need custom Excel, Google Sheets, Python, database, or automation logic.

Which columns should I compare?

Start with URL, status code, final URL, indexability, canonical, title, meta description, H1, robots directives, schema, and selected internal-link or content fields.

How do I compare Screaming Frog exports in Excel?

Place the two exports in separate sheets, use the URL as the key, retrieve previous values with XLOOKUP, and create flags for changed fields and missing URLs.

How do I compare large crawl exports?

Use Python or another data-processing system to outer-join the crawls by URL, normalize selected fields, identify changed columns, and export only rows requiring review.

Can crawl comparisons catch every website change?

No. They show the difference between two crawl states. A change that appears and disappears between the crawl dates may not be visible. Use more frequent scripts or URL monitoring for important pages.