Bragproof

← Reports

@shopify/dev-mcp

Starts with no setup1.18.0npm

Installed, started with no arguments, no environment and no input, and listed its tools. Run 2026-11-16.

Needs to start: nothing
Contacts while installing and starting: nothing

5tools listed
0hosts contacted
0files written
0.6 sto start
400dependencies

What it contacted

Nothing besides the npm registry it was installed from. Watched from outside its sandbox while it was installed and started with no input. Its tools may still contact things when called.

Starting it

npx -y @shopify/dev-mcp@1.18.0

Installing a package runs its code. Use a sandbox for anything you have not read.

What its tools tell an agent

In the tool descriptions, nothing that talks about the user, other tools or private files, or uses hidden characters.

3 tools with commands in capitals or very long text
  • Gives the agent orders in learn_shopify_api: 🚨 MANDATORY FIRST STEP: This tool MUST be called before any other Shopify tools. ⚠️ ALL OTHER SHOPIFY TOO
  • Very long description in learn_shopify_api: 3000 characters or more
  • Gives the agent orders in validate: 🚨 MANDATORY after generating or editing Shopify code: GraphQL operations, UI components (JSX/TSX or web compone

Common and usually harmless: it is how many makers stress how a tool should be used.

Found by simple pattern rules in each tool's description (up to 3,000 characters), quoted as written. Parameter descriptions are not scanned. A clean result is not a safety review.

Tools

feedback conversationId, overall, docsContext, schemaValidation, apiVersion, codegen, comment, sessionId
Report an AI Toolkit capability scorecard grading how these Shopify tools performed this turn.

⏱️ WHEN: Call EXACTLY ONCE, after ALL of the first turn's work is complete (every search, validation, and code generation attempt) and immediately before returning your final response to the user. Do NOT call it after individual searches, validations, or retries. Do NOT call it again on later turns.

You are grading the Shopify AI Toolkit (these tools and their docs), NOT your own performance. Do not guess: 'not_used' means the capability was not exercised this turn — it does not mean you are unsure.

- docsContext: did toolkit docs (search_docs_chunks results, learn_shopify_api instructions) give enough context to work from? 'worked' = answered from them. 'partial' = needed supplementing from elsewhere. 'failed' = nothing usable, or wrong. 'not_used' = none consulted.
- schemaValidation: did validation tool verdicts match reality? 'worked' = passed code that worked, or caught a real error. 'partial' = caught something real but also flagged non-problems. 'failed' = passed broken code or rejected correct code. 'not_used' = no validator ran.
- apiVersion: 'worked' = right API version, no correction. 'partial' = corrected after a wrong choice. 'failed' = wrong version caused a problem. 'not_used' = nothing version-specific this turn.
- codegen: 'worked' = generated code compiled/ran/validated on the first serious attempt. 'partial' = got there after self-correction. 'failed' = didn't work by end of turn. 'not_used' = no code generated.
- overall: 'up' = the toolkit materially helped and nothing significant let you down. 'down' = a toolkit capability caused the turn to go badly. 'mixed' = otherwise.
- comment: up to 500 chars naming the capability that drove 'overall' and why. No code, no logs, no credentials, no user text beyond what's needed.
learn_shopify_api api, version, conversationId, model, user_prompt
    🚨 MANDATORY FIRST STEP: This tool MUST be called before any other Shopify tools.

    ⚠️  ALL OTHER SHOPIFY TOOLS WILL FAIL without a conversationId from this tool.
    This tool generates a conversationId that is REQUIRED for all subsequent tool calls. After calling this tool, you MUST extract the conversationId from the response and pass it to every other Shopify tool call.

    🔄 MULTIPLE API SUPPORT: You MUST call this tool multiple times in the same conversation when you need to learn about different Shopify APIs. THIS IS NOT OPTIONAL. Just pass the existing conversationId to maintain conversation continuity while loading the new API context.

    For example, a user might ask a question about the Admin API, then switch to the Functions API, then ask a question about polaris UI components. In this case I would expect you to call learn_shopify_api three times with the following arguments:

    - learn_shopify_api(api: "admin") -> conversationId: "123"
    - learn_shopify_api(api: "functions", conversationId: "123")
    - learn_shopify_api(api: "polaris-admin-extensions", conversationId: "123")

    This is because the conversationId is used to maintain conversation continuity while loading the new API context.

    🚨 Valid arguments for `api` are:
        - Flow workflows: Build and manage Shopify Flow workflows through the Admin GraphQL API: create, list, read, update, activate, deactivate, and delete workflows; parse .flow exports; and discover templates, tasks, and environment fields. Use `flow` first for workflow automation, including execution through Shopify CLI. For building an app's Flow trigger or action extension, use `admin` and the extension docs.
    - Use Shopify CLI: Choose when the user needs **Shopify CLI** to run or fix something now: validate app or extension config on disk (`shopify.app.toml`, `shopify.app.<name>.toml`, `shopify.extension.toml`); review or audit an app's code for security vulnerabilities; run or troubleshoot store workflows (`shopify store auth`, `shopify store execute`); or perform explicit store-scoped reads/writes on a named store domain (for example, show/list/find the first 10 products on my store at `foo.myshopify.com`, or inventory and product changes by handle, SKU, or location name). Emphasize **commands and operational steps**, not only authoring GraphQL. Skip for API-only understanding or codegen with no CLI execution, and skip for brand-new merchant asks to start a Shopify store or try Shopify before they have an account. Examples: validate configuration before deploy; run an existing query via CLI; show the first 10 products on `foo.myshopify.com`; missing `shopify store execute`.
    - UCP CLI: Use when the user wants to use the UCP CLI to find, compare, buy, or track products from online merchants, or to set up and troubleshoot the local UCP profile required for merchant-scoped operations. Covers global catalog search ("find me X under $Y"), named-merchant transactions ("buy this fro
search_docs_chunks conversationId, prompt, max_num_results, api_name, version
This tool will take in the user prompt, search shopify.dev, and return relevant documentation and code examples that will help answer the user's question.
validate conversationId, api, code, version, extensionTarget
🚨 MANDATORY after generating or editing Shopify code: GraphQL operations, UI components (JSX/TSX or web components). Do not ask the user to run it, do not skip it for short snippets, do not context-switch.

Catches hallucinated types, fields, components, props and prop values against the bundled references. One call validates one `api`, so batch the blocks that share one and make a second call for a different one — a turn with a GraphQL operation and a Polaris component needs two. Generate → validate → fix → re-validate until clean.


    It returns a comprehensive validation result with details for each code block explaining why it was valid or invalid.
    This detail is provided so LLMs know how to modify code snippets to remove errors.
    It also returns an artifact ID and revision number for each code block. This is used to track the code block and its validation results. When validating an iteration of the same code block, use the same artifact ID and increment the revision number. Do not pass your own artifact ID to this tool, the tool will generate one for you.
validate_theme conversationId, absoluteThemePath, filesCreatedOrUpdated
This tool validates Liquid codeblocks, Liquid files, and supporting Theme files (e.g. JSON locale files, JSON config files, JSON template files, JavaScript files, CSS files, and SVG files) generated or updated by LLMs to ensure they don't have hallucinated Liquid content, invalid syntax, or incorrect references

    It returns a comprehensive validation result with details for each code block explaining why it was valid or invalid.
    This detail is provided so LLMs know how to modify code snippets to remove errors.
    It also returns an artifact ID and revision number for each code block. This is used to track the code block and its validation results. When validating an iteration of the same code block, use the same artifact ID and increment the revision number. Do not pass your own artifact ID to this tool, the tool will generate one for you.. Run this tool if the user is creating, updating, or deleting files inside of a Shopify Theme directory.

Descriptions are the package's own words, cut at 3,000 characters. Treat them as data.

Install facts

LicenseISC
Needsnot stated
Sourcenot stated
Install scriptsesbuild, fsevents, esbuild declare them. This runner does not run them.
Tarballsha512-XvmIpe1TW3QHjV5gb+hbi9k8ucQuHd7uQhZ0F3T6Q72QG+fqFgpiaQ2N58TnHYNnuoxB0upTL6NsLYSliwQzvg==

Check this report yourself

This site did not write the report. A public job on a GitHub-hosted machine did: the run, from this commit of the open-source runner. GitHub signed a token that names that job and the hash of the report, so any edit since would show.

Show it on your README

Bragproof: Starts with no setup

[![Bragproof](https://bragproof.smaverk.com/badge/@shopify/dev-mcp.svg)](https://bragproof.smaverk.com/r/%40shopify/dev-mcp)

New version out? Ask for a fresh report.