A recommendation, measured honestly.

Protocol, not a coverage claim

This is Mentionloom’s intended measurement protocol, version 1.1. The public previews remain illustrative. Actual report coverage must identify the implemented sampling, classification and collection methods; this page is not a claim that every planned capability is available today.

The initial beta targets explicit OpenAI and Perplexity API/search surfaces. Coverage and availability will be confirmed with each invitation. API measurements do not reproduce the consumer ChatGPT or Perplexity experience. Claude and Gemini are planned coverage; other provider logos in previews are illustrative.

Four states, one state per brand

Classify the meaning of the answer, not a logo, list position or name match. Retain the passage supporting the classification, classifier version and confidence. Flag ambiguous answers for review instead of turning uncertainty into a confident label.

The two recommendation metrics

Both metrics use the same defined report. A competitor qualifies when shortlisted or recommended. Count a lost answer once, even when several competitors qualify. Multiple brands can be shortlisted in the same answer, so shares across brands need not add up to 100%. Lost Recommendation Share is not necessarily one minus your Recommendation Share: an answer can recommend nobody.

State the denominator. Report scheduled probes, completed answers, failed or unusable responses, retries and exclusions separately. A network failure is not an absent brand. A valid answer that recommends no product remains an observed answer. Comparisons must use a consistent, disclosed question and engine mix.

Worked example: fictional brands, not a benchmark

Suppose a ten-answer sample includes three answers shortlisting Acme and two explicitly recommending it. Acme’s Recommendation Share is 5 ÷ 10 = 50%. If four of the other answers recommend a competitor while Acme is absent or merely mentioned, Lost Recommendation Share is 4 ÷ 10 = 40%.

“Acme exists, but is unsuitable for this use case” is a mention, not a shortlist. “Acme and Northstar both meet these requirements” shortlists both. “For this team, I recommend Acme because it meets the stated requirements” is an explicit recommendation. These sentences are invented classification examples, not provider answers.

Sampling and uncertainty

Target three to five independent runs per approved question per engine per cycle, subject to a stated budget. Record the exact question version, intent stage, surface, model, region, language, date and run index. Repeat the same protocol before comparing cycles.

Report the sample size and an uncertainty or noise band with its calculation method. Repeated answers to one question are not equivalent to independent buyers or a representative survey of all demand. A simple binomial interval may understate uncertainty when answers cluster by question or engine. Use a justified grouped analysis where appropriate and disclose small-sample limitations; do not attach an arbitrary ± value.

Keep Discovery, Comparison and Decision views separate. Preserve a versioned question set; show additions and removals instead of silently changing the denominator. No intent-weighted composite should replace the disclosed unweighted metric.

What an auditable probe retains

Each record should include workspace, question and version, intent stage, provider, engine/model, surface, region, language, timestamp, run index, raw answer, citations, brand and competitor states, classifier version, confidence, tokens, search/tool calls and estimated or actual cost.

Citations are evidence associated with an answer. They do not reveal all information used by the model or its hidden ranking process. Distinguish a source actually cited in the answer from a relevant source found during later research.

Three signals, three different jobs

Answer recommendations are the primary measurement. AI referrals are attributed sessions or conversions, an outcome signal. AI crawler requests show that a page was fetched, a diagnostic signal. Neither referrals nor crawler activity is included in Recommendation Share.

Unattributed visits, private conversations and uncited source use remain outside the observation. A crawler fetch is not proof that a product was recommended, and zero tracked referrals is not proof of zero recommendations.

Improve, then re-measure

Record the shipped change, affected URLs, hypothesis and date. Plan a roughly 20% untreated question set for intervention experiments. Compare changes in the treated and control groups while recording other content and provider changes. Questions may share sources, so spillover can weaken that comparison.

One before/after cycle does not establish causation. A consumer-surface divergence sample is a planned validation step, not a currently promised monthly service. Publish its actual collection dates and coverage when it exists.