Questions to Ask an Attribution Vendor (+ Scorecard)

The scorecard, not the checklist

Most "how to choose an attribution vendor" content is a feature checklist written by people who sell attribution vendors. It compares dashboards and integration counts and asks whether the tool "fits your stack." That is the wrong instrument. It tells you what the tool looks like, not where its numbers come from.

This is a weighted scorecard instead, backed by a five-question demo audit. You score seven categories from 1 to 5, weight them, and total to 100. The weighting is the opinion: the categories that decide whether the numbers are trustworthy at all carry the most, because a beautiful dashboard on biased data is still biased.

The idea that carries the whole thing: a vendor is only as independent as its data supply chain. Any tool that ingests platform-reported conversions and re-serves them through a prettier model inherits the platform's bias by construction. Dedupe logic and a data-driven model can rearrange that bias across channels, but they cannot remove it. You cannot de-bias a number downstream of the party that profits from inflating it. Call that pattern independence laundering. The scorecard exists to catch it.

One guardrail up front, because this audience will otherwise stop reading. No attribution vendor measures incrementality. Attribution allocates credit for conversions that already happened. It does not tell you which conversions would not have happened without the ad. That is a separate discipline: holdout and geo experiments, and marketing mix modeling at the macro level. A high score finds the least-biased, most-auditable allocator, the thing your incrementality tests and MMM triangulate against. It does not crown a truth machine.

How to score

Run the five-question demo audit below, then score each criterion on this scale. Weighted points for a criterion are (your score ÷ 5) × that criterion's share of its category weight. Total the seven categories to 100.

The seven categories

Each category holds two or three scored criteria. Score every criterion 1–5; the category weight is split across its criteria.

Reading the total

One override on the total: any score of 1 in Data completeness or Independence is a hard disqualifier whatever the total says. A vendor that launders platform conversions or won't define its coverage denominator has failed the audit even if it scores well everywhere else.

The rest of this article is the demo audit that produces the scores: the five questions, in the order to ask them, with the good answer, the bad answer, and the tell for each.

Question 1: Demand a coverage number (scores category 1)

Start with coverage because a beautiful model on partial data is a confident guess. Ask what fraction of conversions the tool actually observes, then refuse the first answer, because the first answer is always near 100 percent.

The number is meaningless without a denominator. Coverage of what: sessions, known users, or conversions? Those are different questions with wildly different answers. A tool can observe 98 percent of sessions and still miss a third of conversions if the conversions happen where its tracking does not reach.

The move that turns the question into a real audit: ask them to show you observed conversions versus your own backend conversion count for one week, live in the demo. You know how many orders, signups, or bookings you had last week. Make them put their observed number next to yours. The gap is your real coverage. A vendor that will not run that reconciliation in front of you has just told you the coverage number is a marketing figure.

Coverage matters for a specific reason, and it is not the reason vendors imply. Higher coverage means less missing data. It says nothing about whether the ad caused the conversion. Server-side, first-party collection lowers your dependence on platform-reported conversions and captures roughly 30 to 40 percent more touchpoints than client-side tracking loses to ad blockers and iOS privacy features. That is real, and it improves both the coverage number and the dedupe that depends on it. It does not make the allocation causal. Keep the two ideas apart, or the coverage question quietly mutates into "high coverage equals correct attribution," which is false.

Question 2: Can you see the model, and can you change it

Every vendor sells a model. The question is whether you are allowed to look at it.

Ask two things: can I see the model, and can I edit the weights? A good answer shows you the logic and lets you touch it. A bad answer is some version of "our algorithm is proprietary; you cannot view or edit the weighting." That leaves you holding a number you cannot audit and cannot reconcile against your own MMM or your holdout tests.

There is an honest cost on this side too, so do not oversell editability as pure virtue. A fully editable model can be misconfigured by a buyer into something worse than a sensible default. Pair the question with a second one: does it ship with a defensible default, and does it show you what changed when you edit it? Editability plus a good default and a visible diff is the thing you want. Editability with no guardrails is just a different way to get the wrong number.

Question 3: How dedupe actually works

When Meta claims a $180 purchase under a 1-day view-through window and Google claims the same $180 purchase under a 30-day click window, somebody has to decide who wins. That decision is dedupe, and it silently moves credit between channels. Ask how the vendor makes it.

There are three broad answers. Deterministic matching resolves the conflict on a shared identifier: an email, a user ID, a login. High confidence, lower reach, because not every journey carries a shared ID. Probabilistic matching uses device, IP, and behavioral signals to extend past that boundary. Higher reach, lower precision. Both are legitimate; the tradeoff is precision versus coverage, and the right mix depends on how much of your journey is logged-in.

Neither of those is the answer that should worry you. The answer that should worry you is the third one.

The practical disqualifier is simpler than any of this. When a vendor will not tell you its dedupe method, it is usually because the answer is "we take the platform's word for it." A vendor that cannot articulate how it resolves a double-claimed conversion is a vendor that does not resolve it.

Question 4: Who pays whom

Every vendor says it is independent. Independence is a claim about the data supply chain and the money, not about the logo on the slide, so ask about the money directly.

Do you have a revenue-share, data-licensing, referral, or co-marketing relationship with any ad platform you report on? Do you resell or embed platform-reported conversions as ground truth? The laundering pattern is the combination: independent branding on the outside, platform-reported conversions treated as the source of truth on the inside. A vendor can be genuinely neutral and still ingest platform data as one signal to reconcile. The problem is the vendor that ingests it as ground truth and then charges you for a "neutral" read on top.

Expect a smart pushback here, because it is partly right: "everyone ingests platform data, you cannot avoid it." True. There is no platform-free attribution vendor, and pretending otherwise is its own kind of dishonesty. The distinction that survives the pushback is what the vendor does with the platform's number. A tool that collects your events server-side and treats each platform-reported conversion as one signal among several to reconcile is categorically different from a tool that treats the platform's conversion as ground truth and models on top of it. The audit's job is to locate the vendor on that spectrum, not to demand a purity that does not exist.

Question 5: Export rights

The last question is about the day you leave, which is the day the vendor has the least incentive to help you.

Ask whether you can export raw, event-level data, not aggregated reports, on day one and on the day you cancel, in a format you can query without them. The answer you do not want: "You can export aggregated reports as CSV; raw event data stays in our platform."

Here is the one-week reconciliation query from Question 1, written against a warehouse you control. If the vendor cannot give you the raw events to run something like this, that is the answer to Question 5.

-- Reconcile vendor-observed conversions against your own backend count,
-- from raw event-level data synced to your warehouse.
WITH vendor_observed AS (
  SELECT
    DATE_TRUNC(event_ts, WEEK) AS week,
    COUNT(DISTINCT conversion_id) AS vendor_conversions
  FROM vendor_events
  WHERE event_type = 'conversion'
  GROUP BY 1
),
backend_truth AS (
  SELECT
    DATE_TRUNC(created_at, WEEK) AS week,
    COUNT(DISTINCT order_id) AS actual_conversions
  FROM orders
  WHERE status = 'completed'
  GROUP BY 1
)
SELECT
  b.week,
  b.actual_conversions,
  v.vendor_conversions,
  SAFE_DIVIDE(v.vendor_conversions, b.actual_conversions) AS coverage_ratio
FROM backend_truth b
LEFT JOIN vendor_observed v USING (week)
ORDER BY b.week DESC;

Aggregated CSVs are a hostage situation dressed as portability. You can see the summary but you cannot rebuild your history, run the query above, or leave without losing your baseline.

What the audit does not do

This audience deserves the limitations stated plainly, so read them before you run the script on a live vendor.

Passing all five questions does not make a vendor accurate. It makes the vendor auditable and clean-sourced. Those are different properties. The audit finds the least-biased allocator, and that is worth finding, but it is not truth.

No attribution vendor measures incrementality, full stop. MTA allocates credit for conversions that occurred; it cannot tell you which were caused by the ad. Only experiments and, at the macro level, MMM approach the causal question. There is no true number here, only better and worse lenses. Any sentence implying the audit surfaces "the real number" is wrong, and a measurement-literate buyer will dismiss the whole exercise if you let that sentence in.

The classic evidence is worth carrying into the room. In a set of 15 large-scale Facebook field experiments, Gordon, Zettelmeyer, Bhargava and Chapsky found that common observational and attribution methods overstate ad effectiveness by roughly a factor of three for purchase outcomes, and by much more for cheaper conversions like registrations and page views. In eBay's paid-search experiment, Blake, Nosko and Tadelis found that when eBay stopped bidding on its own brand keywords, essentially all of that traffic arrived anyway through organic and direct, which means the attribution report had been crediting brand ads for close to zero incremental sales. A perfectly attributed channel can be worth nothing. That is the gap between allocation and cause, and no vendor audit closes it.

A few more honest edges. Deterministic dedupe is not free: high-precision matching lowers coverage, so you can dedupe cleanly and still be blind to the 30 to 40 percent of the journey with no shared identifier. Dedupe method is a question about how bias enters, not a box that, once ticked, delivers accuracy. Server-side collection reduces the platform-bias problem but does not eliminate it. And model editability, as noted, has a real downside if it ships without a defensible default. The audit is a filter for supply-chain cleanliness and auditability, not a certificate of correctness.

Run the five questions, score the seven categories against the scorecard at the top, and total to 100. The vendor that answers all five well is the lens your MMM and your holdouts get to argue with. The vendor that dodges even one is selling you a platform's homework with a nicer chart on top, and it will show up as a 1 in the category that matters most.

Holly Mehakovic

Co-Founder, mbuzz

Holly Mehakovic is Co-Founder of mbuzz. With 10+ years in marketing including roles at Westpac, Avon, and Forebrite, she's obsessed with making measurement actually useful.

Harvard Extension School Forebrite Westpac Avon