What a vendor grade covers
We grade how much control customers have over their software, using the same six dimensions across industries. Each assessment names the product, plan, region and access routes it covers (such as APIs, partner integrations, data exports and customization tools). A product grade does not extend to every product sold by its vendor.
An assessment can include several purchasable routes. Paid add-ons are disclosed, and their commercial barriers cap the data-access score relative to the lowest paid plan. Combining capabilities from several routes does not imply that they are all included on that plan or available through a single API.
Each answer retains its product, route, region, commercial entitlement, source clause and checked date. Rights established for one product, region or agreement do not transfer to another. Partner terms do not establish customer termination rights. Industry comparisons and vendor reviews explain the scope of their evidence; this methodology defines the common grading rules.
For each question, choose the highest anchor directly supported by applicable evidence on the stated route. Restrictions qualify that capability in its own criteria; an available export does not establish unrestricted AI processing, and a warehouse read does not establish writes. Unknown contract terms remain unresolved even when an integration works technically.
Provider specifications and applicable legal terms can establish documentary answers. Provider-authored procurement disclosures are also eligible. Marketing pages establish advertised availability only. Historical documentation and third-party commentary cannot establish a current score. Recheck evidence within 90 days; do not infer permission from silence.
| Record | Required information |
|---|---|
| Assessment scope | Vendor, product/version, customer plan, region, route, use case, test profile, assessment date and assessor. |
| Permission and third parties | Credential owner, acting identity, scopes, consent, revocation, registration, partner review and permitted downstream use. |
| Read coverage | Objects, required fields, identifiers, relationships, attachments, historical range, expected totals and recovered totals. |
| Action and continuity | Write operations, retries, validation, audit records, changes/deletions, webhooks or polling, recovery and deprecation. |
| Logic and customization | Customer-owned formulas, rules, automations, dependencies, config export, API/schema documentation, own-tool coding, reusable validation cases and external deployment rights. |
| Cost and limits | Lowest paid plan, read/write entitlements, mandatory upgrades, customer and partner fees with separate payers, setup/review fees, recurring minimums, usage units, currency, dated quotes and destination cloud costs. Calculate first-month and recurring costs for a declared workload. |
| Time to usable access | Complete-request timestamp, agreement/security/pilot milestones, activation and first successful authorized query timestamps, provider waiting days versus customer engineering hours, published deadline versus observed duration, failures and sample size. |
| Query capabilities | Native API versus replica SQL, supported filters/projections, pagination, joins/aggregations, permitted tables, schema documentation, freshness, query/row limits and costs on the same commercial route. |
| Exit conditions | Output format, schema/manifest, cancellation window, retention, termination rights, portability tests and migration assumptions. |
| Evidence and outcome | Source URL/title/date, document or live-test status, reproducible steps, redacted observed result, criterion rationale, conflicts and unresolved questions. |
Public agreements may differ from signed customer terms. These are assessments of documented conditions, not legal opinions or verified tenant behavior. Conflicting or inapplicable evidence stays unknown. A hyphen leaves a summary value blank; source details explain why.
From answers to overall grades
Each criterion has five named answers worth 0, 2.5, 5, 7.5 or 10. Its enum answer selects the score. Source counts, route counts, partner counts and vendor popularity do not enter the formula.
Technical read capability = (read interface + bulk delivery + query expressiveness) ÷ 3
Data access = minimum of technical read capability and access cost barrier
Other dimensions = average of their two criterion scores
Overall = sum of six dimension scores ÷ 6
Dimensions have equal weight. Data access has three technical inputs and a commercial cap; the other dimensions have two equally weighted inputs. An add-on cannot earn an A for usable access merely because it supports powerful queries. These are editorial weights for customer control, not empirically optimized predictors. Every assessed vendor uses the same questions and anchors.
Cost, onboarding time and queries
The top cost anchor requires structured access on the lowest paid plan with no fixed access fee. Published usage-only charges qualify; collect the actual bill for a declared workload to check whether light use stays affordable. A one-time setup fee caps access at 7.5, a plan upgrade or separately licensed add-on at 5, and a negotiated access project at 2.5. These anchors measure the commercial barrier, not the size of an unpublished fee.
Pair each capability with its own entitlement. Partner license fees, customer add-ons and destination cloud bills have different payers. A fee for one API cannot be assigned to an unrelated warehouse route. If capabilities require several paid routes, use the most restrictive required commercial anchor; document simpler alternatives separately.
Collect time from a complete access request to the first authorized query returning usable customer records. Separate provider waiting time from customer engineering time, including agreements, security review, pilot testing and required implementation. Record published deadlines and measured days separately. Company-age eligibility is not onboarding duration. This rubric does not yet score timing; a measured time cap requires comparable observations, a common test workload and a new rubric version.
Query evidence must specify filters, pagination, projections, joins and aggregations. SQL on a delivered database copy earns 7.5 for query expressiveness; SQL or equivalent queries against operational data earns 10. Record freshness, table permissions, row limits and query charges. A provider’s internal SQL database or generic warehouse export claim does not establish a customer SQL endpoint.
Provisional scores and ranges
An unresolved answer remains null and retains its weight. Its possible score spans 0-10. The lower bound assigns 0 to unresolved inputs and the upper bound assigns 10; neither is a claim that the missing fact is false or true. Run the full formula at both bounds, including the commercial cap. The displayed provisional point is their midpoint. Because the cap is nonlinear, this is not always the same as substituting 5 for unresolved inputs.
Dimension high = dimension formula with unresolved inputs at 10
Overall low/high = average of six dimension low/high values
Provisional point = (overall low + overall high) ÷ 2
Publish a provisional overall grade only when at least 75% of the 14 answers are supported and every dimension has a supported input. A complete documentary grade requires all 14. The range is a sensitivity bound for missing information, not statistical confidence. Close estimates with overlapping ranges cannot establish which vendor is conclusively better.
For example, a dimension with one supported 7.5 and one unresolved answer has a range of 3.75-8.75 and a provisional point of 6.25. The supported criterion remains 7.5; the unresolved answer is not stored as 5. Live-test results are excluded from these documentary estimates until a separate tested profile is adopted.
Letter grades
The letter comes from the unrounded overall or dimension score. One decimal is for display only. Provisional letters can change as missing answers resolve. An A describes strong control under the stated rubric; it does not certify complete data recovery.
| Letter | Score /10 | Interpretation |
|---|---|---|
| A | 8 to 10 inclusive | Customer control |
| B | 6 to below 8 | Published conditions |
| C | 4 to below 6 | Material restrictions |
| D | 2 to below 4 | Severely restricted |
| F | 0 to below 2 | Unavailable or prohibited |
ONE SIXTH OF THE OVERALL GRADE
Data access
Can you retrieve the data your workflow needs?
Read interface
What structured read interface is documented?
- 0.0/10 · No external read path
- 2.5/10 · Report outputs only
- 5.0/10 · API for one business domain
- 7.5/10 · Structured APIs across business domains
- 10.0/10 · Queryable database or replicated schema
Bulk delivery
How can customers retrieve data in bulk?
- 0.0/10 · Bulk retrieval expressly unavailable
- 2.5/10 · Manual report download
- 5.0/10 · Batched or iterable API requests
- 7.5/10 · Supplier-delivered database extract
- 10.0/10 · Recurring feed to a customer-selected data environment
Query expressiveness
What queries can a customer issue outside the provider UI?
- 0.0/10 · No external structured query path
- 2.5/10 · Fixed object or report retrieval
- 5.0/10 · Documented structured API filters
- 7.5/10 · SQL against a customer-controlled database copy; freshness depends on delivery
- 10.0/10 · SQL or equivalent joins and aggregations against the operational system
Access cost barrier
What extra commercial condition applies to the routes supplying the assessed read capability, relative to the lowest paid plan?
- 0.0/10 · No purchasable access path
- 2.5/10 · Access requires an individually negotiated project or agreement
- 5.0/10 · Higher plan, separately licensed add-on or fixed recurring access fee
- 7.5/10 · Lowest paid plan includes access; a published one-time setup fee applies
- 10.0/10 · Lowest paid plan includes access with no fixed access fee; published usage-only charges are eligible
ONE SIXTH OF THE OVERALL GRADE
Third-party accessibility
Can you authorize the tools you choose?
Third-party eligibility
Who decides whether a chosen third party may connect?
- 0.0/10 · Chosen third parties prohibited
- 2.5/10 · Provider-selected integrations only
- 5.0/10 · Separate provider consent or partner agreement required
- 7.5/10 · Open registration under published review criteria
- 10.0/10 · Customer authorization without discretionary provider approval
Downstream permission
What permission is needed for the documented downstream use?
- 0.0/10 · Declared downstream use expressly prohibited
- 2.5/10 · Provider tools only
- 5.0/10 · Separate consent for the third party or declared use
- 7.5/10 · Published purpose and confidentiality boundaries
- 10.0/10 · Customer-directed processing without separate discretionary consent
ONE SIXTH OF THE OVERALL GRADE
External action
Can external tools perform the operations you need?
Write interface
Which external write operations are documented?
- 0.0/10 · Assessed route is expressly read-only
- 2.5/10 · Named operations in a limited domain
- 5.0/10 · Create and update in one business domain
- 7.5/10 · Create and update across business domains
- 10.0/10 · Customer-configurable write interfaces with validation
Custom application path
How can customers connect an application they build or choose?
- 0.0/10 · Custom applications expressly prohibited
- 2.5/10 · Provider-built applications only
- 5.0/10 · Approved third-party applications using fixed interfaces
- 7.5/10 · Customer applications using fixed APIs without provider selection
- 10.0/10 · Customer-configurable interfaces for custom applications
ONE SIXTH OF THE OVERALL GRADE
Continuity
Can the integration stay correct over time?
Refresh mechanism
What mechanism supports repeatable extraction of changes?
- 0.0/10 · No repeatable extraction path
- 2.5/10 · Manual extraction
- 5.0/10 · Scheduled feeds or recurring extracts
- 7.5/10 · Documented updated-since filters or webhooks
- 10.0/10 · Database change-data capture
Limit visibility
How are integration limits disclosed?
- 0.0/10 · Assessed integration cannot operate within the stated limits
- 2.5/10 · Governors or limits stated without quantities
- 5.0/10 · Numeric usage or polling guidance
- 7.5/10 · Service-specific limits disclosed in runtime headers
- 10.0/10 · Published numeric quotas and recovery instructions
ONE SIXTH OF THE OVERALL GRADE
Data exit
Can you move the system’s data elsewhere?
Export custody
Where can a usable data copy be delivered?
- 0.0/10 · No copy may leave the provider
- 2.5/10 · Rendered reports only
- 5.0/10 · Structured file or API object download
- 7.5/10 · Database extract delivered to the customer
- 10.0/10 · Feed into a customer-controlled store
Cancellation access
What retrieval rights are documented at cancellation?
- 0.0/10 · Retrieval expressly prohibited at cancellation
- 2.5/10 · Extract before customer cancellation; limited conditional access if provider terminates
- 5.0/10 · Exit extract offered; assistance or retention can cost extra
- 7.5/10 · Defined retrieval window for either party’s termination
- 10.0/10 · Defined window and full export without an exit charge
ONE SIXTH OF THE OVERALL GRADE
Business logic and customization
Can you understand, reuse and extend how the system works?
Rights to custom logic
What rights or portability are documented for custom workflow artifacts?
- 0.0/10 · Provider owns custom workflow artifacts with assignment of remaining rights
- 2.5/10 · Provider owns artifacts but grants reuse outside its runtime
- 5.0/10 · Customer owns definitions; external execution not established
- 7.5/10 · Customer-owned definitions export in a reusable format
- 10.0/10 · Customer-owned definitions can execute independently
What still needs testing
The documentary grade answers what is offered and what conditions apply. It does not measure recovered records, files, relationships, history, write success, safe retries or recovery from outages. A second test bench collects these results in a real customer environment with permission and a declared sample.
Download live-test template ↓24 live-test and policy checks
Data access
- Record coverage: Expected and recovered records by required object, including pagination and restricted objects.
- Fields and relationships: Expected and recovered required fields and foreign-key links. Record missing relationships separately.
- Attachment coverage: Expected and recovered files, file metadata and record-to-file links. Verify the bytes, not just download URLs.
- Historical coverage: Expected and recovered historical events, transactions and versions within the declared time range.
Third-party accessibility
- Third-party eligibility: Who can register, eligibility rules, reviews, contracts, rejection reasons and required partner status.
- Customer authorization: Acting identity, credential owner, grant scopes, revocation, token lifetime and shared-password requirements.
- Permitted use: Terms covering downstream processing, AI, competing tools, redistribution and customer-directed use.
- Access cost and entitlement: Plan uplift, setup/review charges, per-connection fees, minimums, metering units and a dated quote.
External action
- Write operation coverage: Required create, update and delete operations versus those completed on the declared objects.
- Write permissions: Operation scopes, acting identity, customer approval and audit attribution.
- Safe retries: Declared duplicate/retry scenarios versus correct final states. Check idempotency and uncertain timeouts.
- Validation and errors: Declared invalid, conflicting and unauthorized requests versus correct rejection and inspectable errors.
Continuity
- Change capture: Expected updates and deletions versus observed events or incremental-query results in the declared interval.
- Failure recovery: Rate limiting, expired tokens, missed events and interrupted pagination versus correct recovery without silent loss.
- API change policy: Versioning, deprecation notice, support windows and migration commitments.
- Continuing and expanding access: New locations, accounts and third parties; suspension, cancellation, appeal and transition terms.
Data exit
- Exit package coverage: Expected migration objects, records, relationships, files and history versus the exit package. Keep an object-level manifest.
- Reusable format: Machine-readable formats, field definitions, identifiers, schemas and proprietary readers.
- Access during exit: Export availability, notice, retention, post-cancellation retrieval and the declared migration schedule.
- Exit cost: Export fees, required renewal, professional services, format conversion and metered extraction cost.
Business logic and customization
- Inspectable business logic: Formulas, automations, validation, derived-field rules and calculation dependencies available to customers.
- Rules and workflow export: Expected and recovered customer-owned formulas, rules, automations and dependency definitions in a reusable representation.
- Customer-built applications: API/schema documentation, own-tool access, custom UI support, permitted coding/AI tools and required vendor runtimes.
- Independent execution: Whether exported customer logic can be run elsewhere, which dependencies remain and what deployment terms allow.
Declare expected objects, fields, file bytes, relationships, operation types and time windows before testing. Preserve the expected and recovered manifests, procedures, observed failures and redacted result artifacts. Counts alone do not prove that the correct items were recovered.
Measured coverage uses five levels: 0% recovered or tests passed; More than 0%, below 50%; 50% to below 90%; 90% to below 100%; 100% of the declared sample. These measurements are separate from the published documentary grade. Customer data and credentials never appear in public evidence.
Corrections and revisions
Every published grade retains its rubric version, assessment profile, route scopes and date. Changes to anchors or weights require a new version and recalculation of every vendor in the comparison. Resolve conflicting evidence before scoring that input; a fresh check date does not make a historical specification current.
Version 0.4 adds query expressiveness and moves commercial entitlement into a cap on data access. Earlier version 0.3 averaged technical read and bulk capability without that cap; its scores are not comparable to this version. Published assessments have been recalculated under the current formula.
Vendors can supply evidence and challenge factual errors under the same standard as other contributors. Payments and commercial relationships do not change weights, suppress restrictions or buy a better grade. See the ownership disclosures. The site is owned by Permute, so it is not financially independent of an integration provider.