The Hotel AI Vendor Evaluation Scorecard: 50 Questions to Ask Before You Sign

The Demo Problem: Why Most Hotels Buy the Wrong AI

Hotel AI sales cycles in 2026 look uncannily like the SaaS gold rush of 2014, only with higher stakes and far better marketing. A regional director sits through a fluent demo from a vendor whose product has been in market for nine months. The dashboard sparkles. The forecast accuracy slide claims a number nobody has independently audited. The reference call comes from a hotel that has been live for ninety days and is still in the honeymoon phase. Six weeks later, a five-year contract is signed, and twenty-two months after that the property is paying for a system that runs in parallel with the workflow it was supposed to replace.

This is not a story about bad vendors. The hospitality AI market has dozens of genuinely excellent companies. It is a story about a buying motion that has not caught up to the complexity of the product category. Hotels evaluate AI tools the way they evaluate vacuums — they watch a demo, they ask about price, they ask for references, and they sign. AI is not a vacuum. It is a long-term commitment that touches your data, your guests, your staff, your distribution, and your liability surface. Buying one without a scorecard is malpractice.

What follows is the scorecard. Fifty questions, organized into the seven domains that actually predict whether an AI deployment will succeed: company viability, integration depth, data ownership, security and compliance, model performance, contract economics, and implementation reality. Use it as a takeaway template, force every vendor to answer every question in writing, and weight the scores. The framework will not eliminate failed deployments — nothing does — but it will surface the red flags before you sign rather than after.

Domain 1: Company Viability (Questions 1-7)

You are buying a multi-year relationship with a software company, not a feature. The most common failure mode in hospitality AI is not that the product breaks; it is that the company gets acquired, pivots, or quietly dies, and your contract becomes orphaned. Eighty-two percent of hotels are expanding AI investment in 2026 according to Canary's survey of 400+ hotel tech leaders, which means the market is flush with venture-backed entrants chasing the same dollars. Many of these companies will not exist in their current form by 2028. Your job is to figure out which ones.

Ask for the company's most recent annual recurring revenue (ARR), customer count, and net revenue retention. Ask how much runway they have at current burn. Ask for their three largest customers by ARR and contact those customers directly without the vendor on the call. Ask about board composition and whether any institutional investors have signaled a target exit timeline. None of these questions are rude. They are the same questions any sophisticated buyer asks during a Series B due-diligence process, and they should not surprise a vendor capable of supporting your business for five years.

The seven viability questions surface the patterns that matter:

#	Question	Why it matters	Red flag answer
1	What is your current ARR, customer count, and net revenue retention?	NRR < 100% means more customers leaving than expanding	"We don't share those numbers"
2	How many months of runway do you have at current burn?	Less than 18 months = high acquisition or shutdown risk	Anything < 12 months
3	Who are your three largest hospitality customers by ARR?	Concentration risk and reference quality	Refusal or all customers under 6 months
4	Have you been approached for acquisition in the past 12 months?	Acquisition almost always changes product direction	Vague non-answers
5	What percentage of your engineering team is dedicated to hospitality?	Horizontal vendors deprioritize hotel use cases	"We serve many industries"
6	What is your product roadmap commitment for the next 18 months?	Tests whether they have a strategy or just react	"It depends on customer demand"
7	Who is your CEO and what is their hospitality background?	CEOs without industry context build the wrong things	No domain experience at the top

A vendor that answers all seven cleanly is not necessarily the right pick — but a vendor that fumbles three or more is a vendor whose roadmap and survival you cannot underwrite. Walk away.

Domain 2: Integration Depth (Questions 8-15)

Thirty-eight percent of hoteliers in the 2026 NYU / Stayntouch Hotel Tech Outlook cite integration as their top operational pain point, and it is the single largest predictor of whether an AI vendor will deliver value. An AI model is only as smart as the data it sees. A revenue management AI cut off from the PMS is guessing. A guest messaging AI without CRM access is broadcasting. A predictive maintenance AI that cannot write back to your CMMS is a glorified alert engine.

The right questions move past "Do you integrate with Opera?" — that answer is almost always yes — and force the vendor to specify the depth of the integration, the cadence of data flow, and the contractual commitments around uptime. There are four levels of integration depth in this market, and a hotel that signs at the wrong level will spend years compensating with manual workarounds:

Integration level	What it means	Typical latency	Use cases that work
L1 - File import	CSV/SFTP batch, manual triggers	Daily or weekly	Historical analytics only
L2 - Polling API	Vendor pulls data on a schedule	5 - 60 minutes	Reporting, marketing segmentation
L3 - Webhooks + REST	Real-time push from PMS to vendor	< 60 seconds	Guest messaging, dynamic pricing
L4 - Bidirectional certified	Read AND write back, vendor-certified	< 5 seconds	Automation, workflow replacement

Most AI vendors will quietly position at L2 while implying L4. The integration questions force them to commit:

1. Question 8: Which PMS, RMS, CRM, and POS platforms are you officially certified with, and which are "in development"?
2. Question 9: For each certified integration, what is the data refresh frequency — real-time, near-real-time, or batch?
3. Question 10: Can you read and write back to the source system, or is integration one-directional?
4. Question 11: What is the typical integration setup time, and is integration billed separately?
5. Question 12: Who owns the integration when it breaks — you, the PMS vendor, or me?
6. Question 13: Do you charge the PMS vendor for the integration, or do they charge you? (Hidden cost driver)
7. Question 14: How do you handle PMS version upgrades — automatic, scheduled, or manual re-validation?
8. Question 15: Will you provide a written guarantee of integration uptime in the SLA?

Question 13 is the silent budget killer. Several major PMS providers charge AI vendors per-property or per-transaction fees that get passed to the hotel either explicitly or through inflated SaaS pricing. Hotels that do not surface this dynamic during diligence routinely discover their "$24,000/year AI tool" is actually $38,000/year after PMS integration surcharges. Force the answer.

Domain 3: Data Ownership (Questions 16-22)

Your guest data is the most strategic asset your hotel owns, and AI vendors have every commercial incentive to use it for purposes you did not contemplate when you signed. Some vendors train their general models on your guest history. Some sell anonymized aggregates to third parties. Some retain a perpetual license to your data even after you cancel. The contractual language buried in the data processing addendum will tell you everything if you read it carefully and almost nothing if you do not.

The data ownership domain is also where 2026's regulatory pressure is intensifying fastest. GDPR enforcement has expanded, California's CPRA is in full effect, and several U.S. states have passed AI-specific consumer protection laws that require hotels to disclose when guest-facing decisions are AI-driven. Vendors that wave off these questions are vendors who will leave you holding the regulatory bag.

The seven data ownership questions:

#	Question	What "good" looks like
16	Who owns the guest data the system collects and processes?	Hotel retains full ownership; vendor is processor only
17	Do you train your general AI models on my data?	No, unless I opt in with separate compensation
18	What is your data retention policy after contract termination?	All data returned in standard format, deleted within 30 days, certificate of destruction provided
19	What subprocessors do you use, and how am I notified of changes?	Current list provided, 60-day notice on additions, right to object
20	Where is my data stored geographically?	Specific regions named; data residency commitments in writing
21	Can I export my full data set at any time in a standard format?	Yes, JSON or CSV, full schema documentation included
22	Do you sell, license, or share aggregated data with third parties?	No, or with explicit opt-in and revenue share

Independent operators are especially vulnerable on Question 17. Many AI vendors include a default clause that permits training on customer data for "service improvement" purposes — language broad enough to cover almost anything. The model that learns from your booking patterns today becomes the competitive advantage your vendor sells to the hotel down the street tomorrow. Strike the clause or rewrite it to require explicit opt-in.

Domain 4: Security and Compliance (Questions 23-30)

The average cost of a third-party vendor breach reached $5.08M in IBM's 2025 Cost of a Data Breach report — and breaches originating in vendor systems cost approximately 40% more to remediate than internal incidents. Hotels concentrate enormous quantities of payment data, government ID information, and behavioral profiles, which makes the hospitality vertical a perpetual target. An AI vendor is, in security terms, a new attack surface attached to your most sensitive systems.

The non-negotiable baseline in 2026 is SOC 2 Type II. Anything less is a vendor either too small or too immature to be trusted with guest data. Beyond SOC 2, the AI-specific controls matter: multi-tenant isolation (your data is logically separated from other hotels', not just in the same shared model), encryption of sensitive fields at rest and in transit, complete audit trails for any AI-driven decision that affects a guest, and an immediate kill switch that lets you disable the AI's autonomous actions if something goes wrong.

The eight security questions are absolute minimums:

23. Provide your current SOC 2 Type II report and the date of your last audit.
24. Are you PCI DSS compliant for any system that touches payment data? Provide AOC.
25. How is data encrypted at rest and in transit, and what key management practices do you follow?
26. How do you ensure multi-tenant isolation — what prevents data bleed between hotels?
27. What is your incident response process and notification timeline if my data is compromised?
28. Do you carry cyber liability insurance, and what are the policy limits?
29. What human-in-the-loop controls exist for AI decisions that affect guests directly?
30. Is there an immediate kill switch I can activate if AI behavior becomes unacceptable?

Question 29 deserves emphasis. In 2026 we have moved past the era when AI was a recommendation engine that humans approved. Many vendors now deploy agentic AI that takes autonomous action — sending guest messages, adjusting rates, dispatching work orders. A kill switch is no longer optional. If a vendor's product can act without human approval, you must have a single button that disables it instantly without their involvement. Verify this in a sandbox before signing.

Domain 5: Model Performance and Transparency (Questions 31-36)

This is the domain where vendor marketing is at its most aggressive and most meaningless. "97% forecast accuracy" sounds impressive until you ask the next question: accuracy against what baseline, measured over what period, on what segment of business, and how does that compare to the naive baseline of last year's actuals? Most vendors cannot answer. Many have never measured. The ones who can are the ones who deserve your business.

The six performance questions force vendors out of demo theater and into honest measurement:

#	Question	Acceptable answer pattern
31	What baseline do you compare your AI's performance against, and how was it constructed?	Compared to LY actuals, current vendor output, or naive forecasts — with methodology documented
32	What is the typical accuracy on transient, group, and ancillary business separately?	Segment-level breakdown; not a single blended number
33	How does the model handle low-data scenarios — new properties, post-renovation, post-disruption?	Explicit cold-start strategy with fallback logic
34	How often is the model retrained, and on what triggers?	Scheduled cadence + drift-based retraining
35	Can you provide explainability for any specific AI decision after the fact?	Yes, with feature attribution or rule traceability
36	What is your published rate of false positives / hallucinations in production?	A number, with methodology — not "very low"

Question 36 is the test that separates serious vendors from theatrical ones. Any vendor running production AI knows their hallucination or false positive rate. Vendors who do not measure it either do not have AI in production at scale, or they do and they are hiding the number. Both are disqualifying.

Domain 6: Contract Economics (Questions 37-44)

The headline SaaS fee is rarely the largest line item in the true cost of hotel AI ownership. The eight questions below are designed to surface every hidden cost driver and every contractual asymmetry that vendors use to lock in revenue. The principles are simple: term commitments should match the value-delivery curve, price escalators should be capped, termination rights should be symmetric, and any number that compounds (overage fees, usage-based pricing, per-property surcharges) should be modeled out three years before signing.

The Total Cost of Ownership (TCO) model that every hotel should build before signing:

TCO line item	Year 1	Year 2	Year 3	3-yr total
Base SaaS license	$24,000	$26,400 (+10%)	$29,040 (+10%)	$79,440
Implementation / onboarding	$15,000	$0	$0	$15,000
PMS integration surcharge	$6,000	$6,000	$6,000	$18,000
Training (staff turnover-adjusted)	$8,000	$4,000	$4,000	$16,000
Internal labor (project mgmt + IT)	$22,000	$12,000	$12,000	$46,000
Usage / API overage allowance	$3,000	$4,500	$6,000	$13,500
True TCO	$78,000	$52,900	$57,040	$187,940

A vendor quoting "$24K/year" is actually proposing $63K average annual cost. Build this table for every shortlist vendor before you read another marketing deck.

The eight contract economics questions:

37. What is the minimum contract term, and what are the renewal terms?
38. What is the annual price escalator cap, and is it CPI-linked?
39. What termination-for-convenience rights exist after the initial term?
40. What are the exact triggers and dollar values of SLA credits?
41. How is usage measured, and what overage fees apply if I exceed thresholds?
42. Are professional services billed by deliverable or by hour, and what is the not-to-exceed?
43. What is the liability cap, and does it apply to data breaches?
44. Will you accept a most-favored-nation clause for properties of similar size?

Question 39 is the single most important contract concession to fight for. Without termination-for-convenience after an initial period — typically 12 months of a multi-year commitment — you become locked into technology that may be commercially or technically obsolete before the contract expires. Top enterprise SaaS counsel routinely advise that termination-for-convenience be standard in any AI contract over 18 months because the underlying technology is changing too fast to commit blindly. Hold the line.

Domain 7: Implementation Reality (Questions 45-50)

Vendors will tell you implementation takes "4-6 weeks." In reality, most hotel AI deployments take 90-180 days to reach steady-state value, and the difference between vendors who acknowledge this and vendors who do not is the difference between a project that succeeds and a project that quietly stalls in month three. The final six questions surface the operational realities that determine whether your team will actually use the system you bought.

Implementation realism matters disproportionately for hospitality because of staff turnover (average front-of-house turnover exceeds 70% annually at many properties), shift-based scheduling that complicates training, and the multi-stakeholder approval cycles that slow workflow changes. A vendor whose implementation methodology does not acknowledge these realities is a vendor whose implementation will fail.

Implementation phase	Realistic duration	Critical success factor
Contracting + procurement	3 - 6 weeks	Legal alignment on data, SLA, exit
Integration + provisioning	4 - 10 weeks	PMS vendor cooperation; sandbox testing
Configuration + tuning	3 - 6 weeks	Property-specific rules; historical data validation
Training + change management	4 - 8 weeks	Department champions; in-shift coaching
Pilot to full production	4 - 8 weeks	KPI scorecard; weekly review cadence
Total to steady state	18 - 38 weeks	—

The final six questions:

45. What is your realistic 90-day, 180-day, and 12-month implementation timeline for a property of my size?
46. Who specifically from your team will be assigned to my implementation, and what is their experience?
47. How do you handle change management and staff training given hospitality turnover?
48. What is the cadence and format of post-go-live success reviews?
49. What KPIs do you commit to in writing, and what remedies apply if we miss them?
50. Can I speak with three reference customers — one happy, one mid-deployment, and one who churned?

Question 50 is the underrated one. Every vendor will give you happy references. Demand the mid-deployment customer (still in the trenches, can speak to what's actually hard) and the churn customer (the most honest signal you will ever get). A vendor who refuses Question 50 is a vendor who knows what those calls would reveal.

How to Use the Scorecard

The scorecard is most powerful when used as a structured RFP rather than an interview script. Send all fifty questions to each shortlist vendor in writing, with a 10-business-day deadline. Score each response on a 0-3 scale (0 = does not answer or red flag, 3 = answers cleanly with specifics). The weighting we recommend for general-purpose hotel AI vendor evaluation:

Domain	Questions	Weight	Max points
Company viability	1-7	15%	21
Integration depth	8-15	20%	24
Data ownership	16-22	15%	21
Security and compliance	23-30	15%	24
Model performance	31-36	15%	18
Contract economics	37-44	10%	24
Implementation reality	45-50	10%	18
Total	50	100%	150

A vendor scoring below 70% of available points in any single domain is a vendor with a structural deficiency, regardless of overall score. A vendor scoring above 85% overall and above 70% in every domain is a vendor worth negotiating with. Anything in between requires a judgment call — and that judgment should be informed by reference checks, sandbox testing, and a pilot of no more than 90 days with a defined kill-switch.

Hotels that institutionalize this process see two changes. First, the vendors they buy from are demonstrably better — measured 12 months later by NPS, ROI, and renewal intent. Second, the conversation with the vendor changes. A vendor who has just answered fifty rigorous questions in writing arrives at the contracting table with a different posture than one who closed a deal off a single sparkling demo. They know they are being held to a standard. That standard is what produces partnerships that last.

Hotels beginning this evaluation process often benefit from a structured technology audit that maps existing systems, gaps, and dependencies before any vendor conversations start — explore our Hotel Technology AI Audit & Roadmap service →

Frequently Asked Questions