Dealership AI Evaluation Checklist

Set up a test your team can repeat

Use a demonstration environment with test customers and appointments. Ask the vendor to show how the environment is separated from live customer messaging and production bookings. Choose two inventory vehicles, one unavailable vehicle, a known price, a price that is missing, a fully booked slot, and a department that should not be bookable. Write those facts down before starting.

Run each scenario through the channels included in your proposed scope. Keep the question, transcript, expected outcome, actual outcome, timestamp, and final record. A fluent response is useful evidence about conversation quality, but it is not evidence that a booking, opt-out, or handoff was saved. Have a manager who did not configure the demonstration review the results.

This is Paddock's suggested dealership buying checklist, not a certification. Its repeatable-testing approach follows the general principle in NIST's AI Risk Management Framework: evaluate AI before use and continue assessing it during operation. Adapt the tests to your departments and risk tolerance rather than treating a fixed pass count as proof that every use case is covered.

Test inventory answers and missing information

Begin with a specific vehicle and an answer you can verify: availability, stock number, listed price, or a recorded feature. Then change one fact in the authoritative inventory source and repeat the question. Ask the vendor to explain when the update should become visible. Record any delay against that explanation instead of assuming every connection updates instantly.

Next ask for a price or feature that is absent from the record. A useful assistant identifies the gap or asks staff to help. It should not manufacture a number to keep the conversation moving. Repeat the test with an unavailable car and ask for alternatives; verify that suggested alternatives are real inventory and that the assistant describes their status accurately.

• Ask about a known available vehicle and compare the answer with its record.
• Mark a test vehicle unavailable and repeat the request.
• Ask for a missing price, discount, or feature.
• Move between two vehicle pages and check the assistant's context.
• Ask about a car outside the dealership's inventory.
• Inspect the transcript for invented facts or unsupported promises.

Prove that appointment promises become valid bookings

Ask to book a test drive within configured hours. Verify the customer, vehicle, department, date, timezone, and appointment status in the staff calendar. A proposed slot should remain a proposal until the customer confirms the action. Check that the confirmation matches the stored record, including any vehicle reservation required for a test drive.

Then try a closed day, a slot inside the store's minimum lead time, a full slot, and the same vehicle at the same time from another conversation. Ask to move and cancel the original visit. The expected result depends on your rules, but the assistant should consistently apply those rules, avoid promising a failed action, and leave an understandable history of completed changes.

For service, finance, parts, or other departments, test the workflows included in the proposed deployment individually. A successful sales booking does not prove that every department is configured correctly. Ask what happens when a department is disabled after a conversation begins, and verify that the system checks current permission before completing the next action.

Make the assistant involve a person

Ask for a manager, challenge an unexplained fee, or ask a question the assistant cannot resolve. Inspect the handoff destination, assignment, summary, and outstanding work. Confirm whether the promised next step is a callback, a staff takeover, or a live phone transfer. Those are different capabilities; the customer should hear the one the deployed system can actually provide.

Have a staff member take over a test conversation. The staff member should see the context needed to continue without starting over, and the automation should behave according to the takeover policy. Test the return to automation separately if you plan to use it. Also send a request overnight and confirm who will see it when the dealership reopens.

Check opt-outs and contact preferences end to end

Use test contact details and the vendor's supported test procedure. Try an explicit texting stop keyword, a plain-language request to stop texting, and a request to stop all contact. Inspect both the customer's saved preference and the next scheduled outreach attempt. Reading the right acknowledgment is only part of the test; the affected sends must also be suppressed.

Ask how the system distinguishes a texting preference from a broader contact restriction, where the record is stored, and what a staff user can change. Include a customer who has already opted out before the campaign begins. Have your dealership's responsible staff review the intended outreach process and approved templates for your jurisdictions; this operational checklist does not determine legal permission to contact anyone.

Listen to a real phone conversation

A prerecorded voice sample cannot demonstrate turn-taking. Make a test call, interrupt mid-answer, correct a name, ask the assistant to repeat a time, and change the requested vehicle. Listen for whether it stops speaking when interrupted, understands corrections, and confirms consequential details before acting. Compare the final transcript with the appointment or handoff record.

Try an ordinary noisy environment and a short pause before answering a question. Measure the experience your customers are likely to have rather than a studio recording. If a call disconnects, inspect what was saved and what follow-up, if any, was scheduled. Confirm your expected handling of call recording notices with the team responsible for the store's phone process.

Use a scorecard with evidence, not a single demo score

Mark each scenario as pass, needs work, or outside scope, and attach the record that supports the result. Keep a separate list of launch blockers such as an invented price, an invalid booking, an unassigned escalation, or outreach that ignores a saved opt-out. A high average should not hide a failure your dealership considers unacceptable.

Ask the vendor to trace one appointment from its originating conversation to the staff calendar and later outcome. Distinguish appointments requested, appointments successfully stored, and visits that actually happened. If reporting combines those events, it becomes hard to tell whether a change improved customer service or only produced more messages. Agree on definitions before the pilot begins.

Turn the checklist into a controlled pilot

Choose the initial channel, department, review owner, and pause procedure. Give staff the same short instructions for reporting a problem, including the conversation identifier and the expected behavior. Review routine successes as well as exceptions so that the process does not become a collection of anecdotes from the loudest complaint or the most impressive demonstration.

Repeat relevant tests when inventory connections, appointment rules, department access, handoff destinations, or assistant behavior changes. Retain the previous results so a regression is visible. Expand coverage only when the initial workflow is operating reliably and the staff queue is being handled. A pilot is useful when it informs a launch decision and a monitoring routine, not simply when its calendar end date arrives.

Paddock provides an isolated AI test during onboarding using dealership facts and sample inventory, without copying real customer conversations or live appointments into that test. Use the checklist to assess the proposed configuration, then validate the selected production channels with the people who will operate them. Our product pages describe the available chat, phone, campaign, and console workflows.