|8 min read

Reviewed by LeadPilot Editorial

24/7 Customer Support Chatbot: What to Automate and What to Keep Human

How to scope a 24/7 customer support chatbot: which questions to automate, how to design the overnight handoff, and how to test it before launch.


In this guide, a 24/7 customer support chatbot is software that answers customer questions on your website at any hour and passes anything it can't resolve to a person. As a working recommendation, let it answer on its own when three conditions hold. The answer already exists in content you've published. The answer is the same for every customer. A wrong answer would cost you little. For everything else, the chatbot's job is to collect the details and hand off cleanly, not to improvise.

This guide is for small and mid-market B2B teams (SaaS, agencies, consultancies, professional services) deciding what a round-the-clock chatbot should handle. It covers how to scope the bot, how to design the overnight handoff, and how to test it before customers see it. The frameworks here are the author's recommendations. Check them against your own inbox and chat logs.

What "24/7" actually changes

Without a chatbot, a question that arrives late at night waits until someone is back at a desk. With a chatbot, the visitor gets an immediate reply. That reply is only valuable if it's accurate and if the conversation ends somewhere useful.

Being available around the clock doesn't make a bot good. It raises the stakes on three design choices:

  1. Scope. Which questions the bot may answer on its own.
  2. Grounding. Where its answers come from.
  3. Escalation. What happens when the bot reaches the edge of its scope and nobody on your team is online.

If a bot disappoints in testing, check these three choices first.

Step 1: Sort your questions before you configure anything

Pull a sample of recent questions from your support inbox, contact form submissions, sales emails, and any existing chat logs. Use a sample large enough that patterns repeat. Then sort each question with three tests. This is a recommended working method, not a standard.

Test 1: Is the answer published? If the answer lives on your website (services, process, general turnaround, plan structure, integrations you list, policies), a bot can reflect it. If it lives only in someone's head or in a private system, the bot has nothing reliable to draw from.

Test 2: Is the answer identical for every customer? "What does your onboarding involve?" is the same for everyone. "Why was my invoice higher this month?" depends on one account's records. Questions that need account data, a judgment call, or an exception belong with a person unless your bot is securely connected to the relevant records. Verify any such connection with your vendor before relying on it.

Test 3: What does a wrong answer cost? A slightly vague answer about office hours costs little. A wrong answer about a refund, a contract term, a security practice, or a compliance commitment can cost real money or trust. Put high-cost topics on a "never answer, always escalate" list.

A working sort

Question typeHypothetical exampleSuggested bot behavior
Published, universal, low risk"Do you work with agencies?"Answer from your content
Published but needs a next step"How do I get started?"Answer, then offer the next step you choose
Account-specific"Where is my deliverable?"Collect details, escalate to a person
Policy or legal"Can I get a refund?"Answer only if the policy is published and clear; otherwise capture the question and escalate
Urgent or emotional"Our site is down."Escalate immediately, with the clearest possible context
Out of scopeQuestions unrelated to your businessSay so politely and offer a human contact route

The examples are hypothetical and meant only to illustrate the categories. Build your own table from your real questions.

Step 2: Ground answers in content you control

A chatbot that answers from general knowledge can sound confident while being wrong about your business. A safer design restricts answers to material you've approved: your website pages, documentation, and any written instructions you provide.

That design has a direct consequence for your content. If the bot can only be as accurate as what you've published, then outdated pages produce outdated answers. Before launch, review the pages the bot will learn from:

  • Do pricing, service descriptions, and timelines match what you tell customers today?
  • Are there old pages that contradict newer ones?
  • Do your policies (refunds, cancellations, data handling) exist in writing anywhere public?

Where a policy isn't published, decide now whether to publish it or to route that topic to a person. Don't leave it to the bot to fill the gap.

What the bot should do when it doesn't know

This is the most important behavior to test. A good support bot says plainly that it couldn't find the answer, then offers a next step, such as capturing the question for your team. A bad one invents something plausible. When you evaluate any tool, ask a question you know isn't covered on your site and watch the response.

Step 3: Design the overnight handoff

Plan the moment the bot says "let me get someone to help you" at a time when nobody is available. Decide in advance what happens next.

Set an honest expectation. Tell the customer a person will follow up, and say when, but only promise what your team can deliver. If nobody works weekends, the bot shouldn't imply a weekend reply.

Collect only what the follow-up needs. Decide the minimum your team needs to act, typically a way to reach the customer and a clear description of the issue. If your chat analytics show visitors leaving at the contact-details prompt, test removing a field. Ask one question at a time, and only when the answer changes what happens next.

Pass the context along. The person picking this up the next morning shouldn't have to re-ask what the customer already said. The handoff should include the contact details, the customer's question in their words, what the bot already told them, and why it escalated.

Separate urgent from routine. Decide in advance which topics should trigger a faster path, such as a phone number or an on-call alert if your team has one. Don't let an outage report sit in the same queue as a question about invoicing formats.

Assign an owner. Every escalated conversation needs a named person or role responsible for reading it. If nobody owns the queue, the chatbot just delays the unanswered question.

Step 4: Test before customers see it

You don't need special tooling to run a useful pre-launch test. Here is a reader-run check with clear inputs and observable results.

Inputs: A set of real questions from your inbox, including some you expect the bot to answer, some it should escalate, and some it should decline.

Process:

  1. Run each question through the bot as a customer would, using natural wording, not copied phrasing from your site.
  2. Label each result: correct and complete, correct but unhelpful, wrong, or appropriately escalated.
  3. For escalations, open the resulting handoff and check whether someone could act on it without asking the customer anything again.
  4. Repeat with a few deliberately tricky inputs: a question with a typo, a vague two-word message, a question that combines two topics, and a request that should never be answered automatically.

Decision rule: Any "wrong" label on a high-cost topic is a blocker. Fix the source content or move that topic to the escalate-always list, then retest. If you see many "correct but unhelpful" results, check whether the pages the bot learns from are missing the detail the question needed.

After launch, keep reading transcripts on a regular schedule. Questions the bot couldn't answer are a to-do list for your website content. Whether the bot reduces your support workload is something to confirm in your own numbers, such as how many questions reach your team and how long they take to resolve. Don't assume it.

Where LeadPilot fits

LeadPilot is an AI sales agent for your website, built around pre-sale conversations: visitors with questions about your business who may become customers. These are the mechanics that map to the design principles above:

  • It learns from your website. You enter your URL, and LeadPilot reads the accessible pages to build a knowledge base from your real content. Answers stay grounded in business content you approve.
  • It admits gaps. If an answer is missing, LeadPilot says so instead of inventing it. For example, when asked about refunds with no published policy, it says it couldn't find one and offers to capture the question for your team.
  • It engages visitors 24/7. The agent is available whenever a visitor lands, including outside your business hours.
  • It qualifies without turning chat into a form. It asks one relevant question at a time, and only when the answer changes fit or the right next step.
  • It captures contact details and passes along context. When a lead is captured, LeadPilot sends the contact details, buying context, and conversation to your team.
  • You choose the outcome. You can set the agent to capture a lead, start a signup, or book a call, and add a booking link for when intent is clear.

LeadPilot has a free plan that needs no credit card, and it works with public website pages. If your biggest need is answering prospects' questions and routing the serious ones to your team overnight, that fits.

If most of your questions turn out to be account-specific, test whether any tool, including LeadPilot, handles them to your standard before you commit. The sort in Step 1 will tell you which category most of your questions fall into.

Build your LeadPilot agent to see how it answers questions from your own website content and what the handoff looks like for your team.

Questions to ask any vendor before you commit

Whatever tool you evaluate, put these questions to the vendor and verify the answers in a trial:

  • What content does the bot learn from, and can I control it?
  • What does the bot do when it has no answer?
  • Which topics can I block from automated answers?
  • What exactly does my team receive when a conversation escalates?
  • Where can I read full transcripts?
  • What happens to conversations that start outside my team's working hours?

If the vendor's answers are specific and the trial matches them, you have what you need to make the decision. If they stay vague, treat that as information about the tool.

Put your website conversations to work

Build a LeadPilot agent from your website and review it before you publish.

Start Free