Measuring AI Business Impact in Sales and Customer Service
Evaluating an internal AI assistant through causal analysis, adoption research, and data pipelines spanning large-scale sales and service operations.
Context
We evaluated an internal AI assistant across sales and customer service for a payroll and HR services provider. Working within the data and AI function, we connected executive questions, data preparation, causal study design, and reporting. The evaluation distinguished tool adoption from business impact, accounting for differences in who used the assistant and the work they handled.
Approach
For sales, we applied staggered-adoption models and placebo checks to longitudinal representative data. For service, we built a three-stage DuckDB pipeline combining large-scale monthly interaction extracts, usage telemetry, and independently scored call-quality data.
We linked prompts to interactions through identity matching and time-zone-aware attribution windows, calibrating precision and recall. The evaluation addressed treatment timing, changing product availability, measurement consistency, and differential attrition. Sensitivity tests and agent-exclusion checks examined whether conclusions depended on particular observations or analytical choices.
We distinguished directly observed outcomes from proxy measures and assessed whether subgroup evidence could support broader conclusions.
Results
Leadership received impact studies spanning 40,000+ sales representative-months and 70+ million source service interactions, a 25-slide adoption report, and reproducible pipelines for AI investment review. Documentation made assumptions, validation checks, and the scope of inference explicit. Reporting distinguished supported findings from claims the analysis could not establish.
