AI Deployment FAQ

A foundation model is a large pre-trained model developed on broad datasets by an AI vendor or research organization. A fine-tuned model starts from a foundation model and is further trained on organization-specific data to improve performance on domain-specific tasks. Fine-tuning can improve accuracy on specific task types but requires sufficient labeled training data, careful evaluation to avoid overfitting, and a process for managing model versions as the fine-tuned model is updated over time.

Infrastructure requirements depend significantly on the deployment model. Cloud-hosted AI platforms (SaaS or managed API) require reliable internet connectivity, API authentication and rate limit management, and integration with the organization's identity and access management systems. Self-hosted or on-premise deployments require GPU or accelerator compute for inference, storage for model artifacts and data, a container orchestration environment, and monitoring infrastructure. For organizations with data residency requirements, infrastructure location in Canada may be required.

Model versioning in production requires recording the model artifact, training data snapshot, evaluation results, and deployment date for each version deployed. A version registry — whether provided by the AI platform or maintained separately — allows rollback to prior versions when a new version underperforms. Versioning also supports audit requirements in regulated environments by enabling reconstruction of what model version was operating at any given time.

Pilot duration varies by use case complexity, but most enterprise AI pilots run for four to twelve weeks to ensure exposure to representative operational conditions, including any seasonal or cyclical variation in inputs. Pilots shorter than four weeks often do not surface edge cases or performance issues that appear with broader operational exposure. The pilot evaluation criteria and decision criteria for proceeding to broader rollout should be defined before the pilot begins rather than at its conclusion.

Governance structures for enterprise AI typically include: an accountable owner for each deployed system (a business leader with accountability for outcomes, not only a technical owner); a defined escalation path for issues and errors; a model review cadence for assessing ongoing performance; a documentation standard for model capabilities, limitations, and intended use; and an incident response process for safety-critical or compliance-relevant failures. In regulated industries, governance requirements may also include regulatory reporting for automated decision systems that affect individuals.

Production AI system failure handling should be designed before deployment, not improvised when a failure occurs. Key elements include: monitoring alerts that trigger when performance degrades below defined thresholds; a documented fallback process (typically the pre-AI manual workflow) that can be activated rapidly; a communication protocol for notifying affected teams; an investigation process for root cause analysis; and a post-incident review that produces remediation recommendations. The fallback process should be tested periodically rather than only relied on when a real failure occurs.

Success metrics should include both technical and business outcome indicators. Technical indicators include inference latency, error rates, and model performance metrics (precision, recall, or regression accuracy depending on task type). Business outcome indicators should be tied to the specific objectives the deployment was designed to address — processing time reduction, error rate improvement, cost per transaction change, or customer satisfaction where applicable. Override rate (how often human reviewers override AI recommendations) is a diagnostic metric that provides signal on model reliability and user trust that neither technical nor business metrics capture directly.

Vendor risk assessment for AI vendors should examine financial stability (particularly for startup vendors without established enterprise customer bases), the vendor's data processing practices and willingness to sign appropriate data processing agreements, the existence of enterprise-grade SLAs for uptime and support response, reference customers in comparable sectors or of comparable organizational complexity, and the vendor's development roadmap and history of delivering on announced capability timelines. For mission-critical deployments, continuity planning should include a documented process for migrating to an alternative platform if the vendor relationship is disrupted.