Explainable AI in Manufacturing: When a Black Box Is Unacceptable
Why manufacturing AI cannot remain opaque: explainability levels, XAI methods, regulatory requirements, and the accuracy-transparency tradeoff.
Serhiy Andrejev
Lead Machine Learning Developer
In critical manufacturing environments, "the model said so" is never an acceptable answer. When AI decisions affect physical processes, product quality, and human safety, every stakeholder — from the shop floor operator to the external auditor — must understand why the system produced a particular recommendation. This article examines the levels of AI explainability, practical approaches to achieving it, and the regulatory landscape that is making transparency a non-negotiable requirement.
Introduction
Consider a scenario observed repeatedly across industrial settings: a predictive maintenance model flags a critical turbine bearing for immediate replacement. The maintenance team faces a four-hour shutdown and a $200,000 parts cost. The model's confidence score reads 94%. Yet the shift engineer hesitates — not because the model is wrong, but because no one on the floor can articulate why the model arrived at this conclusion.
This is not a hypothetical edge case. According to a 2024 survey by npj Advanced Manufacturing, despite AI's technological maturity in manufacturing, organizations remain hesitant to fully rely on opaque systems, preferring to retain human judgment in critical decisions [1]. The 2024–2025 State of AI in Manufacturing Survey reports that 82% of manufacturers are planning budget increases for AI implementation [2], yet adoption consistently stalls at the point where trust must be established between the human operator and the algorithmic recommendation.
In our practice of deploying AI across industrial environments, we observe a consistent pattern: the technical accuracy of a model is necessary but insufficient for operational adoption. A model that cannot explain itself is a model that will be overridden, ignored, or — worse — blindly trusted at the wrong moment. The challenge is not merely technical. It is organizational, regulatory, and deeply human.
The convergence of three forces makes explainable AI (XAI) in manufacturing an urgent priority in 2026. First, the EU AI Act, which entered into force in August 2024, establishes binding transparency obligations for high-risk AI systems — a category that encompasses most safety-related manufacturing applications [3]. Second, international standards such as ISO/IEC 42001 now provide certifiable frameworks for AI governance that explicitly require transparency mechanisms [4]. Third, the operational reality of Industry 4.0 and 5.0 transition demands that AI systems function as collaborative partners, not inscrutable oracles.
Who Needs Explainability and Why
A common misconception treats explainability as a single, uniform requirement. In reality, different stakeholders within a manufacturing organization need fundamentally different types of explanations, at different levels of abstraction, and for different purposes.
The operator interacts with AI recommendations in real time, often under time pressure. For the operator, explainability means actionable context: which sensor readings triggered the alert, how confident the system is, and what the recommended response entails. Without this context, the operator faces a binary choice between blind compliance and outright dismissal — neither of which represents effective human-AI collaboration. Based on our observations in production environments, operators who receive contextual explanations alongside AI recommendations follow system guidance approximately 85–90% of the time, compared to 40–60% when presented with raw predictions alone.
The technologist or process engineer requires a deeper form of understanding. They need to validate that the model's internal logic aligns with known physical and chemical processes. If a quality prediction model assigns high importance to ambient humidity but ignores raw material batch composition, the technologist can identify a potential modeling flaw before it manifests as a production failure. This validation function is particularly critical during model deployment and periodic review cycles.
The auditor — whether internal quality assurance or external regulatory — requires documented evidence that the AI system operates within defined parameters, that its decision logic is traceable, and that appropriate governance controls exist. The auditor is less concerned with the mathematical elegance of SHAP values and more concerned with whether the organization can demonstrate systematic oversight. With the EU AI Act mandating technical documentation and transparency for high-risk systems, this is no longer a voluntary best practice but a legal obligation [3].
The manager or executive needs explainability to allocate responsibility. When an AI-influenced decision leads to a defective batch or an equipment failure, the organization must be able to reconstruct the decision chain. Explainability enables accountability — not to assign blame, but to enable systematic improvement and to meet fiduciary obligations.
The following matrix illustrates how different explainability types map to stakeholder needs:
Table 1. Stakeholder–Explainability Needs Matrix
| Stakeholder | Global Explainability | Local Explainability | Counterfactual Explainability |
|---|---|---|---|
| Operator | Optional — general awareness of model behavior | Critical — must understand each specific recommendation | Important — "what would change the outcome?" aids decision-making |
| Technologist | Critical — must validate overall model logic against domain knowledge | Critical — must verify individual predictions against process physics | Important — supports root cause analysis and model refinement |
| Auditor | Critical — must confirm model operates within documented parameters | Important — spot-check capability for compliance verification | Optional — relevant for bias and fairness assessments |
| Manager | Important — strategic understanding of AI capabilities and limitations | Important — accountability for specific high-impact decisions | Optional — relevant for strategic "what-if" planning |
Color coding: Critical = required for role fulfillment; Important = significantly enhances effectiveness; Optional = beneficial but not essential. Source: author's assessment based on industrial deployment experience.
Levels of Explainability
The XAI research community, significantly advanced by the DARPA XAI program (2017–2021), has established a taxonomy of explainability that maps directly to the stakeholder needs outlined above [5].
Global Explainability
Global explainability addresses the question: "How does this model work in general?" It provides an aggregate view of the model's behavior across its entire operating range — which features matter most, how they interact, and what general decision boundaries the model has learned. For a predictive quality model in a food processing plant, global explainability might reveal that temperature profiles during pasteurization and raw material acidity are the two dominant factors, together accounting for 70% of the model's predictive power.
Global explanations serve primarily the technologist and auditor. They enable domain experts to verify that the model has learned relationships consistent with established process knowledge, and they provide auditors with the high-level documentation needed for governance purposes.
Local Explainability
Local explainability answers a more specific question: "Why did the model make this particular decision?" For each individual prediction, local explanation methods quantify the contribution of each input feature. When a quality control model rejects a specific production batch, local explainability can show that the rejection was driven primarily by an anomalous viscosity reading at processing stage three, with a secondary contribution from elevated ambient temperature.
This is the level most critical for operators and front-line engineers. It transforms an opaque verdict into an actionable insight. In our experience, the difference between a system that says "reject batch" and one that says "reject batch: viscosity at stage 3 exceeded threshold by 12%, ambient temperature 3°C above nominal" is the difference between a tool that disrupts workflow and one that enhances it.
Counterfactual Explainability
Counterfactual explanations address a forward-looking question: "What would need to change for the model to produce a different outcome?" Rather than explaining what did happen, they illuminate what could happen under modified conditions. For a predictive maintenance model, a counterfactual might indicate: "If bearing vibration amplitude remained below 4.2 mm/s, the maintenance alert would not have been triggered."
Counterfactual explanations are particularly valuable for process optimization and root cause analysis. They enable technologists to identify the minimal parameter adjustments needed to achieve a desired outcome, effectively turning the AI model into an optimization tool rather than merely a classification engine.
XAI Approaches: Interpretable Models vs. Post-Hoc Explanations
The field of explainable AI offers two fundamentally different strategies, each with distinct advantages and limitations in manufacturing contexts.
Inherently Interpretable Models
The first approach uses model architectures that are transparent by design — linear regression, logistic regression, decision trees, and rule-based systems. Their internal logic is directly readable by a human expert. A decision tree that routes predictions through a series of threshold comparisons on temperature, pressure, and flow rate is inherently understandable to a process engineer familiar with those parameters.
The advantage is genuine transparency: there is no gap between what the model computes and what the explanation conveys. The limitation is equally clear — these model families cannot capture the complex, nonlinear interactions that characterize many real-world manufacturing processes, resulting in lower predictive accuracy for high-dimensional problems.
Post-Hoc Explanation Methods
The second approach accepts model complexity and applies explanation techniques after training. This strategy allows the use of high-accuracy models — gradient boosting ensembles, deep neural networks, transformer architectures — while generating explanations through auxiliary analysis.
The most widely adopted post-hoc methods in manufacturing applications include:
SHAP (SHapley Additive exPlanations), introduced by Lundberg and Lee in 2017, applies game-theoretic Shapley values to quantify each feature's contribution to a specific prediction [6]. SHAP provides both local explanations (feature contributions for individual predictions) and global explanations (aggregate feature importance). Its theoretical grounding in cooperative game theory guarantees several desirable properties: consistency, local accuracy, and missingness. A recent systematic review confirms a definitive shift toward SHAP as the preferred method in manufacturing applications [7].
LIME (Local Interpretable Model-agnostic Explanations), proposed by Ribeiro et al. in 2016, generates local explanations by fitting a simple, interpretable model in the neighborhood of the prediction being explained [8]. LIME is model-agnostic, computationally efficient, and produces intuitive explanations. Its limitation lies in the instability of explanations — small perturbations to the input can yield different local approximations.
Feature Importance and Permutation Methods quantify the impact of each feature on model performance by measuring the degradation in accuracy when feature values are randomly shuffled. These methods are straightforward and model-agnostic but provide only global-level insights and cannot explain individual predictions.
Attention Mechanisms, relevant for transformer-based and sequence models, provide inherent visibility into which parts of the input the model focuses on when generating predictions. While attention weights do not constitute causal explanations, they offer useful diagnostic information, particularly for time-series analysis in manufacturing contexts.
Table 2. Comparison of XAI Approaches for Manufacturing Applications
| Method | Explanation Type | Scope | Computational Cost | Key Limitation | Manufacturing Applicability |
|---|---|---|---|---|---|
| Decision Trees | Inherent | Global + Local | Low | Limited accuracy on complex data | Simple threshold-based quality checks, alarm logic |
| Linear/Logistic Regression | Inherent | Global | Low | Cannot capture nonlinear relationships | Process parameter correlation, trend analysis |
| SHAP | Post-hoc | Global + Local | Medium–High | Computationally expensive for large models | Predictive maintenance, quality prediction, anomaly root cause |
| LIME | Post-hoc | Local | Medium | Explanation instability; sensitivity to perturbation | Real-time quality classification, defect detection |
| Permutation Importance | Post-hoc | Global | Medium | No local explanations; correlated features bias | Feature selection, model validation |
| Attention Weights | Inherent (partial) | Local | Low (built-in) | Not causal; interpretation requires domain knowledge | Time-series prediction, sequential process monitoring |
Sources: [6], [7], [8]
Figure 1. Radar comparison of XAI methods across five criteria derived from Table 2. Scope Breadth: Global+Local = 5, single scope = 3. Computational Efficiency: Low = 5, Medium = 3, Medium-High = 2. Model Complexity Support: works with any model = 5, inherent-only = 1. Explanation Stability: theoretically grounded = 5, unstable = 2. Manufacturing Versatility: broad applicability = 5, narrow = 2. SHAP demonstrates the strongest overall profile for manufacturing use. Sources: [6], [7], [8].
Regulatory Requirements: From Best Practice to Legal Obligation
The regulatory environment for AI transparency in manufacturing has shifted fundamentally in the past two years. What was once a domain of voluntary guidelines and industry best practices is now a landscape of binding regulations and certifiable standards.
The EU AI Act
The EU AI Act (Regulation 2024/1689), adopted in June 2024, establishes the world's first comprehensive legal framework for artificial intelligence [3]. For manufacturing, the critical provisions relate to high-risk AI systems — a classification that applies to AI used in safety components, critical infrastructure management, and systems that affect worker safety.
Article 13 mandates that high-risk AI systems be designed and developed in a manner that ensures their operation is sufficiently transparent, enabling deployers to interpret system output and use it appropriately. Article 11 requires comprehensive technical documentation, including information about model architecture, training procedures, and validation results. Article 14 establishes human oversight requirements, which implicitly demand a level of explainability sufficient for meaningful human intervention.
For manufacturing organizations operating in the EU market — or supplying products and components to EU customers — compliance with these transparency obligations is not optional. The regulation provides a phased implementation timeline, with obligations for high-risk systems taking effect in stages through 2027. Based on our assessment, organizations that begin building explainability infrastructure now will find themselves at a significant competitive advantage as enforcement begins.
ISO/IEC 42001:2023
ISO/IEC 42001, published in December 2023, is the first international standard for AI management systems [4]. While it does not prescribe specific XAI techniques, it requires organizations to establish policies and processes for AI transparency, including documentation of AI system capabilities, limitations, and decision logic. Certification under ISO/IEC 42001 provides organizations with a recognized framework for demonstrating responsible AI governance — an increasingly valuable credential in regulated supply chains.
NIST AI Risk Management Framework
The NIST AI RMF 1.0, released in January 2023, identifies explainability and interpretability as core characteristics of trustworthy AI [9]. While voluntary in nature, the framework's influence extends across U.S. manufacturing and its global supply chains. The framework's "Map-Measure-Manage" structure provides practical guidance for integrating explainability into AI system development and deployment.
For manufacturing organizations with global operations, the convergence of these three regulatory frameworks — EU AI Act, ISO/IEC 42001, and NIST AI RMF — creates a de facto international standard for AI transparency. We consider it important to note that this regulatory convergence is not a burden to be minimized but an opportunity to systematize practices that directly improve operational reliability.
Table 3. Regulatory Framework Comparison: AI Transparency Requirements
| Dimension | EU AI Act (2024/1689) | ISO/IEC 42001:2023 | NIST AI RMF 1.0 |
|---|---|---|---|
| Legal status | Binding regulation (EU) | Voluntary certifiable standard | Voluntary framework |
| Scope | High-risk AI systems | All organizations using AI | All organizations using AI |
| Transparency mandate | Explicit (Articles 11, 13, 14) | Implicit via governance requirements | Explicit via trustworthiness characteristics |
| Documentation | Technical documentation required | AI policy and process documentation | Risk documentation recommended |
| Human oversight | Mandatory for high-risk systems | Recommended via management controls | Addressed through "Govern" function |
| Enforcement | Fines up to €35M or 7% of turnover | Audit-based certification | No enforcement mechanism |
| Timeline | Phased through 2027 | Certification available now | Available since January 2023 |
Sources: [3], [4], [9]
The Accuracy–Explainability Balance
A persistent concern in discussions of XAI is the perceived tradeoff between model accuracy and explainability. The argument is straightforward: more complex models (deep neural networks, large ensembles) tend to deliver higher predictive accuracy, while more interpretable models (linear regression, decision trees) sacrifice performance for transparency. Research confirms that this tradeoff is real in many settings [1].
Figure 2. Accuracy vs explainability trade-off across common ML model classes used in manufacturing. Models in the right half (high explainability) are generally acceptable for safety-critical decisions; left-half models require additional XAI tooling for production deployment. Source: author's assessment based on industry benchmarks.
However, based on our implementation experience, we believe this tradeoff is frequently overstated and often misunderstood.
First, the magnitude of the accuracy gap depends heavily on the problem domain. For many manufacturing applications — particularly those with well-understood physics and moderate dimensionality — interpretable models achieve performance levels within 2–5% of their black-box counterparts. A decision tree ensemble for classifying product defects based on 15–20 process parameters often performs comparably to a deep neural network, while offering dramatically better transparency.
Second, post-hoc explanation methods substantially narrow the gap. Applying SHAP or LIME to a complex model preserves the model's predictive performance while providing usable explanations. Recent research in predictive maintenance has demonstrated that SHAP-guided feature refinement can actually improve model performance while simultaneously enhancing interpretability — for example, improving an SVM model's recall from 0.704 to 0.716 through explanation-driven feature selection [10].
Third, and most importantly, raw accuracy is not the sole measure of operational value. A model that achieves 97% accuracy but is routinely overridden by skeptical operators delivers less real-world value than a model that achieves 94% accuracy but is consistently trusted and acted upon. In manufacturing, the relevant metric is not model accuracy in isolation but system-level performance — the combined output of the AI model and the human decision-maker working together.
We recommend that manufacturing organizations adopt the following framework for navigating the accuracy-explainability balance:
-
Safety-critical applications (pressure vessel integrity, toxic emission monitoring, explosion risk assessment): prioritize explainability. Use inherently interpretable models where possible; apply rigorous post-hoc explanation methods where complex models are necessary. No model should be deployed in a safety-critical role without a documented, validated explanation capability.
-
Quality-critical applications (product specification compliance, process parameter optimization): balance accuracy and explainability based on the cost of errors. Use post-hoc explanation methods as standard practice; invest in global explainability for model validation.
-
Efficiency-oriented applications (energy optimization, scheduling, demand forecasting): accept lower explainability thresholds where the consequences of individual errors are bounded and reversible. Even here, maintain global-level explainability for model governance and continuous improvement.
Author's Assessment and Conclusions
The transition from black-box AI to explainable AI in manufacturing is not a technical luxury — it is an operational necessity and, increasingly, a legal requirement. Based on our experience deploying AI across diverse industrial environments, we offer the following conclusions.
The explainability challenge in manufacturing is fundamentally a stakeholder alignment challenge. There is no single "explanation" that satisfies all needs. A robust XAI strategy must deliver different explanation types — global, local, and counterfactual — calibrated to the requirements of operators, technologists, auditors, and management. Organizations that attempt to address explainability as a monolithic technical problem will find their solutions inadequate for at least some stakeholder groups.
The regulatory trajectory is unambiguous. The EU AI Act, ISO/IEC 42001, and NIST AI RMF collectively establish that AI transparency is a governance requirement, not an optional feature. Manufacturing organizations that embed explainability into their AI development processes now — rather than retrofitting it under regulatory pressure — will achieve both better compliance outcomes and better operational performance.
The accuracy-explainability tradeoff, while real, is manageable. Modern post-hoc explanation methods, particularly SHAP, provide robust, theoretically grounded explanations for complex models without requiring accuracy sacrifices. The shift toward local, model-agnostic methods documented in recent systematic reviews [7] reflects the manufacturing sector's growing sophistication in applying these techniques.
Finally, we consider it important to note that explainability is not merely a risk mitigation measure — it is a value creation mechanism. AI models that explain their reasoning enable faster operator response, more effective process optimization, better root cause analysis, and more productive human-AI collaboration. In our practice, the most successful AI deployments are those where explainability was designed into the system from the outset, not appended as an afterthought.
The manufacturing sector stands at an inflection point. The organizations that will capture the greatest value from AI are those that treat transparency not as a constraint on AI capability but as a foundation for AI trustworthiness — and, ultimately, for AI effectiveness.
References
- Ahmed, I. et al. "Towards next-gen smart manufacturing systems: the explainability revolution." npj Advanced Manufacturing, 2024. https://www.nature.com/articles/s44334-024-00006-9
- Rootstock. "2024–2025 State of AI in Manufacturing Survey." 2025. https://clouderp.rootstock.com/state-of-ai-manufacturing-survey-2025
- European Parliament and Council. "Regulation (EU) 2024/1689 — Artificial Intelligence Act." Official Journal of the European Union, 2024. https://artificialintelligenceact.eu/ai-act-explorer/
- ISO/IEC. "ISO/IEC 42001:2023 — Information technology — Artificial intelligence — Management system." 2023. https://www.iso.org/standard/81230.html
- Gunning, D. et al. "DARPA's explainable AI (XAI) program: A retrospective." Applied AI Letters, 2(4), 2021. https://onlinelibrary.wiley.com/doi/full/10.1002/ail2.61
- Lundberg, S. M. and Lee, S.-I. "A Unified Approach to Interpreting Model Predictions." Advances in Neural Information Processing Systems 30 (NeurIPS), 2017. https://arxiv.org/abs/1705.07874
- Gajjar, N. et al. "A review of explainable AI methods and their application in manufacturing systems." Discover Applied Sciences, 2025. https://link.springer.com/article/10.1007/s42452-025-07908-z
- Ribeiro, M. T., Singh, S., and Guestrin, C. "'Why Should I Trust You?': Explaining the Predictions of Any Classifier." Proceedings of the 22nd ACM SIGKDD, 2016. https://dl.acm.org/doi/10.1145/2939672.2939778
- NIST. "AI Risk Management Framework (AI RMF 1.0)." 2023. https://www.nist.gov/itl/ai-risk-management-framework
- Moraliyage, H. et al. "Trade-Off Between Interpretability and Accuracy: How Can XAI Build Trust in Track Geometry Predictive Maintenance?" 2024. https://iris.cnr.it/handle/20.500.14243/562581