Satya Nadella, CEO of Microsoft, in remarks about the challenges of governing advanced artificial intelligence, emphasized that trustworthy AI systems should provide a minimum level of trust instead of relying on complete user trust. He raised this issue in a post published on the social network X.
Challenges of AI Governance
Nadella pointed to the governance challenges associated with advanced AI systems, stating that modern systems lack the mechanical clarity of older computer systems. He said: "While engineers in the past could link software operations to specific code paths, this clarity does not exist in contemporary boundary architectures."
Read more: Oman and the launch of a strategic program for advanced air transport tests
He continued: "When traditional software systems were established in the economy over the past few decades, we had tools and capabilities to track behaviors to a specific code path. This type of mechanical understanding in today's advanced AI systems, which have far greater capabilities than traditional software systems, is distant from us. We cannot attribute behaviors and model outputs to specific training data inputs or model weight settings. However, we engage these complex systems and grant them access to our most sensitive data, allowing them to take critical actions on our behalf!"
The Need for an Engineering Approach to Governance
Nadella further warned that one cannot rely solely on assurances from developers and should not view these deployments as black boxes. He emphasized that companies cannot delegate responsibility for automated actions to others and must separate the provision of raw intelligence from ultimate operational power.
He added: "We need to move towards constraints and governance with an engineering approach instead of solving the hard problem of coordination. We must surround uncertain models with robust and deterministic system designs, human controls, and reliable operational procedures, and establish industry standards where existing standards are inadequate."
Nadella then referred to risk management and suggested that closed and open weight models be considered potential internal threats. He stated that any agent with a score can fail or be compromised, and therefore organizations must isolate models through proven security practices at the organizational level.
Nadella noted: "Controls that govern what a model can access and what actions it can take must be external to the model. This is based on an information security principle that dates back to the 1970s, which is that a program should not be able to circumvent or manipulate the mechanisms that enforce its permissions."
He continued: "Today, this means separating the model from the system that coordinates its work and also the operational space that defines what actions it can take. This also means externalizing controls and safety measures."
Nadella introduced key principles for observing and governing these networks, including model diversity, tamper-resistant registration, continuous verifiability in specific cases, independent auditing, reliable cut-offs, and incident disclosure at the industry level.
Read more: Removal of the Bitchat app from the Apple Store in India · Microsoft unveils new Copilot features for workplace automation
