Ensuring algorithmic transparency through public contracts?

Various approaches to algorithmic transparency implementation

The ongoing discussion on algorithmic transparency has many strands and dimensions, and most of them are already well covered in this symposium. Against the big ideas developed at the constitutional level, discussion of implementation mechanisms may seem niche or fringe, but it is important. I am interested in how to deliver the desired level of algorithmic transparency in the context of public-sector AI adoption. To me, ensuring algorithmic transparency requires effective regulation of access to the internal workings of an algorithm and, especially, source code. Meaningful access to the source code can require access to related know-how and technical documentation. Such algorithmic transparency is primarily constrained by economic factors, such as commercial secrets, and the costs of disclosing information.¹Kossow, Windwehr and Jenkins, 2021. It is related to but distinct from algorithmic explainability, even if the latter affects algorithmic transparency more broadly.

For the purposes of this discussion, it seems clear that the adoption of AI in the public sector cannot be entirely opaque, but also that it will not be entirely transparent. Whichever way it is determined, a balanced approach to competing (public and private) interests in algorithmic transparency will have to be implemented. Implementation mechanisms can range from general legislation to voluntary codes of practice, covering many intermediate instruments such as sectoral legislation or statutory guidance. A hybrid approach is, of course, possible—and the proposed EU AI Act is an example of a tiered choice of implementation mechanisms dependent on AI use. An alternative tiered approach could be built around the institutional characteristics of the organisation adopting the AI. The choices of implementation mechanisms are not endless, but they are varied.

Regulating AI transparency by (public) contract

There is an emerging line of thought that suggests that the introduction of contractual obligations in public contracts is a plausible way forward. Such obligations would establish a nuanced regime of algorithmic disclosure to the public buyer and, in turn, to the public (which was the approach followed in contracts for covid-19 tracing apps2Palmiotto Ettore, 2022). Contractual obligations would facilitate the setting of different rights to access and disseminate information on the workings of the relevant algorithm to different recipients and for different purposes, such as: 

1. the administration of internal processes of the public sector entity in relation to the use of the AI solution (ie management of the information in a closed environment); 
2. the management of the contract, including its retendering (which could be in a semi-closed or permissioned environment); and/or 
3. the publication of relevant information to comply with mandatory or voluntary algorithmic disclosure obligations, or with open source policies (open environment). 

Roughly, this is the pragmatic view of the European Commission, which is sponsoring the development of standard contractual clauses for the procurement of ethical AI. It is also the general approach advanced by prominent European³See eg Martini, 2019. and US scholars⁴See eg Coglianese & Lampmann, 2021. In this post, I offer some observations on the use of public procurement to ensure algorithmic transparency through contract, taking the prism of the top layer of procurement regulation: trade law.

Procurement as a (limited) exception to algorithmic secrecy in trade law

A first observation is that the role of procurement for the implementation of horizontal policies (eg green or social procurement, or now procurement for digital governance) is generally constrained by international trade rules. Given the protective approach towards algorithmic secrecy in most recent trade agreements,5Słok-Wódkowska & Mazur, 2022 it is interesting to see that procurement-related algorithmic disclosure is emerging as an exception to such opacity. An interesting recent analysis6Irion, 2022 shows that source code protection is potentially excluded in the context of public procurement in all emerging treaties. Generally, the treaties either do not prohibit, or have an explicit exception for, source code transfers in the context of commercially negotiated contracts—which can in principle include contracts with the public sector. More clearly, under what can be labelled as the ‘EU approach’, there is an explicit carve-out for ‘the voluntary transfer of or granting of access to source code for instance in the context of government procurement’ ⁷See Article 8.73 EU-Japan EPA; similarly, Article 207 EU–UK TCA; and Article 9 EU-Mexico Agreement in principle. This means that the EU is clear in the general approach to facilitating the use of procurement as a mechanism to implement algorithmic transparency. This approach is being extended to other players⁸Perhaps unsurprisingly, the UK; UK-Japan CEPA and may well become the global standard.

However, things are never entirely straightforward, and this approach leaves a few tricky issues open. For example, what are the allowable (negotiated) limits to procurement-related transparency? It does not seem entirely outlandish to suggest that, being an exception to the general protection of source code under the relevant trade agreements, procurement-related disclosures should be strictly limited to what is necessary to ensure the full enjoyment of contractual rights by the public buyer and compliance with legal requirements. In the EU, there would be additional arguments along those lines based on the Trade Secrets Directive.⁹See Maggiolino, 2019. That could be seen narrowly, eg in relation to internal uses only ((i) above) and eg to allow compliance with legal requirements in relation to contract management ((ii) above). Such uses would in principle seem relatively unproblematic. 

It would however require a rather expansive view to construct the procurement-related exception as covering the subsequent public disclosure of algorithmic information, and in particular the source code, at least where this is not strictly legally mandated ((iii) above). Such an expansive view would exceed eg the boundaries of algorithmic transparency foreseen in the proposed EU AI Act.¹⁰See Art 64. Ultimately, the boundary between trade law permitted procurement-related disclosure, on the one hand, and trade law prohibited general access to or transfer of source code, on the other, can become contentious—at least where public disclosure of algorithmic information is quite detailed, and in relation to open source policies. Functionally, procurement-related disclosures could generate the same technology transfer effect that the general trade rules protecting algorithmic secrecy seek to avoid. This thus requires some further thought.

Algorithmic transparency is a procurement governance requirement, but also a legal one?

There should be no question that (some level of) algorithmic transparency will crystallise as a fundamental digital procurement governance requirement.¹¹See eg Sanchez-Graells, 2022. However, one of the key issues in establishing the scope of contract-based (permissible) procurement-related algorithmic disclosures may concern the legal status of such disclosures. Where such disclosures are legally mandated, there is a strong argument to fit them under the exception to trade law-enabled algorithmic opacity (and, similarly, trade secrets protection¹² Maggiolino, 2019.). However, where such disclosures are voluntary and simply a matter of best practice, the issue can become contentious (as above). It is thus relevant to take a closer look at the emerging differential status of various types of procurement-related disclosures.

First, it is worth highlighting that, under EU law, there is an absolute obligation for public buyers to access source code and to ensure they have the right to communicate it to third parties in the context of generating a level playing field for the award of public contracts. This was recognised by the Court of Justice of the European Union (CJEU) in a software procurement case, establishing that, to ensure compliance with the general principles of procurement law, contracting authorities must have access to the source code, they must communicate it to potential service providers, and ‘access to that source code [must] in itself [be] a sufficient guarantee that economic operators interested in the award of the contract in question are treated in a transparent manner, equally and without discrimination’ (see here for discussion). Under EU law, the disclosure of source code in controlled/permissioned environments for the purposes of contract management (above (ii)) is thus a legal mandate that should be covered by the contract-based trade exception.

By contrast, the broader disclosure of source code to comply with eg open source procurement policies is particularly challenging. It seems tenable that mandatory disclosure clauses would probably exceed the remit of an exception based on ‘voluntary’ transfer in the context of procurement.¹³See eg Słok-Wódkowska & Mazur, 2022, although this is not seen as a problem by all commentators, see eg Dorobantu, Ostmann & Hitrova, 2021. The imposition of the use of open source code as a procurement condition (so that any amendments to the code then also become open access) is also potentially problematic, at least in some trade proposals. This is bound to become contentious and will require detailed case-by-case analysis of the legal constraints on the procurement-related publication of algorithmic source code (above (iii)).

Second, the situation is also not so clear in relation to public disclosures short of disclosing the source code itself. A particularly tricky scenario would emerge from the proposal by Leslie and Kazim in this symposium to release executable versions of algorithms being used by the public administration (see part I and part II). They recognise that such disclosures could allow for some degree of reverse engineering of the relevant algorithms ‘thus potentially giving rise to intellectual property issues’. In my view, such risk of impingement on proprietary algorithmic opacity would in principle clash with trade rules, at least where the disclosure of executable versions solely relied on contractual obligations imposed by the public buyer. This stresses the importance of clarifying the legal status of public disclosures to generate algorithmic opacity short of disclosing the source code.

Taking the proposed EU AI Act, for example, (still partly undefined) transparency obligations will vary depending on the type of use. High-risk AI systems will be subject to transparency obligations,¹⁴Art 13(1) AI Act. as would some AI systems intended to interact with natural persons.¹⁵Art 52 AI Act. However, this leaves transparency requirements for other types of AI systems/uses to the potential development of codes of conduct,¹⁶Art 69 AI Act. or national legal requirements (eg in the context of the self-organisation and regulation of the public sector in particular). 

This can create a variety of approaches and opinion on whether contractual obligations linked to public disclosures fit or not the trade exemption, which will also depend on the delineation given to the undefined concept of ‘source code’ in those treaties, eg whether the protection is of the ‘strict’ source code only, or of all know how and information relating to the source code which revelation can compromise the interests of the rights holder¹⁷See Irion, 2022—which in part relates to emerging approaches not only to protect the source code, but also ‘algorithms expressed in that source’.¹⁸See eg Słok-Wódkowska & Mazur, 2022. In the EU context, this can be further complicated by the protective system of the Trade Secrets Directive,¹⁹Editorial note: see also the post by Ida Varošanec in this symposium. which explicitly refers in its preamble to the heightened risks of excessive disclosure in procurement settings.²⁰For discussion, see Maggiolino, 2019. This type of assessment can be particularly tricky in relation to voluntary disclosure approaches—such as the algorithmic transparency standard currently piloted in the UK. The simplest way out of this would be to legally mandate compliance with harmonised algorithmic transparency requirements across the public sector on public interest grounds. However, this would run against the flexibility and nuance of approach sought with the ‘transparency by contract’ approach.

Can public buyers deal with this?

Given the emerging role of procurement as a tool of algorithmic transparency/governance, it is worth considering whether public buyers are in a good position to play their part. This should be assessed by reference to a rather straightforward implication of the analysis above, which is that the emerging regulatory landscape is complex (even if only the trade dimension is considered) and that flexibility breeds complexity. If algorithmic transparency requirements could be formulated as hard, simple rules, the challenge would be smaller than seeking to create tailored approaches that require the exercise of discretion and eg the management of different tiers of access to information. Using public contracts to design and operate governance mechanisms requires expertise and capacity. Moreover, discharging the role of algorithmic transparency gatekeeper triggers additional obligations, eg to secure that information and protect it from cybersecurity threats. Holding algorithmic information generates governance risks and, ultimately, the issue of managing algorithmic transparency is closely interlinked with many new digital governance requirements facing public buyers.²¹See eg Sanchez-Graells, 2022.

Unfortunately, most public buyers are currently ill-prepared to effectively fulfil their digital governance roles due to the digital skills gap in the public sector,²²See eg this concerning recent report. as well as broader power imbalances between the (not almighty) public buyer and private vendors in highly-concentrated industries. This generates the risk that algorithmic governance by contract may be quickly growing into a giant with feet of clay—which highlights the importance of quickly building up public sector digital capacity.²³See eg recent Canadian recommendations.

Conclusion

The observations above show that careful and detailed analysis is required before a specific implementation mechanism is chosen to deliver algorithmic transparency—or digital governance, more generally. When it comes to the use of procurement to ‘regulate AI by contract’, a complex set of issues quickly emerges (even by only considering one of the multiple layers of regulation). Effectively managing those issues will require a quick expansion of the digital capability of the public sector and, in some cases, the creation of new regulatory bodies—or at the very least, an expansion of the oversight powers and material means of existing regulators. Discussions of digital constitutionalism need to (also) pay attention to these implementation issues if the goals and ideals developed in this field are to gain traction in practical terms.

Acknowledgements

This post has been developed in the context of a British Academy Mid-Career Fellowship to support the project Digital technologies and public procurement. Gatekeeping and experimentation in digital public governance.

Suggested citation

Albert Sanchez-Graells, ‘Ensuring algorithmic transparency through public contracts?’ (The Digital Constitutionalist, 24 November 2022). Available at https://digi-con.org/ensuring-algorithmic-transparency-through-public-contracts/