Navigating AI in procurement

Learn more about AI in procurement.

On this page

Overview

Artificial intelligence tools, technologies and systems (collectively referred to as AI) create new opportunities for procurement. They also introduce new risks that buyers need to identify and manage.

This guidance helps buyers understand how AI can affect procurements, and what actions to take at each stage of a procurement process.

It covers:

  • procuring goods or services that include AI
  • using AI to support procurement activities
  • managing other risks where suppliers use AI in bidding or contract delivery.

This guidance applies broadly to procurement involving AI, but it is not an exhaustive list of all legal, policy or risk considerations. Buyers should read this guidance alongside other relevant policies.

As AI continues to evolve, this guidance will be updated.

Know when you are procuring AI

AI is usually procured in four main ways:

  • acquiring complete, ready-to-use AI or software that includes AI;
  • engaging suppliers to develop bespoke AI;
  • modifying or updating existing software to include AI; and
  • contracting a supplier where AI is an essential part of delivering the services.

When buyers know they are procuring AI, they can apply the relevant policies and manage AI appropriately. If buyers are not aware they are procuring AI, they can be exposed to AI risks. This includes when:

  • suppliers add AI to existing goods or services after contracting
  • suppliers use AI during service delivery that is unknown to the buyer.

See section ‘Managing additional AI risks in procurement’ below.

Procuring AI

Buyers should consider the following actions when procuring AI:

Procurement planning

  • Read existing AI policies and guidance relevant to your procurement (see AI policy and governance resources). Check regularly to stay up to date with current requirements and best practice.
  • Consider what risks may arise when suppliers use AI to prepare offers for procurement opportunities (see Managing additional AI risks in procurement).
  • Establish a suitably skilled project team. This includes experience with AI and AI policies and guidance. Engage with stakeholders that have expertise in:
    • AI ethics and human rights
    • cybersecurity
    • information security and data governance
    • intellectual property and copyright
    • IT systems infrastructure
    • law
    • privacy
    • sustainability.
  • Complete the VPS AI Assurance Framework to evaluate your AI use and to identify key risks. Include the completed framework with your procurement plan and formal approvals.
  • Undertake other assessments as identified by the VPS AI Assurance Framework. For example:
    • a privacy impact assessment to identify and mitigate privacy risks arising from AI in your procurement. Refer to the guidance offered by the Office of the Victorian Information Commissioner
    • a security risk assessment to identify AI-specific information security risks related to your procurement
    • update completed assessments as required throughout the procurement.
  • Where relevant, consider any opportunities for collaboration with other agencies. Agencies may have shared or overlapping needs for AI. Refer to the collaborative procurement guide for more information.

Market engagement

  • Where available, purchase AI through an existing Whole of Victorian Government purchasing arrangement (State Purchase Contract or Register).
  • If going to market, ensure that your procurement documentation requests relevant technical information to assess AI risk and suitability for your intended use case.
  • Consider requesting evidence of the supplier’s AI governance, such as internal policies, plans or processes, to ensure their AI use is lawful, fair, transparent and aligned with your agency’s policies.
  • When evaluating supplier offers, seek support from your multidisciplinary team to understand any risks or issues associated with the AI.
  • Ensure contract clauses address the identified AI risks. Seek assistance from legal advisors.

Contract management

  • Manage all identified risks related to the AI throughout delivery. This includes risks such as model drift, bias and hallucination. Periodically review the VPS AI Assurance Framework to assess new and recurring AI risks (e.g. quarterly).
  • Monitor supplier performance under the contract, including the reliability of the AI. The contract should require the supplier to address these issues as they arise and have controls in place to manage them (e.g. continuous monitoring, validation, human oversight).

Using AI to support a procurement process

  • While you may use AI to support you in the procurement process you remain accountable for the decisions and outcomes you make. Ensure that:
    • you have oversight and can explain the rationale behind your decisions or recommendations; and
    • you can identify and implement appropriate corrective actions where the AI being used is not working as intended, or is generating inaccurate outputs.
  • As part of good governance practices:
  • Where a buyer’s use of AI could significantly affect suppliers, buyers should disclose this in the procurement documentation.
  • Make sure information used with agency approved AI does not exceed the protective marking deemed appropriate for that AI. Refer to the Administrative Guideline for the safe and responsible use of Generative Artificial Intelligence in the Victorian Public Sector and the Practitioner Guide issued by the Office of the Victorian Information Commissioner.

Managing additional AI risks in procurement

Buyers who are managing existing contracts or approaching the market should be aware of these additional risks.

Supplier AI use to prepare offers

Suppliers are increasingly using AI to prepare offers for procurement opportunities. Buyers should consider what risks may arise from supplier use of AI. This could include:

Information security risks

A supplier may misuse information in procurement documents. This could include using information to train AI, or entering sensitive information into publicly available AI. If this is a concern, buyers should consult legal and information security advisers and include appropriate safeguards in procurement documentation. This could include restricting the use of information with publicly available AI.

Supplier capability risks

AI use may make a supplier appear more capable than they are. Buyers may need to do additional due diligence to assess the credibility and capability of the supplier to deliver the requirements of the procurement. Due diligence activities could include site visits, presentations, reference checks and seeking clarifications from the supplier.

Fraud risks

This includes using AI to produce false but realistic AI-generated documents, like certificates of compliance or insurances. If fraudulent documentation is suspected, check publicly available databases or contact the relevant organisation to verify authenticity.

Unknown AI use by suppliers during service delivery

Buyers engaging the market should establish whether suppliers intend to use AI to deliver goods or services, either now or in future. This is increasingly important as AI becomes more accessible and more integrated into suppliers’ business practices.

AI may also be introduced into existing goods or services during the life of a contract without the buyer’s knowledge. This may create risks if the contract does not include appropriate safeguards for AI use.

Buyers can safeguard against these risks through their procurement documentation:

Invitation to supply

  • Include a requirement for suppliers to disclose if and how AI is proposed to deliver the goods and services.
  • Buyers may instead prohibit any use of AI to deliver the goods and services.

Contracts

  • Include a schedule of permitted AI uses. Suppliers can only use the AI as stated in the contract.
  • Use contract terms that prevent suppliers from introducing new AI without approval from the buyer.
  • Use contract terms that require the supplier to provide a way for the buyer to opt-out or disable the new AI.

If AI is a concern for an existing contract, consider whether the contract should be varied to minimise any risk. Seek assistance from legal advisors.

A buyer may also use an existing contractual right to audit or request information about the supplier’s use of AI under the existing contract.

Example:

  • A buyer engages a professional services supplier to deliver a report. The buyer expects the report to be prepared by the skilled personnel listed in the tender submission.
  • The buyer receives the report and notices serious issues with its content including factual errors and fake references.
  • The supplier later admits that AI hallucinations led to these issues. The supplier did not have approval to use AI to write the report and did not maintain proper oversight of the AI.
  • This can result in legal issues, data security risks, significant rework and project delays.

Resources, tools and support

AI policy and governance resources

When procuring or using AI in procurements, agencies are encouraged to apply this guidance with (as applicable):

Office of the Victorian Information Commissioner
Public Record Office Victoria
Independent Broad-based Anti-corruption Commission

Tools and support

Procurement Guidance:

Innovation Network communities of practice for procurement and AI:

For technical questions about the AI policies and guidance, please refer to the contact details contained within them.

For all other enquiries please contact the DGS goods and services procurement policy team.

Updated

Back to top