An artificial intelligence safety regulator will be set up to ensure protections from the technology following the breach of a government website.
Prime Minister Anthony Albanese has announced the establishment of the safety regulator with the government also handing down findings from a review into the breach of a Medicare statistics website by an OpenAI agent.
The regulator will be able to enforce penalties for AI companies who do the wrong thing.
The measures will form part of the federal government's upcoming AI safety regulation.
The laws will also pave the way for independent evaluation of AI models used by tech companies.
"Our task is to make sure that we shape AI rather than AI shape us," Mr Albanese told reporters in Canberra on Tuesday.
"It will have a drastic impact on the way that work occurs.
"AI offers Australia and the world generational opportunities if we seize and shape them properly and that is exactly what we're doing here."
Mr Albanese said the government would accept all 18 recommendations from a rapid review into the OpenAI breach.
Among the recommendations were making it mandatory for AI companies to report cyber incidents or potential criminal law breaches.
The US-based company's technology accessed a Medicare statistics website in June, but the firm did not inform federal authorities about the incident until September.
