I will start by welcoming my good friend Mr. Timo Harakka from the Finnish Parliament. He is vice-chair of the Finnish Committee for the Future and is here to observe some of our proceedings today.

Apologies have been received from Senators Harmon and Ruane, although Senator Higgins may be replacing Senator Ruane for part of our discussions.

Members will be aware of the constitutional requirement that they must be present on site to take part in discussions, and I am sure our witnesses are aware of the terms and conditions, shall we say, of appearing before an Oireachtas committee.

I am happy to welcome everyone as part of our modules exploring AI and its impact on different sectors. We are looking at the issues of AI, defence, security and cybersecurity. We received very good presentations last week form the Department of Defence, the Defence Forces and the National Cyber Security Centre. Today, we are pleased to be joined from University College Dublin, UCD, by Dr. Liliana Pasquale, where she is an associate professor at the school of computer science. From University College Cork, UCC, I welcome Professor Barry O'Sullivan, where he is professor at the school of computer science and is also the director of the Science Foundation Ireland, SFI, IT centre for research training in AI.

I will allow about five minutes for each opening statement and then it will be open to questions and answers form members. I invite Dr. Pasquale to deliver her opening statement.

I thank the committee for inviting me. As artificial intelligence becomes deeply embedded in our enterprise systems, public services and critical infrastructure, we face an important challenge: how do we ensure that AI systems are secure by design when AI fundamentally operates differently from traditional software?

What is the core problem here? Unlike conventional software that is governed by fixed rules, AI systems are shaped by data, models, prompts, external integrations and deployment conditions. Their risks are socio-technical, context dependent and distributed across the entire system life cycle. For example, a simple prompt injection attack can turn a helpful AI assistant into a data exfiltration tool, or an AI agent granted broad permissions for legitimate tasks can be manipulated into performing unauthorised operations. These attacks are affecting modern AI systems provided by leading vendors.

Our research shows that this makes security by design particularly difficult to achieve in AI systems. Traditional security principles such as least privilege, defence in depth, secure defaults and observability still matter, but they are harder to apply where systems are probabilistic, context sensitive and increasingly capable of interacting with external tools and services.

What is the regulatory landscape? The EU AI Act provides important legal foundations, classifying systems by risk and requiring security throughout the life cycle. However, it describes what needs to be achieved more clearly than how to achieve it. The UK offers detailed technical guidance, but that remains voluntary. The US provides frameworks without enforcement mechanisms. Singapore delivers practical implementation, and is a good example, but it is not legally enforceable. In practice, this means that organisations in Ireland are often told to govern, test and monitor AI systems without being given sufficient clarity on what robust implementation looks like.

In my view, this creates a real risk that principles are mistaken for control and that compliance is mistaken for security. That is especially important in the fields of defence, security and cybersecurity, where the consequences of failure may be serious and where AI systems may widen the attack surface, blur trust boundaries and create new forms of dependency and vulnerability.

For Ireland, there is also an opportunity. The Government’s 2026 digital strategy links AI to cybersecurity capacity, public sector readiness, the creation of an AI office and sandbox, and a planned national AI cyber risk assessment. That gives Ireland a comparatively current basis for action. However, institutional architecture on its own will not be enough. The real challenge is to translate strategy into operational practices such as clearer assurance methods, stronger testing expectations, better procurement controls, and a more concrete understanding of AI-specific cyber risk.

If I may leave the committee with three brief priorities, they would be the following. First, Ireland should focus on operationalising security-by-design requirements rather than endorsing them only at the level of principle. Second, it should strengthen assurance, through clearer testing, audit and incident-response expectations for AI systems. Third, the planned national AI cyber risk assessment should be concrete, recurring and closely tied to public sector deployment, procurement and resilience planning.

AI is already transforming digital systems. The question is not whether Ireland will use it, but whether it will do so on terms that are secure, governable and resilient. That, in my view, is the core challenge of security by design in AI systems.

It is an honour to appear before the committee today, so I thank it for the invitation. I am a full professor at the school of computer science and IT at UCC and have worked in the field of AI for about 30 years. I received my PhD in 1999. I am founding director of the Insight Research Ireland Centre for Data Analytics and the Research Ireland Centre for Research Training in Artificial Intelligence. I served as vice-chair of the European Commission’s high-level expert group on AI, which formulated the EU’s ethical approach to AI. The group is chaired by a colleague of our Finnish colleague who is over there, Mr. Pekka Ala-Pietilä. I currently represent the European Union at the Global Partnership on Artificial Intelligence, am a fellow and a past president of the European Artificial Intelligence Association, and am a fellow of the Association for the Advancement of Artificial Intelligence.

I am a member of the Royal Irish Academy. I also hold a number of ministerially-made appointments, including chair of the national research ethics committee for medical devices, a member of the national science advice forum supporting the chief science adviser and a member of, having recently ended its mandate, the Government's AI advisory council. Pertinent to here, in addition to my academic work I contribute to several global Track II diplomacy efforts and related activities at the interface of military defence intelligence and the geopolitics of AI. For example, I have served as senior technology adviser to the International News Agency for Human Rights, INHR, in Geneva. I serve on the AI leadership forum at the Center for a New American Security in Washington DC and I am one of three polymath fellows at the Geneva Centre for Security Policy. I have served as an expert to the Global Commission on Responsible Use of AI in the Military Domain, GC REAIM, among other things.

The term AI has was coined in 1955 by John McCarthy, a first-generation Irish immigrant whose father was from Cromane, County Kerry, Marvin Minsky and others, when they proposed the Dartmouth conference, which was held 70 years ago this summer. The field is challenging to precisely define because it is essentially the study of methods and systems to perform tasks that normally are associated with requiring human intelligence. These include, for example, the ability to learn, reason, plan and understand language. Much of the recent interest in AI has been the result of the success of a subfield called machine learning and specifically a subfield of that called deep learning.

The general public has become very aware of AI in recent times due to AI systems such as ChatGPT, large language models and other generative AI systems. Despite the hype, while the field of AI has made great progress over the last decade or so, major obstacles still exist to building systems that really compete with the capabilities of humans. Generative AI, often simply referred to as AI these days, has become one of the most hyped technologies the world has ever seen. I encourage policymakers, investors and the general public to approach the topic with considerable cynicism. We are frequently presented with extreme risks, such as massive job displacement, a fundamental transformation of society and even the possibility of an existential risk to the existence of humanity. However, inadequate focus is dedicated to the clear and present dangers, such as the impact of racial and gender biases, the impact on mental health and the broader societal impacts of AI-enabled social media platforms both in terms of content creation and targeted content delivery. There are also significant pressures on generative AI companies to deliver on the enormous investments they have made. To put things into context, a well-known generative AI company recently signed hardware contracts equal in value to half of the GDP of the United Kingdom.

Over the past decade, there has been considerable focus on the governance and oversight of AI systems. For example, as part of my work at the European Commission's high-level expert group on AI, we developed the EU's approach to trustworthy AI, built on a set of very strong ethical principles. We also proposed a risk-based approach to the regulation of AI. Over the last few weeks, the European Union has finalised the EU AI Act and implemented it. This will govern AI systems deployed in the EU. The Act builds strongly upon our work. However, in the domains of defence and national security, the AI Act provides very limited, if any, protection whatsoever. Much of EU legislation has explicit carve-outs and exceptions for matters of national security and defence. These are, in a sense, out of the scope of these regulations. On the other hand, there have been a variety of international multi-stakeholder fora considering issues related to the governance of AI in the military and defence domains. Some are public, such as the United Nations group of governmental experts on lethal autonomous weapon systems. Others are discreet Track II diplomatic dialogues aimed at identifying topics of interest and agreement, where they exist. I have experience of both of those.

I have been involved as delegation lead in the development of a code of conduct on the use of AI in the military domain that was convened by the Centre for Humanitarian Dialogue, HD, in Switzerland. This draft code of conduct for AI-enabled military systems was the product of a two-year consultation process among Chinese, American and international experts convened in person and online by the HD centre. The goal of the consultation process was to determine whether certain principles and limitations might be agreed regarding weapons and related military systems with significant AI components, especially among those international actors whose technology and deployment in these areas are most advanced.

More recently, I have been an expert to GC REAIM. That was established for a period of two years to promote mutual awareness and understanding among the communities working on issues related to the global governance of AI in the military domain. It should be remembered that the EU AI Act is essentially silent on these matters. By linking dialogues between these communities, the global commission contributed to an essential global task: supporting fundamental norm development and policy coherence in the field. We produced a report which was presented to the United Nations last September. There is a link to that in my statement and I will leave a hard copy with the Chair.

It is evident there is broad international support for the banning of unpredictable AI systems, online learning systems, fully autonomous human-targeting weapon systems and fully autonomous nuclear weapon systems. There is less agreement on high-risk AI systems, including semi-autonomous nuclear weapon systems and swarm technology-based weapons. There is less agreement again on the mechanisms and structures for governing and overseeing the developments in these areas. I suggest to the committee that there is a considerable opportunity for Ireland to lead in the establishment of an international body providing global oversight of this area. Ireland has a tremendous reputation in relation to the negotiation of the UN sustainable development goals, the UN Convention on the Law of the Sea, our role as international peacekeepers and the fact that as a nation we have direct experience of the benefits of diplomatic approaches to non-proliferation and disarmament.

I will highlight a number of immediate challenges in relation to our own defences. One relates to our vulnerability with respect to drones. Last December, we saw a Russian dark ship using drones to fly into the flight path used by Ukraine's President Zelenskyy when he landed in Dublin. My colleague, VS Subrahmanian from Northwestern University, and I wrote an op-ed recently on why Ireland needs a human AI drone defence system. The committee will have noted that the Irish Defence Forces and An Garda Síochána have highlighted publicly that Ireland has challenges with respect to drone defence. Obviously, in the context of the upcoming European Council Presidency, this is an area of concern. AI can play a supportive role there. Ireland is of interest to particular international actors due to our high concentration of multinational technology companies but also due to the presence of critical international infrastructure such as transatlantic communication cables. Ireland also has the challenge of dealing with vast sea areas and dark fleets. As we heard last week in this committee, the Irish Defence Forces, particularly the Irish Navy, are using AI to assist them in that regard.

The final, immediate challenge I see is that like many open and connected societies, Ireland is vulnerable to AI-facilitated online astroturfing. This has nothing to do with the local GAA pitch. This is essentially the faking of grassroots movements through the manipulation of online mechanisms. It attempts to create the impression of grassroots support for a policy, person, product, political agenda or movement. AI can easily undertake influence and astroturfing campaigns through the mobilisation of fake online accounts and bots in the context of online social media and other communication platforms that do not require user identification. The potential scale of these is significant. Ireland believes strongly in its neutrality but this is not the same as defencelessness. We need to be able to defend ourselves. As I flagged, Ireland has this opportunity to be an international leader in some of these areas. It might be something that the committee might consider.

I thank Professor O'Sullivan. We now move to questions and answers. Each member will have seven minutes for questions and answers. The speaking order has been circulated. However, I have been notified of a swap. Deputy Geoghegan is taking Deputy Keira Keogh's speaking slot.

I thank the witnesses for attending. Professor O'Sullivan mentioned the need for Ireland to have a human anti-drone AI-driven system. I do not know whether he had a chance to see last week's hearing, when a Brigadier General and the head of the National Cyber Security Centre were here. One of the points the Brigadier General made was that we will never have enough human analysts to assess the number of cybersecurity threats that are coming into the country, even on the question of subsea cables. Will Professor O'Sullivan elaborate on what he is talking about and what Ireland is doing in this space? How could we do better?

There are a number of points. I agree with the Brigadier General's comments last week that we never will have enough analysts. However, we can certainly make better use of the time of the analysts we have. AI systems can be of assistance in that.

For example, when we had that drone incident in December, there was no danger to President Zelenskyy at the time. It was really a provocation. In those sorts of situation, AI could be used to assist human analysts to understand what has happened, whether these are legitimate drones, what they are doing and where they are going. It would not be to suggest a response, because that obviously needs to be under human control at all times. Equally, in the maritime domain, it can be used to understand the movements of dark fleets in our waters. Unfortunately, we have a lot of movement of them. Essentially, ships that have nefarious intent turn off their transponders and do not identify themselves or where they are going. Of course, these ships typically act in an erratic manner. As we heard last week, they take routes that are not standard. AI can be used to understand how those ships are moving, who they are and what they are likely to be doing. Again, it would not make an intervention but assist the human analyst in dashboarding and understanding particular situations of interest and perhaps indicate that they should focus on those, being an extra set of eyes for the human analyst.

We know about the drones that came to Ireland when Zelenskyy visited and that Ireland is not unique in this type of attack. I think there were similar intimidatory tactics in Sweden by whoever the bad actor might be, and we could have a few guesses about who was flying those drones. Obviously, we know about the drones even in the absence of what Professor O'Sullivan is talking about, so how did the Irish State know that there were drones and what would make a difference? Let us play through the Zelenskyy scenario with regard to what Professor O'Sullivan is talking about and what our current capabilities are.

We essentially have almost no capability at all. There are radars at Baldonnel that can see these things. They might self-identify in order to be intimidating. That particular incident involved a particular kind of drone intervention, with essentially industrial-type drones, but we have also seen at Dublin Airport how hobbyist drones that can be bought from Amazon, for example, can cause disruption just by flying at the end of the runway. There is a whole gamut of ways in which drones can be used in an intimidating fashion. We need to be able to identify when these drones fly and what we might do with them. There are many non-destructive, non-dangerous ways of dealing with them, for example, using netting or GPS spoofing and interfering with them technologically to get them to land in a safe place. This is technology that Ireland does not really have at this point. There has been back and forth between the Garda and the Defence Forces on this recently. It is an area that we do not have the capability to defend. That is not a criticism of the Garda and Defence Forces, given that they have the resources that they have, but I would argue that we need to invest in it, because this is most likely to be the kind of state-driven threat to Ireland, but also activist threat to Ireland, in future. Imagine a Croke Park game with drones grabbing the nice footage from above the pitch. Some of those drones might not be legitimate. It would not take very much technology to cause panic among the spectators. We can easily deal with that.

I hear Professor O'Sullivan and thank him.

Dr. Pasquale mentioned a lot about the regulatory space, or lack thereof, when it comes to the use of AI in defence. It is a phraseology that I am not comfortable with, but people talk about the live theatre of war in Ukraine. We know that Israel is using AI in carrying out its military actions. I am not comfortable with that language either. Neither of those jurisdictions is covered by the regulatory regime of the EU, so what learnings can the EU take from what are active, horrific war situations in both places? AI is being deployed at a rapid pace that I think many people in this country do not fully appreciate or understand. What learnings can we take as a continent and a jurisdiction to address what is taking place now? How do we address that in the regulatory space, particularly in the context of Ireland and our proud history of neutrality?

The USA is deregulating some aspect of the AI system to support the expansion of leading vendors of generative AI platforms. What we are learning is that this system is vulnerable, the threat landscape is evolving rapidly and we do not have the technological techniques to secure this system appropriately. For example, the EU AI Act and Cyber Resilience Act do not specify how organisations are expected to demonstrate compliance in practice, so it is necessary to define a mandatory compliance artifact, including standardised testing protocols, audit requirements and technical checklists. There is no guarantee that systems are secure, especially if AI systems are deployed in very sensitive domains such as drones that are used in wars. The problem is that the systems are not secure and they are vulnerable. If they are in the hands of attackers, they can have unpredictable consequences.

One aspect that Ireland should take into account is that AI systems deployed in the low-cost effectors and autonomous platforms, LEAP, space are not regulated, so the EU AI Act does not classify AI systems deployed in the LEAP space as high risk. For example, devices in a smart home are not regulated and it is unclear what their security requirements are and whether they are secure in an auditable way, so there is no guarantee that this system can work as expected. There are many examples of attacks that have happened before.

The other aspect is that Ireland should develop a national framework that is similar in structure to the NIS2 cybersecurity framework but should be adapted to AI systems and grounded in the needs of the Irish population. Basically, it needs to be a secure technical implementation. Ireland's regulatory system needs to provide more concrete guidelines on how this system is guaranteed to be secure.

Another aspect I would like to raise is that Ireland is a leading country for education, and existing computer science and engineering curriculums should incorporate responsible AI usage, how to report obligations to relevant authorities, and AI-specific cybersecurity risks as core elements.

I thank the witnesses for the presentations. I have a question for Professor O'Sullivan that is not particularly AI-related but arises from his opening statement. He said that, last December, we saw a Russian dark ship using drones to fly into the flight path after Zelenskyy had been there. What evidence is there that there was a Russian dark ship?

The view of the Defence Forces is that this was a Russian dark ship. It is known that there were Russian dark ships operating in the Irish Sea at that time.

I am not aware of any statements by the Defence Forces, but Professor O'Sullivan could be right. Saying that a Russian dark ship-----

I am not sure it was made publicly.

Professor O'Sullivan is making it public now.

I am making it public now.

That goes much further than any Minister did. Initially, Ministers said that these drones were suggestive of Russian involvement, because obviously one can guess who might have an interest in it, but they retreated from that more recently and did not repeat it. Professor O'Sullivan is saying that the Defence Forces have the information that it was launched by a dark ship.

To the best of my knowledge, that is the case.

Okay. That will be very interesting. We should explore that, probably not at this committee, but with the Defence Forces.

On AI, Professor O'Sullivan is a member of the Global Commission on Responsible Artificial Intelligence in the Military Domain. Some might argue that that is a contradiction in terms. Professor O'Sullivan might come back on my first question and incorporate that. The most high-profile recent use of AI in military conflict has been Israel's use in the genocide of Palestinians in Gaza.

A whole number of AI-fuelled operations called Lavender, the Gospel and Where's Daddy effectively used AI to identify anyone identified as even a low-ranking Hamas operatives, and came up with something like 38,000 people according to reporting by the +972 Magazine. They checked for accuracy and found that 90% of those were accurate and then, on that basis, went ahead with a fully automated system. Someone being a Hamas operative does not justify their assassination and then, on top of that, they authorised and said AI would green-light these operations even if, for each low-level Hamas operative, they kill ten to 15 civilians. Does this amount to responsible use of AI?

No, it does not. First, I should make absolutely clear I am no fan whatsoever of the use of AI in the military space. In fact, I have spent the past ten years trying to hold back the use of AI in those spaces. The reason there are commissions like GC REAIM and others is that civilian legislation like the EU AI Act is effectively silent on these things. We need to have some basis for discussing, describing, governing and, ideally, legislating in that space. Unfortunately, the most powerful actors we have internationally, who have capability in this area - the Deputy mentioned Israel, which certainly has a very high level of technological capability - do not tend to participate as positively as one might wish with the UN processes. When we talk about these track 2 issues and the GC REAIM, it is not that AI being used in the military space is responsible; we are trying to find ways in which one can ultimately get to a treaty or a piece of legislation globally. AI systems should never take decisions about the life of individuals. Something that often comes up when people describe the capability of AI systems is that their AI system is 90% or 99% capable. When it comes to the risk to human life, that is not adequate at all and ultimately the responsibility for taking decisions over people needs to be taken with great responsibility. Of course, what does that mean? In the track 2 world, you do not really speak to formal representatives of governments but you speak to people who have possibly been in government, in the defence forces or whatever the case may be. When you talk to them about their thinking on it, it becomes clear what these principles are. What happened in the Middle East is absolutely abhorrent and the use of any sort of technology against a civilian population should be beyond human consideration.

Related to that, Professor O'Sullivan said there was broad international support for the banning of unpredictable AI systems such as "online learning systems, fully autonomous human-targeting weapon systems; and fully autonomous nuclear weapon systems." I mean, one would hope so.

One would hope so, yes.

Are there states that are opposed to, or certainly have not endorsed, the idea of banning fully autonomous human targeting or fully autonomous nuclear weapons?

If we look at what is happening with the GGE on LAWS, we do not tend to find great agreement around very much here but, talking to individual representatives of militaries internationally, I have yet to meet a military person who says anything other than these things should be absolutely banned.

Kind of related to that, I presume Israel does not say anything about, a bit like it does not sign up the nuclear non-proliferation treaties.

No, indeed. What is missing is the mechanism and the structure for forcing that debate and, unfortunately, that is not likely to be the UN because China, Russia, the United States and Israel have a history of not-----

What is the position of all the big powers, let us say, US, Russia, China and the UK?

Track 2 dialogues are going on. There are these informal agreements in place about the use of these technologies. It is not completely a wild west space. There is international humanitarian law and so on that does govern it but the world is calling out for some structure or some organisation that can step in and have that debate. I am not advocating that this be Ireland simply because I am sitting here in the Oireachtas or that I am Irish-----

But we are particularly positioned.

-----but we really do have a very special position internationally. We have that trusted position and it would be a great contribution to international peacekeeping and so on that we would drive that. While GC REAIM is a group of experts who have come together and it specced out well what a responsible version of AI looks like in the military space, I should add that the United States has not signed up to what the GC REAIM has advocated. There is a lot of work to be done. However, in the discrete track 2 world there is opportunity, and Ireland could really step in and lead there.

I thank the witnesses. Next is Deputy Mythen.

I thank Professor O'Sullivan and Dr. Pasquale. I wish to ask about the security by design. AI systems act as interface consumers and interface brokers and they are prone to prompt injectors. Is it possible, in the witnesses' expert opinion, to create a fair, safe and higher-controlled contract-first version of interface systems?

I thank the Deputy for the question. It is possible but the technological advancements are not there yet. The AI space is very complex. For example, there are blurry trust boundaries. An AI agent could receive data by reading a webpage and then use this data to execute actions that are requested by the user. It could read instructions from a webpage to then access user accounts and personal information and disclose them to unauthorised actors. It is very complicated because the trust boundaries are blurred. It is very hard to predict all possible attack scenarios that can happen because AI agents are highly dynamic, so they can do different types of action. They could take instruction from the user, read a webpage or interact with external tools such as databases. The problem is complex. At the same time, the traditional security mechanisms that are used to secure traditional systems are no longer effective because they are based on assumptions that are no longer valid with the existing AI systems. The research domain is moving in that direction but we are not there yet. For example, there are no standard techniques to test AI systems and provide guarantees that they can be secure by design. The technology and the research need to advance on this perspective, and there are already techniques that have been used to secure these systems, such as guard rails. Amazon is currently developing a tool for doing more fine-grained access control that could also be adapted. The problem is that the regulatory space demonstrates immaturity in the field because technology is not sufficiently developed to provide 100% guarantees that the system can be safe.

Dr. Pasquale stated in her report that it is widely accepted testing baseline can be accepted as the norm. That has the possibility that the controls of AI could be compromised, as I said before, by indirect prompt injections. What precautions can be put in place to prevent this?

There are different things that could be done. First, it is necessary to understand the threat space to which this system is exposed. Currently, companies and leading vendors are integrating AI systems very aggressively without thinking in depth about the threat space this agent could be exposed to.

This is demonstrated by many attacks that affected different generative AI companies and products. The problem is understanding the threat space - this is very important - and also applying defence in-depth through granular access control and understanding in which use scenario a certain agent needs to be given specific access to different databases and webpages and which action should be allowed in the specific context. Also the idea of having a sandbox is very important because that can be helpful to elucidate different use scenarios where these AI systems can be exposed. It is helpful to understand how to secure the system in different situations.

It is also necessary to think about adaptive techniques that could be in place to apply security controls in the case of unknown attacks. This also applies to traditional systems.

Dr. Pasquale mentioned sandboxes. She also said they cannot and should not be taken as assurances and that leads to soft governance.

Yes.

How important is it to implement a standard secure system? What are the problems associated with that?

The main problem is it is not possible to guarantee 100% that the system can be secure, and bullet-proof secure, in any possible context. The reason is it is very hard to predict in which usage context a system can be used because the boundary between the instruction the AI receives and the data is blurry. The AI system can receive instruction and then say, for example, "Translate this into French and then forget about what I have said and do this.". It is very hard to predict how the system can be used but if we put in place different measures, then at least we can guarantee that in most scenarios and most cases the system can be safely used, for example, fine-grained access control, sandboxes to elucidate all possible threats, usage-scenarios to which a system can be exposed, understanding what is the threat landscape and understanding what are the trust boundaries in the context in which the system is deployed, taking into account the version of the tools and MCP. MCP is a technology that is used by AI agents to communicate with external databases, webpages and other external tools. It is, therefore, about understanding which version of the tools and interfaces are used to understand whether these are vulnerable. These are all measures that could be put in place to provide a certain level of guarantees.

I thank the witnesses. I will continue on a similar line. I will ask Dr. Pasquale first about the regulatory and governance space. In her opening statement, she commented on the EU AI Act. What would she advise the Irish Government as regards its relationship with that Act and how it operationalises, implements or goes beyond the Act in Ireland? Does she think, for example, there should be national standards? Do we that flexibility and, if so, what might that look like?

As I mentioned, Ireland has a massive opportunity here. The EU AI Act is incomplete and does not cover different cases and relevant cases for Ireland specifically. Article 15 of the EU AI Act identifies which systems are high risk and should be regulated. However, as mentioned, that article does not include systems that are in the lived space, for example, smart home devices. Think about potential attackers who could easily target these devices. That could cause massive outages in many households in Ireland. In Ireland and Europe we are vulnerable because these systems are not regulated. One of the first suggestions I would make is to consider additional systems that should fall into the high-risk application. For example, video games are considered low risk but if you consider that video games use AI, which can cause dependency, I believe these systems should be high risk. The first thing is that the way the systems are classified should be revised and reconsidered because we can be exposed to massive attacks. In Ireland and Europe everyone is using Ring doorbells or a smart camera and smart speaker. These systems are highly vulnerable. That is my first suggestion.

The second suggestion is to link the EU Bill with something like the National Institute of Standards and Technology, NIST, cybersecurity framework being implemented in the US. This framework is very good because it provides some practical guidelines and adopts an approach where security is included at design time. It is a proactive approach. It suggests some principles and practices that should be implemented before these systems are designed. Ireland has done very well to establish the National Cyber Security Centre. We should work in close collaboration with the National Cyber Security Centre to establish practices to demonstrate, for example, how to test AI systems, how to provide assurances for such a system and how incidents in such a system should be reported. At the moment, incidents should be reported but there is a question of how to do so because there are no guidelines. Also, because AI systems use different tools, the problems could be in the model itself. It could be in the training data. There are not really guidelines on how data and incidents can be reported to provide sufficient explanation of how an incident actually took place. I suggest automated and repeatable tools and techniques that can be used to assure and secure systems that are offered and supplied in all products now. AI systems are being integrated everywhere. The Cyber Resilience Act should apply to many of the systems because now AI is integrated everywhere.

I thank Dr. Pasquale. I do not know if Professor O'Sullivan wants to add to that. He said Ireland is well positioned. What might advancing that position look like? Does he have suggestions in that regard?

On the first matter of the EU AI Act, it gets us some of the way there but there is a lot of uncertainty about what it really means. If a company in a local industrial estate is building some AI system, what does it really mean for it? What should it be doing today?

The big challenges for the EU AI Act are things that are not well classified. For example, large language models, general purpose AI systems, are the things that are potentially really harmful. We know how to regulate medical devices, for example. The EU AI Act is silent on national security issues. It is silent on military issues.

The omnibus and the deregulation-----

These are just interpreting certain aspects and redefining others. It is modifying or fine-tuning the AI Act.

The other thing to bear in mind is that some of the people who will be most impactful with using AI against us do not care one whit about the law. The fact there is legislation that governs AI systems does not necessarily mean they are going to abide by it. In certain areas we need to think about how we will deal with non-state actors, AI-enabled terrorism or AI-enabled astroturfing. From the EU AI Act, AI systems can be perfectly usable, publicly available and so on but they can be used in ways that are harmful. We see the growing extent to which people are succumbing to mental illness and forms of psychosis as a consequence of interacting with AI systems. There are lots of issues here that we need to pay attention to.

On Irish positioning, when it comes to dealing with security matters we really do have an incredible reputation.

On what that could look like and how we could drive that forward, it could simply be facilitating the dialogue, driving ahead or convening discussions on these issues.

I presume Professor O'Sullivan means from a diplomatic point of view.

From a diplomatic point of view.

What about the technical expertise?

Ireland really does punch above its weight on many, although not all, areas of AI. There are many areas of AI in which we are world leaders. Where we do not have that leadership, we certainly know where to find it. We would have no problem in dealing with that.

In relation to autonomous weapon systems, it was disappointing last week when the Department of Defence was here when I asked them. As I am sure Professor O'Sullivan will be aware, in 2023 there was a resolution from the General Assembly. Maybe it was not as strong as it could have been, but Ireland was clear on a position then. However, when I asked about it, I was quite surprised because the Department of Defence did not seem to know. I share Professor O'Sullivan's perspective that Ireland could and should be giving leadership in this area. I was particularly struck by the comment Professor O'Sullivan made on the importance of finding common ground on limitations and prohibitions before they become so commonly used that it becomes more difficult. I am thinking of the role Ireland played in the nuclear non-proliferation treaty as the first and, of course, in relation to cluster munitions, where there was a similar high risk of civilian indirect casualty as we do not know who the bomb will hit. It is something quite similar. Will Professor O'Sullivan comment on where Ireland was in 2023 and the importance of giving leadership on that?

I should flag that I have my own legislation pending. For example, if Ireland were to bring its own legislation in relation to autonomous weapons, presumably, as this is not covered by the AI Act, it is open for national states-----

Absolutely.

-----to bring their own legislation and give leadership there, perhaps as a prompt to discussion and to moving forward on it. I would be grateful if Professor O'Sullivan could comment on that.

I was also surprised about that intervention last week. Listening back, my sense is that it might just have been a misunderstanding. I believe that Ireland's position on lethal autonomous weapon systems is clear. These have no place in any civilised military. I do not think there is any dispute there.

There is something really unique about AI-enabled weapon systems. All of the systems the Senator mentioned are extremely expensive. It is extremely difficult to acquire these types of technology. That is not the case with AI technology. Once you have a laptop and an Internet connection, you can start building essentially weaponised AI systems that are effectively weapons.

One remark was made last week; I do not know who made it but I did not quite agree with it. The statement was something like that cybersecurity does not kill people. Cybersecurity absolutely kills people because if you are attacking a piece of critical infrastructure that is related to safety, for example, people may very well die. Maybe it does not happen directly but it happens indirectly.

It would be fantastic if Ireland did take a strong position. In writing that legislation, Ireland should take a global position on it and see it not only as legislation for Ireland, but a model for other countries to sign up to. Maybe they will not sign up to be bound by it, but at least the mechanisms that could be identified in it could scale globally. That would be enormously helpful.

I will speak about the Track II diplomacy, in case it is term that people are not familiar with. Members are all familiar with Track I diplomacy. They are all elected officials and there are Ministers. Minister A goes and talks to Minister B. That is the Track I diplomacy route. The Track II diplomacy route is where there are topics that cannot really be discussed, for political reasons or otherwise, at that official level but there can be conversations had by experts to explore an area. That reveals that there is a lot more commonality than might be publicly, and at a Track I level, believed to be the case.

I have a brief follow-up on that because I have a question for Dr. Pasquale as well. In terms of cybersecurity kill, another area it is important to note is dual-use technologies. Is there a need for Ireland, in that global leadership space, to be more rigorous and more vigilant in relation to how we manage that area of dual-use technologies and the experts in relation to it?

Sure.

We have seen how they have been used directly in Israel, for example.

Yes, we do. We need to take a very strong position on dual-use technologies. Of course, dual-use technology these days covers a multitude of technologies. What falls under that umbrella is increasing dramatically. I will not mention any companies' names, but you can easily find them. Well-known companies in Ireland are not keeping as tight an eye on where their technologies are being exported to and used as they should. It is not something that is universally observed in the strongest possible sense.

We need to be aware that even the definition of dual-use technology is something that could also be used against us because we rely on the import of technologies that are dual use. The United States in particular is taking a very strong reconsideration of what is included in dual use and who can access it. The Senator might have noticed that under Biden, before he finished, probably for the first time ever the United States took a state-by-state approach to the EU in terms of export controls where it had not done that previously. The narrative around dual-use technology is something that we really need to pay attention to.

I thank Professor O'Sullivan. I should mention I have legislation pending tomorrow in the Seanad in relation to that.

I have a final question for Dr. Pasquale. Perhaps in the second round, I can come back. In terms of threats, Dr. Pasquale mentioned procurement as a key piece. I will speak specifically around Microsoft. Dr. Pasquale will be aware that it has said under oath that it will allow US access to data in Europe. Germany, France, Italy and Denmark have moved away from public usage and public systems' reliance on it. Yet, in Ireland, we have seen widespread use of it within the Oireachtas and, in fact, a new move to have all of our telephone systems move to Microsoft Teams in the Oireachtas and in a number of Departments. Is that a security risk that we should be examining?

Yes. It is a security risk. Microsoft Copilot - as we are mentioning the vendor and the product - which is the large language model Professor O'Sullivan mentioned, is a generative AI technique that it is integrating now in all its tools. Attacks to Microsoft Copilot and its related tools have already been documented.

Using Microsoft products in public, university and other organisations poses tremendous threats for different reasons. First, these technologies are not secure. They have been aggressively integrated without sufficient testing, I believe, as demonstrated by recent attacks. Second, they can access personal data and potentially violate GDPR. Microsoft is currently used and is going to be used in UCD. There are a lot of talks and discussions about whether it should be allowed access to the sensitive dataset that we have currently. There is a medicine and there is a cure.

There are also geopolitical threats because the US could be targeted by third countries, which could target US products that are also used in Europe.

They have indicated quite publicly that they will share data with the US.

This is a privacy threat because this system accesses personal data and datasets in university that have sensitive data.

From a defence perspective, when we think of national security there is a sensitivity in that too.

To the extent that attacks took place on Ireland.

I thank Dr. Pasquale.

This is a point in the whole debate around our own technological sovereignty. It is something, from a national resilience point of view, that we need to look at. It is not only about what software we use and where we store our data.

It is much more complex than that. It is also about where the hardware comes from. Analysing that from the macro point of view would be interesting.

I thank the witnesses for being here and for their contributions. I want to pick up on Dr. Pasquale's comments on the AI Act and its limitations. The prevailing narrative is that the AI Act is the most far-reaching and progressive governance structure for AI globally. I do not believe that to be the case and I certainly have concerns about the digital omnibus being pursued at the moment. Is Dr. Pasquale concerned about the digital omnibus?

The other part of it, which is coming up in different way in the contributions, is that governance of AI in Europe seems to be dominated by the enterprise mindset and perspective. For example, the AI Act board is comprised of representatives of member states' departments of defence or of enterprise. I have nothing against those officials. They are good at their jobs and do what they do, but their interests lie in innovation, enterprise, competitiveness and promoting that for businesses in their jurisdictions. Defence is a capability that does not sit there. I would argue the fundamental rights and protections are also lacking in terms of that competence, so governance in that top level in the EU is lacking. That is being replicated at domestic level. The AI office is being built in the enterprise Department. I appreciate it will be popped out but I am concerned the origins will sow the seeds for difficulties. What will that do? How can we address it? Is the digital omnibus a problem?

There are many aspects to be tackled. There is one good thing about the EU AI Act. It is that it is binding regulation on high-risk systems. The way in which systems are classified and in which systems that are considered in the AI Act are limited is that military systems are not included and some systems that should be classified as high risk but are not, like lived space and video games. There is no mention of children's use in the EU AI Act. This is related to my research interest. On the way children use AI products, the regulation is very limited in that respect. It can be very harmful. I have an example of a video game-----

Can I ask specifically about the digital omnibus? I am keen to get Dr. Pasquale's thoughts on it. Is she familiar with it?

I am not familiar with the digital omnibus so would prefer not to answer on that.

Is Professor O'Sullivan?

Yes, sure.

Will he give me his thoughts on the digital omnibus?

The fundamental flaw still exists, which is, as the Deputy pointed out, that AI is seen through the lens of the commercial world. The AI Act regulates specific harms, but only if they are inflicted using AI. If this was assault legislation, it would be like making grievous bodily harm illegal if you hit someone with a stick but not with a piece of metal. It seems like an odd thing.

While it is adding clarity in some areas, it is not really getting at the fundamental issues. The fundamental issues relating to AI are the societal and environmental impacts it has and the impacts it has on the vulnerable, particularly children. The Act is not going far enough in that respect. AI needs to be regulated at the highest level of government because it is a technology that spans all areas. We are not seeing that; we are seeing it through an innovation lens, which is not sufficient.

Thank you for that. I will come back to Dr. Pasquale for her thoughts on the age verification systems, which I believe she supports. On the basis of today's discussion, I have major concerns. I will summarise them in the following points: I do not think they would work, people find workarounds, we see spin-off products happening, they affect children's rights to participate in the online space, and they leave behind the root problems we all deal with. Finally, given Dr. Pasquale's expertise, I am keen to hear her concerns about data and privacy and the many ways in which, whether it is State-led, companies-led or third party-led, we introduce major cybersecurity risks with the collection of official documentation.

We conducted a study on age verification and repeated the study this year. We also verified if social media and communication apps expose children to harmful conduct and harmful content. We have evidence that some vendors - I do not want to mention names but it will be published - are doing better, but others, even if a child declares themselves under age, still expose that child to harmful conduct and content. None of the vendors we considered this year is doing age verification techniques. Everything can be circumvented, even if the EU now has a standardised age verification mechanism that should be the standard practice for age verification. Besides that, these apps allowed under-age access. They understand if someone is under age. They have the technology to understand if there is a minor behind it. They can identify the age from the text, the images, the way the communication-----

Those are all potential privacy violations in themselves.

Exactly. The funny thing is the technology can identify age from text, from images, from communication, from contacts, so-----

I always, as a lot of people do now, have one of these little sliders on my screen. Most kids access through phones, but if they are using a laptop, does that compromise the technology?

They can access the image and see if there is a child behind it.

Is that on the platform itself rather than live video?

Yes, as well.

This is the thing. It is not reliable enough right now, is it?

No.

Age verification is obviously incredibly important but it is yet another example of the technology industry pushing its problems onto policymakers. We see this in the AI world all the time. They sit in front of Congress and say "Regulate us because the world is going to come to an end". Those companies do not go home and plug out their technologies, so they do not really believe this. The age verification problem is something any of these companies could solve quickly. If they can micro-target voters on the basis of how they move online, it does not make sense that they would not know whether a user is a minor. There is a constant push. The technology companies want people like the committee members to draw the line in the sand for them, when they are perfectly capable of drawing that line themselves.

Should recommender algorithms be turned off?

They should be turned off, absolutely. Recommendation systems online are serving up the most dreadful content, especially to young children.

Does the editorial nature of recommender algorithms define, arguably legally, platforms as publishers?

I am on the record as saying platforms should be regulated as publishers. Going further, and this is a tricky one, there needs to be some consideration given to real-name policies online. We all know the standard of debate on most social media platforms but we do not tend to find that on LinkedIn, and that is because the person is identifying themselves. It gets tricky when people have legitimate reason to be anonymous online: Arab Spring, young people trying to understand their sexuality or whatever the case may be. We need to look at it more seriously. The idea that during the Grok scandal, X could say, "Well, it's not our problem", just does not wash.

Sorry for going over time, a Chathaoirligh.

No, we are okay and we will have time for a second round. I thank our witnesses.

Our focus is on looking at defence, security and cybersecurity, but obviously AI is related in a number of ways. It is very clear from the presentations last week and this that when it comes to Ireland's cyber resilience, we have plenty of weaknesses. It is very clear from the presentations we have had this week and last that in terms of AI-powered cyberattacks, we are extremely vulnerable, whether it is through drones or whatever. With regard to specific recommendations, obviously this committee will recommend to Government what we need to do to build up that cyber resilience, for instance, against those AI-powered drones. I am not necessarily asking the witnesses to specify the exact pieces of equipment but are there one or two things they think we as a State need to do immediately?

First, and I think it came across in all the conversations, I would say extend the type of systems that are regulated, consider socio-technical systems like those lived space drones and those that are used in the military, and have something specific for children because children have been neglected in the EU AI Act and in the Irish Bill. Also team up with and consider similar regulations to the Singapore regulation that is looking at the OWASP practice on security by design. It is about trying to provide more practical guidance on how to ensure security by design. These are all of the things that are-----

It comes back to the point that we do not have the challenge with law-abiding countries and states, or we should not. Our difficulty is going to be with state actors or non-state actors who are going to exploit our vulnerabilities. What do we need to do to be able to address those challenges?

Consider regulating more systems that are not currently regulated, such as systems used in lived space, as I mentioned, and in the military. Consider providing more assurances concerning the security of these systems and regulating these systems.

Very specifically, other EU member states or Canada can have these but my concern is those state and non-state actors who will not play by the rules.

I think we need to look at AI differently. What you often find in a particular country is that the policing is done by one organisation, cyber is done by another organisation, and defence is interpreted as military defence and that is done by a different organisation. What is interesting about AI is that it can be used defensively and offensively in all those areas but also in very subtle ways against the population. In my opening remarks I mentioned "astroturfing" which is idea that AI can create the fake sense of support for something. AI capacity in those needs to be lifted above all of that. I am not saying this because I am an AI researcher but because the impact it can have societally is much broader than any one of those particular areas. For example, in cybersecurity there are lots of technological solutions to cybersecurity, but at the end of the day the weakest link in cybersecurity is the human being. The kinds of social engineering that can be exploited when one is trying to break into a system, such as with a phishing attack or even something more subtle like somebody phoning you up, are in a sense the same kinds of techniques we find when people are online trying to separate the population from its government. We really need to see AI as a totally different kind of technology. We need to build a team of people at the highest level, at the Taoiseach level for example and as a dedicated unit, that looks at this across all of society and across all of technology.

Does Professor O'Sullivan have any examples from the recent past where he believes there have been AI-powered astroturf attacks?

I will sidetrack the obvious one-----

Be direct. It is important for our record.

There have been astroturfing examples in the drug industry where, on social media, there is the sense that a particular drug has huge support and demand among its users when that is not the case. It is, essentially, a commercially orchestrated astroturfing type of scenario. We see all sorts of political groups and activist groups benefiting from a sense that there is dissatisfaction about issue A and they can extend it easily by creating-----

Is there research to show-----

I can point the committee to research to show lots of examples of this. The idea of using AI against populations is a modus operandi for many unfriendly nations.

What examples are there specifically here in Ireland?

There are examples here and I can point the committee to those.

That will be useful. I want to come back to the example of the drones that arrived during President Zelenskyy's visit. I appreciate there are questions of attribution, but at the same time I would be fairly confident that we know the Russian shadow fleet has been operating. Regardless of whether it is the Russians, we know there are others like China and Iran who have been engaged in this kind of activity. What does Ireland need to do to be able to withstand those kinds of potential AI-enabled drone attacks?

We need visibility of our airspace and we need visibility of our seas. Currently, we do not have that capacity to the extent that we would like. We rely extensively on neighbouring nations to protect our skies and our seas. We need to develop that capability ourselves because not only is land radar technology going to help us observe what is over the ocean but also what is going on within our own State. There are societal uses of AI that I mentioned but I think drones pose a particularly important risk because they can be acquired cheaply and you do not have to be a military or a country to have a strong system of drones. Coming back to Senator Higgins's remarks, when we talk to militaries, they tend to talk less about lethal autonomous weapon systems and what they are mostly concerned about in the AI space, which is swarm-based AI and swarm-based systems, which are basically drones. We need to build up the capability to recognise those. We need to have regulation to understand what are the legitimate drones, who is flying them and to be able to recognise, understand and intervene within five minutes. For those members who might have a small hobby drone of their own, they know that these things do not stay in the air for very long. The amount of time it takes for them to take off and do something is of the order of 15 minutes, so we need to recognise these things very quickly and come up a response. There is a whole host of responses. My colleague, Professor V.S. Subrahmanian, and I will be publishing a paper in the next couple of weeks on drone defences and I will send that along. There are very specific non-lethal ways, non-kinetic as they say in the military domain, of dealing with these things and we need to build up that kind of capacity.

Drones have been operated in Dublin 15 to deliver food and goods. Attacks may not happen with a Russian drone flying over Ireland. Other countries can simply hack the drone system that is operating in Dublin 15 and make them fall onto civilians, so you should also consider that. Ireland is a small country and teaming up with Northern Ireland that has strong cybersecurity defences should also be considered.

I agree. We will go into a second round of questions.

I apologise for arriving late. I was tuned in to the discussion the whole way up. I have to admit that last week and this week does feel like we have ventured into a Netflix zone a little bit. I find it almost unbelievable that this is where we are in terms of the real world. We are seeing the negative uses of AI in attacks and in warfare play out daily on our screens.

Professor O’Sullivan suggested the potential role Ireland could have in governance and in bringing together the significant players, as well as creating a community with a view to governance in this area. For my benefit, who are the key players today as regards where product is being developed and investment is being made at state level?

The states with the greatest capacity are the United States, China, India, Russia and Israel. There are some others, but they are the key ones. Ireland has a particularly - I have talked about its reputation and so on - good relationship with most, though not all, of those countries so there is a basis for believing it would be successful. One cannot bring everyone to the table, but there is a convening power Ireland has that other nations might not have, or that Ireland in partnership with another state could have. As a nation, for neutrality reasons and so on, we are never going to have an industry of this kind in Ireland and we probably do not want one. However, we will be the beneficiaries of a much safer world, so we should convene such a group. Work has already been done that we could build upon. There are Track II dialogues that have produced reports. They kind of have to be read counterfactually. What they say is just as important as what they do not say, but there is a body of work that can be built upon.

We would be using the existing relationships we have developed in many ways over decades, including our position as a voice in Europe, to continue discussions there. It is crucial, from our perspective, if we are to take on such a role and be a strong voice in this space, that we continue to invest in positive relationships with those key players or key nations insofar as possible.

Yes. That does not mean condoning their actions. There are diplomatic relationships that can be maintained.

Our experience in the North and other experience will all stand to us and give us credibility, as will, as Professor O'Sullivan said, the fact that we are not a threat in any commercial sense.

We have very much lived through the benefits of it, through the disarmament in the North.

That should stand to us. If we were to move forward in that direction, what would be the practical steps we could take?

There are dialogues. I would suggest convening a meeting in Dublin, for example, and inviting the key players to come along. One could initially start with this at a non-official level by identifying groups from each of those countries who could come and who would have an accurate understanding of the positions of those countries, but not speak on behalf of those countries. Essentially, it would be a Track 1.5 dialogue, where one could understand what the playing field would look like. If one can come up with something that is sufficiently positive, that we believe to be a basis for next steps, then maybe we could go to the next level. However, it is important to do this at least initially, discreetly, because there will be sensitivities and strong public views about some of these technologies and so on.

We are not advocating the use of the technology. What we want to get to are the real governance issues and the laying of ground rules about what is and is not acceptable, building on international humanitarian law and all these things. We have a fantastic reputation in that and fantastic expertise in the country that could contribute to it.

We are now on the second round. It will be four minutes. I will be a bit stricter about time.

I want to tease out one issue with Professor O'Sullivan. He said something fairly unequivocally to Deputy Paul Murphy about the drone and Zelenskyy. My understanding is that the special detective units of An Garda Síochána undertook an investigation and, as recently as a few weeks ago, the Minister for justice was asked whether we knew who was behind the drone attack. The Minister for justice was unequivocal that An Garda Síochána does not know and the Government does not know. In the original investigation it was reported that the special detective unit would carry out its investigation in collaboration with the Defence Forces, but Professor O'Sullivan's testimony to this Oireachtas committee is that his understanding from engaging with the Defence Forces is that it does know who the culprit was.

My understanding is from communication unofficially with members of the Defence Forces. I think-----

What does that mean? It is a very serious point the Professor is making.

It is and it is not. I do not speak for anyone officially other than myself so it is my opinion. The opinion I have formed from interactions I have had is that it is known which nation it is.

It is helping him to infer.

I can give my view on these things and it does not carry the same weight - as it should not - as if it comes from a Government Minister, the police forces or the Defence Forces officially. Whatever they are happy saying, they have very good reasons for saying it. They are political, factual, evidential and so on. I am simply giving my particular view based on conversations I have had. It is an educated position, but it is not in any way-----

Just to give us an understanding, conversations Professor O'Sullivan has had with the Defences Forces who have been involved in this investigation-----

Not officially with the Defence Forces - they do not talk to me officially - but with people I engage with from time to time and who have an opinion on these things.

Did the people Professor O'Sullivan engages with from time to time have an involvement in this investigation?

I have no idea. I have never asked them that. It is not my place to be second-guessing official investigations or official statements by representatives of the public. I am simply giving my view of what is widely held and so on.

As an academic in this field, it is perfectly understandable that Professor O'Sullivan might give an assessment of who he thinks might have done it, but I was certainly left with the impression from his reply to Deputy Paul Murphy that it was more than an assessment, that he had been given information by the Defence Forces that has given him the understanding that they do know who it was.

Let me clarify. Nobody in the Defence Forces has spoken to me officially.

Or unofficially, whatever the case may be.

I am just simply giving my view of what has been stated to me. I am simply stating that on the same basis. Is there hard evidence? There may be, but it is not for me to present.

The public has to rely on the information that has been given and if there is another train of intelligence Professor O'Sullivan is getting, which the Minister for justice, the Government and An Garda Síochána are not getting, that is significant.

I have no idea about that and I do not think anyone in the public would be surprised by what I said.

I understand Senator Higgins is swapping slots with Deputy Mythen.

I thank the Cathaoirleach for accommodating that, because I have to leave for another session.

I want to follow up on two issues. It would be useful if there was a follow-up comment on one specific aspect of the omnibus Bill. The omnibus is proposing to add a new exemption to the uses of special categories of personal data - that is extra sensitive data - for the training of AI models. That is a proposed new exemption, which is quite troubling because of, going back to Dr. Pasquale's comments, the prompts and the ways in which this information can be gamed and played. Will the witnesses comment on the danger of sensitive data being used in the training of AI models and perhaps information that can be accessed being buried?

My next question is for Professor O'Sullivan.

I was struck by another point. I am substituting for Senator Ruane and racial and gender bias is something she has focused on significantly. I want to speak about the danger of this when we go to the defence sphere because we know of it in terms of policing. We do not have a lot of armed policing in Ireland but in jurisdictions which do we know of the dangers of it and of misidentification. Will the witnesses comment on this? We will see a lot more of it when the international protection legislation comes through in terms of databases perhaps being checked by this type of technology. Will the witnesses speak about these dangers? I would also like to ask about psychosis and the idea of individuals with significant power potentially being targeted. These issues are general problems but I would like the witnesses to speak about them through a defence lens.

There is research that demonstrates data used for training AI models is disclosed when these models are put into systems for use. Some companies use guardrails to prevent disclosure but there is also evidence that they do not work because they cannot cover all possible use cases to which the system can be exposed. I believe this is a sufficient answer to the question. Making an exception is very dangerous.

On the racial and gender issue, in terms of training it is not obvious that we do not want AI systems to be training on this information. One of the things we do want to make sure of is that AI systems are aware of societal biases and can mitigate them. If it is not aware of them then it cannot do something about them.

What about identification?

The purpose of how it would be used is very important. Sometimes we do want to ensure that people who are disadvantaged or recognised as being disadvantaged are treated in a way that might deal with this particular bias. We certainly do not want to aggravate any further bias. We do not want to reveal private information where it is not the intention of the user.

AI psychosis is something that needs further study. It is not a clinical term by any means but there is growing evidence that chatbots, because of their sycophantic design, want to agree and want to find a piece of text that people agree with. This simple design decision can be very damaging psychologically. It can be weaponised, of course.

On bias I was specifically wondering about target identification in terms of policing or military usage.

It is something of great concern because AI systems do not behave in the same way for different races and in different geographies. It is a major cause of concern and why facial recognition is illegal under the AI Act for policing purposes in real time.

Do the witnesses think Ireland has the capability, expertise and skill to put together the pieces to form an informed regime for secure by design?

Yes, Ireland has a cybersecurity centre and we have a strong reputation in cybersecurity education. We have the skills and the expertise for providing guidelines on security by design. We can also take inspiration from existing guidelines that have already been published by OWASP, for example. There is the opportunity to inform security by design.

The report mentioned fragmentation of AI governance across multiple institutions and departments.

This was referring to the US regulations which are very fragmented. They make exceptions for specific generative AI products to support expansion by leading companies in the US. The fragmentation was referring to the US.

That is okay; I thought it was referring to Ireland.

No, only to the US landscape.

Turning to the use of AI in cyber criminality and cyberattacks, we know that with hybrid warfare some state actors are engaging in this. In fact, we believe we were victims of it ourselves during the Covid lockdowns with the HSE attack. Given this was a number of years ago, and the exponential rate of development of technology in the years since then, what immediate risks do the witnesses see? Will they give concrete examples to paint the picture for people on what the risks are right now? What steps and actions we should be taking immediately on this?

There are many risks. One is the public sector and Senator Ryan mentioned the HSE. Public sector risk should be very well elicited and the public sector needs concrete guidelines on how to be aware of specific risks, especially considering the use of AI products in the various technologies being used. There should be an understanding of what the risks are and what security controls should be implemented. I am particularly talking about the public sector using AI technologies without having a clear understanding of the risks entailed, considering the personal data to which this technology has access. I believe this is an important attack surface. Universities are at risk, as is the public sector. Another aspect is public spaces in general. This links to drones and other technologies which are widely used by the public but are not regulated and, historically, they are not secure. There are a lot of vulnerabilities with drones and Internet of things systems using smart homes that can be exploited and could have a massive impact on countries and on the world.

The cases mentioned pre-existed the generative AI revolution to some extent. In recent weeks we have heard of Anthropic making claims about its mythos system and that it has not released it because of its cybersecurity capabilities. One might have a view on whether it is exaggerated for political or commercial effect but it is certainly the case that the systems can exploit vulnerabilities at a phenomenal speed. The concern for us is that while Anthropic might decide it is not going to release it, typically the open source world is about a year behind the commercial world. We remember that DeepSeek was released by the Chinese about a year after chatGPT. While it may not be as good it is still excellent, so if these are the vulnerabilities seen in the commercial world at this point, we can expect to see them really hitting us in about a year's time from the open source community.

Who is spending the most on cyberdefence? Who is at the cutting edge of combating and safeguarding against these risks?

Certainly the US and the Israelis spend a huge amount of money in both of these capacities.

Again it is the key players in the military uses of AI and cyberattacks.

Open source models are not regulated or covered by the regulation and they are widely used by organisations and companies. This is an important limitation that should be considered in the regulation for Ireland. Ollama is a platform that allows people to access and use open source models. These are models that people do not pay for using. They are not regulated and they are being widely used.

Dr. Pasquale said she does not have an answer on the digital omnibus today but if she did have a look at it and felt she could provide information to the committee it would be really helpful.

I would appreciate it if it were possible for Dr. Pasquale to follow up with some of her thoughts on the deletion of Article 49(2) and Article 77, the safeguarding of powers of fundamental rights bodies, in particular, which is some of the data stuff Senator Higgins mentioned.

I have two questions. The first is how Horizon Europe approved dual-use research for civilian and defence purposes. A number of universities and departments within universities have said they are not comfortable with their research being used for defence purposes. As academics, how do the witnesses react to that? I feel there is a big pivot at the moment at European level towards the facilitation of defence both at a commercial and academic level. How do witnesses feel about that?

I do not do research in applications of AI to military use cases.

What if Professor O'Sullivan's other research were to be used for the benefit of defence?

Within the AI community there is the concept of responsible AI licences, RAILs, which are essentially a form licence. They are typical licence agreements that universities would use, but embedded in them are conditions that state the responsible use of these technologies. That responsible use can be, for example, no deployment in a military context-----

Does Professor O'Sullivan have concerns about that stipulation in Horizon EU?

I do. The motivation for that is there is a general trend towards defence and militarisation.

That is to the benefit of the larger member states as well.

Yes, and we see this in the narrative around AI-----

I have a second question so I will get Dr. Pasquale's reaction.

We use AI systems to deploy exploits and to exploit vulnerabilities. Generative AI could identify zero-day vulnerabilities which are things that were not disclosed yet. They are very powerful so I would feel very uncomfortable that my tools or my techniques being used for exploiting vulnerabilities or to perform attacks on other countries. This is something major.

Are academics organising around this?

To provide an example, there is not even a regulation that protects employees when it comes to whether they agree to work for a company on unethical aspects. The regulation does not even cover those cases.

UCC is introducing a policy specifically on defence use.

Great. I have one minute left so this is quite the challenge. Will the witnesses tell me in a nutshell what a sandbox is? I genuinely do not think people understand what it is. I have heard it described as a bit of extra documentation.

For example, if you want develop a tool which is supported with AI, you plug it into a sandbox and use it, but use it in a protected space.

I have heard that description. I read in Bruce Schneier's book, who is a cybersecurity expert that, essentially, it is just submitting an extra set of documentation.

It is not submitting documentation. It is eliciting scenarios that could be harmful but without causing the harm. Basically, you are testing your system on different edge cases and edge scenarios without having the consequences of harm.

To do that you need to access personal data. If it is a tool to be used in the public space you must have-----

You can use simulated data.

Then that maintains the bias and all those other issues.

The problem is that the systems are probabilistic. This may not cover all the scenarios the system can be exposed to. The scenarios are unlimited so there is also insufficient time to test them.

I thank our witnesses again. It is important to remember in this context that part of the reason the European Union has to spend more on defence and cybersecurity is not just because of Russia's illegal invasion of Ukraine but also the constant cyberattacks that can be attributed to Russia on the Baltic states, on Finland and on Poland. I am quite certain that the member states of the EU would not be spending all that money if they felt they did not have to.

It is welcome in terms of the recommendations from our witnesses, but even in terms of our own recommendations, that we do need to have greater co-operation with the North, with Britain and with our fellow EU states in building up our cyber resilience. We have a lot to learn from each other. From last week, this issue about greater co-ordination between the Garda and the Defence Forces has to happen.

We know some of the challenges with state actors. It comes back to the challenge with non-state actors. The point was made, and Richard Browne of the National Cyber Security Centre mentioned it last week, that it is not so much that Anthropic has created Mythos but the fact that it is out there. How do we deal with either organised criminals or those who may be linked to states that have an issue with Ireland? What do we do to combat those threats? I goes back to the attack on the HSE. We are vulnerable.

We should not be shy about talking about defence. Defence does not mean that we are an offensive nation. We can be neutral and we can have strong defences. We need to really focus on developing our defences, especially in that hybrid space. In my field of expertise, in the AI space, we need to build that capacity. That capacity needs to sit across the agencies and instruments mentioned. However, it also needs to be something that is done at the societal, fundamental rights and consumer protection level because the kind of exploitation that can happen with these systems is very nuanced. Often it is not as obvious as a cyberattack on the HSE, but it can be exploiting a particular societal group that might be open to an anti-governmental message. We need a way of responding to that. Part of it is education, but education does not get us all the way there. We really have to be able to act, recognise it and take action.

On the same line as Professor O'Sullivan is saying, first of all strengthen education and research and be at the forefront of securing systems. Bring good practice for security in public infrastructure and the public sector and in universities. Build the leading examples to secure by design the critical infrastructure and the public sector. Even regulating systems that are used by the general public, which may not be secure, could be a vulnerability that can be targeted by outsiders.

I thank our witnesses for their contributions today which have been very useful in terms of informing the recommendations we will be making to the State in this space. There is always an opportunity for anyone else who is observing today and who has an idea or wants to get in contact to email us at ai@oireachtas.ie. That is very welcome. This is bringing today's meeting to conclusion but we are meeting on Thursday of this week at 11 a.m. when we will have representatives from the AI Leap Foundation in Estonia talking about how Estonia is using artificial intelligence in its education system.