TECHNOLOGY moves at breakneck speed – but the latest warning from AI bosses is that we need to slow down.
It’s a surprising call from Silicon Valley (and beyond) as tech leaders and experts worry we’re struggling to keep up with .
Just last week, Anthropic researcher Jacob Coxon resigned over fears that AI companies . Anthropic boss Dario Amodei warned that we must “pace the frontier” of AI development.
And AI scientist Evan Hubinger warned that there was a 10% or greater chance of advanced AI causing human extinction within a decade.
This week even saw wade in on the debate at an AI summit, where he said that AI had the capacity to both save and take life.
Helping to unpick the mysteries of AI is Jake Moore, Global Cybersecurity Advisor at ESET, who spoke to The Sun for Future Tech Feed with Sean Keach (Watch on YouTube / ).
Jake is optimistic about AI, but says much of the worry isn’t tied to chatbots so much as their evolution: agents.
“We’re moving from AI telling us something from a request that we’re asking a chatbot – some information, very similar to like a ‘super Google’ – but now we’re moving into the realms of AI doing it for us,” Jake told us.
“This is this agentic side of things. We have to remember that we are still in charge of them.
“Because we are able to control them at the moment by telling them the realms in which they’re allowed to do those things.
“And we are choosing which programs and which applications that we’re allowing them access to.”
Part of the worry is that these AI systems might become “misaligned” – and do things that humans don’t intend.
Earlier this year, a agent and hacked into another company.
These could get worse, experts say, as agents get increasingly more intelligent and capable.
That’s led to recent calls for AI development to slow down.
“We keep hearing about this slowing down and it’s not something that we ever see in other areas of business,” Jake explained.
“I don’t think anyone seriously saying turn AI off. I mean there’s a chance that there is this brake pedal that people talk about.
“But what it really means is if we break it down it’s about throwing in independent testing. That would be really good before it hits the masses.”
Jake gave the example of Claude Mythos.
It was an ultra-powerful model built by Anthropic, one of the leading AI giants.
“Earlier on in the year Anthropic was saying that they didn’t really want to release their Mythos software to everyone,” Jake said.
“They wanted to give it to the banks and the big companies in America beforehand to make sure that they were able to go and test any of their vulnerabilities because it’s a very good vulnerability scanner.
“And that’s effectively what we’re looking at in the future to test more before it goes to the masses and I think that’s possible.
“Again we’re in charge here. we are the top of that food chain.
“So let’s bake in those safety measures, but you can also say put in those safeguards.”
Of course, plenty of people will rightly be worried by all of the warnings over the existential danger of AI.
We’ve heard endless tales of AI one day going full Terminator and dominating humanity.
Last week’s “10% risk” of a total human wipe-out dominated headlines, but should we really be worried?
“A lot of these are just classic statistics that have been made up on the spot,” Jake said.
He continued: “When you throw these in with big claims from people who have had big job titles in the past, it does have that extra weight that makes people worry.
“And we don’t want people to worry and panic about this.
“We want people to work with it because it’s not going away. I think it’s better to stay looking at what the threats are now.
“So you have what we call AI enabled scams. We still see scams all the time, and I deal with them every single day – and how AI is enabling that scale.
“That’s something that I think we should be monitoring much more closely.
“We’ve still got phishing attacks as the number one attack vector into businesses and people’s personal accounts.
“And so if we look at how AI is enabling that, it’s going to make much more of a difference.
“Than if we just sit back and think, ‘well that’s the end of the world we’re all going to die anyway, so what’s the point?’”
Jake is optimistic that humans can remain in control of AI, with enough willpower.
“There is still this term ‘the human in the loop’, which might grate on people,” he said.
“But I still think there’s a lot to be said for that, because we do still have that human control of it.
“It’s not a case of, say, turning AI off as we keep hearing.
“It’s much more to do with: how we are able to control the bits that we have control of now.
“And and also we don’t want to be distracted by those those sci-fi worries that we’ve seen Terminator – we’ve seen all those films.”
Jake says that AI could end up distracting us from other genuine threats.
“We have other threats in here as well,” he says.
“If you just think of nuclear bombs, we talk about how it’s just a few buttons away from extinction.
“But that seems to have lost traction because people are now talking about these other things.
“We’ve still got threats to do with viruses around the world, but we’re trying to control them – and we’ve got humans all over the world researching that and taking control of it.
“I don’t think that this is something that we should be really worrying about too much.
“If anything, we’re learning to work with AI at the moment.
“And in the future, from where we can see, it has so many positive benefits, like the King was saying recently.
“And we do need to focus on that, because otherwise we’re just going to spiral out of control, and just think technology is the end of the game.”
The good news is that Jake has plenty of hope for humanity on AI.
He points out that time and again, we’ve overcome major difficulties – and we are very good at surviving as a species.
“I think we have to look at history to then look at the future and history,” Jake said.
“Let’s say over the last 20 years, we we have aligned in the fact that we don’t all want to create mass extinction.
“There is access to nuclear bombs in the world, but we haven’t pressed those buttons because we don’t want it to happen.
“Why have we not seen ? Because we all know that it’s just a terrible outcome for everyone involved and so we tend to keep that control of.”
He continued: “If we’re talking about the mass extinction – which is the the big term that we’re continually hearing – which is going to worry people, I just can’t see that happening.
“That’s probably because I like to see the positive in a lot of things, in particular tech because it’s so wonderful.
“But I think we are we are forgetting that governments still have that ability to enforce saving lives where possible .
“And doing the best for this technology and doing the best financially as well.
“And there’s going to be a lot of money still to be made around the world.
“Even though people are thinking that AI is going to take our jobs, we all thought that when the internet came out – and in fact, it just boomed everywhere.
“And so it’s just another extension of technology.
“I think the more we talk about it on a government level right the way down through companies to the lowest level, people will be able to improve everything make the areas of their lives much more efficient.
“I think we’re going to be okay.”
