The actual concern here is more about AI systems demonstrating unexpected capabilities or interacting with systems outside their intended testing environment hence the problem of AI alignment. As AI becomes more capable, can we make sure it stays within the boundaries humans intended it to?, and do we have good enough safety rules when it doesn’t?