A Quote from The Turing Test
Tom: Imagine you are in a room. In this room you are passed Chinese sentences trough a slot in the wall. Inside the room is an instruction book written in English. This instruction book tells you which Chinese words to pass back trough the slot in the wall as a response. By doing so you have a conversation in Chinese.
In the Chinese room, because the response you pass back trough the door are the correct responses the person on the other side of the door is convinced you are a native Chinese speaker.
Ava: Well, they're wrong. But the person stuck in the Chinese room is not aware of the conversations content.
Tom: This is a problem with the Turing Test. A computer can pass the Turing Test having convinced a human they are having a polite conversation. While the computer has no idea that a conversation has taken place.
Ava: What if both people passing Chinese words are reading from the instruction books?