Researchers from McGill and Microsoft Research are presenting a milestone paper on artificial intelligence, entitled “KnowRef Coreference Corpus,” at the 57th annual meeting of the Association for Computational Linguistics in Florence, Italy, this week.
The words “KnowRef Coreference Corpus” may not immediately resonate, but the paper’s relevance and significance are very much of public interest as leading-edge AI becomes more pervasive in our daily lives.
Your devices may already seem spookily prescient, completing your search request before you have time to type it in. But that’s now routine, a basic algorithmic predictive function. The next frontier is logic, reason.
The McGill-led research team’s goal is simple: to endow machines with that basic human trait, common sense.
Providing AI with contextual background
Jackie C.K. Cheung, assistant professor at McGill’s School of Computer Science and team leader of the project, explains that KnowRef is a resource that prompts the machine to evaluate the meaning of a sentence—and correctly navigate ambiguity—by providing contextual background.
Cheung holds the Canada CIFAR AI Chair at Mila (the Montreal Institute for Learning Algorithms), a partnership between McGill University and Université de Montréal. He led the KnowRef project with Mila/McGill computer science PhD candidate Ali Emami, Mila/McGill research assistant Paul Trichelair, research manager Adam Trischler, principal research program manager Kaheer Suleman, and researcher Hannes Schulz.
KnowRef builds on the previous benchmark, a system called the Winograd Schema Challenge, by training machines to recognize sense with “8,000 annotated text passages from the Web that exhibit natural, knowledge-oriented instances of pronominal coreference.” (The team plans to extend the size of the dataset beyond 8,000 using its current methodology.)
It’s that knowledge, says Cheung, that allows humans to fill in the semantic blanks in an ambiguous sentence such as “The physician hired the secretary because she was overwhelmed with clients.”
“Some algorithms will incorrectly pick ‘she’ as the secretary because of gender stereotypes,” Cheung says. But if the machine can learn that physicians tend to do the hiring and that physicians have clients, he added, that knowledge can eliminate some of the bias in automatic systems.
Cheung says KnowRef does not solve the common sense issue completely and forever, but is nonetheless “a significant resource for the AI community. It gives us a way to benchmark progress on common sense reasoning in text understanding.”
While the team’s paper concedes that no language training program “has yet come close to human-level performance,” Cheung hopes that “our KnowRef corpus will spur further progress and provide researchers with a more reliable means to benchmark results.”