From the PBS NOVA program I gathered that Watson 'reads' the questions. No image to text conversion is required. He answers in voice. Actually understanding his answers is not an issue. A few things that tripped him up: The clue had '40's in the question. Watson misinterpreted that to mean 1840's. Another set of clues was asking about the month that certain holidays fell in. From the clues, Watson was giving the actual name of the holiday. Watson was able to teach himself from a few correct answers given by the humans that the answers needed to be a month. Once he learned that he started getting the rest of the category right. The aim of the IBM'ers is to not have to have Watson 'learn' anything during the show, but be able to better recognize what the clues are asking. As zero said, the most challenging hurdle for Watson is the 'human' interpretations that we take for granted as humans. Assuming they just give the Jeopardy! categories and clues as they normally would, not catering them to Watson at all, we should have a very tight game on our hands! When IBM first brought in the Jeopardy! gang for a demo, Watson failed greatly. It seems now they've developed Watson to perform at a like level to the top Jeopardy! winners. Something interesting: like the regular Jeopardy! contestants, Watson needs to stand alone, ie: not reference 'the internet'. Therefore 'the internet' was loaded into Watson's guts. For example, millions of news articles, encyclopedias, the entire imdb... he has the computing power of 6000 home computers, iirc.