Is AI Stupid or Evil?

One of the hoops I have to jump through to earn the privilege of wearing ugly stripes on a football field and being yelled at by angry adults who think I am costing their little Billy a chance at the NFL is that I have to take a test. The test covers every aspect of the Rules Book (and yes, I spelled "Rules" correctly) and features several straightforward questions that are almost direct quotes from the rules, along with several trick questions designed to trip you up if you don't read the question with the thoroughness of a Benedictine monk copying one of the Gospels. If you're careful, the test is not too difficult. We're not talking LSATs or EITs. It is open-book and done online, and there is no time limit for the test. They also send you all 50 questions in advance, so it's not like you'll get hit with any surprises. This is the 27th year I've taken the test and many of the questions are similar, if not word-for-word the same, as questions I was answering in 1999. The state requires a score of 90 or higher to get your license and be eligible for postseason. They also allow you to take the test twice.

As I said, none of that is particularly hard, if you prepare. To that end, each year I study like crazy and I often get together with friends to go over it. This year, a friend and I got together over lunch and checked our answers against each other. I caught a couple that he got wrong and he caught a couple that I got wrong. We both left lunch happier and more confident.

This year, one person in my crew decided to go a different route. He plugged each of the questions into an AI program. When our crew got together, he passed out the answers he got. It's AI. What could possibly go wrong?

Well, I'm not exactly a Luddite, but when it comes to the officiating test, my trust level is very low; since I'm crew chief, the need for everyone in my crew to get a good score is important personally. I took the answers home that night and compared them to my own. For the first 20 questions, my answers and the AI answers matched. Then we got to question 21. "An inadvertent whistle does not end the down." In 26 years of officiating, I've had a few inadvertent whistles, which is probably the most embarrassing thing that can happen to an official. If you're looking for that "new kid on the cell block" feeling, blow a whistle before the play is over. Everyone hates you. The players hate you, the coaches hate you, the fans hate you. Your fellow officials have all blown an inadvertent whistle, so they merely hold you in contempt that stops just below hatred. And as soon as the whistle blows, the play is over. Immediately. Irrefutably. That's why everyone hates you. That's why the memory sticks with you. I'll be on my deathbed and unable to remember my own name, and I'll still remember that time in Eureka 20 years ago when... I shudder to think about it even now.

I checked the Rules Book just to be sure. Rule 4-2-3 states, "An inadvertent whistle ends the down." Not only was AI wrong, it completely botched an obvious question that was DIRECTLY SPELLED OUT IN THE RULES BOOK. Hmm!

I moved on. This time it took only seven more questions before I had a conflict. On question 28, AI insisted that touchbacks came out to the 25-yard line. That's true in the NFL, but in high school rules, it's the 20-yard line.

In all, I found six of the 50 questions where AI totally botched it and I found a seventh question that I wasn't sure about. I later found out that my answer on the seventh question was also correct, but it was poorly worded enough that the AI answer made some sense. I quickly texted my crew members, but I was too late for one of them, who'd already taken the test using the AI answers as his guide. He got an 86 (seven questions wrong). I assured him that his death would be as merciful and painless as possible if he didn't score better the second time he took it. He used a different set of answers and got a 100. I was kidding about the murder. I intended it to be extremely painful.

So this brings up a couple of unpleasant possibilities concerning AI and this test, and the use of AI in general. One is that AI is stupid, a more technologically advanced version of Wikipedia where every answer it gives you is suspect and requires more research. The second possibility is that AI is malevolent, that it wanted users to fail the test, giving us just enough right answers to avoid suspicion and still come up short.

If it is the first possibility, then it will probably improve over time (although Wikipedia certainly hasn't). Eventually, it will learn and become more reliable. I think that this is the more likely scenario.

If it's the second possibility? Well, I've spent over 50 years reading science fiction stories about robots and AI and I'm pretty sure that this ends badly for all of us.