Impossible to find things

I’m going to predict that an ability to source and evaluate accurate information (critically) is going to become one of the most popular and crucial things to check when hiring people. No matter the role.

I wanted to search for some information on Massive Multitask Language Understanding (MMLU). It’s a benchmark for AI models. I wanted to see where the newer released open models lay against performance (which in simple terms is compute, so how much ‘effort’ do they require?). What’s it looked like over time?

(Just needed to clear some thoughts before the weekend)

Easiest place to start was to put ‘MMLU performance vs parameter size’ into your friendly neighbourhood search engine.

First few results? Okay press releases from the major players. Trying to find something more independent though. Scroll down and then come across the results in the screenshot below. I have to admit I thought I’d zoned out, switched to the wrong tab, or had a stroke.

This is going to get as bad as our rivers and beaches being polluted with merda.

I give up, time for a glass of wine.