Microsoft says its Agent Mode in Excel has an accuracy rate of 57.2 percent in SpreadsheetBench, a benchmark for evaluating an AI model’s ability to edit real world spreadsheets.
It generates 42.8% bullshit.
They probably view that as a statistic worth bragging about. It’s not. If Excel got calculations right 57.2% of the time it would be completely worthless.
I asked copilot to look through my every spreadsheet and find how many instances of a category occurred. I was curious to see if it was any good. Gave me 2 different numbers. Neither were correct.
Copilot: Putting the “Artificial” in Artificial Intelligence.
The tech behind LLMs could have just been Clippy and everyone would be happy.
Did you read the next sentence? Humans only get like 72% right. It’s not far off at all.
I wonder where that “human accuracy” statistic is coming from. Plenty of people don’t know how to read and interpret data, much less use excel in the first place. There’s a difference between 1/4 of people in the workforce not being able to complete a task, and a specialized AI not being able to complete a task. Additionally, this is how you get into the KPI as a goal rather than a proxy issue. AI will never understand context isn’t directly provided in the workbook. If you introduced a new drink at your restaurant in 2020 AI will tell you that the introduction of the drink caused a 100% decrease in foot traffic since there’s no line item for “global pandemic”. I’m not saying AI will never be there, but people using this version of AI instead of actual analysis don’t care about the facts and just want an answer and for that answer to be cheap.
As I’ve said many times, though not in this topic - AI is a tool to be used, and using it is a skill that needs to be learned.
For your pandemic example, that’s something that you would need to provide the AI with the context of. The joke of a “prompt engineer” being a job soon actually has merit, in that you want people who know how to use their tools the best. It’s constantly learning through iteration to give the AI a specific instruction set to get the results you want/need.
Depending on where you go to school, 70% is passing while 50% is not. While “not far off,” one is a C, the other a F.
That’s not at all what this means. In this instance, 70% is basically “human level”. For AI to already get 57% it means that it’s approaching the same level as people do in Excel.
Just keep regenerating data until it’s something the stock holders like. Doesn’t matter if it’s BS. They’re already accustomed to that.
So it achieved the actual proficiency of a middle manager…
Decades ago. The company that replaced it’s CEO with a LLM thrives.
Nice. Basically a coin flip
Slightly better than Vegas. Unfortunately, plenty of people are okay with Vegas odds.
Not enough accuracy to be useful. Not enough bullshit for politics.
The best cancers of both worlds.
Oh it’s going to do it for Word too?
Prompt: Termination letter telling my boss and bosses to kindly go fuck themselves and make it professional
The best you can do in any job is to care as little about them as they care about you.
They will barely read it, and they won’t care nearly as much as you do.
I resign my position as a [position], effective [DATE].
They’re out smarting the sheet that’s for sure.
So let me fast forward a bit, ->underpaid stressed out techworkers in the global south pretending to be AI for incompetent upper management in wealthy countries?
Not related but does global south refer to south of the equator or just everything south of north America?
I don’t know if it is a perfect term, but it doesn’t literally refer to any specific “South”, rather I think it is a reference to the coincidence that many of the heavily industrialized empires of the 18th, 19th and 20th centuries have been in the northern hemisphere, and the general colonial power dynamic therein set up has lead to the term “Global South” meaning pretty much anywhere that has gotten the short end of the colonialism stick, vs the long end.
https://en.m.wikipedia.org/wiki/Global_North_and_Global_South
Excel is one place where AI makes sense. All the data is there, in a nice structured and typed format with headings etc. Easily verifiable and to provide the reasoning for its work.
LLMs can’t count. Can’t add. Can’t deal with actually large datasets
How is excel a good fit for vibe-coding?
This isn’t just an LLM. It uses excel functions and features to do the counting and adding and dealing with large data sets.
It’s not “vibe coding” as much as “vibe performing steps in excel”.
Also LLMs absolutely can deal with large data sets anyway. Not sure where you got that from.
LLMs lose context over a short session. They all have input limits. Very small input limits usually. Best it can probably do is suggest formulas for you based on your natural language, maybe some copy/paste. Which means it can beat a 9 year old, great news everyone! Or show a help article on pivot tables (which the help function already does!)
Excel is very simple to work with, hence its ubiquity. LLMs also get shit wrong about half the time, way more than half with difficult things ime. Meaning they cost experienced operators time, a few studies are showing this now with coding. And are expensive as fuck. And slow as fuck. And reduce capacity for learning. Meaning they actually cap what excel can achieve, as the user won’t grow at the same rate, renoving the one advantage excel actually has: the learning rate is phenomenal
The C-Suite which insisted on this integration is basically an subservient idiot themselves at this stage who doesn’t understand their product, their market fit, or their userbase. They should replace thenselves with an LLM
I think you didn’t even read the article or read up about this integration.
This isn’t just an LLM, it’s Agentic AI.
AI is a tool that needs to be learned how to be used properly. Anyone can pick it up and get results that are “good enough“, but in the right hands what can be done is incredible - just like with any tool.
Look at something like minecraft as a perfect example of what can be done with a tool in the right hands.
Most people don’t understand “AI” as it is, and mistakenly think it’s just a school assignment cheating tool and a chat bot that makes things up.
People in here have been saying since LLMs can’t do maths perfectly it’s terrible for numbers, but they can’t see that it doesn’t need to do maths here because it’s in excel and excel has formulas and functions that can.
It’s crazy how the mere mention of AI makes some people lose any and all semblance of critical thinking and intelligence.
Excel is very simple to work with
Ok so your idea of excel is just what your average person might do at home with it - that’s not what or who this stuff is for.
And it will fuck up around half of even the simple formulas. This is really bad, and the idiots in charge should feel bad. Excel basically runs the world and they are about to fuck it up
Until it starts pulling data from nonexistent worksheets
You tell it not to.
I swear none of you guys have even attempted to use AI to do data analysis. I have, I built a MCP and integrated a copilot agent into Teams which has access to specific database data, and refined the rules for it to the point where the CFO rigorously tested it (and still does) and trusts the results it returns.
It could be good to layer in standard machine learning (ML), and it already does have some features (like line of best fit).
However, in today’s context AI means LLMs, and that is not a good fit due to its unpredictability.
