Huawei caught faking benchmarks

Performance benchmarks are a weird phenomenon, but consumers absolutely make purchase decisions based on numbers, especially when they look larger. So, what happens if they're a lie? Well, we just found out, with Huawei's latest phone.

Huawei is a huge proponent of pushing big fancy benchmark numbers to sell devices, and it's been caught red-handed messing with its latest devices lying about how they work. Essentially, it's using phone performance profiles to inflate those numbers:

[Huawei's] newer devices come with a benchmark detection mechanism that enables a much higher power limit for the SoC with far more generous thermal headroom. Ultimately, on certain whitelisted applications, the device performs super high compared to what a user might expect from other similar non-whitelisted titles. This consumes power, pushes the efficiency of the unit down, and reduces battery life.

Essentially, that boils down to this: Huawei detected a benchmark was running, and took off the performance restrictions that would throttle performance when the phone gets hotter, or drains too fast. 

It's analogous to the Volkswagen diesel scandal, except without the industry regulations that are there to protect consumers and an industry scandal, because they could basically rig this up any way they wanted without generally getting caught. 

The test was only discovered to be bogus when Anandtech found a discrepancy between private and public versions of their benchmarking tools, and Huawei later claimed that it planned to release this "performance mode" all along.

Essentially, the company lied to get to the top of the benchmark list, got caught, and won't admit it was ever at fault. The funniest thing in all of this is Huawei used "AI" as a excuse, saying that its "intelligent" software is at fault for throttling the software, and should be considered a feature.

While this won't matter in America, where benchmarks are less important, it will matter on a global scale in markets where those numbers do matter (largely Asia). Companies that rate phones have actually delisted the product from their indexes until the restrictions are removed on the software mode, which is a decent blow given how many potential buyers will be looking for the rank.

What's wild, to me, about this isn't even any of the cheating, it's that benchmarks matter -- and Huawei's sort of right. Even if it is laughable, AI tools are becoming a core part of software on our devices and is being used to manage everything from our battery usage to screen brightness. How do we measure a benchmark in a world where most users just want to get to the end of the day?

Benchmarks are relatively useless on paper, but it's incredible that a vendor went to these lengths just to market to those who do care, and frankly, it's not going to be the last time we ask these questions, because it's going to get messier as time goes on.


Tab Dump

Apple killed a well known malware app
Turns out the 'Adware doctor' was a pretty shitty doctor, and sent your data to China. Go figure.

Verizon's ad boss, Tim Armstrong, is ready to hit the eject button
Tim Armstrong was CEO of AOL, before it sold to Verizon and merged into Oath (the worst-named company ever). Now he wants out! It's almost like he got rich, then bored, but had to wait out some sort of agreement before he could leave...

Microsoft announces a Surface event for October 2nd
A new Surface Studio (the iMac killer) or a better Surface Pro? Time will tell, but this one will be really interesting, given how mature the Surface line is now.

British airways was hacked and lost so much data
OK, this blows: BA was attacked over the space of a week and lost financial data of its customers, along with a ton of personal information. This will be a fun exercise in will GDPR mean they have to pay people money for their lost data? In theory, the law requires compensation, financially in this situation.