This past Saturday, we got tagged by LINE Skis in a social post (along with SKI, Powder, and Freeskier magazine), about a “blacked-out” topsheet snowboard test that Snowboarder Magazine put together.
The post went:
“Well done Snowboarder Magazine!
Completely even playing field. No favorites, no tuning wars, no advertising dollars from competing brands and no sponsored / influenced testers. This would be a great idea for skiing.
Do you agree? SKI Magazine, FREESKIER Magazine, Blister Review, Powder? We think adding something like this is a great idea.”
So since we were asked to weigh in, we did, and we replied that, while this was certainly an interesting experiment being done by Snowboarder Magazine, we don’t actually believe that blacking out topsheets would affect the results of our testing at all, and that there were far more important factors when it comes to creating a “completely even playing field.”Such as:
(1) Not accepting advertising dollars from any of the companies whose products are being reviewed.
(2) Not having sponsored skiers reviewing skis.
(3) Testing product over multiple days, and in multiple conditions.
(4) Finding skilled reviewers, not just skilled skiers. (I.e., being really good at skiing doesn’t automatically mean that you’re really good at teasing out and / or articulating the nuances of how a given ski performs.)
(For more on how and why we do things the way we do, you can check this out.)
But having said all that, we’re still very curious: assuming that the 4 criteria above are in place, do you think that blacking out topsheets would make for a much more objective test?
Because we could certainly start working to put such a test together.
Only thing is, the more time you spend thinking about this, the more the logistical issues start to pile up.
(1) It would be incredibly easy to identify certain well-known skis just from their shape, even if you blacked-out the topsheets. This seems like a really big problem.
(2) For this to be an apples-to-apples test, you’d need to pick a category of skis. It would be pretty useless to review a blacked-out 118mm-wide pow ski vs. a blacked-out 93mm-wide carver vs. a 105mm-wide charger. So you’d have to pick a category. Which is fine, except then it would seem that you would pretty quickly encounter the first problem we listed — some of the shapes would be obvious.
(3) So to counter that, you could have a bunch of companies all make a brand-new ski — e.g., either a touring ski, or a jib ski, or a directional charger. But they’d have to (a) want to do that and have the time and resources to do that; (b) they’d need time to develop prototypes and test the new ski so that it didn’t perform like garbage — so we’d have to wait something like 6-18 months to do this test. And then (c) I guess they’d all then release for sale to the public all of these production skis in the same category?
But this is why we think there are other more impactful steps that can be taken to trying to create a level playing field — which we have gone to great lengths to create — and have turned down a lot of advertising money to try to create.
And the more I think about this, the more it seems that the blacked-out topsheets test really makes the most sense if (a) you are taking money from advertisers (and therefore you need to find ways to remove the financial conflict of interest), and / or (b) your reviewers are working for (or are sponsored by) ski companies — so you need to make sure they aren’t being homers for their products, and haters on everyone else’s). But again, we don’t think you should be doing (a) or (b) in the first place if you’re really trying to create a level playing field.
So anyway, that’s our take on the blacked-out topsheet test. But we’re curious what you think? Worth doing? Not worth doing? Do you think we should put a test like this together? Any thoughts on the logistical issues?