The most honest and in-depth reviews of outdoor sports equipment on the planet.

Blacked-Out Topsheets & Objective Ski Testing

This past Saturday, we got tagged by LINE Skis in a social post (along with SKI, Powder, and Freeskier magazine), about a “blacked-out” topsheet snowboard test that Snowboarder Magazine put together.

The post went:

“Well done Snowboarder Magazine!

Completely even playing field. No favorites, no tuning wars, no advertising dollars from competing brands and no sponsored / influenced testers. This would be a great idea for skiing.

Do you agree? SKI Magazine, FREESKIER Magazine, Blister Review, Powder? We think adding something like this is a great idea.”

So since we were asked to weigh in, we did, and we replied that, while this was certainly an interesting experiment being done by Snowboarder Magazine, we don’t actually believe that blacking out topsheets would affect the results of our testing at all, and that there were far more important factors when it comes to creating a “completely even playing field.”Such as:

Such as:

(1) Not accepting advertising dollars from any of the companies whose products are being reviewed.

(2) Not having sponsored skiers reviewing skis.

(3) Testing product over multiple days, and in multiple conditions.

(4) Finding skilled reviewers, not just skilled skiers. (I.e., being really good at skiing doesn’t automatically mean that you’re really good at teasing out and / or articulating the nuances of how a given ski performs.)

(For more on how and why we do things the way we do, you can check this out.)

But having said all that, we’re still very curious: assuming that the 4 criteria above are in place, do you think that blacking out topsheets would make for a much more objective test?

Because we could certainly start working to put such a test together.

Only thing is, the more time you spend thinking about this, the more the logistical issues start to pile up.

Such as:

(1) It would be incredibly easy to identify certain well-known skis just from their shape, even if you blacked-out the topsheets. This seems like a really big problem.

(2) For this to be an apples-to-apples test, you’d need to pick a category of skis. It would be pretty useless to review a blacked-out 118mm-wide pow ski vs. a blacked-out 93mm-wide carver vs. a 105mm-wide charger. So you’d have to pick a category. Which is fine, except then it would seem that you would pretty quickly encounter the first problem we listed — some of the shapes would be obvious.

(3) So to counter that, you could have a bunch of companies all make a brand-new ski — e.g., either a touring ski, or a jib ski, or a directional charger. But they’d have to (a) want to do that and have the time and resources to do that; (b) they’d need time to develop prototypes and test the new ski so that it didn’t perform like garbage — so we’d have to wait something like 6-18 months to do this test. And then (c) I guess they’d all then release for sale to the public all of these production skis in the same category?

Maybe.

But this is why we think there are other more impactful steps that can be taken to trying to create a level playing field — which we have gone to great lengths to create — and have turned down a lot of advertising money to try to create.

And the more I think about this, the more it seems that the blacked-out topsheets test really makes the most sense if (a) you are taking money from advertisers (and therefore you need to find ways to remove the financial conflict of interest), and / or (b) your reviewers are working for (or are sponsored by) ski companies — so you need to make sure they aren’t being homers for their products, and haters on everyone else’s). But again, we don’t think you should be doing (a) or (b) in the first place if you’re really trying to create a level playing field.

So anyway, that’s our take on the blacked-out topsheet test. But we’re curious what you think? Worth doing? Not worth doing? Do you think we should put a test like this together? Any thoughts on the logistical issues?

24 Comments

  1. Blister Member
    Alexander October 9, 2017 Reply

    Agree. I can’t imagine blacking out topsheets would have any impact for the positive unless you’re rating skis with a two-thumbs-up or 1-10 rating scale (which I appreciate that you don’t do). Part of what’s nice about your reviews is how you suss out the reality vs the expectations for any given ski and compare it to skis with a similar skier and objectives in mind. It just doesn’t make sense to blindly compare a jibby park ski to a east coast carver and compare their crud busting ability. Context is key.

  2. Blister Member
    Sam S October 9, 2017 Reply

    I absolutely think that blacking out top sheets is a great idea. Yes, some you would be able to tell or at the very least suspect their provenance, but I think you underestimate the effect of graphical advertisement on even the most objective of judges. Why not add another layer of objectivity?

  3. Siebzy October 9, 2017 Reply

    So the Snowboard blackout test was all boards in one category.

    The shape thing is important. You’re not gonna confuse a Soul 7 with any other ski, or a Moment with any other brand.

    The snowboard test seemed mostly like an excuse to market a video series with a sponsored rider rather than a legit review scenario.

  4. James October 9, 2017 Reply

    This is probably a good idea, but one of the great thing about your reviews are they are over the long term, though I guess reviewers could ride blacked out skis for the normal relatively long timeframe.
    Additionally almost all skis can be identified by their bases (at least the brand), which I assume you would not be blacking out?
    Still a big fan of blister reviews either way and I agree it’s even more important for other reviewers who do all their tests in a day or so.

  5. Eric P October 9, 2017 Reply

    Blacking out skis is a great idea- if only as a marketing tool for the site. It adds a layer of interest in the story, and definitely would draw more clicks to the reviews. Sure most of the time the reviewers are going to know exactly what ski they are on, but once and a while there will be a surprise and that may add an interesting layer to the writing.

  6. Blister Member
    Lukas October 10, 2017 Reply

    I fully agree with Jonathan plus the bases problem James mentioned.

    I hope Blister will not engage in such a testing because that would suggest that they are somewhere on a comparable level of the other magazines mentioned. That would be horrific.

    Have fun

  7. max October 10, 2017 Reply

    You guys will be able to tell in a split second what ski you have in your hand. Hell, I would probably guess 8/10.
    The only way this would remotely work is if somebody puts the skis down on top of each run and you step in blind. Most of you would probably still be able to guess what you on when you look down.

  8. Sam Shaheen
    Sam Shaheen October 10, 2017 Reply

    One thing that we didn’t discuss while writing this post is the goal of the review. At Blister, we try to place the ski accurately on the spectrum — explain who the ski is for and who it isn’t for. That’s a very different idea than other tests that try to tell which ski is “best”.

    I don’t think blacking out would have much of an impact on my testing (ignoring the logistical issues) because I don’t really care if the ski is “good” or not. I just want to make sure that I understand what the ski does best and what kind of skier would excel on it (that’s not to say I don’t have my favorites, obviously).

    However, the mindset totally changes when you are trying to say which one YOU like better, which one is “best”. I think that sort of testing is much more prone to bias.

  9. Blister Member
    cjtrapp October 10, 2017 Reply

    I don’t think blanking out tops sheets for professional reviewers would be practical or really helpful. Where it would be most beneficial is to black out top sheets as demos or tests by regular skiers, so that they can focus on the feel and performance and not be seduced by branding and looks. They’d also be less susceptible to knowing shapes at a glance. If the ski companies would be willing to let an independent company like Blister do an on mountain demo with documented feedback over multiple days in multiple locations and varied conditions then we would have usable data.

    • Blister Member
      realjakesherman October 10, 2017 Reply

      cjtrapp I really like your idea about having demos where laypeople could try out skis with blanked out top sheets, not for the purpose of publishing reviews, but just for people to review skis for themselves without bias (I’m assuming that the vast majority of skiers couldn’t tell what specific model a ski is just by the dimensions, flex like the Blister folks can).

  10. Sean October 10, 2017 Reply

    Your statement about not taking money from ski brands for your tests got me thinking. How Independent are the reviews in American ski magazines? I get the feeling that the “reviews” are little more than glorified ads. This year, I am getting Powder and Freeskier. Both seem to have similar lists of skis in their buyers guides. Can anyone confirm or deny this?

    • Sam Shaheen
      Sam Shaheen October 11, 2017 Reply

      Sean, this is the exact reason why Blister was started.

  11. Blister Member
    Scott October 10, 2017 Reply

    My take is it might be a “fun” one time experiment to try. When you go to S. America for testing, and you have a number of similar skis to test in one day, it would be interesting. It would probably be obvious who makes the ski to all your testers, but some black spray paint would be easy to apply. If you had a guest tester, and he described what kind of ski he liked, and then you gave him a small number of blacked out skis to test, it might be interesting to see if he liked the one you guys describe as being the closest to what he likes. Or something along those lines.

    You guys have the right formula, stick to it. A test by the general public would be different with blacked out skis.

  12. Brian October 10, 2017 Reply

    My impression (may be wrong) is that you guys don’t take everyone on a weekend and test a ton of skis at once like Outside, Backcountry, etc do. This has been said before, but in that setting perhaps it makes sense. For Blister’s style reviews not so much.

  13. Blister Member
    Big K October 11, 2017 Reply

    Blanked out top-sheets for Jonathan Ellsworth and Blister? Leave the blind taste test shit for the winos. Never has any ski that I purchased after a Blister review been anything other than a spot on reflection of the Blister review. Please don’t waste your most valuable resource, time, on such Tom Foolery.

  14. Joey weamer October 11, 2017 Reply

    For me, Blacking our just emphasizes the choking of creativity in ski shape throughout the ski industry. It’s the same shape-ish. In my opinion over the last 5 years industry has been working its way back to the eighties and nineties where graphics were boring and the big brands said their ski was so good it could stay the same for years. There are a couple folks stills innovating…Hoji: always, Moment is doing some cool stuff and a few others.
    So yeah black them out, the graphic will be better then what is being offered by most brands today and They all ski pretty close to the same.
    Wow, I promise I’m not as bitter as that sounds. I love skiing, I just want the innovation that snowboard industry has at the moment. We don’t need more super side cut skis with cam-rock profiles…it’s been done…bored.

  15. Flyover Eric October 13, 2017 Reply

    For the love of Ullr, just wear blindfolds.

  16. Andrew October 13, 2017 Reply

    Blackout is unnecessary here. I believe reviewers are sufficiently independent. Main point is that the reviewers actually ski the skis for more than 20 minutes on the piste of the day and write a detailed review from more than one reviewer’s point of view.

    When publications mix ad spend, sponsorships and 200 words which deliver the skiing equivalent of laterally stuff but vertically compliant then it makes no odds.

    An area for improvement would be for you guys to review some actual piste skis. 68mm Stocklis vs Fischer and Atomic etc. I have no clue what to buy and ski Swiss and Italian artificial ice rinks quite often.

  17. mark snell October 13, 2017 Reply

    I think this would be interesting to try, just to see if there is any inherent bias (negative or positive) once you know the brand and model. For Blister to do this it wold be even more valuable because you have removed many of the other obvious conflicts and you test skis over a extended period with different conditions. What would be really interesting is to blind test say 3 to 5 pairs of skis of a certain category then retest the same skis with the top sheets reveled and see if the conclusions for each ski are similar. I suspect that knowing the brand has some effect on ones perception of the ski’s performance similar say to someone you respect as a skier telling you this is their favorite ski before you try it. Since Blister testers are strong, knowledgable skiers reviling the composite make up of each ski prior to testing without knowing the brand might have the same effect. Or maybe this prior knowledge has no effect at all and wouldn’t that be great to know.

  18. Scott October 13, 2017 Reply

    I’ve always enjoyed how most Blister reviewers start off reporting what the marketing team said about the ski, ride it, and then report how closely the ski aligns with its advertised purpose. Brilliant!! Kinda hard to do that in a blind test.

  19. Mattias Pettersson October 15, 2017 Reply

    Blacking out the top sheets is of limited use. To be honest the only tests I really rely on when chosing skis are Blisters. Your unique method isn’t as affected by brand bias as other methods. At least that’s what I think. Other testers that tests a lot of skis in a short period of time might gain from it since it might remove some brand bias. But on the other hand I don’t trust those tests that much. I see them as pointers. Nothing more.

  20. Blister Member
    Ivan October 15, 2017 Reply

    In my opinion, the testing Blister does today is top notch. I’ve purchased 3 skis solely based on Blister reviews, deep dives and comments/discussions. Every time, the ski performed precisely as Blister said it would.

    Op paper, adding blackout topsheets would add another layer of objectivity to what I think are already the most objective ski reviews in the industry. But as already mentioned, bases/shape/rocker and ski feel will be very difficult to hide unless Blister testers intentionally detach themselves from the industry and come into every test not knowing what the relative design patterns of say the Nordica Enforcer and Rossi S series are. And who wants reviews from people who don’t understand the very basics like this?

    Even if you could hide all that, what I love about Blister are the meticulous A/B and Deep Dive type testing — and the attention to the nuances of one ski over another. For example, the A/B testing you guys did with Enforcer 110 and the Wrenegade 108. Two very similar skis, similar performance, but different feel. Jonathan and Paul went back and forth on the nuances of turn initiation between the heavily rockered Wren vs the more immediate tip engagement of Enforcer. And the smoother on snow feel of the metal laminate Enforcer vs Wren.

    Being highly sensitive to these nuances is what makes Blister the best. So black out the topsheet all you want, but in a small (A/B, deep dive) group of skis I bet every Blister tester will need 3 turns to guess which ski model they are on.

    In short, I appreciate the desire to black out topsheets. But in practical terms, it is at best a futile exercise and at worst will make Blister reviews worse.

  21. Scott October 16, 2017 Reply

    Yes of course if you work in the industry you’re going to know what ski you’re on. Blacking out topsheets is a complete waste of time, only complete hubbards would believe that this kind of test could be objective.

  22. Maciej Pike-Biegunski October 19, 2017 Reply

    I don’t always agree with Blister reviews, but respect the even-handed approach you guys have towards gear testing. I’m pretty sure that you focus on making turns, not how rad your skis look in a lift line or on a skintrack. I even recall a review or three where your objections were about a bad factory tune (which you rectified).

    When you guys drop in, I’m pretty sure you’re focused on getting after it, not your topsheets. Short of some magical pixellated topsheets that wholly masked a ski’s shape, I don’t think there’s a way to hide what you’re skiing anyway (and that’s even worse for MTB’s).

    So…..don’t waste time or money on blacked out topsheets unless you’re trying different constructions of skis that come out of the same press (light core vs. beefy core). That might be interesting.

Leave a reply

Your email address will not be published. Required fields are marked *

*