Our Process at Tastemaker is Complicated. Here’s the short(-ish) version.
We are often asked what goes into the testing process behind our critiques, and we love to share about it, because, more than anything, we’re proud of how much work goes into making a Tastemaker review as comprehensive as it can be. While every product is unique, and requires tailor-made criteria, we can provide the general blueprint on how we’re able to produce as much unbiased and useful data as we do.
Selection and Preparation
A Tastemaker review starts with a choice. We select the products we review usually on the basis of their relevance to the larger market, whether that means the product is an industry dominator or an industry disruptor. We believe that highlighting what’s most relevant to a sustainable, great consumer experience is important, but we also think properly assessing products that are on the forefront of the market because of hype or market share is equally important to making sure our audience can make an informed decision.
We do often seek out other specific products that we have interest in for any number of reasons, but most usually for innovations we haven’t seen to keep our audience on the cutting edge of being informed. Our other focus is ensuring that our understanding of the market is based on what the future might bring, so we try to seek out products that have not yet been released but we can reasonably predict to be a major part of the conversation in a product space. We have ongoing first-access relationships with certain brands to review new release products before they become available to consumers so our audience can have information day-one of release, while the brands themselves can get nuanced feedback before roll-out.
Once we’ve selected a product we want to review, we assemble a team of multiple individuals all with specific expertise relating to the product or market sector, usually from in-house reviewers, but also oftentimes in collaboration with vetted in-network product experts who are not affiliated with any brands. After these individuals have met and discussed preliminary expectations to establish a baseline, they are tasked with building an itinerary for product testing that allows them to fully test all product capabilities, while also collecting standardized performance metrics to measure against peers.
Whenever we receive a product from a use category we have not yet encountered, we do research and speak with experts within the field to establish benchmark testing that we can use for current and future releases, in order to standardize the metrics by which peers are measured against one another. We believe that one of the biggest issues with the vast majority of review content is lack of consistent standards, so we’ve attempted to root our grading system for products within a space entirely in consistent standards to avoid bias compromise and subjective argument.
Intake and Verification
Usually the above part of the process happens in tandem with our actual acquisition of the product and the intake process. Products are often shipped directly to us from brands to avoid distributors who may alter the standard consumer experience. When receiving a product, we assess all attributes of the product before testing to ensure that metrics provided by manufacturer advertisements or statistics stated on storefronts are accurate, as well as photographing and documenting build elements and materials to measure wear later during our testing process.
During our reception, we will collect data on the packaging, materials and other accessories potentially provided by the product in a bundle, in order to record specific data points on the footprint of the product’s shipping and storage, as we believe acknowledging the often forgotten element of packaging and presentation is key to fully understanding the environmental impact a product has throughout the entirety of its lifecycle. Brands that put particular effort into minimizing impact while maintaining class, design and functionality in the branding of their shipping and unboxing are unfortunately not commended for their thoughtfulness.
After all these data points are collected and recorded, products are disseminated to team members, where they are again inspected to ensure no changes have occurred during transport or allocation. Products that require certain standards for transport are also given special consideration to ensure their journey aligns with the manufacturer’s intent, while also documenting the impact special arrangements may have ecologically.
Testing
During testing we employ a rigorous and dynamic process that includes well-over three hundred (and growing) different points on which performance, quality, features, sustainability and even aesthetics are graded throughout the duration of the testing process. At every instance of testing, measurements and circumstances are recorded to be compared against other reviewers from the team to vet out biases and inconsistencies. We do not test in a vacuum either, choosing to test products in the intended environment alongside extreme examples to give use cases for every possible buyer. If you might be interested in a product in a particular environment, we hope to be able to provide data specific to your intended application.
Products with special use cases to focus on are often pre-designated, or testing is modified to accommodate specific new situations. With lifestyle products, particularly outdoor uses, often modifications to the testing process can cause the duration of the testing period to be significantly extended. In these cases, we will delay release of final results in order to ensure any scores or assessments are reflections of the full suite of features. We always prioritize painting a full picture over rushing out reviews, but if we believe preliminary results are an accurate microcosm, we will release with the caveat of preliminary results being later being publicly modified to reflect final summaries.
When a product starts the testing phase it can often expect to be put through daily use, if applicable, all the way until the testing period ends for all members of the team. For lifestyle products, reviewers will often use them in structured environments as well as their own day-to-day use to gather additional data and potentially discover hidden or unexpected distinctions or criticisms to be had with product design.
Release, Revisions and Post-Release
Once a testing period has ended, time is then devoted to compiling observations and determining what measurements and takeaways are accurate and most relevant to the overall performance of the product. Criticisms, distinctions and general observations are all compiled and deliberated on to decide what elements should be points of focus within the public review.
The crucial part of this discussion becomes a comparison of findings and results within the single product and the larger space of peers, during which the team will decide, if it has not been decided by prior testing, which elements are most relevant to the consumer experience. These elements are then directly contrasted against other empirical measurements from previously tested offerings from peers, and a set of scores representative of performances within the space are designated for the product.
At this time, a final score can be compiled, though it will still be subject to change with new peers. After this, reviews are written, reviewed, and finally, released to the Tastemaker site and other venues.
There is a post-review process, as revisions are often released to the official review page as well as our dedicated revisions blog. Official updates are often generated as new accessibility or feature functions are added in updates, while we may choose to make revisions as new information results in the product’s performance being altered from where it was at the time of release. Some of our revisions are also prompted by our readership using our revisions form to connect us with aspects we had not previously considered as important to the experience, but our audience has mentioned are important to their enjoyment of the product.
Our process is ever evolving to become more exact and efficient, we look forward to releasing more updates as we can to maintain transparency. If you would like a look at some of the testing data we generate, checking attached videos on specific reviews may yield an in-depth look. Media outlets interested in more information may reach out at media@tastemaker.online.