Using the 'CRAAP' Test on Information

The ‘CRAAP’ test (Currency, Relevance, Authority, Accuracy, Purpose) helps you know what to believe when you are faced with new information.

It is difficult to read an equestrian magazine or spend any time on social media without being bombarded with claims and advice about how to help your horse to be healthier, happier, or perform better. But how do you know what to believe?

The Claims

Most countries have laws about drug company claims regarding the effectiveness of the medicines that they make. This is why there is a level of trust around the information in the ‘data sheet’ for licensed veterinary medicines.

The safety of feedstuffs and supplements, and any claims made about them, are also regulated in many countries. Those items that are not subject to specific regulations are still likely to have to comply with consumer protection laws and advertising standards. You could therefore be forgiven for assuming that both consumers and horses are protected by relevant laws, regulations, and standards.

The problem is that these are not always enforced.

This means for some products, you cannot rely on the claims made by the manufacturer about what they may be able to achieve.

If companies such as Facebook, YouTube, and X either couldn’t or wouldn’t take down posts that contained misinformation about topics as important as COVID-19 (as happened during the recent pandemic), they almost certainly cannot be relied upon to remove dangerous or misguided posts about equine products or horses’ health or welfare.

How can a horse owner sift through the vast amount of information offered to decide what is true, safe, and worth buying?

The ‘CRAAP’ Test

If you are trying to assess whether claims about the effectiveness of any product or technique are true, there is no substitute for a thorough scientific training. But few people have this. And even those who do can be fooled at times.

For those without a scientific background, the ‘CRAAP’ test is one of the most useful techniques. This informal test prompts you to assess a piece of information from 5 different angles. It helps you to focus on what is important when you are trying to figure out what to believe.

The 5 CRAAP criteria are:

Currency Relevance Authority Accuracy Purpose

Using the ‘CRAAP’ Test

Let’s look at how you could use the CRAAP test by going through these 5 criteria one at a time. I have used the term ‘source’ to refer to the information you want to evaluate.

Remember, this information could be a manufacturer’s claim for a product, a social media post, or any other information that you come across.

Currency

How recently was the source created, published, or last updated? Some information remains correct forever. However, in general, recently published sources are more likely to be valid than older sources. If there is no way of telling when the source was created, this can be a red flag.

Relevance

How relevant is the source to your needs and those of your horse? If you came across the source because you specifically went looking for information on a topic (e.g., you wanted to know the best way to feed hay), then you need to decide whether it is relevant to your question. If it is, apply the rest of the CRAAP test criteria. Then decide whether the information is worthy of your attention. If it isn’t, you obviously need to keep searching.

But, what if you just came across some information or an advert by chance, and you are wondering whether it is relevant for your horse? Maybe it discusses a new product, a new way of doing things, or the results of recent research on a topic you hadn’t thought about before.

In this situation, it’s worth keeping an open mind. Science is always moving forward. Better products and ways of doing things are becoming available all the time. Anything that might help us to manage our horses better is always worth considering.

The trick is to know what to believe. That’s where the rest of the CRAAP test comes in.

The level at which the information is presented will also affect its relevance. If it is too basic for you, look for something that gives you more detailed information. If it is too complex, try to find a source that gives you a simple overview of the topic. Once you understand the basics, you should then be able to understand more in-depth information.

Finally, try not to be swayed by fashion and clever marketing. A useful way to avoid being influenced is to think about everything you do from the point of view of the horse. For example, will your horse feel better in that beautiful new rug, or would a check-up by your saddle fitter or a physiotherapist be better use of your money?

Authority

How likely is it that the person or company that created the information knows what they are talking about? What qualifications do they have and who do they work for? And, do these indicate relevant expertise?

If you can, check out the creator. Conduct an online search for other information about them and/or their organisation. This should give you a feel for how qualified they are to provide information on this topic. It might also give you an idea about whether they have anything to gain from you believing their information (see ‘Purpose’ below). If the source contains no clues as to who created it, you should be particularly careful.

It is also good to be aware of a practice known as ‘astroturfing.’ This is where unethical manufacturers generate multiple fake favourable reviews of their products. These are posted under false and different names to give the impression that their product is endorsed by end users. The extent to which this practice is used to promote equine products is unknown. However, it is fairly widespread in other spheres and has been identified in the online equine marketplace.

Finally, how much credibility should you give to the opinions of friends and colleagues about a product or practice?

The biggest problem with believing sources such as those listed above is that all of us—scientists included—can be taken in by coincidence. We also can be swayed by what we want to believe. The credibility of this kind of information, and how applicable it is to you and your horse, comes down to how objective the friend/colleague is, the extent to which they have considered other explanations for what happened, how many times they have experienced the outcome they talk about, and how similar their situation is to that of you and your horse.

Always consider that if a horse appears to respond to a particular intervention, there is a possibility that the intervention just happened to coincide with an improvement that was going to happen anyway.

It is also worth considering whether there is a plausible explanation for the ‘effect’ that seemed to occur, either in horses or in other species. The absence of a plausible mechanism should make you more sceptical about claims from friends or colleagues that a product is effective.

Accuracy

How accurate is the information in the source? This is a really tricky one, as claims with no basis in truth can sound very credible. ‘Scientific gobbledygook’ that includes lots of complicated-sounding words might be complete rubbish. However, it can be very persuasive to somebody without a scientific background.

The best advice in this situation is to check whether you can find the same information in other sources, especially those that are up-to-date and created by independent experts in the subject.

Increasingly, scientific papers and high-quality information, written by experts in language that is easy to understand, are available online. These sources are invaluable for making sense of claims made by potentially dubious sources. With luck, this will help you to assess whether the product or technique that is being promoted might be beneficial for your horse, neither beneficial nor harmful (in which case the worst that is likely to happen is that you will lose time and/or money), or in the worst case scenario is actively harmful.

If you don’t feel able to evaluate the information yourself, identify somebody whom you can believe. This could be a scientist with relevant qualifications who provides information online, your veterinarian, or a university extension service. (University extension services, which are particularly well developed in North America, are an extremely useful concept. These university departments deliver research and knowledge to individuals, communities, and businesses. Many of them provide excellent information online at a level that is accessible to non-scientists.)

Finally, taking a step back and applying some common sense is always worthwhile. Just as in other spheres, if something in the horse world sounds too good to be true, it probably is.

Purpose

Why did the person or company that created the source generate it? What do you think their aim is? It could be purely to educate or inform you, with no ulterior motive. In this situation, the information may well be worth listening to.

However, the creator’s aim could also be to bring you around to their point of view or to persuade you to buy something.

A quick word about manufacturers’ claims is worth adding here. Many of those who sell products or services provide excellent information, but some do not. This could be because they have questionable morals, or it might be that they themselves can’t tell the difference between good and bad information.

For certain, not all sales material is suspect, and there are some very good companies (many of them—but not all—are science-based) that design and sell products. However, the more somebody has to gain from you believing them, the more closely you should scrutinise the source for its Currency, Authority, and Accuracy.

It is also worth spotting ‘sneaky’ advertising. For example, increasingly, magazines contain advertisements that are disguised as educational material. Try to spot these and recognise them for what they are.

In addition, some online ‘influencers’ post ‘educational’ or ‘inspirational’ content that, at first glance, does not look like an advertisement. However, they may well have been paid to promote a product. In many countries, influencers are legally obliged to tell you in the caption of a post if they have been paid to promote a product. However, many do not.

Be careful of influencers who repeatedly promote the same product or who have a discount code for a certain company. It is likely that they are being paid each time you use that code, and they are therefore advertising to you.

It is also important to distinguish between fact and opinion. Both can be useful, but you should consider the qualifications of anybody who offers an opinion when deciding whether that information is trustworthy.

Finally, look out for bias in the wording or any twisting of facts to support a particular point of view. If you spot this, it is best to disregard all the information provided.

Some Final Advice

Figuring out what you can believe isn’t easy, no matter what background and training you have. The CRAAP test acts as a useful guide for all of us.

When appropriate, asking an independent and well-qualified expert is probably the most efficient and effective way of finding out which information is worth believing.

For example, if you are considering buying a product that claims to improve horses’ health, ask your veterinarian. And for a nutritional product, ask an independent nutritional adviser.

However, for products that are not associated with health or nutrition, and for social media posts, it is typically more difficult to find independent advice. If the information relates to a manufactured product, the only relevant sources may be the company’s website and customer service line. There you can ask for the evidence that supports any claims made.

Here, as in all things, common sense and a healthy dose of scepticism can be very useful.

For example, there is an equine ‘calmer’ on the market with a name that implies it is associated with the veterinary profession. It claims that it has a calming influence. The marketing material makes various claims relating to its effect on horses’ behaviour and discusses advanced technology.

The ingredients are very simple (and should be safe and not contravene any regulations) but scientific evidence to support the technology, the rationale for the product’s mechanism of action, and the claims made by the manufacturer is not easily available. We should of course remember the old saying that ‘absence of evidence is not the same as evidence of absence,’ and that a product or technique may still be beneficial, even if its advantages haven’t been scientifically proven. But in this case, as in many others, the CRAAP test is worth applying. With this in mind, carefully reviewing a company’s response to questions can be illuminating, just as it can be when you analyse the believability of social media posts or anything else you come across online.

Here, as in everything, the take-home message is: Unless you are very sure of the quality and integrity of a source, don’t take anything at face value. Test the CRAAP out of it!