Is All Research Good Research?

I recently came across a published report from an advertising & communications agency that conducted research in regard to mobile technology usage in a specific target audience category.  The findings were impressive – reflecting that mobile communications is being utilized by a majority of the audience through smart phones and other devices.

I lost interest in the study when I finally realized how the research was conducted – agency staff interviewed 25 participants at a trade show.

Is this valid research?  The answer is both yes and no.  From a qualitative standpoint, the research can validate potential trends in the category as most qualitative projects can do.  However, from a quantitative perspective, the research has no statistical validity since the total population of the target audience is just under 1 million.  Can a mere 25 research participants represent the entire category? 

The research was obviously conducted by the agency with the specific purpose of promoting its services while positioning the agency as a thought leader in the category.  But unfortunately, the report, published as a white paper, loses credibility when the research methodology is in question.

This is a good example of misuse of market research.  No true trend lines can be established based on qualitative studies.  Qualitative research, including in-depth interviews and focus groups can uncover certain behaviors and perceptions.  However, those market trends can only be confirmed through an adequate representation of the target audience in quantitative studies.

Ironically, the above agency is promoting social media and personal communication technologies to reach this specific target audience.  It would have been credible research if they used that same technology to survey a large number of participants to provide statistical representation of the target audience.

The important point is to know the limitations of the different research methodologies and don’t embellish findings to attempt to prove your point.

There is no replacement for objective, factual and thorough research.  It is critical to understanding the dynamics of your market.