You will be redirected back to your article in seconds
Skip to main content

Institute Issues Contradictory Sourcing Reports; Is the Research You’re Buying Worth the Price?

In a hyper-competitive business environment a hefty premium is put on timely information immediately translatable into practical action. But how do top executives appraise the reliableness of the data they pay dearly for? And are they adequately aware of the methodologies and statistical assumptions built into the models used to gather and parse the data dumps they purchase?

Consider a recent study issued by Consero, an organization that describes itself as “an international leader in creating high-level, invitation only events for senior executives.” The study is part of their Fall 2013 Procurement and Strategic Sourcing Data Survey. The crux of the report’s findings is that 67 percent of “surveyed Chief Procurement Officers say their departments are more focused on building collaborative relationships with suppliers than on obtaining the lowest costs.” If true, these are fascinating, and for anyone who covers the sourcing beat, wildly counterintuitive results.

According to Consero, this analysis is based on a survey with eleven questions tabulated from the responses provided by forty-two Chief Procurement Officers (CPO). The description of the study on the company’s website does not provide those questions or what kinds of responses were permitted, information essential to determining the ultimate value of the responses.

The survey is also based on a very small statistical sample. While forty-two might seem like a healthy grouping, a quick google search turned up several studies that surveyed anywhere from 250 to 500 CPO’s. One research report composed by the IBM Institute for Business Value interviewed 1,128 CPO’s.

Related Stories

And who exactly are these CPO’s? A study published in March 2007 by CAPS Research discovered that 21% to 30% of CPO’s had limited experience in supply management and procurement work. It also found that these executives tends to be incredibly mobile, rapidly jumping from one company to the next in search of a more profitable employment. Does this information mitigate the value of the results delivered by Consero in any way? It’s not clear since these issues are never addressed.

Now consider another analytical study produced in April 2013, also by Consero. The white paper found that “over 50 percent of Chief Procurement Officers said their company pursues short-term savings from suppliers that undermine long-term value.” In this report, only 35 percent (versus the September report’s 67 percent) stated that their “procurement strategies” emphasized long-term collaboration over short-term savings. Coincidentally, this data was also culled from forty-two respondents. Were they the same ones interviewed this month? If so, what would account for such a swift and comprehensive revision of their procurement policies? If not, what actionable conclusions could any top level sourcing executive infer from these two contradictory studies, understood in tandem?

The confounding nature of these anecdotal reports is paradoxical since polling was invented to render common opinion transparent. In 1940, George Gallup raised the question: “ What is the common man thinking?” It occurred to him that this wasn’t always clear but was also information essential to the proper functioning of representative democracy. This inspired him to devise a polling method that would “provide a continuous chart of the opinions of the man on the street.” If meaningful consent in a modern democracy demands that elected officials know what their constituents desire, then polling seems like a necessary instrument of political communication in between elections. America apparently owes a great debt to Mr. Gallup.

But any anecdotal report, however translated into the language of mathematical precision, is rife with problems. Opinion polling depends on the human formulation of questions to be interpreted and then answered by other human beings. Just as the questions posed can be loaded with hidden premises and implicit assumptions, the responses solicited can be more or less honest, clear, and well-considered. It seems methodologically cheap to proudly claim scientific exactitude after counting the yeas and nays generated by the hidden complexity of these exchanges. Measuring what are basically anecdotal reports with numbers doesn’t magically transform a species of hearsay into irrefutable evidence any more than it would my mother’s homespun grapevine of gossip. The ambiguous contours of human language resist the charms of arithmetic.

Of course, this doesn’t mean that corporations shouldn’t pay for research conducted by reputable think tanks. Sometimes, even often, the information generated by these institutes is scientifically rigorous and strategically invaluable. However, the buyer should also always beware, since not all reports are created equal, and September’s cutting-edge findings might defeat the prevailing wisdom in April. Maybe even from the mouths of the same forty-two executives.