IT Analyst Ethics: Advocacy vs. Objectivity
Introduction
There seems to be some confusion about the role of information technology (IT) analysts amongst the technology press and amongst some IT buyers when it comes to IT analyst objectivity. And that confusion stems from the failure of each group to understand the difference between "advocacy" and "objectivity". In this article, Clabby Analytics (that’s me) explores these differences — and how they relate to IT analyst ethics.
The Role of the Press The Role of the IT Analyst Community The Rub: When Objectivity, Advocacy and Ethics Clash In many cases reporters first ask IT analysts to help build an objective baseline (reporters ask, for instance, for a research analyst to describe overall trends in a given industry). Later in the interview, reporters then ask IT analysts to advocate (or, to put it differently, to demonstrate a clear bias) a pro or con position on a given topic. It is at this point that an IT analysts become particularly vulnerable to criticism about his or her motivation for making statements that show a clear bias — as well as criticism about his or her integrity, objectivity, and ethics. I call this the "advocacy/objectivity dilemma". The Advocacy/Objectivity Dilemma In each case, questions about a given research analyst’s ethics can be raised — especially if that research analyst has had any kind of financially beneficial relationship with the vendor in question (consulting services, subscriptions, reprints, etc.). It is valid to question the ethics of a given IT analyst — especially if he or she has had a financially beneficial relationship with a given vendor. But questioning ethics on this basis alone can get very confusing (for instance, I’ve performed consulting and/or have sold report reprints to almost every major vendor in the information systems marketplace. Hence, almost everything I say could be challenged from an objectivity or ethics perspective). Yet, I should still be allowed to express my opinions based-upon solid research… Questioning a research analyst’s objectivity and ethics only on the basis of whether that research analyst has ever done any work with a given vendor is short-sighted. That analyst’s objectivity and ethics should be adjudged by the methodology used to produce reports and opinions. Escaping the Advocacy/Objectivity Dilemma 1. Research methodology; 2. Business model; and, 3. Relationship to the user and vendor communities. When it comes to evaluating an analyst’s research methodology, I look for two things: 1. In qualitative analysis I want to know:
a. Is that analyst conducting primary research with end users, IT executives, business leaders, college professors and other germane b. Does that analyst have any hands-on or real-world experience with the product or service in question; and, 1. In quantitative analysis:
a. What were the sample questions; and, b. What was the sample size. When it comes to IT analyst business models, I want to know where the IT analyst’s revenue is coming from (because the source of revenue may or may not influence that analyst’s research conclusions). And if the IT analyst is conducting research on behalf of vendors, I just plain want to know that what I’m reading may or may not have been influenced by the vendor in question. (In other words, I want the IT analyst to identify up-front that the research paper was sponsored research). Incidentally, I have no problem with sponsored research — I’ve learned an awful lot about the IT industry by reading sponsored research papers. And as for IT analyst ethics with regard to this practice — read the next section on advocacy to understand my position on this type of research.
Real-world Objective Advocacy Positions A few years back the Itanium Solutions Alliance posted two articles by IDC comparing Intel’s Itanium architecture and market position to Sun and IBM. I took issue with the IDC whitepaper on Itanium versus IBM — and ultimately produced a presentation that explored ten reasons not to Itanium. IBM bought the rights to distribute that white paper. (Note that readers now have an IDC piece that advocates the use of Itanium — and a counter piece by me that describes why not. This kind of balanced give and take makes me ethically comfortable when advocating one vendor’s products over another).
Last year Gartner Group advised mainframe owners to consider moving off of mainframes due to an impending (alleged) skills shortage. I talked with mainframe owners, college professors, and employment agencies and concluded that Gartner’s advice was unfounded. Again, you, the reader, gets two points of view (a good thing); and again, I am ethically comfortable with this type of advocacy. Last year I produced the same kind of counter opinion work in the blade server space. IDC produced a whitepaper extolling the virtues of Hewlett-Packard’s Virtual Connect architecture; I wrote a strong argumentative report that counters IDC’s claims. And again, I am ethically comfortable having done so. In each case there were already whitepapers available on the Internet that, in my opinion, favored one vendor’s products or services. My reports countered the opinions of other analysts reflected in those reports. What is transpiring here is a kind of dialog in which two sides of an argument are represented. By contrasting these arguments, reporters and readers can then arrive at some version of the truth. This is why Clabby Analytics (my firm) loves to write counter-opinion advocacy reports. Advocating for Advocacy In my own IT analyst practice, I teach; I write product overviews (informational reports); and I write reports that advocate certain actions (I call these "Advisories") and/or that advocate the position of a particular vendor. Is it possible to advocate a position about a particular vendor’s product, service, or strategy — yet remain objective? As long as your reports are based on a solid research methodology — you can definitely produce objective reports even when advocating for a particular cause. But, be aware that when an analyst writes these kinds of reports, that analysts sets himself or herself up for criticism for being biased — and scruples, ethics, and integrity are all called into question.
When reading articles in the technology press, what readers (including myself) expect is objectivity. The primary role of the press is to present readers with objective new stories. To build an objective story, members of the press need to understand the pros and cons of a given product, service, or event. Above all, reporters are not to take sides (which makes me wonder why some have taken to blogging…). The role of reporters is to assemble opinion in order to help readers reach an objective opinion.
To build a really good story, reporters need to find expert sources who are willing to express positive opinions, and expert sources who are willing to express negative opinions on the topic at hand. By comparing and contrasting these opinions, reporters are able to present both sides of an argument. And by comparing/contrasting source opinions, reporters are better able to deliver some degree of objectivity to his or her readers. One of the main roles of the IT research analyst community is to present objective expert opinion — based-upon primary research and/or experience. But, unlike the press community (which should remain impartial and objective at all costs), research analysts are also encouraged to express opinion (in other words, IT analysts should be able to advocate one position over another — and justify their opinions based-upon a solid research methodology or experience). A research analyst can express an objective opinion based-upon research — but research analysts are also given the luxury of advocating one position over another based-upon research. This is the primary difference between the press and analyst community.
More often than not, when a reporter asks a research analyst (expert source) to advocate one position over another (particularly when the reporter is asking the expert to take a pro or con position on a vendor’s product or service), the reporter secretly wonders "why is this person saying this?" Or, stated another way: "is this research analyst (my source) expressing an objective opinion — or has this research analyst’s opinion been influenced by some form of dealing with the vendor in question?" And this is where objectivity, advocacy, and ethics clash.
This advocacy/objectivity dilemma is a "no-win" for IT research analysts — a true "damned-if-you-do, damned-if-you-don’t" scenario. Here’s why:
In a perfect world, all IT analysts would conduct research, concur on all of the same findings, and thus be able to provide reporters and readers with 100% objective assessments on any and all topics in question. But a strange thing happens in the area of research: researchers can collect the same data from the same people or systems, at the same time — yet arrive at two different conclusions. And, interestingly, the analysts who arrived at these diametrically opposed conclusions may each believe that his or her opinion was arrived at objectively.
Assessing Objectivity
I personally adjudge the objectivity of a research analyst by looking at three key factors. The analyst’s:
individuals in order to form his or her opinions — or is that analyst speaking primarily from his or her past experiences; and/or
To avoid this dilemma, many of my reports advocate a perspective that is different from that of a competing analyst. For example:
All IT analysts have opinions. Hopefully those opinions are based on solid and thorough research. IT analysts frequently express their opinions in the form of research reports. Some of these reports are educational; some are informational; and some are instructional. And some of these reports are written in a manner that advocates a certain action and/or advocates a position that favors a particular vendor.
As for Clabby Analytics — I’m proud to advocate the positions I take. I can back them up with solid research, as well as with 30 years of industry opinion. I’m no apologist for advocacy — I’m proud of the advocacy positions that I take.
Summary Observations
A funny thing happens in the reporting process when it comes to establishing the credibility of a news source. Real-world customer sources are automatically considered by readers to be valid and objective. After all, who can argue with the direct results of a given customer’s experience with a product or service? On the other hand, vendor quotes are almost always discounted as biased. After all, vendors have a clear self-interest when supplying source information (we all know they’re trying to sell us something) — right? But here’s where it gets really tricky: how should readers react to IT analyst quotes — especially if they don’t know the analyst, the analyst’s firm, or the analyst’s "agenda"? My advice for reporters and for IT buyers is to not focus on IT analyst ethics — but rather on that analyst’s research methodology. Are the research methods of that particular analyst based-on real world interviews with IT buyers and/or hands-on experience with specific products or services? If so, I’m going to trust that analyst’s opinion more than the opinion of research analysts who are simply espousing opinion based upon "industry experience". Also consider this: there is nothing wrong with an IT analyst taking an advocacy approach in reports that he or she publishes. Advocacy reports create an opportunity for dialog — and through this dialog reporters and readers can arrive at some version of the truth. Clear value can be derived when subject matter experts present competing perspectives.
