Paul Penrose - Finextra - London 18 December, 2007, 13:26 0 likes

The Finextra news desk tends to take a lot of these survey-style reports with a pinch of salt. Those that ask too many leading questions, or feature small or self-selecting samples tend to get passed over - and believe me, there's a lot of this junk research about.

While we're not too keen on Gartner's penchant for extrapolation  - which produces the headline-pleasing multi-million dollar loss figures - the sample size is bigger than average, and the questions leave little to interpretation.

All the same, I've got agree with your general drift. Too many research houses seem to come up with a crowd-pleasing headline and then frame the research to deliver the requisite results.

Gartner has been had up by this Community in the past for some of its general purpose research notes on MiFID and M&A activity in the sector, among other things.

Glad to see the Community being used as a forum to question and challenge assumptions. Keep up the good work.  

Beth Robertson - First Annapolis Consulting - Linthicum 19 December, 2007, 15:33 0 likes

The fact is that there is no single source of hard data for an analyst firm to use to develop estimates such as those Gartner recently developed for phishing losses. As an analyst, one has to make assumptions based on the best available market data and what seems reasonable. The problem is that what is reasonable to one analyst is not to another, and some firms do go for the headline grabbing numbers. Numbers can then become "institutionalized" to a certain extent as they are cited again and again in media reports. At some point you may see that the WSJ or some other credible news source cites phishing losses at $3.2B - the Gartner number. At that point, no one is really questioning who originally developed the number or the associated methodology for doing so - it is all viewed as fact - whether the numbers are realistic or not.

Gary's point and Paul's are important for users of analyst firms and their market research products to understand. Numbers such as phishing loss estimates are not hard facts; the methodology varies and should be questioned. Sound methodology makes for well-supported estimates that can lead to better understanding of and planning to address business needs.