Eric T. Peterson's Web Analytics Blog - Latest Comments

Re: Top 5 Metrics You’re Measuring Incorrectly … or Not

Jon Narong — Wed, 01 Oct 2014 19:17:29 -0000

To be blunt, it's a bit ironic because I have rarely come across anyone who thinks comScore is truly "accurate" in any unique visitor metric. But the unfortunate truth is that it is still the gold standard in certain industries.

Re: Top 5 Metrics You’re Measuring Incorrectly … or Not

Eric T. Peterson — Wed, 24 Sep 2014 00:17:21 -0000

Sergio,

Good to hear from you. Some thoughts:

1) Jodi has a long-standing reputation in this industry and is about as smart as they come. I don't think she needs anyone to "take up the glove" on her behalf. As she chose not to respond I am sure there is a good reason ... perhaps simply that as a representative of comScore she knows that debates like these are a slippery slope indeed.

2) I don't disagree that the metrics you and she mentions are suffering from technology-induced issues and I did not say or imply that; I disagree with the fearmongering given that the issues are well-known in the industry.

3) I don't know of ANYONE who has been fired because of inaccurate data. Do you?

4) As far as entering a new era in which everything is sampled in some type of value exchange ... I guess you could say that Facebook and Amazon already do that, but how do you extend it to the masses of web sites? Value exchanges only work if the company can afford to provide something of value AND use the data they gather as a result.

As an aside, it's kind of sad that Jody didn't respond and defend her position. We used to have nice debates in the Analytics industry ... people would express opinions, others could comment and disagree, and we would flesh out our thinking together. Now everyone seems so concerned with how they will come across or who they might offend ... it kind of sucks. I just see people hiding on social networks that nobody uses whining and pretending some great mass of people agree with them but are "afraid" to say anything because Avinash or me or Adam or Jody will take offense. Grow some balls and take a stand, I say.

Anyway, you asked. ;-)

Re: Top 5 Metrics You’re Measuring Incorrectly … or Not

Sergio Maldonado — Mon, 22 Sep 2014 17:30:10 -0000

Hi there, Eric. Let me take up that glove on Jodi’s behalf.

Having read her post, only discovered through yours:

I believe she is 100% right here: Unique Visitors, Conversion Rates, Engagement metrics and Actual Impressions are becoming weaker by the minute.

We have put religious faith on these metrics (Have you tried asking people in Media to dump Unique Visitors!?) when in fact they represent some of the biggest fictions of our times. Unique Visitors were first based on IP addresses (memories of your WT years?), then IP/User Agent combos, then cookies and DeviceIDs (while those lasted)… then dysfunctional panels where members of the media industry are suspiciously over-represented. I wonder how many people have been fired and hired in the name of Unique Visitors!

Jodi does stop short of saying hybrid panel/census solutions are the answer, which also goes to her credit. Others would have gone far beyond. But she probably knows all too well that not even comScore has a real answer to these limitations.

I would take it even further by bringing other facts into the mix: third party cookie rejection rates (there goes comScore’s), legal compliance tolls on first party cookies (as they also require “prior consent” under many circumstances in Europe - here: http://www.sweetspotintelli... ), “privacy-powered” mobile phones, incognito browsing, perishable content...

I would say we are entering a new era in which everything is sampled, some people will share a “360 view” of themselves when given something in return, and customer journey or conversion rates at campaign or click level will become a thing of the past. SO, Yes, very similar to the panels of old times (i.e. comScore), but in a much more open and transparent way in which panelists select themselves. Other sources of information like VoC, CEM, etc. will complement this and bring us a much more rational, aggregated view of it all.

If I had to amend something in Jodi’s post I would only change the title: Top 4 Metrics (Aren't #4 and #1 the same?) You May Want To Dump (since there is NO way you can ever measure them “correctly”).

There, you have it. Your debate :)

Re: Top 5 Metrics You’re Measuring Incorrectly … or Not

Eric T. Peterson — Fri, 05 Sep 2014 10:04:04 -0000

Well, I guess when the DAA President writes an article in Forbes that says "your metrics are bad, and you should feel bad" my instincts are to call her out on that and see if she wants to discuss or defend that decision.

Apparently she doesn't.

Would you like to defend her argument? I don't know you or where you work, but perhaps more than just a five word response is in order? Up to you.

Re: Top 5 Metrics You’re Measuring Incorrectly … or Not

marktristan — Fri, 05 Sep 2014 05:46:20 -0000

Bit of an overreaction Eric.

Re: Top 5 Metrics You’re Measuring Incorrectly … or Not

Eric T. Peterson — Thu, 04 Sep 2014 11:14:19 -0000

Agree, although I do understand why she likely wrote the piece, having worked for a few vendors myself. Honestly if Jodi wasn't the President of the DAA I would have simply ignored the piece like I do every other "Five Reasons Your Analytics Are Broken" post.

It is also interesting that she has not responded at all so no opportunity for debate. Too bad.

Re: Top 5 Metrics You’re Measuring Incorrectly … or Not

cleveyoung — Thu, 04 Sep 2014 11:07:16 -0000

I was disappointed in her blog, which I read on LinkedIn before I saw your post. Pretty much all competent, experienced digital analysts I work with are at least aware of the issue, even if they aren't sure how to address it. And it is a HARD issue which no vendor has a good answer for yet. And probably won't anytime in the near future. I clicked to her article expecting to get an interesting perspective upon some aspect of our complex industry. Instead all I found was some generic platitude which sounded like it was written by a junior PR person, not the industry professional Jodi is.

Re: The Recent Forrester Wave on Web Analytics … is Wrong

vbhagat — Wed, 03 Sep 2014 15:32:06 -0000

Eric - I agree with your sentiments and many of those voiced on this thread. Today, we published an alternative Digital Analytics Software Rankings based upon 230 authenticated reviews by real end-users. We take a market-segment view - small businesses, mid-sized companies, enterprises, and remove subjectivity by plotting vendors on two simple axis - likelihood to recommend based upon aggregate ratings by customers in that segment vs. estimated adoption (based upon Builtwith.com website usage data) in that segment. For adoption rates, we use data for penetration of top 10k websites for the enterprise, 100k for mid-mkt and 1m for SMB. We also present the pros/cons of each product based upon a distillation of what's said in reviews. We believe that rankings can help narrow your list, but at the end of the day you need to find the right tools or combination of tools for your use case. Our findings contradict Forrester's in several respects.

Here's our blog post on the vendor rankings: http://blog.trustradius.com...

Article published today by CMS Wire - In Web Analytics Ranking World, Another Choice Emerges
http://www.cmswire.com/cms/...

Re: The Recent Forrester Wave on Web Analytics … is Wrong

Eric T. Peterson — Tue, 17 Jun 2014 11:45:09 -0000

Approve—
Eric T. Peterson
(858) 663-6079

Re: The Recent Forrester Wave on Web Analytics … is Wrong

Arpit Srivastava — Tue, 17 Jun 2014 05:20:51 -0000

No doubt both the giants are making a steady progress I remember when we internally decided to go with the Adobe's enterprise web analytics offering 2.5 years back, GA was no where in terms of custom dimensions, visitor segmentation, attribution but as on date they have moved on to pretty robust offering and I think lot of these have been derived from Adobe & the rate at which GA is building is perhaps higher then Adobe. But with feature additions & customization comes the complexity. And the end of day we need good team technically & analytically strong enough to utilize these tools optimally.

Re: The Recent Forrester Wave on Web Analytics … is Wrong

Roman Rock — Mon, 09 Jun 2014 14:57:59 -0000

I am not so much shocked by the fact that GA is a strong performer depending on the context that you are using to grade the relevant platforms. In my mind there is a leader in this market space and that is Adobe there rest trail them by a distance. Trying to review just the product on it is own is not how most of us would select an analytics platform in this day and age. You want tools at easily integrate with other platforms with low effort and cost, this is where platforms such as Webtrends would barely even be a contender for me. They charge for everything, integrating is complex and it offers almost no flexibility with limited data visualisation making it almost impossible to drive a data adoption. Compare this to Adobe where most integration as drag and drop, on the fly correlations and advance segmentation, simple visuals and cloud integration, how are they even close?

Re: The Recent Forrester Wave on Web Analytics … is Wrong

Sergio Maldonado — Tue, 03 Jun 2014 14:48:53 -0000

This is fun :) Well put.

Now, Does it really come as a surprise? How many years have we seen a couple of old timers ranking high on the Wave and not raised a hand? (we all here have deployed and maintained all of these tools extensively… we were all well aware of the differences!)

I guess it gets more surreal over time, as there is no “wave" when you only have 2 players.

In AT’s defense I will say they have owned the French market for many years. But Webtrekk is even stronger in Germany and there is not a trace of them on the report.

Seriously, Forrester should pull the plug on this particular Wave.

Re: The Recent Forrester Wave on Web Analytics … is Wrong

Robert Jackson — Wed, 21 May 2014 09:58:34 -0000

The relationship between vendors, research companies, brands and consultants is an absolute game of thrones. Forrester have for me, seriously damaged their credibility by downgrading Google. I remember a similar situation last year when a leading industry figure promoted a white paper for a Tag Management Solution which lauded features which just didn't exist.

Re: The Recent Forrester Wave on Web Analytics … is Wrong

Eric T. Peterson — Tue, 20 May 2014 10:55:22 -0000

Thanks Stefan.

Re: The Recent Forrester Wave on Web Analytics … is Wrong

Campbell Macdonald — Mon, 19 May 2014 19:17:45 -0000

Fair enough. I think the broader point is: what is the methodology for inclusion/placement in the Wave? They have to look at "objective" measures like revenue, # of customers, etc. And then they need a feature check box. Whereas some software that is exploding in usage and easy to use (e.g. Optimizely) makes me question the approach.

Models are wrong? Or is it just data that's not well behaved?

Re: The Recent Forrester Wave on Web Analytics … is Wrong

Eric T. Peterson — Mon, 19 May 2014 17:58:11 -0000

I stopped short because I don't personally have any evidence that the "relationship building" has any impact on inclusion/placement in the Wave.

I never worked at Forrester, but during my tenure at Jupiter Research it was clear that it was not a "pay to play" model and we treated all vendors equally in the web analytics constellation. I also made sure that I fully understood the market dynamics and spent a ton of time "ground truthing" the results the model gave back so that we could be confident that we had painted an accurate picture of the landscape at the time.

Because, you know, sometimes the models ... are wrong.

Re: The Recent Forrester Wave on Web Analytics … is Wrong

Campbell Macdonald — Mon, 19 May 2014 15:37:00 -0000

Eric:

Great article calling out what is clear to everyone else. You stopped short of asking "How does Forrester get results like these?" Look at the "relationship building" that each of these companies do with Forrester (attending their conferences, sponsoring conferences, buying speaking spots at conferences, buying reports like these to share with prospects, etc) all in the name of providing objective assessments of the industry.

Re: The Myth of the “Data-Driven” Business

Eric Winter — Mon, 19 May 2014 11:46:44 -0000

While I don't think the word "driven" within the word "data-driven" is bad compared to "opinion-driven" or "ego-driven", I do see marketing causing an overreaching misapplication of the word. Thank you for the push back on that.

So what does data-driven look like?

As I see it, effective data-driven teams have three capabilities:
1) They can unearth and properly frame questions whose answers impact organizations' abilities to achieve their missions.
2) They know how to bring data to the service of answering those questions.
3) They can effectively prioritize data efforts to get good "bang for the buck".

Just for fun, what DID you end up coming with for CBS News?

Re: The Recent Forrester Wave on Web Analytics … is Wrong

Eric T. Peterson — Mon, 19 May 2014 11:18:20 -0000

Ole, thanks, but then we either have A) Forrester up-weighting the least scalable aspect of vendor offerings (people and consulting) or B) giving credit for new products that have not yet been released into customer's hands.

Either way I still do not believe that AT Internet is a "market leader" in web and digital analytics ...

Re: The Recent Forrester Wave on Web Analytics … is Wrong

Ole Bahlmann — Mon, 19 May 2014 03:37:22 -0000

I can't speak for SAS but I have worked with AT Internet for the last 3 years and was a happy customer (I recently changed jobs so I'm not personally using it anymore). I would agree that their strength is more on people and consulting than on the current tool. But having seen and used their upcoming new products I'd guess those are already positively impacting this study.

Re: The Recent Forrester Wave on Web Analytics … is Wrong

Eric T. Peterson — Fri, 16 May 2014 16:44:38 -0000

I'd like to see a show of hands ... who does have tremendous experience with AT Internet (dubbed a Market Leader by Forrester) ...

...

I'll just be here waiting for the legions to come to AT and SAS's defense. Can someone bring me some water after four or five days? Thanks. ;-)

Re: The Recent Forrester Wave on Web Analytics … is Wrong

Andrew Garibay — Fri, 16 May 2014 16:24:37 -0000

I agree with your sentiments regarding Leaders and Strong Performers. I believe that in the right hands, either of the two are capable of pulling tremendous insights, especially with a bit of programming experience. I don't have a tremendous amount of experience with AT Internet or SAS Institute's offerings, but I would argue that the tool is less important than the analyst having a sound understanding of terms/definitions, setup (execution), the questions they are asking, and whether or not their analysis accurately addresses those questions.

Re: Average Page Views per Visit

Martin W — Wed, 20 Nov 2013 05:01:36 -0000

I'll disagree with @Aaron here. Pageviews per visit is more indicative of the visitors' engagement than, for example, bounce rate or time on site. For businesses I know, larger organic pv corelate rather directly to better conversions/income over time, and that doesn't only go for advertising-driven sites. Larger pv also translates to returning visitors.

Surely, it could be argued that by introducing counter-productive interactions you could boost your pv at the expense of conversions, but this should be easily avoided by common sense and some A/B testing.

Re: The problem with “Big Data” …

randschulman — Wed, 23 Oct 2013 01:36:37 -0000

I like that Eric, "Little big data"! Yep, start small. Unfortunately, I haven't seen too many successes yet with either "real-time" analytics (an oxymoron) or making sense of multi-channel data in the customer decision journey (with smaller systems). I like what Tealium is doing as it's action oriented, not "analytics" intensive.

Re: Social media is like coffee …

Tristan Jutras — Sat, 09 Feb 2013 20:47:23 -0000

Hi Eric. Thought-provoking article. You actually inspired me to get off my butt and write something in response:
http://tristanjutras.com/po...
Thanks for the nudge!