Content Fairy – Guy Redwood's Blog

The two day Tobii User Meeting in Frankfurt was fantastic. We were hoping it would be special – and can now confirm it was. 60 pioneers of eye tracking for usability studies, sharing stories and ideas over 2 days and much beer.

I’ve got many pages of notes, which I plan to place on here over the next few weeks.

The first thing I’d like to mention in this quick review is that Tobii is a great company with a world class team. It was great to place faces next to email adresses and confirm that they only seem to employ great people. Anne and the team did a first class job with the event and the hospitality was second to none.

The audience had a good blend of academia and commercial usability people. The latter was very strong in their pressence and passion. Our main competitors in the UK had all fielded a person to attend – so we enjoyed putting faces next to the names and learning a little more about the personalities behind the brands.

A few highlights, without too many spoilers:

1. There was much debate about the correct use of ‘Current Think Aloud’ and ‘Retrospective Think Aloud’ protocols in illiciting feedback from users in testing with eye tracking equipment.
2. The validity of a study showing ‘F’ pattern findings reported by a famous usability group was questioned by the majority of the attendees. Throughout the 2 days, there were many heckles of ‘I can see an F’ when various heatmaps were presented.
3. The infamous ‘golden triangle’ found by a different company, also received similar attention.
4. There was no agreement on a minimum sample size when conducting usability tests with eye tracking.
5. Testing methodologies need be fluid to match project and customer needs.
6. The software used for usability testing has a long way to go.
7. The new Tobii Studio Software – looks very good and very useful.
8. There is no alternative to a skilled usability practioner when testing.

One of the more memorable moments was watching a video of a Lemur with a head mounted eye tracking unit. Whilst Tobii don’t build head mounted units, it was a fun thing to watch. Study details here and some video here – we had the pleasure of watching live tracking footage. Lemurs like to watch tails and heads… aparently!

A few months ago I was looking for web session playback solutions for a particular project that may have benefited from it and put together the list of technologies at the end of this page. I haven’t played with many of them but have spoken to some vendors and got inside info on the tech. Recording people use your website sounds very cool – but once you ask a few questions about the reality, the gloss begins to dull rather quickly. One of the ideas is that when somebody calls the customer service team and says ‘I tried to place an order on the site but it didn’t work’ you could go back and possibly replay that user trying to place an order and learn how to make the website more useable.

Unfortunately,  there are huge flaws in these technologies and you need to understand that before deciding if they are suitable. In the right situation they are awesome – but unless the vendors find new ways to record – they’re usefulness is increasingly limited.

These technologies work in two ways:

1. Method 1 – javascript tags placed on a page pass back mouse coordinates and other session info allowing the session to be replayed and analysed. Just like Google Analytics.
2. Method 2 – a proxy server/packet sniffer that sits on the webserver network storing the packets of data that each user downloads in their session.

It’s interesting to note that many banks use method two to record every interaction with their websites, just as they record your telephone call when you talk to the call centre.

The idea is that you can watch a recording of a person using your website, from a few days ago, and see the mouse move on screen and watch the user scroll etc. Which is great – you get a playback of the flow of a user through your website and that’s it. Some packages will produce heat maps of where the mouse has been on a screen and also where people are clicking.

At this point I feel we need to start educating people in what they are actually getting. There is no beneficial relationship between the position of a mouse and what a user is looking at. In a nice review of TapeFaliure on the Big Green Blog, (go watch the video) Marshall writes:

kinda like a cheap man’s replacement for eye-tracking, if you ask me

This is wrong and unfortunately, many people are starting to believe this. We conduct eye tracking studies all the time and we see no relationship between what a user is looking at, reading, scanning etc, and where the mouse pointer is on the screen. The only time the mouse and eye meet are when they user goes to click on a button/link or scroll without using a wheel mouse.

The second problem with this technology is that Method 1 plays back sessions against the current website – they basically overlay your mouse and screen coordinates over your current website. Most websites change regularly, so playback is possibly only realistic for pages where there is no change, maybe your checkout pages?

The third problem is that these methods struggle to cope with dynamic page scripting which is commonly referred to as AJAX. As more sites move away from ‘pages’ and start to utilise dynamic content, these playbacks also become void – as do the popular approaches to traffic analysis.

There are some privacy concerns with this technology, but your average user is not bothered whilst they remain unaware. We track lots of data through analytics already, but when you mention to people that we could track everything, even your mouse movements, typing etc… they show a new increased level of concern.

Would I use this tech myself in projects?

Yes, indeed. I can see huge benefit in showing heatmaps of mouse clicks on a website. I can also see huge benefit in playing back sessions for customers that have good old fashioned non-ajax reliant websites. But as I mentioned at the start of this post, look beyond the gloss and understand the limitations.

But non of this comes close to the value you get from observing a few ‘real’ users in an eye tracking studio.

List of vendors/products:

1. Crazy Egg – http://crazyegg.com
2. TapeFailure – http://www.tapefailure.com
3. ClickTale – http://www.clicktale.com
4. ClickDensity – http://www.clickdensity.com/
5. Tealeaf – http://www.tealeaf.com
6. Foglight – http://www.quest.com/foglight
7. Metronome (formerly BeatBox) – http://www.metronomelabs.com

The last 3 are the big tools for those big jobs.

To sum up, this reminds me of the days when I met Interwoven in their early years, pre 2000 I think, when they were getting a lot of interest in their content management system because the sales team had a very sexy demo. They could show you content editing in context – which was amazing. You could show a site owner how they could browse through their website with their browser and at any point click on the edit button and then just change some words. CMS at that point, typically involved geeky administration screens, but the Interwoven Teamsite demo made the system look so easy to use for real users. However, there was a flaw in this approach as this did not work, with data driven sites, where content was pulled from other sources, like databases. Projects found this out when they actually started to implement their CMS – but the demo was so good, people just wanted it.

You need to look beyond the gloss and understand how this technology can help.