ePortfolio Analytics: what to do with it all that data?
Recently I was at a conference (AMEE 2012) in Lyon on the subject of education in medicine and the healthcare professions where I presented an analysis of web usage data derived from Google Analytics and from a snap-shot (96 hours) of internal tracking data (i.e. data collected within our system).
Google data is vast but difficult to unpick the differing activities and page flow of the various different types of users that use ePortfolio. Internal tracking data is also big (which was why it was necessary/practical to use a snap shot of data) but allows for more granular analysis of the type of users and be able to map what they do during the first few steps into their ePortfolio account. The presentation can be found at this link or at the end of this piece:
As a web designer my first instinct was to try to understand what users most want to do, or rather, what they actually do irrespective of how the site is laid out and what their training programme might want them to do.
The conclusion drawn from the data was perhaps predictably that trainees mainly log in to check up assessment form completions and request new assessment forms (56% of the trainee log ins went to directly to this activity). So to me as a designer it would seem wise to improve access to these areas for users so making their life a little easier. A simple starter for ten there. A broader interpretation might be that e-portfolios (especially professional e-portfolios) are inclined to be box ticking facilities: important to a professional body such as a Royal College who need to know that their trainees are safe to practice medicine, but less satisfactory to trainees who want something less prescriptive [no pun intended]. Exactly what trainees want is something that we are starting to get a better idea of more recently thanks to the wonders of social media. There is a balance between college needs and trainee needs in the design of an e-portfolio and possibly this type of data can help inform this debate.
There were of course deeper issues relating to the underlying process of learning and development as doctors progress through training. Differences were seen between trainee activity as they become more senior (from graduate Foundation years through to specialty training) that related engagement with curricula and log forms that were not necessarily reflected in their respective training programmes.
So to summarise, while I’m happy to ask simple questions about user flow, user experience and so on, questions about the educational implications of this data have not been well addressed as it’s not really in my sphere of knowledge (or in my job description). I would see the research I presented at this conference as a “this is the kind of thing we can do” exercise that should lead on to better designed questions that will allow us to understand how best to develop an e-portfolio that supports effective learning and development through the effective delivery of a training programme.