E-assessment harder to deliver than e-learning?

I have another reflection on the learning technologies conference which is about the creation of e-learning in comparison to developing e-assessment. In this context the e-assessment I am referring to is not simple testing but rather the more complex process, of capturing evidence of competence and then having multiple people interacting with that evidence to confirm its validity.

On the eve of the conference City & Guilds/Kineo announced the launch of the Adapt Learning Open Source Framework. At face value it appears to be another ‘open source kit’ on the market and a generous offer from Kineo because in effect it empowers people to build their own e-learning.

At the conference itself I watched a presentation from Epic, like Kineo another successful e-learning company which showed how they use Moodle; Sharepoint and Drupal to create e-learning content for a range of customers. All this software is open source and therefore Epic’s skill is in adapting the software and in creating plug-ins to create solutions that meet the specific requirements of their customers. Their income comes from maintenance contracts and the adaptations of the software and whilst their customers are not tied into licensing specific software they are ‘tied’ to those capable of adapting the open source software.

In both cases I doubt that any of the e-learning is underpinned by the sort of comprehensive and complex database that is required to deliver e-assessment particularly where this a highly complex rule base for the assessment.

To illustrate the difference there are two ways to ensure compliance using electronic methods. First take one of the examples of e-learning used by Mark the presenter from Epic. If for example you are a member of the customer services team working on Jet 2, one of the ways your understanding of the duties you are required to perform can be tested, is by asking questions about a number of scenarios and then tracking your responses. Another way is for someone to actually record how they saw you perform, to cross reference this against specific standards and then to have their judgements checked twice to make sure it was fair and equitable.

I am not implying that either of these two processes are better or worse. What however should be self-evident is that one of the processes is more complex.

The failure to understand the complexities involved in assessing competence electronically, is in part down to the industry itself which still sometimes places the ‘recommendations’ provided by Linked in, on the same level as thorough assessment that required rigorous evaluation and cross checking.

Nevertheless those who have tried to build an e-portfolio for genuine and rigorous assessment know how complex the process is and the type of sophisticated database that is required. It is not something that is easy to build from a ‘do-it-yourself’ kit.

E-assessment; a work in progress?

Just been again to the Learning Technologies conference at Olympia. I wrote a blog about the event last year. This year’s reflection on the event is different from last year’s, where the overarching impression was of every supplier offering everything and as a result it was really hard to distinguish, what were the particularly special offers being made by each provider.

At this year’s conference there was certainly more activity both in terms of participants and exhibitors, however it would again be hard to identify any supplier who was offering anything that was outstanding.

Indeed in terms of two of the presentations I attended by suppliers we know well, namely City & Guilds/Kineo and Atticmedia they both promised something radical, but with respect to both organisations they were launching products that were to come.

City & Guilds/Kineo new product was interesting in that it combines e-learning materials alongside an e-portfolio with a comprehensive reporting tool. A concept we have consistently argued has the potential to transform the market. Indeed it was a point I personally made to the executive team at City & Guilds about five years ago.

We were informed that this new product was being used from this Monday, in which case it was regrettable that the only screen shot provided was of the opening page. It was perhaps because they have not yet quite finished the e-portfolio element, which we know from experience is often far more difficult to develop than any e-learning materials.

It was clearly also strange that City & Guilds/Kineo had opted not to use their own e-portfolio company, Learning Assistant to deliver this solution but rather relied on the Totara LMS system to provide this functionality. To my knowledge the Totara e-portfolio is still a ‘work in progress’ and it is not something that can be created in months!

Atticmedia another company we have also worked with, were announcing a new project called ‘Mojo Central’. It is a new mobile and social learning platform. Again it was not really a launch not least because this development was announced back on October 31st. What they were actually appealing was for volunteers to try out their Beta version in the summer. Um? I really like Attic but they have a bit of a track record of talking about systems that never quite get finished.

Maybe this is a trick I have missed, which is that you no longer have to wait until you have a real site but rather it is sufficient to tell people about some great ideas you have….so here goes. We are working on a tool that enables staff development to be planned and its impact tracked and one that will enable work experience placements to be created and closely matched against the needs of individual students. We are also continuing to work on our upgraded e-portfolio for vocational qualifications and exploring whether we can generate the reports necessary to claim funding.

All due to be delivered in the Spring. The difference from some of the launches above is that we will deliver.

Online CPD continuing to make progress in Ireland

It is a year almost to the day since we first launched our first CPD site in Ireland with the Irish Institute of Radiographers and Radiation Therapists. I was back last Saturday in the Gibson Hotel where the launch took place at their annual conference.

The Gibson Hotel is itself an interesting place. It is located at the Point which is at the end of the Red Luas line next to the O2 arena. The location of both buildings is a little incongruous. They face onto the docks on one side and on the other the development that was meant to link these buildings with the main financial centre in Dublin has stopped, hopefully temporarily.

This structural picture in some senses reflects the challenges that we and the professional bodies we work with in Ireland, have faced over the past few years.

Health professional bodies in Ireland have been raising their game over the past few years. They want to be at the forefront of best practice. However the economic conditions in Ireland which have led to reductions in public spending, have made this harder to achieve.

However there have been factors that have mitigated the impact of the economy on the development of effective health professional bodies. The HSE, the Irish equivalent to the NHS clearly views them as partners in driving up professional standards. As a result they have encouraged and supported professional bodies to play a leading role in the development and delivery of CPD by directly funding CPD officers and programmes.

Their regulator CORU has proven to be equally helpful. Initially I sensed that professional bodies were nervous of CORU. They thought that they would take over responsibility for CPD. There were also concerned about other matters. Those professional bodies who have the title ‘Registered Member’ for members who have earned a higher status within their profession, recognise that this might need to change when in effect all members become registered with CORU.

However although these concerns remain the overall impression I have is that CORU is keen to forge a strong relationship with all professional bodies. Indeed arguably the professional bodies offer them the means to drive up engagement with CPD, which is so key to the effective operation of the Registers and improving patient safety.

At this year’s IIRRT conference I spoke to Margaret Murphy who is a Patient Advocate for the World Health Organisation. She received a standing ovation for her presentation. We both agreed that because Ireland is a small place, surrounded by water, it is a great place to drive change. It. There are relatively few universities that need to be influenced and relatively few ‘top people’ who need to be persuaded to bring forward change. Her primary concern is patient safety which ultimately is the objective of CPD.

At the conference it was heartening to hear practitioners sharing their best practice; it was great to watch the ‘coalface’ work of practitioners being shared and celebrated and it was encouraging to hear a profession welcoming the challenges of regulation. Over the last year it has been great to have been a part in this albeit even if it is a small way.

Completing the Circle

The NET2013 conference for heath professionals from across the world was a fitting event at which I was able to talk about for the first time, the recently announced acquisition of a majority share holding in this company by the VIA partnership.

At the conference it was clear that there was substantial interest in the presentation of the e-AoPP e-portfolio we have developed with Southampton University, for the assessment of nurses, midwives and other health care professionals when they are on placement. This interest was generated by the early results from the roll out of the system which indicates that users are finding it easy to use and difficulties accessing the system have been overstated not least because of the range of devices being used to access it. The quality of assessments are being enhanced because practice educators have more time to reflect on their judgements and tutors are alerted when target dates in the placement are not being met.

The e-AoPP also has an innovative way of capturing feedback from other professionals and patients. They are invited by student nurse/midwife to make a comment by e-mail and this is then seamlessly added to the correct section in the portfolio.

The delivery of the project comes at a time when there is a national interest in creating records of competence and linking these to the introduction of national standards. There has been an announcement from the Nursing and Midwifery Council today, confirming that nurses and midwives will be required to gather and submit their evidence of their professional development online.

As a result we find ourselves back at the place where the company first started, which is to be engaged with a project that has the potential to make a significant impact on the care delivered by nurses and midwives.

Over 10 years ago we were contracted to deliver what became recognised then, as the largest e-portfolio project in the world, which was the Learning Zone created for the Royal College of Nursing. It was a visionary project designed to enable nurses to meet their regulatory requirement. For various external and internal reasons its impact was not as great as first hoped. Being involved in the work with Southampton and seeing its potential has brought us back to what has often felt like unfinished business.

At the time we created the Learning Zone we had access to the resources of our then parent company Axia Netmedia Corporation. It is therefore fitting, as we stand on the brink of developing similar work, we have again formed a partnership with a larger company and we now finally have the opportunity to close the circle.

Why in e-learning/e-assessment does it matter so much what it is, rather than what it does?

I am subscriber to many e-learning forums. In my experience too often the discussion is about systems rather than learning. The discussion that is most likely to generate responses is one that starts ‘I about to purchase a Learning Management System, Any recommendations’. What that follows is a mixture of name checking of various LMS systems interspersed with the suppliers recommending their wares. It is a rare for someone to ask the killer question which is what do you want it for?


Today we had an e-learning guru declare how impressed he was that someone had found a way of using Word Press as an LMS. There was little discussion about what transformation in learning had happened as a result of this discovery. The point he was trying to make was that wasn’t it good that someone had got hold of some open source software and come up with a ‘free alternative’ LMS. Aside from the making the obvious point that nothing is free i.e. there was a cost for working out how to use word press in this way, what was the actual impact on learning.

I know of one learning institution that is making creative use of share point to create what someone might want to call an LMS. They would actually reject such a suggestion and rather focus on what it has enabled them to do. This is to create a learning environment that places the learner at the centre of the process. It is not about managing the learner but enabling the learner to manage their own learning, the system is built around a learner not the course they are on. Furthermore in creating such a solution they are not wedded to one approach; they are quite happy to use proprietary software if this fits the bill.

I have written in this blog before how I find it difficult to understand this desire in e-learning to create solutions that do everything. It is hard to imagine in what other field of ‘engineering’ is this approach taken. It seems perfectly OK for someone who is manufacturing a car or constructing a building to source the expertise and the materials from different places. So what’s so different about designing learning systems?

Just because you are using sales force to manage your customers/members of your organisation what is the logic to extending sales force to create an e-portfolio? Why is there such a need to have an open source portfolio if your LMS is open source?

Having an integrated offer is a right and proper objective but if it is all sourced from one place are we sure that we are making the best offer to our learners/users? Maybe there is real value to be had from ‘shopping around’ and filling a shopping basket full of items that have been brought not because of the packaging but because they actually deliver what your learner’s need.

We need to stay out of our silos

Every Friday night for the past two decades I have gone out with the same group of mates who have often delighted at pouring scorn on e-learning and e-portfolios. This week was different one of them has just joined Doosan, a South Korean company and he confessed to be enjoying his e-learning course that was inducting him into the history of the company and also showing him how to interact Koreans. Quite important given they are now his boss. He was not just enjoying it he could see its validity.


Early on in the week I was with the Group Director for E- learning for a large college and he was describing how e-learning was having a transformational effect in his organisation. As a result he was being enabled to harness expertise from a number of sources, including the private sector to create a full range of learning opportunities. Incrementally his team have been able to encourage each Department, to see the benefits of e-learning and they are now transforming the way that learning is delivered and tracked.

What had been clearly critical to this success was that e-learning was not seen as a separate activity from the development of the technical capacity and resources of the College, all this activity is being co-ordinated from one place, by someone who had the vision to see the links and with support from the executive to bring all the links together.

In one sense that is an obvious development but if you were to go around any learning institution in this country and track e-learning projects you are likely to find lots of ‘pet projects’ happening; a number of different e-learning products procured to do the same thing and little evidence of a positive benefit on the whole institution. You might also find that these initiatives were taking place with at best minimal support and at worst outright opposition, from those responsible for technology in the institution.

As someone going in from the outside into these institutions, the one person I am least happy to see at any demonstration is the representative from the IT department. As one of ‘their number’ was clearly pleased to remark recently ‘these e-portfolios were meant to be the big new thing 10 years ago and they have not gone anywhere’ It them became clear that any notion of the transformational effect of technology on learning was clearly not on his agenda.

Just over a decade ago I was manager of the East Leeds Family Learning Centre which was the venue for one of the first job guarantee programmes with Tesco. This was where unemployed people were offered a job, on the basis of their potential to be a good employee and provided they completed a training programme. The success of the programme was lauded because it got lots of unemployed people back into work. That was the end objective and it was spectacularly delivered.

However it was the easy and natural access into other services that ‘seamlessly’ delivered the result. What made the difference, was that a number of the successfully applicants, motivated by the guarantee of getting a job were able to quietly access the centre’s literacy and numeracy programmes, so that they were able to address the key issue that had stopped them making progress before. The ability to read and write. What mattered was that when the store opening was delayed, so that it opened after the summer holidays, we were able to harness the work of the Council’s Early Years’ service to create over 60 child care places. A process helped by the fact they were already in the building. It was the full integration of service delivery that made the difference.

E-learning and e-assessment will only be successfully delivered when it becomes a fully integrated part of curriculum delivery not something that is an occasionally added on and made available. The full extent of what it can offer will only be realised, when it is seen as more than just making learning environments electronic but where there is a recognition that it can transform the way that those learning environments are structured and delivered.

One would have hoped that the organisation like JISC that have asked to make this transformation would have done so by now, but for all the reasons described in this blog this is never likely to be the case. Like most things it is probably down to individual centres to pick up the e-learning ball and run with it in the hope that organisation like JISC might eventually catch up and others might look outside their silos.

It is OK to make a mistake!

The proposal from the Francis Report to introduce a duty of candour into the NHS is a challenging one. My only experience of working in the NHS was one summer placement. It was generally a happy experience but I remember one morning when cold tea was served to the residents on the ward. One of them let us know what he thought and I pacified him with an apology. I was then asked to come into the charge nurse’s (as they were called then) office and I was told that we should never admit to a resident that we had made a mistake.


This is a flippant example in comparison to the serious issues faced by the Stafford enquiry. However I use it a symptomatic of a culture where a lack of openness and honesty was encouraged and expected. Something that appears to have continued.

When I am talking about e-portfolios one of the benefits that I often highlight is how transparent they are. Not because they are ‘open’ documents, because ours are certainly not that, but because those with permission to do so are able to review the whole process. It is not easy to ‘hide’ what you don’t want those involved in the assessment to see.

For some this can become a significant barrier to an adopting an electronic system. They fear that somehow their ‘mistakes’ will be published over the web and even if this is not the case the fact is that there is now evidence that I have not got it always right. For professionals the can fear revealing their mistakes.

However when I am internal verifying vocational qualifications, evidence of students getting ‘it wrong’ and assessors highlighting this, is exactly what I want to see because each occasion provides an opportunity for deeper learning. In the same way the most powerful CPD records are often those where someone has reflected on their existing practice, perhaps as a result of a learning intervention or sometimes not, recognised a ‘mistake’ or improvement and implemented it.

Even when this happens there is a reluctance to share that learning experience as if in some senses it erodes your reputation as a practitioner, whereas in many cases, it is the person who has made a mistake and has reflected and learnt from it, is the one that you most want to see.

It is time we started seeing transparency/candour as an asset not as a threat.