NHS ePortfolio

Discussion and articles specifically about NHS ePortfolio

Under pressure: how the ePortfolio system performed during ARCP

During the summer the NHS ePortfolio experiences a significant increase in the level of usage and activity in the lead up and during the ARCP process for Foundation and Specialty trainees.

User activity

As shown in the Figure 1 below, the number of page loads due to extensive user activity peaks towards the end of May with 3.1 million page views across the entire ePortfolio. Over half of these are specifically Foundation ePortfolio users.

Of interest is the average page load times that peaks in mid-June. This is particularly marked in Foundation-specific data and can be accounted for by the change in the type of pages being accessed during June compared to May. During ARCP reviews the server will be generating a higher proportion of complex report pages that require the server to have to process more data.

(a) All users

(b) Foundation users

Figure 1. Page views (blue lines) and average page load time (red line) for all users (a) and just Foundation Users (b)

Server performance

The performance of the server (Figure 2) showed spikes in the data transfer rate (blue lines) and measured overall server response time (elapsed time, red lines) with a peak coinciding with the date of highest usage (22nd May). Greater daily spikes are seen seen during June while transfer rates flatten from the end of May onwards. The occasional transfer rate troughs coincide with scheduled downtime for software updates. Users who may have  experienced delays in opening pages or submitting forms during May and June may have been subject to network issues related to local IT systems or infrastructure that we can’t detect or influence.

Performance Summary May June 13

Figure 2. Server performance during May and June 2013: data transfer rate (blue lines) and overall server response time (red lines)

Summary

Overall, there was no reported or detected performance problems with the ePortfolio that could be attributed to high levels of general activity (as seen prior to ARCP) or by high levels of complex processing activity (as seen during ARCPs). This can for the most part be attributed to recently implemented server architecture that can be “powered up” during peak times but also to ongoing efforts by the development team to identify and remove software and data “bottle necks”.

 

 

Outcome of the NHS ePortfolio Review by NES

NES ePortfolio : Outcome of 1st Phase : Ownership and Management of NES ePortfolio

[Download to read the full document]

“In June we announced our intention to undertake a review of the ownership and management of the NES
eP1. We decided to undertake this review in recognition of the rapid growth of the eP, and concerns that
opportunities for its further development might be constrained due to the financial and governance
arrangements which apply to NES as a Special Health Board.

“We have completed the first phase of our review and have undertaken an option appraisal based on the
three options we identified in our paper;

  • Option1 – Do Nothing
  • Option 2 – Further development of eP within NES
  • Option 3 – Joint venture between NES and a third party.

“By our deadline for receipt of initial proposals we received 7 submissions from third parties to develop a
joint venture with NES. The submissions received were generally all of a high quality and followed a number
of constructive meetings with nearly all of the third parties who submitted proposals. The review project
team also developed Option 2 to support the further development of eP within NES.

“The outcome of the first phase of the review has been a decision by NES to continue to manage the
development of the eP in-house. In summary, we concluded that meeting the future needs of trainees,
students and other eP stakeholders was best protected by retaining the eP within NES. However, the review
also high-lighted that there were aspects of the future development of the eP which NES was not best
placed to manage and we may consider partnership arrangements in due course to address these issues
more effectively.

“We are very pleased that ePortfolio will remain within NES (Option 2) and a Short life Working Group
consisting of a number of senior managers is being assembled to develop a sustainable plan that will ensure
ePortfolio continues to evolve and adapt to meet the needs of healthcare staff. [Continued..]

[Download to read the full document]

NES ePortfolio 2013: Deploy 13th August 2013

As you are aware, we work closely with the UKFPO each year to implement improvements to the core content of the FP e-portfolio provided by the NES ePortfolio team. This year, the changes relate to improved usability and streamlining the completion of forms on the system – they do not affect the educational structure of the e-portfolio in any way.
We are pleased to confirm that the 2013 e-portfolio core content updates have been completed and have now been through user acceptance testing by colleagues from the UKFPO and foundation schools. The updates have been scheduled for deployment on the evening of 13th August 2013. We are disappointed the changes will not be implemented from the start of the foundation year, but believe it is better to complete final quality checks before releasing the new software in order to minimise any risks to users.
All existing e-portfolio content (as per FP2012) will remain in place to support the incoming FP2013 cohort and all existing users. Forms such as the Initial meeting with Educational Supervisor, Induction meeting with the Clinical Supervisor, PDPs and all other forms used at this time will therefore not be affected and should be used as per the usual FP requirements/framework. Please also be assured that all forms and curriculum items completed before 13th August will not be affected by the updates since the changes this year are more ‘behind the scenes’.

We are confident that the thorough checking and testing processes this year will result in a smooth implementation of the 2013 changes to the foundation e-portfolio.

Why do we update at 10pm on a Tuesday?

When is the best time to perform site updates and do maintenance that requires the ePortfolio to go offline for up to an hour? The graph below shows the activity of the ePortfolio during a normal 24 hour week day. As you can see the quietest period is between 2 and 4am. So why not do this work then?

24hour Analytics

ePortfolio over a 24 hour period (Wednesday to Thursday)

The problem is that that we haven’t sufficient resources such that we can have highly qualified development staff working a night-shift. Similar issues surround working over weekends. The site may be offline for 30 to 60 minutes but the technical team involved in the work must prepare code and database scripts, then take the site offline to implement the updates, test the updates, and then monitor and run checks immediately after the site has been brought back online.

Another reason to perform updates in the evening is that if we encounter a significant technical issue with an update we have several more hours to bring about a fix before the majority of users come online at 8am.

There is no ideal time to take the site offline but we have to work within the limits of resources and also ensure that we can minimise the risk of downtime affecting normal working hours.

We endeavour to provide as much warning of downtime as we can, and hope to be consistent in when these scheduled updates occur and so allow users to work around our deployment times.

 

Trainee doctors’ switch to Supervised Learning Events:

The article below was presented in June 2013 as a poster presentation at the 17th Annual International Association of Medical Science Educators (IAMSE) at St Andrews:

INTRODUCTION

The Foundation Programme is a two-year generic training programme which forms the bridge between medical school and specialist/general practice. In August 2012 a change in the approach to trainee assessment was introduced in the form of Supervised Learning Events (SLEs) for 3 key tools, the Case-based Discussion (CbD), Mini- Clinical Evaluation Exercise (MiniCEX), and Direct Observation of Procedure (DOPs). These tools directly replaced the equivalent work-based assessment (WBA) forms that were used in previous years and included Likert scales. SLEs are not individually summative; the intention has been to encourage trainees to use these tools to support ongoing learning, reflection and discussion with assessors and supervisors.

The NHS ePortfolio is a web-based tool used by 15,000 (93%) UK trainees as part of the Foundation Programme that follows graduation from medical school. The purpose of this study has been to compare the level of trainee engagement with the new SLE tools (session 2012 – 2013) with the equivalent assessment tools during the previous year (session 2011 – 2012).

METHODOLOGY

To measure engagement levels we have analysed aspects of behaviour relating to the submission of these forms to their ePortfolio account during a fixed, equivalent period of time, including the time-dependent pattern of form submission. We have compared the number of form submissions made by Foundation Year 1 (FY1) trainees and Foundation Year 2 (FY2) trainees. Data was generated from the NHS ePortfolio database counting the number of forms submitted each day, starting from day 1 at the beginning of the start of the Foundation session through to 6 weeks after the start of the trainees’ third post in May, 43 weeks after start of the training year. The total number of form submissions and the rate of form submission were compared between WBA and SLEs forms.

For reasons of brevity and to avoid data “noise” we analysed data taken from 3 representative Foundation Deaneries: consisting of 1600, 750, and 900 trainees, respectively.

RESULTS

A total of 119,019 form submissions were analysed. Comparison was made between WBA form types against equivalent SLE forms for the same time periods during the training year. This indicated that no appreciable differences exist between WBA and SLE form submission counts were observed with respect to the number of forms submitted or the rate of submission.  Minor variations in number of DOPs versus SLE-DOPS forms were observed though this may be attributed to changes in training requirements and the use of “Signed Procedure” forms.

During 2011 – 2012 SLE versions of MiniCEX, CbD and DOPs forms were not available.  On 1st August 2012 the SLE forms were introduced, replacing their WBA equivalents.

The accompanying figures show the total weekly form submissions during the training year, beginning on 1st August 2011, and with form submission during the training year beginning on 1st August 2012.

DISCUSSION AND CONCLUSIONS

The results presented here have shown the pattern of creation of SLE forms does not significantly differ from the worked based assessments (WBA) forms that they replaced. The total number created would largely be governed by the curricula requirements of the Foundation Programme; in the sample deaneries used in this study 69,000 WBA forms were completed compared to 50,000 SLE forms.

Anecdotal criticism of WBAs has been that assessment (and associated scoring) of procedures may not best support a learning process throughout training as trainees might wait until nearer the end of placements to be assessed thereby gaining better scores. It has previously been observed that the end of each placement is characterised by a significant rush by trainees to submit the required number of WBA forms, commonly within days of the placement end date. With the introduction of Supervised Learning Event (SLE) based forms it might be expected that these tools would result in more measured and timely submission of SLE forms during a trainee’s placement. However, no significant shift in trainee behaviour was observed when examining the number of forms. Indeed, the pattern of SLE form submission was remarkably similar to WBAs.

Of interest was the fact that having previous experience of WBAs did not result in FY2 trainee behaviour when switched to SLEs, as compared to FY1 who were entirely new to SLEs.

To conclude, the lack of any behavioural change in form filling patterns might be attributed to either (i) the lack of training/understanding by trainees (and supervisors) of how to use the SLE tools, or (ii) the nature of form-filling during training in a work environment is be its nature going to result in trainees leaving completion of WBA or SLE forms until late into a particular placement.

Following discussions during poster sessions the issue of the trainees being able to get SLE forms completed by assessors is problematic – the process of trainees having to request assessment/SLE completion using a ticketing system can result the “moment has passed”. The development of a mobile app that allows for offline completion of tickets and forms may have an impact on this so repeating this data analysis in 2014 may be telling

This is available as a Prezi slide presentation at the link

http://prezi.com/7uszv44ilxyp/?utm_campaign=share&utm_medium=copy

and is embedded below:

Acknowledgements

NES would like to thank the Academy of Medical Royal Colleges, the UK Foundation Programme Office and the representatives of the Postgraduate Deaneries for their continued support

 

Accessing ePortfolio – get to the right log in page

It has come to our attention that a number of users, instead of clicking a bookmark to the site or typing https://www.nhseportfolios.org into the address bar of their browser, access our site by searching for terms such as “NHS ePortfolio” in search engines such as Google or Bing, and then clicking links in the search results page these sites choose to provide.

When navigating to the site in this way, the results page provided by these third party sites can often include links to alternate instances of our main site in which users login credentials do not work, but which look very similar to our main site, causing you confusion/frustration.  (Please note, these results vary on a per user basis and are not consistent in their content).

As part of development and testing there exists alternative ePortfolio versions that contain test accounts and test data. If you attempt to log in to one of these alternative sites will find that your user name and password, and your email address, are not recognised.

These alternate sites can be easily identified as they have different addresses which will appear in the address bar of the browser when accessing the site. What you should see is “www.nhseportfolios.org”.

Whilst the contents of the results pages in the search engines are not controlled by NES, we are contacting the major search engines to remove links to alternate instances of our site to reduce this possible source of confusion.

In the interim, please ensure that you are visiting the main site at the following address:
https://www.nhseportfolios.org

If  you continue to experience/report login issues, please first confirm that you are visiting https://www.nhseportfolios.org and, if problems persist, please contact Support and provide a full screengrab where possible to assign with identifying the issue.

Admin’s guide to Foundation ARCP reviewer set up

As part of the Foundation Programmes ARCP process there is a need for individuals to review trainees’ ePortfolios as ARCP assessors.

A role has been provided on the ePortfolio that will allow these individuals this access. However this role will have to be set up on the ePortfolio system by ePortfolio Administrator before the assessor can see an account.

You need to set up the ARCP reviewer role to be located so that they “see” trainees in the same way as Foundation Administrators have access based on their location. See the diagram below that shows how locations are arranged as a tree – by placing the ARCP Assessor on the correct “location” then you establish which trainees are visible to them.

In the above example, ARCP Reviewer X is assigned to Hospital 1 only and can see on trainees based at that location. ARCP Reviewer Y is assigned to Trust C and thus sees trainees at all hospitals location “under” this location, i.e. Hospital 4 and Hospital 5.

1) The first thing to do is make sure you know the location of the trainee in question. The easiest way to do this is to go to the ‘trainee search’ (Home > Select A Trainee) and type their name into the ‘name’ box.

The search results section should include a column called ‘Location’. This is the location that the ARCP assessor must be at or above on the location tree.

2) Next you need to locate the ARCP assessor on the system. The best place to start is the ‘Search Users’ function on the ‘Administer Users’ page (Home > ePortfolio Admin > Users).

2a) Start by searching ‘within your location’. This means searching all the users who are set at the same place as you are. Start you search by typing a name, email, GMC etc into the search fields.

In the example above I have found the user I was looking for. But if you don’t find your user, they may not share a location with you.

2b) Search for the user ‘outside your location’. This will pick up users on the same site as you, but outside your jurisdiction or for users whose accounts are on other sites (e.g. they could be a ‘Foundation Educational Supervisor’)

Perform the search again, but this time select ‘-All Locations-‘ from the dropdown box in the ‘Location’ area.

 

If you still do not find the user, they probably are not on the system you will need to go to step 4) – see below.

2c) Once you have located your user, select them by clicking on their name (this will be blue and underlined) on the search results.

3) Setting up the user’s ARCP assessor role

Once you have selected a user you should see a page like the following:

To examine what roles they currently have, click the grey ‘Admin Roles’ button on the left hand side.  This should take you to a list of their current roles

To enable a new assessor role, click the ‘Add new role’ button

3a) Select the role you wish to add (probably ‘Foundation ARCP Assessor’)

Then, select the location the Assessor needs from the tree.  An assessor will have to be at the same level or above that of the trainee they want to see.  If an assessor needs to see multiple trainees, then put them high enough up the tree so they can see them all.

Click ‘add role’ to finish the process.

The assessor will then be able to access the ePortfolios of all trainees at that location for review.

4) If the user is not on the system you will need to create them an account with an ARCP assessor role using the ‘add new’ user function on the ‘Administer Users’ page

Fill in the new user’s details and then select the ‘ARCP assessor role’ and a suitable location, this is similar to step 3a as detailed above.

The new user’s details will then be emailed to them automatically.

5) Signing: The NHS ePortfolio is designed to allow the FTPD to create the F1/F2 ARCP outcome form. This functionality exists to limit/avoid any confusion as to the official, agreed ARCP review outcome. If for any reason, the FTPD has assigned a deputy a trust/LEP ePortfolio administrator can create the outcome form.

It is possible for a Foundation Administrator to generate, fill and save the form.  However a user with a FTPD role would still need to log in to sign that form after the admin had saved it.  Admins can’t sign it themselves

6) Once the ARCP review has been conducted you will need to remove access by deleting or removing the ARCP assessor role from the user.  A ‘safe’ location to move the role to would be one with no trainees in, maybe an ‘abroad’ location.  Remember you can search for trainees at a given location on the ‘Trainee Search’ page.

Removing the ARCP assessor role is important as otherwise the user may be able to view the portfolios of trainees that they don’t need to.

To delete a role select a user as detailed above, return to the ‘Admin Roles’ page again and view the user’s roles.  There will be an option to ‘edit’ or ‘delete’ the roles present which will allow you to change a location or remove the role.

Please note: If a user only has one role, you will not be able to delete it.  You can move the role to a location with no trainees if you need to restrict access

 

New NHS ePortfolio App

For a number of months now we at ePortfolio have been hearing from our users that the one thing that would make really make a difference in using the ePortfolio for them would be a mobile App.

The ePortfolio team are happy to announce that we have been developing an App and the first version is now available in a ‘beta’ (testing) format.

To install the App use you device’s web browser to go to http://app.nhseportfolios.org where you can set up a short cut on your device. How to achieve this will depend on whether you are using iOS (i.e. iPhone or iPad), Windows or Android operating systems.

The first release of App functionality is built around our ticketing facility and will allow ePortfolio trainees to use a HTML5 compatible mobile device to connect to their ePortfolio account and:
– Raise new ticket requests
– Review existing tickets
– Delete unneeded tickets
– Send ticket reminders

We know that getting internet access in many hospitals and other NHS buildings can be difficult at best and downright impossible at worst so crucially the App will work even if you are not currently connected to the internet via an offline/online sync function.

The ticketing function is just the first of many that we will be releasing.  We are currently working on developing the ability for reflection forms to be available on the App and have lots of plans for additional functionality after that.

If you wish to help us by testing the current beta release version of the App you can access in the first instance by visiting the following URL on your mobile device: http://app.nhseportfolios.org

Please try to access and use the App for a few days or weeks and then let us know how you get on with it.

We have written a feedback form that can be accessed at the following web addresses:
http://tinyurl.com/nesapptest
. Please use this to return any feedback.

If you have any problems using the App please send support requests to app@nhseportfolios.org rather than use our regular support channels.  Please do not contact your College, Deanery, Medical School, or Professional group in regards to the App.

The App, and the API that supports it, have been developed by the ePortfolio team in response to our user’s feedback.  This project is not currently being funded by any of our customer groups, but we hope to work with them in the future to improve our service to our users.