Kenny Page

Unscheduled downtime event – Thursday 20th July 2017

A failure with the networking infrastructure at our hosting provider (specifically, the Microsoft Azure UKSouth region) resulted in the NHS ePortfolios website at http://www.nhseportfolios.org becoming unavailable/unresponsive to all users on three occasions on Thursday the 20th July 2017:

  • At 21:49 to 21:55 (6 minutes)
  • At 22:05 to 22:10 (5 minutes)
  • At 22:29 to 00:58 (149 minutes)

Following identification of the cause of the outage, updates were provided via the @NHSePortfolios Twitter account between 23:13 and 01:05.

No data was damaged / compromised as a result of this incident.

Update, 25 July 2017: Root Cause Analysis provided by hosting provider:

RCA – Network Infrastructure – UK South

Summary of impact: Between July 20, 2017 21:41 UTC and July 21, 2017 1:40 UTC, a subset of customers may have encountered connectivity failures for their resources deployed in the UK South region. Customers would have experienced errors or timeouts while accessing their resources. Upon investigation, the Azure Load Balancing team found that the data plane for one of the instances of Azure Load Balancing service in UK South region was down. A single instance of Azure Load Balancing service has multiple instances of data plane. It was noticed that all data plane instances went down in quick succession and failed repeatedly whilst trying to self-recover. The team immediately started working on the mitigation to fail over from the offending Azure Load Balancing instance to another instance of Azure Load Balancing service. This failover process was delayed due to the fact that VIP address of Azure authentication service used to secure access to any Azure production service in that region was also being served by the Azure Load Balancing service instance that went down. The Engineering teams resolved the access issue and then recovered the impacted Azure Load Balancing service instance by failing over the impacted customers to another instance of Azure Load Balancing service. The dependent services recovered gradually once the underlying load balancing service instance was recovered. Full recovery by all of the affected services was confirmed by 01:40 UTC on 21 July 2017.

Workaround: Customers who had deployed their services across multiple regions could fail out of UK South region.

Root cause and mitigation: The issue occurred when one of the instances of Azure Load Balancing service went down in the UK South region. The root cause of the issue was a bug in the Azure Load Balancing service. The issue was exposed due to a specific combination of configurations on this load balancing instance combined with a deployment specification that caused the data plane of the load balancing service to crash. There are multiple instances of data plane in a particular instance of Azure Load Balancing Service. However, due to this bug, the crash cascaded through multiple instances. The issue was recovered by failing over from the specific load balancing instance to another load balancing instance. The software bug was not detected in deployments in prior regions because it only manifested under specific combinations of the configuration in Azure Load Balancing services. The combination of configurations that exposed this bug was addressed by recovering the Azure Load Balancing service instance.

Next steps: We sincerely apologize for the impact to affected customers. We are continuously taking steps to improve the Microsoft Azure Platform and our processes to help ensure such incidents do not occur in the future. In this case, we will: 1. Roll out a fix to the bug which caused Azure Load Balancing instance data plane to crash. In the interim a temporary mitigation has been applied to prevent this bug from resurfacing in any other region. 2. Improve test coverage for the specific combination of configuration that exposed the bug. 3. Address operational issues for Azure Authentication services break-glass scenarios.

Update for deployment on Sat 13 Jul 2017 (Release 25.0)

Below are the items that are to be included in Release 25.0, scheduled for deployment on Sat 13 Jul 2017.

Update items included in this build are:

  • 23597 – EPORTFOLIO – Update .NET Framework from 4.5 to 4.6.1
  • 23623 – FOUNDATION – HEE – Removal of posts starting on or after 1st August 2017
  • 23655 – EPORTFOLIO – Update libraries
  • 23713 – FOUNDATION – TURAS – Export Rest of UK Users for TURAS

Fixes included in this build are:

  • 23711 – EPORTFOLIO – QA – spQA_InsertAllForms – Update to db structure

Descriptions for these items are as follows:

23597 – EPORTFOLIO – Update .NET Framework from 4.5 to 4.6.1
NHS ePortfolios Internal Item.

23623 – FOUNDATION – HEE – Removal of posts starting on or after 1st August 2017
Within the FOUNDATION site, remove posts (for HEE locations only) that start on or after the 1st August 2017.

23655 – EPORTFOLIO – Update libraries
NHS ePortfolios Internal Item.

23711 – EPORTFOLIO – QA – spQA_InsertAllForms – Update to db structure
NHS ePortfolios Internal Item.

23713 – FOUNDATION – TURAS – Export Rest of UK Users for TURAS
Export details of Rest of UK Foundation users for migration to TURAS.

Update for deployment on Sat 24 Jun 2017 (Release 24.9)

Below are the items that are to be included in Release 24.9, scheduled for deployment on Sat 24 Jun 2017.

Update items included in this build are:

  • 23347 – EPORTFOLIO – Update styling of modal dialog headers
  • 23451 – ICELAND – Apply Bootstrap style to forms within new site
  • 23536 – EPORTFOLIO – Update Move Forms facility
  • 23537 – EPORTFOLIO – Remove ContentLastModified columns from FilledForms table
  • 23550 – EPORTFOLIO – Updates to page footers

Fixes included in this build are:

  • 23556 – DENTAL – Unable to view blank LEP forms [ZEN 33076]
  • 23557 – DENTAL – PAQSOP (VT/LDFT & HTVT) Formatting issues [ZEN 33079]
  • 23592 – MANAGEMENT – Incorrect styling on tabs in error pages
  • 23605 – FOUNDATION – Revert Delete Permissions on Supervision Forms [ZEN 33313]

Descriptions for these items are as follows:

23347 – EPORTFOLIO – Update styling of modal dialog headers
NHS ePortfolios Internal Item.

23451 – ICELAND – Apply Bootstrap style to forms within new site
Within the ICELAND site, ensure the styling rules implemented in all custom forms match the styling rules utilised in other sites.

23536 – EPORTFOLIO – Update Move Forms facility
NHS ePortfolios Internal Item.

23537 – EPORTFOLIO – Remove ContentLastModified columns from FilledForms table
NHS ePortfolios Internal Item.

23550 – EPORTFOLIO – Updates to page footers
NHS ePortfolios Internal Item.

23556 – DENTAL – Unable to view blank LEP forms [ZEN 33076]
Within the DENTAL site, resolve the issue that was preventing the viewing of VT/LDFT and HTVT versions of LEP forms via the Blank Forms facility.

23557 – DENTAL – PAQSOP (VT/LDFT & HTVT) Formatting issues [ZEN 33079]
Within the DENTAL site, resolve styling issues with the VT/LDFT and HTVT versions of the PAQSOP forms.

23592 – MANAGEMENT – Incorrect styling on tabs in error pages
NHS ePortfolios Internal Item.

23605 – FOUNDATION – Revert Delete Permissions on Supervision Forms [ZEN 33313]
Within the FOUNDATION site, revert permissions for the deletion of supervision forms to pre 2016/17 values.

Update for deployment on Sat 10 Jun 2017 (Release 24.8)

Below are the items that are to be included in Release 24.8, scheduled for deployment on Sat 10 Jun 2017.

Update items included in this build are:

  • 22829 – EPORTFOLIO – Introduce “undeliverable” warning flag on ticket overview page.
  • 23377 – DPH – Basic site build – Phase 2 – SAC External Representative
  • 23421 – IoPPN – Change to Supervisor Access to Previous Post [ZEN 32276]
  • 23468 – FOUNDATION – 2017 – HEE HORUS Migration – Extract FOUNDATION database [ZEN 32526]
  • 23560 – EPORTFOLIO – Updates to forgotten password tool [ZEN 33092]

Fixes included in this build are:

  • 23479 – FOUNDATION – Update permission on “Psychiatric and Mental Health Competency” forms [ZEN 32516]
  • 23494 – EPORTFOLIO – Error when completing ticket request sent to yourself [ZEN 32561]
  • 23511 – EPORTFOLIO – Update placeholder text in step 2 of ticket creation process
  • 23524 – PHYSICIAN – Error opening “Supervised Trainees” page for certain users [ZEN 32812]
  • 23528 – EPORTFOLIO – Error when attempting to view a draft form via the my draft forms page/widget [ZEN 32833] / [ZEN 32892]
  • 23547 – EPORTFOLIO – Assessor TicketForms page setting SelectedUser irrespective of current user role [ZEN 32844]
  • 23551 – EPORTFOLIO – Ensure home icon is styled as per other top navigation items
  • 23562 – EPORTFOLIO – Incorrect positioning of checkboxes in association admin feature

Descriptions for these items are as follows:

22829 – EPORTFOLIO – Introduce “undeliverable” warning flag on ticket overview page.
Within all sites, update the Ticket Overview page to contain a flag indicating when delivery of the related email message has not been possible.

23377 – DPH – Basic site build – Phase 2 – SAC External Representative
Within the DPH site, update the forms and facilities that are available to users of the SAC External Representative role.

23421 – IoPPN – Change to Supervisor Access to Previous Post [ZEN 32276]
Within the IoPPN site, update permissions to prevent supervisors from viewing form content related to previous posts.

23468 – FOUNDATION – 2017 – HEE HORUS Migration – Extract FOUNDATION database [ZEN 32526]
NHS ePortfolios Internal Item.

23479 – FOUNDATION – Update permission on “Psychiatric and Mental Health Competency” forms [ZEN 32516]
Within the Foundation site, update the permissions on “Psychiatric and Mental Health Competency” forms to prevent draft forms become locked when moved.

23494 – EPORTFOLIO – Error when completing ticket request sent to yourself [ZEN 32561]
Within all sites, resolve issue that causes an error to occur when attempting to complete a ticket request sent to yourself.

23511 – EPORTFOLIO – Update placeholder text in step 2 of ticket creation process
Within all sites, update the placeholder text within step 2 of the ticket creation process, ensuring correct assessor terminology is utilised.

23524 – PHYSICIAN – Error opening “Supervised Trainees” page for certain users [ZEN 32812]
Within the PHYSICIAN site, resolve issue causing an error to occur when attempting to view the “Supervised Trainees” page.

23528 – EPORTFOLIO – Error when attempting to view a draft form via the my draft forms page/widget [ZEN 32833] / [ZEN 32892]
Within all sites, resolve issue that causes an error to occur when viewing draft forms via the My Draft Forms tab / page.

23547 – EPORTFOLIO – Assessor TicketForms page setting SelectedUser irrespective of current user role [ZEN 32844]
NHS ePortfolios Internal item.

23551 – EPORTFOLIO – Ensure home icon is styled as per other top navigation items
NHS ePortfolios Internal item.

23560 – EPORTFOLIO – Updates to forgotten password tool [ZEN 33092]
NHS ePortfolios Internal item.

23562 – EPORTFOLIO – Incorrect positioning of checkboxes in association admin feature
NHS ePortfolios Internal Item.

Update for deployment on Sat 3 Jun 2017 (Release 24.7)

Below are the items that are to be included in Release 24.7, scheduled for deployment on Sat 3 Jun 2017.

Update items included in this build are:

  • 23208 – EPORTFOLIO – Update our Libraries
  • 23517 – EPORTFOLIO – Update EntityFramework-Reverse-POCO-Code-First-Generator
  • 23534 – RCPI PCS – Increase Personal Library Storage Space from 200Mb to 300Mb
  • 23549 – PHYSICIAN – MOST COLLEGES Changes to ARCP form [ZEN 32188]

Fixes included in this build are:

Descriptions for these items are as follows:

23208 – EPORTFOLIO – Update our Libraries
NHS ePortfolios Internal Item.

23517 – EPORTFOLIO – Update EntityFramework-Reverse-POCO-Code-First-Generator
NHS ePortfolios Internal Item.

23534 – RCPI PCS – Increase Personal Library Storage Space from 200Mb to 300Mb
Within the RCPI site, increase the per trainee limit on for storage of personal library and uploaded items from 80Mb to 100Mb.

23549 – PHYSICIAN – MOST COLLEGES Changes to ARCP form [ZEN 32188]
Within the PHYSICIAN site, update the ARCP Outcome Form to include Medical Ophthalmology in the Specialty being assessed field.

Unscheduled Downtime Event – 12th May 2017 & 13th May 2017

08:27 on Saturday the 13th May 2017

Access to the NHS ePortfolios platform at https://www.nhseportfolios.org has now been restored.

08:25 on Saturday the 13th May 2017

We have now received notification that access to the NHS ePortfolios platform can be restored and are working to action this.

06:55 on Saturday the 13th May 2017

The NHS ePortfolios platform remains offline as part of co-ordinated containment efforts.

Further information will be posted here and via the @NHSePortfolios Twitter account when we have an indication as to when the platform will be re-enabled.

22:55 on Friday the 12th May 2017

The NHS ePortfolios platform still remains offline as part of co-ordinated containment efforts.

Further information will be posted here and via the @NHSePortfolios Twitter account when we have an indication as to when the platform will be re-enabled.

20:55 on Friday the 12th May 2017

The NHS ePortfolios platform remains offline as part of co-ordinated containment efforts following the rapid spread of Malware within associated but disconnected networks.

Further information will be posted here and via the @NHSePortfolios Twitter account when we have an indication as to when the platform will be re-enabled.

18:00 on Friday the 12th May 2017

The NHS ePortfolios team within NHS Education for Scotland have complied with an official request and temporarily prevented access to the NHS ePortfolio platform at https://www.nhseportfolios.org.  This is a precautionary measure, taken to ensure the safety of users’ data.

Further information will be posted here and via the @NHSePortfolios Twitter account when we have an indication as to when the platform will be re-enabled.

 

Update for deployment on Sat 13 May 2017 (Release 24.6)

Below are the items that are to be included in Release 24.6, scheduled for deployment on Sat 13 May 2017.

Update items included in this build are:

  • 20152 – EPORTFOLIO – My Draft Forms Page / Widget
  • 22933 – EPORTFOLIO – Responsive Previews with Form Designer
  • 23194 – RCPI PCS – Update 5 forms to allow selection of 1 to 3 domains [ZEN 30548]
  • 23290 – RCPI PCS – Update to Annual Statement of Participation [ZEN 30548]
  • 23348 – EPORTFOLIO – Update default styling of top navigation
  • 23375 – EPORTFOLIO – Restyle Quicklinks Homepage Nugget
  • 23379 – EPORTFOLIO – Update styling of homepage tabs
  • 23398 – PHYSICIAN – Create new locations and specialty [ZEN 32189]
  • 23399 – PHYSICIAN – MOST COLLEGES CR2017_20 ARCP 2017_Most Colleges_Amendments [ZEN 32188]

Fixes included in this build are:

  • 23364 – EPORTFOLIO – Enable viewing of in-use picklists [ZEN 32000]
  • 23462 – EPORTFOLIO – Updates to linked signature guidance text on form handler
  • 23464 – EPORTFOLIO – Add guidance text in step 2 of ticket creation process [ZEN 32346]
  • 23480 – EPORTFOLIO – Restrict display of LTFT Status within the “Current Post” homepage widget [ZEN 32547]

Descriptions for these items are as follows:

20152 – EPORTFOLIO – My Draft Forms Page / Widget
Within all sites except RCPI and RCPIPCS, introduce a new “My Draft Forms” homepage widget and page that allows users to quickly and easily view a list of forms that have been saved in a draft state and access them to complete them.

22933 – EPORTFOLIO – Responsive Previews with Form Designer
Within all sites, in the self managed form designer, add buttons to allow users to preview the form layout at different screen widths.

23194 – RCPI PCS – Update 5 forms to allow selection of 1 to 3 domains [ZEN 30548]
For all sites, enhance the self-managed form checkbox control to allow restrictions to be placed on the number of selectable items.

23290 – RCPI PCS – Update to Annual Statement of Participation [ZEN 30548]
Within the RCPI PCS site, update the Annual Statement of Participation to ensure that it handles the selection of multiple domains.

23348 – EPORTFOLIO – Update default styling of top navigation
NHS ePortfolios Internal Item.

23364 – EPORTFOLIO – Enable viewing of in-use picklists [ZEN 32000]
Whilst in-use picklists remain non-editable, re-introduce the ability to view the contents of the now non-editable picklists.

23375 – EPORTFOLIO – Restyle Quicklinks Homepage Nugget
NHS ePortfolios Internal Item.

23379 – EPORTFOLIO – Update styling of homepage tabs
NHS ePortfolios Internal Item.

23398 – PHYSICIAN – Create new locations and specialty [ZEN 32189]
Within the PHYSICIAN site, established 14 new locations and one new speciality

23399 – PHYSICIAN – MOST COLLEGES CR2017_20 ARCP 2017_Most Colleges_Amendments [ZEN 32188]
Within the PHYSICIAN, RCR, RCOG, CEM, FICM & CSRH sites, update the ARCP Outcome Form to a new version.

23462 – EPORTFOLIO – Updates to linked signature guidance text on form handler
Resolve typo / styling issues within guidance text relating to linked signatures.

23464 – EPORTFOLIO – Add guidance text in step 2 of ticket creation process [ZEN 32346]
Update field labels in step 2 of the ticketing process, adding extra guidance text to highlight that users can either choose a listed recipient or continue to type the email address of an unlisted recipient as before.

23480 – EPORTFOLIO – Restrict display of LTFT Status within the “Current Post” homepage widget [ZEN 32547]
Hide the FT / LTFT value within the current post widget for users of sites that do not record this value at a per-post level.

Update for deployment on Sat 6 Apr 2017 (Release 24.5)

Below are the items that are to be included in Release 24.5, scheduled for deployment on Sat 6 Apr 2017.

Update items included in this build are:

  • 21247 – EPORTFOLIO – New Forms Admin Page
  • 23134 – CPD – New Limited Administrator Role [ZEN 31043]

Fixes included in this build are:

  • 23454 – EPORTFOLIO – New admin forms page bugfixes

Descriptions for these items are as follows:

21247 – EPORTFOLIO – New Forms Admin Page
Within all sites, update the form administration pages and introduce a new, improved, self managed form designer.

23134 – CPD – New Limited Administrator Role [ZEN 31043]
Within the CPD site, establish a new “CPD Limited Administrator” role.

23454 – EPORTFOLIO – New admin forms page bugfixes
NHS ePortfolios Internal Item.

Updates to display of results when searching for users

As part of Release 24.3 deployed on Saturday the 22nd April 2017, the way in which individual results are displayed when searching for users has been slightly updated, as described below.

In line with other features, a new button entitled “Select User” has been added at the far right of each result, replacing the link that was previously formed by the users name.  The tooltip that was previously accessible by hovering over the users name remains accessible, by hovering over the new “Select User” button.

An information icon has been introduced immediately following the users name that when clicked/tapped opens the “user information card”, providing quick access for both mouse and touch screen users to view the users details without fully opening the user account.

 

 

Fig 1 - BeforeFig 1: Before

Fig 2 - After

Fig 2: After

Update for deployment on Sat 29 Apr 2017 (Release 24.4)

Below are the items that are to be included in Release 24.4, scheduled for deployment on Sat 29 Apr 2017.

Update items included in this build are:

  • 22329 – ICELAND – New site [ZEN 31709]
  • 23231 – EPORTFOLIO – Update internal caching [ZEN 31358]

Fixes included in this build are:

Descriptions for these items are as follows:

22329 – ICELAND – New site [ZEN 31709]
Establish new ICELAND site within NHS ePortfolios platform.

23231 – EPORTFOLIO – Update internal caching [ZEN 31358]
NHS ePortfolios Internal Item.