Monthly Archives: July 2017

Update for deployment on Sat 29 Jul 2017 (Release 25.1)

Below are the items that are to be included in Release 25.1, scheduled for deployment on Sat 29 Jul 2017.

Update items included in this build are:

  • 23302 – NES NM – MENTOR – Alter Nurse Mentor site to READ access only
  • 23539 – EPORTFOLIO – My Tickets To Complete Improvements [ZEN 32895]
  • 23548 – EPORTFOLIO – Updates to the login page
  • 23552 – EPORTFOLIO – Update styling of Alerts homepage widget
  • 23565 – MANAGEMENT – Admin Permissions
  • 23585 – ICELAND – Reorganise Supervision forms and establish MCR Form [ZEN 33249]
  • 23596 – PHARMACY – Enable trainee searching by GPhC Number
  • 23618 – MANAGEMENT – Add dates and times to email report tool
  • 23621 – MANAGEMENT – Error when adding a new account merge to the queue
  • 23635 – FOUNDATION – HEE – New role – Historic Foundation ePortfolio (HEE)
  • 23637 – EPORTFOLIO – Minor Personal Library updates (Homepage Widget & Overview Page)
  • 23656 – MANAGEMENT – Rename Misc menu and establish new Support Tools menu
  • 23659 – MANAGEMENT – Establish new form deletion tool for management administrators
  • 23663 – EPORTFOLIO – Update NUnit to latest version

Fixes included in this build are:

  • 23624 – EPORTFOLIO – Unable to upload profile picture with non-standard characters in filenames
  • 23625 – EPORTFOLIO – Resolve bad link to /Auth/Common/Pages/SelectRole.aspx
  • 23707 – EPORTFOLIO – Fix UI issue in Download Portfolio page

Descriptions for these items are as follows:

23302 – NES NM – MENTOR – Alter Nurse Mentor site to READ access only
Within the NM (Nurse Mentor) site, ensure that all existing accounts are made read-only as data is being migrated to TURAS.

23539 – EPORTFOLIO – My Tickets To Complete Improvements [ZEN 32895]
Within ALL sites, update the “My Tickets To Complete” widget and page to indicate that partially completed tickets cannot be declined.

23548 – EPORTFOLIO – Updates to the login page
Update the styling of UI elements on the Login page to improve consistency.

23552 – EPORTFOLIO – Update styling of Alerts homepage widget
Within ALL sites, updated the styling of the alerts homepage widget to improve consistency with other UI elements.

23565 – MANAGEMENT – Admin Permissions
NHS ePortfolios Internal Item.

23585 – ICELAND – Reorganise Supervision forms and establish MCR Form [ZEN 33249]
Within the ICELAND site, reorganise all supervision forms under new “Supervisor Forms” menu item and establish new MCR form.

23596 – PHARMACY – Enable trainee searching by GPhC Number
Within the PHARMACY site, update the trainee search to allow searching by GPhC Number.

23618 – MANAGEMENT – Add dates and times to email report tool
Within the MANAGEMENT site, update the email report tool to provide dates and times to reduce the requirement to cross reference with other sources.

23621 – MANAGEMENT – Error when adding a new account merge to the queue
Within the MANAGEMENT site, resolve issue casuing an error to occur when queuing an account merge.

23624 – EPORTFOLIO – Unable to upload profile picture with non-standard characters in filenames
Within all sites, ensure that commas in filenames are handled to prevent security warnings occurring when downloading from Google Chrome.

23625 – EPORTFOLIO – Resolve bad link to /Auth/Common/Pages/SelectRole.aspx
NHS ePortfolios Internal Item.

23635 – FOUNDATION – HEE – New role – Historic Foundation ePortfolio (HEE)
Establish a new “Historic Foundation ePortfolio (HEE)” to provide signposting to departed users of the FOUNDATION site from HEE locations.

23637 – EPORTFOLIO – Minor Personal Library updates (Homepage Widget & Overview Page)
Within ALL sites, perform slight updates to the UI of the personal library and associated homepage widget to improve consistency with other UI elements.

23656 – MANAGEMENT – Rename Misc menu and establish new Support Tools menu
NHS ePortfolios Internal Item.

23659 – MANAGEMENT – Establish new form deletion tool for management administrators
Within the MANAGEMENT site, for users of the Management Administrator role, establish a tool to allow form deletion requests to be processed without support scripts.

23663 – EPORTFOLIO – Update NUnit to latest version
NHS ePortfolios Internal Item.

23707 – EPORTFOLIO – Fix UI issue in Download Portfolio page
Within all sites, resolve UI issue that was cropping the action drop down menu in the main download portfolio page.

Unscheduled downtime event – Thursday 20th July 2017

A failure with the networking infrastructure at our hosting provider (specifically, the Microsoft Azure UKSouth region) resulted in the NHS ePortfolios website at http://www.nhseportfolios.org becoming unavailable/unresponsive to all users on three occasions on Thursday the 20th July 2017:

  • At 21:49 to 21:55 (6 minutes)
  • At 22:05 to 22:10 (5 minutes)
  • At 22:29 to 00:58 (149 minutes)

Following identification of the cause of the outage, updates were provided via the @NHSePortfolios Twitter account between 23:13 and 01:05.

No data was damaged / compromised as a result of this incident.

Update, 25 July 2017: Root Cause Analysis provided by hosting provider:

RCA – Network Infrastructure – UK South

Summary of impact: Between July 20, 2017 21:41 UTC and July 21, 2017 1:40 UTC, a subset of customers may have encountered connectivity failures for their resources deployed in the UK South region. Customers would have experienced errors or timeouts while accessing their resources. Upon investigation, the Azure Load Balancing team found that the data plane for one of the instances of Azure Load Balancing service in UK South region was down. A single instance of Azure Load Balancing service has multiple instances of data plane. It was noticed that all data plane instances went down in quick succession and failed repeatedly whilst trying to self-recover. The team immediately started working on the mitigation to fail over from the offending Azure Load Balancing instance to another instance of Azure Load Balancing service. This failover process was delayed due to the fact that VIP address of Azure authentication service used to secure access to any Azure production service in that region was also being served by the Azure Load Balancing service instance that went down. The Engineering teams resolved the access issue and then recovered the impacted Azure Load Balancing service instance by failing over the impacted customers to another instance of Azure Load Balancing service. The dependent services recovered gradually once the underlying load balancing service instance was recovered. Full recovery by all of the affected services was confirmed by 01:40 UTC on 21 July 2017.

Workaround: Customers who had deployed their services across multiple regions could fail out of UK South region.

Root cause and mitigation: The issue occurred when one of the instances of Azure Load Balancing service went down in the UK South region. The root cause of the issue was a bug in the Azure Load Balancing service. The issue was exposed due to a specific combination of configurations on this load balancing instance combined with a deployment specification that caused the data plane of the load balancing service to crash. There are multiple instances of data plane in a particular instance of Azure Load Balancing Service. However, due to this bug, the crash cascaded through multiple instances. The issue was recovered by failing over from the specific load balancing instance to another load balancing instance. The software bug was not detected in deployments in prior regions because it only manifested under specific combinations of the configuration in Azure Load Balancing services. The combination of configurations that exposed this bug was addressed by recovering the Azure Load Balancing service instance.

Next steps: We sincerely apologize for the impact to affected customers. We are continuously taking steps to improve the Microsoft Azure Platform and our processes to help ensure such incidents do not occur in the future. In this case, we will: 1. Roll out a fix to the bug which caused Azure Load Balancing instance data plane to crash. In the interim a temporary mitigation has been applied to prevent this bug from resurfacing in any other region. 2. Improve test coverage for the specific combination of configuration that exposed the bug. 3. Address operational issues for Azure Authentication services break-glass scenarios.

Update for deployment on Sat 13 Jul 2017 (Release 25.0)

Below are the items that are to be included in Release 25.0, scheduled for deployment on Sat 13 Jul 2017.

Update items included in this build are:

  • 23597 – EPORTFOLIO – Update .NET Framework from 4.5 to 4.6.1
  • 23623 – FOUNDATION – HEE – Removal of posts starting on or after 1st August 2017
  • 23655 – EPORTFOLIO – Update libraries
  • 23713 – FOUNDATION – TURAS – Export Rest of UK Users for TURAS

Fixes included in this build are:

  • 23711 – EPORTFOLIO – QA – spQA_InsertAllForms – Update to db structure

Descriptions for these items are as follows:

23597 – EPORTFOLIO – Update .NET Framework from 4.5 to 4.6.1
NHS ePortfolios Internal Item.

23623 – FOUNDATION – HEE – Removal of posts starting on or after 1st August 2017
Within the FOUNDATION site, remove posts (for HEE locations only) that start on or after the 1st August 2017.

23655 – EPORTFOLIO – Update libraries
NHS ePortfolios Internal Item.

23711 – EPORTFOLIO – QA – spQA_InsertAllForms – Update to db structure
NHS ePortfolios Internal Item.

23713 – FOUNDATION – TURAS – Export Rest of UK Users for TURAS
Export details of Rest of UK Foundation users for migration to TURAS.