Unscheduled downtime event – Thursday 20th July 2017

A failure with the networking infrastructure at our hosting provider (specifically, the Microsoft Azure UKSouth region) resulted in the NHS ePortfolios website at http://www.nhseportfolios.org becoming unavailable/unresponsive to all users on three occasions on Thursday the 20th July 2017:

  • At 21:49 to 21:55 (6 minutes)
  • At 22:05 to 22:10 (5 minutes)
  • At 22:29 to 00:58 (149 minutes)

Following identification of the cause of the outage, updates were provided via the @NHSePortfolios Twitter account between 23:13 and 01:05.

No data was damaged / compromised as a result of this incident.

Update, 25 July 2017: Root Cause Analysis provided by hosting provider:

RCA – Network Infrastructure – UK South

Summary of impact: Between July 20, 2017 21:41 UTC and July 21, 2017 1:40 UTC, a subset of customers may have encountered connectivity failures for their resources deployed in the UK South region. Customers would have experienced errors or timeouts while accessing their resources. Upon investigation, the Azure Load Balancing team found that the data plane for one of the instances of Azure Load Balancing service in UK South region was down. A single instance of Azure Load Balancing service has multiple instances of data plane. It was noticed that all data plane instances went down in quick succession and failed repeatedly whilst trying to self-recover. The team immediately started working on the mitigation to fail over from the offending Azure Load Balancing instance to another instance of Azure Load Balancing service. This failover process was delayed due to the fact that VIP address of Azure authentication service used to secure access to any Azure production service in that region was also being served by the Azure Load Balancing service instance that went down. The Engineering teams resolved the access issue and then recovered the impacted Azure Load Balancing service instance by failing over the impacted customers to another instance of Azure Load Balancing service. The dependent services recovered gradually once the underlying load balancing service instance was recovered. Full recovery by all of the affected services was confirmed by 01:40 UTC on 21 July 2017.

Workaround: Customers who had deployed their services across multiple regions could fail out of UK South region.

Root cause and mitigation: The issue occurred when one of the instances of Azure Load Balancing service went down in the UK South region. The root cause of the issue was a bug in the Azure Load Balancing service. The issue was exposed due to a specific combination of configurations on this load balancing instance combined with a deployment specification that caused the data plane of the load balancing service to crash. There are multiple instances of data plane in a particular instance of Azure Load Balancing Service. However, due to this bug, the crash cascaded through multiple instances. The issue was recovered by failing over from the specific load balancing instance to another load balancing instance. The software bug was not detected in deployments in prior regions because it only manifested under specific combinations of the configuration in Azure Load Balancing services. The combination of configurations that exposed this bug was addressed by recovering the Azure Load Balancing service instance.

Next steps: We sincerely apologize for the impact to affected customers. We are continuously taking steps to improve the Microsoft Azure Platform and our processes to help ensure such incidents do not occur in the future. In this case, we will: 1. Roll out a fix to the bug which caused Azure Load Balancing instance data plane to crash. In the interim a temporary mitigation has been applied to prevent this bug from resurfacing in any other region. 2. Improve test coverage for the specific combination of configuration that exposed the bug. 3. Address operational issues for Azure Authentication services break-glass scenarios.

Unscheduled Downtime Event – 12th May 2017 & 13th May 2017

08:27 on Saturday the 13th May 2017

Access to the NHS ePortfolios platform at https://www.nhseportfolios.org has now been restored.

08:25 on Saturday the 13th May 2017

We have now received notification that access to the NHS ePortfolios platform can be restored and are working to action this.

06:55 on Saturday the 13th May 2017

The NHS ePortfolios platform remains offline as part of co-ordinated containment efforts.

Further information will be posted here and via the @NHSePortfolios Twitter account when we have an indication as to when the platform will be re-enabled.

22:55 on Friday the 12th May 2017

The NHS ePortfolios platform still remains offline as part of co-ordinated containment efforts.

Further information will be posted here and via the @NHSePortfolios Twitter account when we have an indication as to when the platform will be re-enabled.

20:55 on Friday the 12th May 2017

The NHS ePortfolios platform remains offline as part of co-ordinated containment efforts following the rapid spread of Malware within associated but disconnected networks.

Further information will be posted here and via the @NHSePortfolios Twitter account when we have an indication as to when the platform will be re-enabled.

18:00 on Friday the 12th May 2017

The NHS ePortfolios team within NHS Education for Scotland have complied with an official request and temporarily prevented access to the NHS ePortfolio platform at https://www.nhseportfolios.org.  This is a precautionary measure, taken to ensure the safety of users’ data.

Further information will be posted here and via the @NHSePortfolios Twitter account when we have an indication as to when the platform will be re-enabled.

 

Updates to display of results when searching for users

As part of Release 24.3 deployed on Saturday the 22nd April 2017, the way in which individual results are displayed when searching for users has been slightly updated, as described below.

In line with other features, a new button entitled “Select User” has been added at the far right of each result, replacing the link that was previously formed by the users name.  The tooltip that was previously accessible by hovering over the users name remains accessible, by hovering over the new “Select User” button.

An information icon has been introduced immediately following the users name that when clicked/tapped opens the “user information card”, providing quick access for both mouse and touch screen users to view the users details without fully opening the user account.

 

 

Fig 1 - BeforeFig 1: Before

Fig 2 - After

Fig 2: After

“No OpenID endpoint found” error creating new link from NHS ePortfolios to e-LfH accounts

UPDATE: 9th November 2016, 8pm.  We have now received notification from e-LfH that the OpenID endpoint has been restored and as such users will once again be able to establish links between their NHS ePortfolios user account and e-LfH user account.

The NHS ePortfolios team are aware that users will presently encounter issues when attempting to establish a new link between an NHS ePortfolios account and an e-LfH user account to enable the exchange of Learning Activity data.

When attempting to establish the links, users are presently witnessing an error message within NHS ePortfolios stating “No OpenID endpoint found”, as shown below:

NHS ePortfolios - No OpenID Endpoint found error.

This issue has arisen following an application upgrade at e-LfH at the end of last week and the e-LfH team are working to resolve this as quickly as they can:

e-LfH OpenID error

If you have already established a link between your NHS ePortfolios account and your e-LfH user account, the ongoing exchange of Learning Activity data will not be affected by this issue.

 

 

Updated Location and Progamme Choosers

Within NHS ePortfolios, there are many occasions where you need to select a location from our “location tree”, e.g. when searching for an individual, or when configuring programmes/posts.

On Saturday the 12th March 2016, Location and Programme choosers were updated to make them quicker and easier to use, particularly on touch screen based devices.

Historically, selecting a location was a two step process.  First, you had to select the top most location in one field, then a second field appeared immediately below which contained the “location tree” itself, from which the location could then be selected:

location-chooser-before

Both of these elements have now been combined into a single chooser that is applied consistently throughout the site:location-chooser-after

To use the new chooser, simply click or tap in the field, and the chooser will appear, then:

  • To expand the location tree, click upon the triangles to the left of the tree items.
  • To select a location, click the round radio button.
  • To de-select locations, click the “clear” button.

Permissions

There will be no changes to the locations to which you have permissions as a result of this change.

The new chooser will contain all locations to which you previously had permission – if previously you had two items in the first field, the new chooser will contain two top level items in a single tree via which you can access all the child locations to which you previously had permission.

Programme Chooser

The programme chooser has also been updated and now behaves in the same manner as the location chooser.

Need help?

If you are having a problem with you NHS ePortfolio account please contact the Support team via the Help section of you account, or email us at support@nhseportfolios.org

Archives

Categories