Blackboard Has Worst LMS Problem in Years, Taking K-12 District Offline for Days

In a post three weeks ago, I described the massive increase in usage of LMS systems experienced as part of the COVID-based transition to virtual instruction. As of April 1, Blackboard reported LMS usage increases of 400% for the Learn LMS and 3600% for Collaborate Virtual Classroom (synchronous video).

In my post I noted that “given the scale and timeframe of the transition, I think it’s actually quite remarkable how few problems we have seen.” While I still feel that is true – overall the LMS market has responded quite well to the crisis – we now have what I believe is the worst Blackboard Learn LMS performance problems in years. Fairfax County Public Schools (K-12) in Virginia, with its roughly 189,000 students, experienced such significant performance issues with Learn that they had to shut down their Virtual instruction for a period of days, and as of the end of Monday this week, they are moving all virtual work off of the Blackboard LMS.

What Happened

As reported by WTOP News:

Leaders in Virginia’s largest school system — Fairfax County Public Schools — are apologizing and promising things will be better when students resume distance learning on Monday.

During a virtual school board meeting that focused on the mishaps and glitches in the distance learning experience, representatives from Blackboard, the system the county uses for online and distance learning, also repeatedly apologized for the platform’s performance, but told board members the school system hadn’t updated the software that runs the virtual system in two years.

The issues led the county to cancel online instruction for the rest of the week to get things back in order.

FCPS Superintendent Scott Brabrand told WTOP that he wanted to apologize to parents and students for the mishaps.

The problems started early last week (April 13) with integration problems with the authorization (sign-in) system that were resolved by FCPS staff, but on Wednesday they started experiencing load issues – where the LMS could not handle the number of concurrent users and was unresponsive, or slow enough to be unusable. The problem was identified, but FCPS and Blackboard decided that they needed time to ensure the problems were fixed and tested, due to heavy customizations of the system made over the past two decades. This decision led to the cancellation of classes for Thursday and Friday of last week.

According to Blackboard representatives, they worked through the weekend testing and confirming the fix. Unfortunately, within a few hours of classes beginning this past Monday (April 20), FCPS experienced a new, mostly unrelated problem that also made the system unusable. That issue was mostly resolved by the end of the day, with a planned upgrade to be performed overnight to ensure system stability. But by that point the FCPS board had had enough and changed plans as announced on the FCPS website Monday night.

We began our second attempt at teacher-led virtual instruction today and continued to experience difficulties with Blackboard Learn 24-7. This is frustrating and disappointing for everyone. We recognize that our students and teachers need a reliable system for virtual learning; therefore, we are going to move away from Blackboard Learn 24-7 as a tool for face to face instruction.   Teachers and students will continue to use  Blackboard Learn 24-7 to access instructional resources and supports. 

Today, with the support and direction of the School Board, I am taking additional steps to improve our distance learning plan for 189,000 students.  First, I am initiating a comprehensive, outside review of the distance learning rollout. We have retained Hunton Andrews Kurth LLP, a law firm with expertise in information technology and cybersecurity.  We anticipate a report from them in the next few weeks to guide our continued work in distance learning. 

The board further announced plans for continuing their virtual instruction by alternative methods.

Our amazing teachers, school-based technology specialists, and other support staff will continue to provide creative instructional opportunities to students through a variety of other tools such as Google Classroom, pre-recorded videos, learning packets, eBooks, and other approved digital resources as well as directing students to programming on Channels 21, 25, and 99. Whether it’s mailing a letter, recording a read-aloud, or calling students at home, our teachers have gone above and beyond to show they care and these interactions will continue.

We will also provide “face to face” instruction to our students as schools begin the transition away from the Blackboard Learn 24/7 system. Schools will be preparing to provide virtual instruction through a secure Blackboard Collaborate Ultra link or through other alternative means. Teachers will contact students over the next several days about virtual learning opportunities and the platform that will be used.   

Blackboard is headquartered in Fairfax County, and FCPS has been a client for nearly 20 years. In January, however, FCPS had already made the decision to transition from Blackboard Learn to Schoology as of Summer 2021. It is not clear if or how FCPS will accelerate that schedule.

Background

I interviewed Tim Tomlinson, Blackboard’s Chief Product Officer, on Tuesday. As of that time, Blackboard had not heard definitive plans from FCPS and had learned about the board’s decision through the website and media.

Tomlinson said this situation at FCPS was by far the worst problem that the company has faced over the past two months of COVID transitions, and he said he personally feels terrible, especially with this happening in Blackboard’s own backyard – colleagues and friends have their children in Fairfax County Schools.

FCPS was running a highly-customized version of Blackboard Learn last updated in 2017 in a managed hosting environment within Blackboards data center. According to Tomlinson, the problems were relating to the LMS itself and not to Collaborate.

The problems were such that they had been addressed in subsequent Learn software releases over the past two years, but it was not clear that these issues would affect FCPS. Once the problems occurred, the challenge was applying fixes could be applied in the specific FCPS environment. This was not just a system using older software, it was a complex system according to Tomlinson.

While COVID-driven transitions have increased system usage, the actual load experienced by FCPS was not unexpected. Blackboard has other K-12 clients with higher usage, and recently they had upsized FCPS’s server configuration based on the forecast load increases. In fact, while recent FCPS usage was higher than typical, it was not the highest experienced by the system. The problem came from the two software flaws.

Risk of Eating Your Cake and Having It, Too

Blackboard is in the process of moving their Learn LMS towards Software as a Service (SaaS, aka Cloud) hosting on Amazon Web Services (AWS). As of our last reporting in November, Blackboard had migrated approximately one third of Learn LMS clients to SaaS hosting models.

Thus far Blackboard still supports older self-hosting models (in client data centers) and managed hosting models (in Blackboard data centers). The problems experienced at FCPS do not impact Learn SaaS deployments, as they have different server configurations and run the latest versions of the software.

This multi-year migration stands in contrast with LMS competitor Canvas – which was built in the cloud – and D2L Brightspace. D2L took a different path and has entirely migrated their client base from self-hosting and managed-hosting to pure AWS-based SaaS hosting over the past 3-4 years. D2L lost some clients along the way based on this approach, but they are now a true cloud-based LMS company.

Blackboard has not been willing to make the difficult decision to force clients off of older hosting models, meaning that the company has increased the number of deployment options they have to support. They will not truly benefit from the move to Learn SaaS until they can retire older models and focus on one code base and one deployment option.

In my opinion, this cautious approach by Blackboard has come back to bite them. While Tomlinson made it clear that the fault was with Blackboard and took full responsibility for the situation, FCPS has made it clear just how difficult it is to support highly-customized, two-year-old software versions in custom data centers. And ironically, by moving to Schoology FCPS will be moving to the cloud without any customizations. Schoology like Canvas only runs with a SaaS-hosting model at AWS.

SaaS-hosting with AWS is not a panacea, however. Instructure has run into several cases of performance issues over the past month, although none have lasted longer than a few hours. And Schoology has had significant problems they acknowledged in a recent company blog post.

During the week of March 30, we experienced three days with intermittent outages between 10am ET and 12:30pm ET that resulted in a lower quality of experience for Schoology than you’re used to, and of a lower quality than you deserve. As a founder, currently SVP Schoology at PowerSchool, and formerly the CEO of Schoology, I am disappointed and take accountability.

This entire situation is unfortunate – for the Fairfax County students and teachers, and for Blackboard as a company. We knew that increased system usage would be a challenge for LMS companies, which have overall handled COVID transitions quite well, but that does not help the students who lost days of school due to the outage.