How to Feel Better About Your Data Warehouse Fiasco

Here’s a little predictive analytics:

About a year ago, I took a swipe at the “$80 million supercomputer to analyze NYC student achievement.” It smelled more like a super sales job than a super useful analytical tool.

At the time I had said:

Teachers are underpaid, hardly appreciated, and overworked. I can only wonder what the half-life is of a system that asks teachers to log on to get information delivered by the “chief accountability officer.”

Well, it appears that things haven’t gone that smoothly with the supercomputer. Today, I received a link from Leonie Haimson, a NYC education advocate, to a story entitled SCHOOLS COMPUTER AN $80M ‘DISASTER’.

Not only has the supercomputer struggled to gain much traction with users (“The school system’s new $80 million computer super system to track student performance has been a super debacle, teachers and principals say”), it has coincided with severe budget cuts.

We see these data warehousing problems all the time with our clients, and the NYC supercomputer displays all the hallmarks:

  • Delivery delays: Nearly six months after the Department of Education unveiled the “first of its kind” data-management system, the city’s 80,000 teachers have yet to log on because of glitches and delays.

  • Bad user experience: Many principals have complained that it runs slowly, lacks vital information, and is often too frustrating to use.

  • Complicated training and set-up: School officials were hoping to have everyone hooked up and trained within months delays in creating IDs and passwords for teachers
  • Trying to do too much, delivering too little: The principal added that she preferred to get student information from a combination of old data systems “rather than wait for ARIS to churn and churn and churn and maybe give me half the report I need.”
  • Massive cost: Complaints about the expensive system—on which nearly $35 million has been spent so far—have gotten louder since the city unceremoniously chopped $100 million from individual school budgets last month.
  • And yet, few success anecdotes to justify the investment: ARIS had already enabled her data team to analyze the performance trends of the school’s many English-language learners.

It does offer one thing that I haven’t seen before: a Chief Accountability Officer.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. All source code is released under a BSD License unless otherwise specified.

6 comments | Show all comments only the last 5 are shown


February 28, 2008
Ross said:

Does these mean that the "CAO" will be the one who has to pay back the $80 mil, he's Accountable right?


February 28, 2008
Lynn Marentette said:

I'm a school psychologist who works with people and data. I've been frustrated with user-unfriendly technology throughout my career. Student data management systems are one of my sore spots.

The quote from the article tells it all: "Bad user experience: Many principals have complained that it runs slowly, lacks vital information, and is often too frustrating to use."

NYC isn't the only place experiencing usability problems with education data management systems.

I work in North Carolina, a state that adopted the NC WISE data management system about 10 years ago, at quite an expense. My school district will be moving to the system next year, so I do not have personal experience with the system. To prepare myself tof the change, I viewed the multitude of questions discussed on the NC WISE Questions/FAQ website. One glance and you will quickly see that ease-of-use was probably not a priority during the system's development: http://www.ncwise.org/ncwiseQuesFAQ.html

By the way, the NC WISE system doesn't play well with data-management systems designed for N.C. special education student information. Many districts use CECAS for handling this data, but as you can see from a recent "bug" list, there are numerous problems: http://www.nccecas.org/systemupdates/openbugs.html

Problems with the NC WISE system have been documented for years. Here are a few examples:

eSchool News article, 2004: http://www.eschoolnews.com/news/top-news/index.cfm?i=35547&CFID=5146403&CFTOKEN=38831309

Byte and Switch article, 2006: http://www.byteandswitch.com/document.asp?doc_id=105150

Note: I returned to school a few years ago to study computers/technology part-time, including HCI, so I know that there is much room for improvement.


February 29, 2008
Annoyed Analyst said:

So...how do I feel better about my data warehouse fiasco? :-)


February 29, 2008
Mary G. said:

Same problem, smaller scale. You’ve opened the door for me to rant a bit about a program that was recently imposed on the special educators in our five town school district at a cost that I have been unable, to date, to track down—but I intend to.

Our small rural district here in Vermont recently bought into a special education management program. As the coordinator of special education services and assistant principal at my school, I have found the program difficult to use and inefficient.

The program, Case-e, was created with little or no understanding of the special education process. The creators didn’t even bother to use the special education terminology we use in Vermont on the countless, seemingly arbitrarily placed, tabs. Each section of an evaluation or individualized education plan is isolated so that the user can’t see the report until it is finished. And, there is no spell check! Really, I’m not kidding. Just as annoying, test scores cannot be shown in columns. Oh, you think I’m making this up? No, the program does not allow it. The support folks from the company said they can’t fix it because it was “created by engineers in California.” What is that suppose to mean? They are willing to sell a program but to be responsible for how it works?

According to their website, the company claims their goal is to “help special educators reduce their time spent on paperwork and administrative duties so they can spend more of their time and energy helping students.” Wrong. The final product is unattractive and difficult to read, often with big blank spaces and sections of the report divided where they shouldn’t be. Not only does the original document take significantly longer to prepare, but also I am ashamed to give it to parents. I always apologize for it and often give them a second copy of a report that I have redone in my old parent-friendly format.

Finally, the program claims to provide “total support… you have all the assistance you need, when you need it.” Not true. If the creators knew anything about teachers, they would know that we often write these reports late at night and on the weekends. Teaching is not a 9 to 5 job. Their technical assistant folks need to keep the same hours that teachers do.

Beyond the cost of this program, this is costing our district many extra hous of professional time. I could go on, but I won’t. Grrr...


March 2, 2008
Mr. C said:

I'm a principal in NYC and can verify for you that ARIS is the fiasco you describe. For a system with very few active users, it's shockingly slow. (What will happen when everyone is actually using it?) Currently, ARIS is a pretty user interface on top of an outdated (and equally atrocious) computer system in which we still do all of our data entry. No new information can be put into ARIS. You can only view the information and produce more attractive looking reports which mostly concern test scores.
The trainings, largely done by IBM staffers with no sense of what we would need/want from a data system, have been a total waste of time. I have spent more time in training about how to use ARIS than I have actually using it for anything to help my students or my school. And yes, I just lost tens of thousands of dollars from my budget.

Your name

Email (optional, will not be shared)

Type the word "juice" (required to confuse the spammers)

Your comment


Add a comment