TALLAHASSEE — A week before the botched launch of Florida's new unemployment benefits website, state senators grilled an agency chief and heard no warning about the chaos to come.
The CONNECT project was well managed and extensive testing showed system failure was unlikely, said Tom Clendenning, director of the Department of Economic Opportunity's workforce services.
"This has been carefully planned out," Clendenning said, smiling broadly during an Oct. 9 Senate hearing. "You can never be too 100 percent bulletproof, so we do have a contingency if in fact the new system isn't ready."
Six days later, the $63 million CONNECT website launched so riddled with technical glitches that it has left thousands of unemployed Floridians without the money they need for food, rent and bills.
The problems were so bad that the DEO began fining the contractor $15,000 a day and federal officials intervened, convincing the state to pay the back claims so claimants could get their money. Two months after CONNECT's debut, so many claims remained unpaid that the DEO hired an extra 330 employees, at a cost of $165,000 a week.
It's clear now that CONNECT wasn't ready and the backup Clendenning promised during testimony never materialized.
Yet DEO officials haven't acknowledged these failures or explained why they didn't better plan for disaster.
Asked repeatedly if there was anything he or state officials could have done differently, DEO executive director Jesse Panuccio spoke only in the most general terms.
"In hindsight, of course, if I knew prior to launch every specific technical problem that had arisen, we would have said, 'Fix the code for those technical problems,' " Panuccio said Wednesday in an interview with the Times/Herald. "I think any sane person would say that. . . . As with any system launch, if you know at the time of launch what you know six months later (from) the time of launch, you would endeavor not to have those issues."
But Panuccio said he didn't know. And for that, he blames the contractors.
• • •
The main contractor of the project, Deloitte Consulting, won the bid to modernize Florida's unemployment compensation system by beating out nine other firms. In early 2011, the company negotiated with Florida that it could do all the work for $39.8 million and finish by December 2012, a deadline it blew — badly.
As contracts go, this wasn't a big one for Deloitte Consulting, a U.S. company that's part of an international British conglomerate better known as Deloitte & Touche. Since 2007, Deloitte Consulting has won $283.4 million in contracts with Florida agencies.
Its interests are protected by one of the most powerful lobbyists in Tallahassee, Brian Ballard, a major campaign fundraiser for Gov. Rick Scott and other GOP officials.
Almost from the start, Deloitte clashed with state officials over the cost and progress of the DEO project.
"Deloitte underestimated the work effort required to accomplish the scope of work for the design phase," DEO project manager Tom McCullion wrote in a February 2012 letter to Deloitte.
Follow trends affecting the local economy
Subscribe to our free Business by the Bay newsletter
You’re all signed up!
Want more of our free, weekly newsletters in your inbox? Let’s get started.
Explore all your optionsIn response, Deloitte offered to hire 40 more staffers and replace several top team leaders, including technical lead Siva Sambasivam, and the lead for testing.
Testing was an important safeguard for a complex project such as this, where more than 2 billion lines of data would be converted.
Deloitte also asked to push back the deadline by 10 months.
The DEO rejected the proposal in June 2012 on the grounds that technical glitches weren't being addressed and proposed fixes were "unrealistic" in price. McCullion put Deloitte on notice that the contract would be terminated unless an agreement was reached on how to conclude the project, alluding to the company's problems in other states, such as California, New Mexico and Massachusetts, with launching a similar system for unemployment benefits.
"Deloitte's demonstrated inability to implement the solution in other jurisdictions has undermined the (DEO's) confidence that Deloitte will successfully complete the (project)," McCullion wrote on June 15, 2012. "The Department contracted for a viable, proven solution. It now appears that the Department is being asked to fund a software development project with limited prospects for success."
One week later, though, the DEO approved Deloitte's final design. On July 13, Deloitte and the DEO signed a new agreement that stated the contractor has "demonstrated its willingness and ability to perform in adherence to the contract terms and condition."
By the time Panuccio, a lawyer by training with no administrative experience, became DEO executive director in 2013, Deloitte again began submitting expensive cost requests.
Asked if he objected to any, Panuccio said only "necessary change orders have a price to them."
• • •
Clendenning's testimony to senators just before CONNECT's October launch was overwhelmingly positive. The system had undergone such thorough testing, as required by the contract, no major deficiencies were detected, he said.
"This is another opportunity for us to tell you how pleased we are with the results," Clendenning told the Senate's transportation, tourism and economic development appropriations committee.
Clendenning pointed to the prognosis from Ernst & Young, which was hired for $155,000 to verify the test results from Deloitte claiming that CONNECT was ready to go.
Ernst & Young concluded "there are no significant project risks."
Panuccio said Wednesday that in the coming weeks, DEO officials will review Ernst & Young's oversight of the project.
"We'll go back and look at that and why Ernst & Young's verification didn't reveal that testing was insufficient," Panuccio said.
Was testing insufficient? It's hard to tell. Deloitte's contract requires that all test scripts and results be retained and made available upon requests. When the Times/Herald made such a request, the DEO said it would cost more than $11,000 to provide the records to redact "potentially confidential information."
Looking for assurances from Clendenning beyond what testing provided, the committee's chairman, Sen. Andy Gardiner, R-Orlando, asked what would happen if CONNECT didn't launch. Was there a contingency?
"We do have a plan, but we don't anticipate that need at all," Clendenning said. "But we do have a plan of going back and bring the (old) system back up if somehow we're not ready."
Only the old system wasn't made available after CONNECT's failed launch. Instead, thousands were locked out because of technical issues and it took weeks to count all of them. On Dec. 20, after concluding that there were 53 technical issues that had yet to be fixed, the DEO withheld $3 million from Deloitte and began fining it $15,000 a day. Last month, DEO hired another contractor, Capgemini, to review the project at a price of $365,000.
With the project in such disarray, why didn't the DEO revert to the old system?
Panuccio said once CONNECT launched, it was too late.
"(The old system) has so many billion lines of records," Panuccio said last month. "Shut off (the old system), turn on CONNECT. All that data has to come over, and the first time a claimant hits that data and changes it, there's no system then to convert it back."
Panuccio finds no fault with how DEO handled CONNECT's failure. He said it wasn't a "total system failure" but rather "discrete problems."
"In this case, the system did launch, it did begin to process thousands of claims," he said. "The issues that came up, like with the PIN number that we dealt with in the first week or two, there was a technical team that flagged the issue, found out what it is, and then worked it like any technical system."
Panuccio blames Deloitte for all the glitches. He promises a day of reckoning will come.
"Right now we're focused on fixing the issues and serving claimants," he said.
Michael Van Sickler can be reached at mvansickler@tampabay.com.