Do CSA Scores Matter?

CSA scores do matter.

Why?

Because significant people think they matter. People who can help you. And, people who can hurt you. While we would like to live in a society based on facts, it’s often perceptions that carry the day.

Read the rest of the article here……

Mark G. Gardner, Chief Executive Officer.  Avatar Management Services, Inc

Fleet Owner: DOT IG faults CSA data quality, interventions

DOT IG faults CSA data quality, interventions

Avery Vise Fleet Owner

In an audit report that focuses primarily on the limited issue of the timeliness and completeness of state- and carrier-reported data, the U.S. Dept. of Transportation’s Office of Inspector General (DOT IG) said that the Federal Motor Carrier Safety Administration has not fully implemented various planned steps to improve the data underlying the Compliance, Safety, Accountability (CSA) program.

The audit report released on March 7 also notes that only 10 states had fully implemented CSA enforcement interventions due to delays in receiving and being trained on new software needed for assessing and monitoring interventions. FMCSA expects to release the software by May 2015.

Given the limited implementation of enforcement interventions to date, the DOT IG did not assess the effectiveness of the interventions. In a footnote, the DOT IG report notes that the National Transportation Safety Board had recommended an audit of FMCSA’s use of focused compliance reviews but did not comment on whether it would be conducting such an audit.

Unlike a recent critical Government Accountability Office study, the DOT IG doesn’t address at all CSA’s effectiveness in identifying high-risk carriers. And unlike with the GAO report, which prompted FMCSA to respond with its own analysis defending CSA, the agency agreed with all of the DOT IG’s recommendations. The DOT IG recommendations are:

• Issue updated DataQs guidance to clarify how data challenges should be reviewed and to provide additional measures to ensure that data challenges are closed consistently;
• Implement the process for deactivating USDOT numbers when carriers do not submit required census data, as outlined in a policy issued by FMCSA in November 2013;
• Develop a comprehensive plan to fully implement CSA enforcement interventions in the remaining 40 states and the District of Columbia. The plan should include an estimated completion date and milestones for releasing Sentri software, developing and delivering training, and using the enforcement interventions, the DOT IG said;
• Update the Carrier Safety Measurement System (CSMS) requirements document to specify all sources of CSMS data – including each of the Motor Carrier Management Information System fields used – and to fully describe CSMS validation procedures;
• Develop and implement a process for managing CSMS system documentation that includes a central file for validation records and testing results; and
• Develop and implement a configuration management policy that includes documentation of system changes and associated testing for CSMS.

Although the DOT IG report was not as critical of FMCSA’s implementation of CSA as the GAO report, the American Trucking Assns. argued that it highlighted significant problems with CSA, including some that ATA had identified in a white paper late last year.

“The Inspector General’s report confirms what industry stakeholders, independent researchers and other government watchdogs have found: there continue to be significant flaws in the data FMCSA is using to evaluate and score carriers under CSA,” said ATA Executive Vice President Dave Osiecki. “ATA continues to support the oversight mission and safety goals of CSA – but FMCSA must acknowledge the program’s many problems – and commit to addressing them.”

ATA cited the DOT IG’s findings that FMCSA (1) had taken “limited action” to address previously identified flaws; (2) had failed to get 40 states and DC to implement CSA interventions; (3) had not followed standard technology validation and testing; and (4) failed to push states to be consistent in correcting reams of inaccurate data.

“The audit found that while FMCSA claims to have enough data to evaluate 40% of the industry – 13% of those companies report not owning or operating any trucks,” Osiecki said.  “Serious inaccuracies like this affect the scores of everyone scored under CSA – since carriers are compared to one another.”

Motor carriers themselves are not blameless, however, Osiecki acknowledged, noting the DOT IG’s finding that many motor carriers need to improve their reporting of operational and exposure data, such as the number of trucks they operate.

ATA said it was disappointed that the DOT IG took FMCSA’s self-assessment of its State Safety Data Quality (SSDQ) system at face value. The SSDQ scores states’ ability to upload timely and accurate data – the very data that drives the SMS.

“While the SSDQ indicates states are performing better than they have in the past, the IG failed to examine under what circumstances a state might obtain a ‘good’ rating,” Osiecki said. “This is a critical link in CSA data quality that deserves greater attention.”

A December 2012 ATA analysis found that FMCSA’s thresholds are so broad that a state could receive a “good” rating even if it underreported crashes by several thousand. In a large state like Texas, for example, the state could obtain a “good” rating for uploading between 12,551 and 25,163 accidents – a range of 12,612 accidents.

Also, FMCSA estimates of total expected crashes is based on the number of known fatal crashes, ATA notes. But ATA argues that fatal crashes may not be a reliable predictor of non-fatal crashes. Thorough examination of state police accidents reports remains the only way to accurately measure state accident reporting to MCMIS, ATA says.

Fleet Owner: Lack of data undermines CSA program, GAO says

Avery Vise Fleet Owner

Safety Measurement System doesn’t capture enough safety-related data on carriers to establish a link between carriers’ SMS scores and their crash risk or to reliably compare carriers to one another, the Government Accountability Office concluded in a report released Feb. 3. In its long-awaited review of the Compliance, Safety, Accountability program, GAO said that even as an internal agency tool for targeting SMS, the program lacked the precision to be as effective as it should be, and it questioned both the way FMCSA presents SMS data to the public and FMCSA’s plan to use SMS scores to determine carriers’ fitness.

“Most carriers operate few vehicles and are inspected infrequently, providing insufficient information to produce reliable SMS scores,” GAO found in the report, which was ordered by Congress in a recent appropriations act. “FMCSA acknowledges that violation rates are less precise for carriers with little information, but its methods do not fully address this limitation.”

Even though FMCSA requires a minimum level of information to generate an SMS score, that minimum still doesn’t generate enough information to produce a reliable SMS score, GAO said. “FMCSA identified many carriers as high risk that were not later involved in a crash, potentially causing FMCSA to miss opportunities to intervene with carriers that were involved in crashes.”

For SMS to be a more effective enforcement tool, GAO said FMCSA should score only carriers with more information than those scored today, allowing it to better identify high-risk carriers likely to be involved in crashes. To test the impact of this idea, instead of the few inspections now required to qualify for SMS scoring, GAO considered an alternative that eliminated the use of safety event groups and set at least 20 inspections or 20 vehicles, depending on the CSA BASIC, as the minimum threshold for a CSA score.

Using this approach, GAO said that 67.1% of high-risk carriers would have had a crash during the period it reviewed as opposed to 39% under FMCSA’s current approach. “This illustrative approach involves trade-offs; it would assign SMS scores to fewer carriers, but these scores would generally be more reliable and thus more useful in targeting FMCSA’s scarce resources.”

Carrier fitness and public awareness

If SMS isn’t precise enough to be used as an effective internal tool, GAO raised further questions about using them for safety fitness determinations as FMCSA currently plans. It said that “basing a carrier’s safety fitness determination on limited performance data may misrepresent the safety status of carriers, particularly those without sufficient data from which to reliably draw such a conclusion.”

GAO also questioned how FMCSA presented SMS data on individual carriers, although it stopped short of suggesting that the agency pull the data altogether.  The watchdog agency noted a contradiction between FMCSA’s official disclaimer about SMS scores being intended for agency and law enforcement purposes and the agency’s own statement that SMS provides stakeholders with valuable information to make safety-based business decisions.

FMCSA itself has released a mobile phone application – SaferBus – to provide safety information, including SMS scores, to consumers selecting a bus company, GAO noted. Meanwhile, the Dept. of Defense has written SMS scores into its minimum safety criteria for selecting carriers of hazardous munitions, and multiple stakeholders have reported that entities such as insurers, freight shippers and brokers, and others use SMS scores. “Given such uses, it is important that any information about SMS scores make clear to users, including FMCSA, the purpose of the scores, their precision, and the context around how they are calculated.”

Industry reaction

Given that criticism of CSA and SMS has grown over the past three years, it’s not surprising that the trucking industry welcomed the GAO report.

“The GAO’s review of FMCSA’s Compliance, Safety, Accountability program was comprehensive, thoughtful and balanced,” said Bill Graves, president of the American Trucking Assns.  “While ATA has long supported CSA’s objectives, we can’t help but agree with GAO’s findings that the scores produced by the program don’t present an accurate or precise assessment of the safety of many carriers.”

“Given GAO’s findings, FMCSA should remove all carriers’ scores from public view,” said Dave Osiecki, ATA executive vice president and chief of national advocacy. “Since scores are so often unreliable, third parties are prone to making erroneous judgments based on inaccurate data, an inequity that can only be solved in the near term by removing the scores from public view.” Osiecki also said that that “it would clearly be improper” for FMCSA to proceed with the rulemaking on carrier fitness determinations based on the data until FMCSA corrects problems that GAO identified.

The Alliance for Safe, Efficient, and Competitive Truck Transportation (ASECTT) – a group that has challenged FMCSA in court over CSA and SMS – said that “the GAO’s own words are a powerful endorsement of many of the arguments [we have] been making for years.” Several years ago, ASECTT’s legal challenges resulted in changes in how FMCSA presents SMS data on its website, and the coalition is still in litigation with FMCSA over SMS and CSA. GAO confirmed SMS’s bias against smaller carriers, ASECTT said.

ASECTT said it agrees with GAO’s recommendation against using SMS as it is today for determining carrier fitness, but the group “continues to also believe that until the system has been fully validated for individual carriers, SMS data should not be presented to the shipping public. The FMCSA should live up to its federal duty to determine which carriers are safe to operate on the highways and shut down the unsafe operators.”

Meanwhile, FMCSA tried to put GAO’s findings in perspective, saying that it uses a “strategic, data-driven approach” to identifying and prioritizing high-risk carriers.

“While we are always looking for ways to improve our safety oversight methods, and will carefully consider the GAO’s latest recommendations, research shows that CSA is already more effective at identifying motor carriers with a greater risk of crashing than the system we replaced in 2010,” FMCSA said. The agency added that under the Obama Administration it has tripled the number of unsafe bus and truck companies taken off the road.