Report recommends sweeping changes to CSA scoring system

Jul 8, 2017 | Industry News

James Jaillet  June 27, 2017


FMCSA says it will respond to the National Academies report within 120 days.

National Academies of Science researchers have issued a Congressionally mandated report recommending that the U.S. Department of Transportation overhaul its Compliance, Safety, Accountability carrier rating system.

The report says DOT needs to make CSA’s Safety Measurement System more fair and accurate in assessing motor carriers’ safety risk, and that data used to create the rankings is in need of “immediate attention.”

Key recommendations from the report, made public Tuesday, include:

  • Reconfiguring the SMS statistical model (the percentile ranking used to target carriers for intervention) with an “item response theory” (IRT) model that more accurately targets at-risk carriers;
  • Making the scoring system more transparent and easier for carriers to replicate and understand; and
  • Departing from using relative metrics as the sole means for targeting carriers.

The NAS urged further study of the impact of the public display of SMS rankings.  Researchers recommend that the Federal Motor Carrier Safety Administration better collaborate with state partners and other data providers to collect more data and higher quality data, specifically related to crash reports and to carriers’ operations (miles traveled, number of power units, etc.).

NAS will provide the roughly 130-page report, “Improving Motor Carrier Safety Measurement,” to FMCSA and to Congress, which called for the report in the 2015 FAST Act highway bill. The act also pulled the SMS’ BASIC percentile rankings from public view.

The law stipulated that the NAS must issue recommendations on how FMCSA can fix the data and methodology issues that have plagued CSA since its 2011 onset — and that FMCSA adopt the recommendations — before the SMS can be made public again.

It’s unclear yet how the agency intends to act on the report’s recommendations or the timeline on which it plans to adopt any CSA reforms. FMCSA confirmed it has received the report and says it will issue a response to Congress within the 120 days allotted by the FAST Act. An agency spokesperson says it is “reviewing the findings.”

The agency has already undertaken an effort to improve crash reporting through its Post-Accident Report Review Subcommittee of its Motor Carrier Safety Advisory Committee. That group is tasked with making recommendations to improve crash-report utility and standardization.

The NAS report on CSA and its underlying data reaches an oft-heard conclusion: While the premise behind CSA’s SMS is sound, FMCSA’s execution of the program was flawed from the ground up. CSA relies on in some ways inadequate data, NAS researchers conclude, which is then funneled into a scoring system that has not been “sufficiently empirically validated.”

FMCSA withdraws rule reforming carrier safety rating system


Findings in other ways reiterate conclusions reached by Overdrive’s multi-year CSA’s Data Trail series, one of many analyses of CSA’s flaws and their impact on carriers. The report notes the chief criticisms of the CSA program: Some BASICs lack correlation with crash risk, data insufficiency, use of relative rankings, use of non-fault or nonpreventable crashes, state variations in inspections and violations, lack of consistency in violation coding, a lack of transparency of the SMS algorithm and the public availability of SMS rankings.

“Conceptually, SMS is structured reasonably,” the NAS report concludes.  However, researchers add, “too much of the detail is ad hoc,” referring to FMCSA’s execution of the program.

The report doesn’t go into depth about its recommendations for a new statistical model to replace the SMS model, but it touts the IRT model as one that has been proven to work in fields such as health care and education to achieve outcomes similar to those sought by the SMS. IRT models are used in health care to assess the quality of care given to patients and in education to assess teachers’ performance. In both fields, IRT models are used to target poor performers for intervention, just as FMCSA does with the SMS rankings.

NAS’ principal recommendation is that FMCSA develop an IRT model over the next two years and then implement it in place of the SMS “if it [demonstrates] to perform well in identifying motor carriers for alerts.” Transitioning to an IRT model, the study’s authors believe, would in part help soothe some of these problems. It would make violation weighting more data-focused (less reliant on severity weights), provide better transparency, account for holes in data and more.

Other NAS recommendations seek to tackle SMS flaws, too. FMCSA needs to work more closely with states to homogenize data reporting and collection, researchers recommend. The agency also should find new sources to assess carriers’ safety risk, such as analyzing driver turnover rates and driver pay methods.

FMCSA needs to build a more “user-friendly version” of the Motor Carrier Management Information System (the data well that feeds CSA) and make its scoring methodology more accessible, NAS says. Such changes would make it easier for carriers to understand how they’re being scored and learn how to improve their scores and, hence, their safety practices.

Before going public with a revamped SMS, the agency should also further analyze the ramifications of publicly displaying the SMS rankings. Shippers and brokers have to varying degrees used the rankings to deny carriers business, while insurers in some cases used the system to charge higher premiums.

What’s your view on the issues identified with CSA? If you’re reading on a smartphone, tap the image to call and leave us a message to weigh in with your story. We’ll round up responses in a special mailbag podcast. Alternately, drop a comment below. If you’re on a desktop, call 530-408-6423. Make sure to tell us your name and state of residence.

Researchers also recommend FMCSA find a way to implement an absolute scoring metric into the SMS algorithm to select carriers for intervention, rather than relying solely on relative rankings. Relative ratings can be a moving target for carriers trying to boost their safety culture.

NAS, established by Congress in 1863 as a private institution, intends to provide nonpartisan and objective research to lawmakers and regulators to help steer policy decisions. Congress funneled the funding for the new study through FMCSA.


About the Author



Doug Marcello The LEAD: Ignoring your data, fearing discovery in litigation, can be fatal to your company. The data is there. Plaintiffs will get it. You must proactively cumulate and analyze it to promote safety and proactively prepare to defend any potential suit....

Roadcheck is May 16 – Download the Inspection Guide

What can you expect duging any roadside inspectin - but especially during the annual Roadcheck blitz?  The inspectin guidelines are published and consistently used throughout North America. Download the guide here.

Preparing your drivers for deposition

Doug Marcello Your driver is to be deposed. They are to be thrust into a foreign world for which they have neither any experience nor concept. And out the other side of the process comes their “sworn testimony”, written in cement, as to the accident, training,...

Driver Survey Results

by David Hollis For the typical trucker who is in the driver's seat of their truck for most of their waking hours while on the road, having that seat be as comfortable as possible is a big deal. That's just one of the things we found out in our recent survey of what...

All 3G ELDs Do Not Comply with FMCSA Regs by 12/31/2022

Verizon is the last network to shut down their 3G network, the sunset date is December 31, 2022.  The other major networks shut down their 3G earlier this year. If you have an ELD system that relies on 3G to operate, it will no longer be compliant.  The carrier has 8...