This week, for the first time ever, the National Highway Traffic Safety Administration released data on crashes involving cars equipped with advanced driver-assist systems and automated driving technology. A lot of headlines — including The Verge’s — focused on the number of Tesla vehicles that crashed, which is understandable because Tesla had a lot of crashes.

But the numbers themselves don’t tell us the whole story. In fact, they don’t really tell us much of any story at all. Not yet.

That’s because we’re still missing a lot of key details, like the number of vehicle miles driven or the prominence of advanced driver-assist technology in each manufacturer’s fleet. NHTSA receives the data through varying sources, including customer complaints and different telematics systems. Depending upon the reporting mechanism, different information may be available to report. Companies were allowed to withhold certain details they consider to be “confidential business information.”

“From a researcher’s standpoint, it would, of course, be wonderful if this data set not only was more standardized but also more complete with fewer redactions,” said J. Christian Gerdes, professor of mechanical engineering at Stanford University and co-director of the Center for Automotive Research. “Practically speaking, however, that is asking a lot.”

In many ways, Tesla is an outlier, not just because it produced the most crash reports of all the manufacturers — 273 crashes, with three injuries and five deaths — but also because of the method it uses to report these numbers.

Unlike most automakers, Tesla knows exactly which vehicles were using Autopilot at the time of a crash. Its vehicles have cellular and Wi-Fi connectivity that automatically report this information back to the manufacturer when a crash occurs. The majority of Tesla’s reports were sourced from telematics, whereas other companies only reported getting a handful of such reports.

Two automakers, Tesla and Honda, also provided a handful of crash reports that preceded NHTSA’s standing general order, which was issued in June 2021. Under the order, companies are required to report crashes within 10 days of the order’s issuance. But Tesla provided several reports of crashes that stretched as far back as 2019, including three of its five fatal crashes.

Honda also generated a lot of reports (90 crashes, the second most). But a closer read of the details seems to indicate that the vast majority of the reported crashes involve situations where a driver-assistance system would not be expected to help, like going over a speed bump, parking, or getting T-boned by another car.

“It would be very helpful to know how many of the Tesla accidents involved situations that would have been a collision regardless of the assistance system,” Gerdes said.

On the autonomous vehicle side, there is a little more uniformity. That’s because autonomous vehicles are not owned by customers but by companies as part of AV fleets. Data collection is pretty similar across the various companies, giving us a more standardized data set. But even so, there are still a lot of nuances.

Waymo, for example, said that one-third of its 60 reported crashes took place when the vehicle was in manual mode. NHTSA’s order requires companies to document crashes when ADAS and automated technologies were in use within 30 seconds of impact. A spokesperson for Waymo said the “crashes we reported in manual fit within the 30-second window of the reporting requirement.”

Another key difference between AV crashes and Level 2 driver-assist crashes is where the vehicle was most likely to be damaged — which can lead to some assumptions about who was at fault. In AV crashes, the vehicle was most likely to be rear-ended by other vehicles, whereas driver-assist vehicles were most likely to be damaged in the front. Also, there were no serious injuries or fatalities reported by the AV companies during the 10-month period.

In a briefing with reporters, NHTSA’s top official stressed over and over the limitations of the data and warned against comparing companies or drawing too many conclusions. “I want to stress to you that this is just nine months of data,” Steven Cliff, chief administrator at NHTSA, said. “And I would advise caution before attempting to draw conclusions based only on the data that we’re releasing. In fact, the data alone may raise more questions than they answer.”

But that likely won’t stop many people from doing exactly that, especially with regard to a company as polarizing as Tesla. Within hours of the report’s release, Tesla critics were hosting Twitter Space discussions and live YouTube chats about what it means for Elon Musk’s long-delayed effort to develop autonomous vehicle technology. (Spoiler alert: they think it’s bad.)

Of course, the report is bigger than just Tesla. Over 100 companies were subject to the new reporting requirements, including Ford, General Motors, Toyota, Waymo, Cruise, and others. NHTSA’s standing general order was worded in a way to apply broadly to the industry in an effort to get as much data as it could. And the agency has promised to update the numbers every month as new crash statistics roll in.

“The data released today is a good start, but it doesn’t provide an apples-to-apples comparison of advanced vehicle safety,” National Transportation Safety Board Chair Jennifer Homendy said in a statement. “What NHTSA provided was a ‘fruit bowl’ of data with a lot of caveats, making it difficult for the public and experts alike to understand what is being reported. Independent analysis of the data is key to identifying any safety gaps and potential remedies.” 

But given the wide disparity between each company’s abilities to obtain and verify crash reports, the data is likely to remain unstandardized for quite some time.

“Standardization would be premature,” said Raj Rajkumar, a professor of computer and electrical engineering at Carnegie Mellon University. “Just the ‘catch-all’ policy where carmakers are required to report all ADAS incidents captures the core information.”