Unlock the potential of your MDF files by processing them with our new cloud-based microservice

Regardless of the source, the thousands of sensor signals from radar and central control units captured by the measurement equipment are often saved in MDF. This allows for efficient storage of large amounts of data, but even files with a measurement duration of just a few minutes already take up several gigabytes in size. When we consider the data collected from an entire fleet, this adds up to multiple petabytes.

Analyzing and simulating these data sets, improving the features, and simulating the new version with locally installed software and resources on a PC is almost impossible. Based on this input the code is updated and simulated with the same test data. But due to the high compute demand and the limited availability of dedicated simulation clusters the feature of each software developer is simulated together with code changes from other developers. This results in feedback loops of several weeks and limited traceability in case of bugs. In addition these on-premises compute clusters quickly reach their limits with increasing data volumes. So, what is the alternative?

The cloud, with its almost limitless scaling possibilities in terms of storage and computing power, can solve this and provide incredible efficiency gains in terms of time and costs. State-of-the-art, cloud-based analysis, and simulation tools are much faster for these automotive use cases. Firstly the files must be uploaded in the cloud. In the next step, the meta data needs to be extracted from the MDF files so that the files can be found again. The signals they contain must then be converted to a big data format like Parquet so that it can be read by these tools at all. In the final steps the data can then be leveraged in downstream uses cases like analytics and simulation in a fraction of the time compared to local on-premises solutions.

The data engineers and cloud architects requested to develop such a cloud-based solution first have to find, evaluate, and define a suitable open-source library as well as design, implement, test, maintain, and update their own work-around for their company. This process can take several months. The first individual and customized data processing solutions, developed, and maintained at a high cost, exist nowadays, but there is no standard service available yet.

We at Bosch Engineering have dealt with this challenge extensively in multiple projects and have come up with an easy-to-integrate microservice: the automotive data transformer. It is a cloud-native measurement data processor for data engineers and cloud architects that allows them to convert MDF files with easy API requests even without several years of experience in automotive calibration.

Our service is fast, reliable, secure, and integrates seamlessly into a centralized data lake based on Amazon Web Services S3. Instead of processing the individual signals sequentially, the AWS Lambda functions are used, which we can scale as required and thus parallelise the signal processing. The Automotive Data Transformer is therefore up to 40 times faster than local applications. Valuable time that you as a software developer can use to gain insights from the analysis and, for example, improve your algorithms for adaptive cruise control much earlier for the next test drive.

The automotive data transformer offers two main features:

• Meta data export – different levels of meta data are extracted, and you get them directly as a JSON response or can save them in JSON format.

• Signal data processing – all or specifically requested signals are transformed and load to the defined file path in the requested cloud storage in either Parquet (most efficient for storage), CSV, or JSON (least efficient) format.

To further simplify your data engineer’s life, creating a data ingest pipeline, we have already implemented resampling and interpolation functionalities. This helps to reduce the file size if a higher sample rate is not needed and unifying the time base for easier further analysis. Otherwise, additional steps in the data pipeline must be integrated. For cloud architects integrating the automotive data transformer as part of their cloud-based analytics tool, we also support requesting only specific time frames to further reduce the output file size.

Since MDF files contain sensitive development data, the customer doesn’t need to provide the files to us. Instead, they can provide access to the automotive data transformer, which can be revoked at any time.