Software-defined vehicle research: 40 arenas, hundreds of suppliers, and rapidly-improved software autonomy
The overall architecture of software-defined vehicles can be divided into four layers:
(1) The hardware platform, heterogeneous distributed hardware architecture;
(2) The system software layer, including hypervisor, system kernels, POSIX, AUTOSAR, etc.;
(3) The application middleware and development framework, including functional software, SOA, etc.;
(4) The application software layer, including smart cockpit HMI, ADAS/AD algorithms, connectivity algorithms, cloud platforms, etc.
In a broad sense, an operating system refers to the middleware based on kernel OS, including system software layer (kernel, hypervisor, middleware), functional software layer (common functional modules and related middleware) and APIs. In a narrow sense, an operating system mainly means the vehicle control OS and automotive OS at the bottom of the system kernel.
The operating system in the narrow sense is the cornerstone of building intelligent connected vehicles, while the operating system in the broad sense is the bridge for application development
Microkernel, only to achieve basic task management, memory management, process communication, etc., and other drivers are defined in the user side to achieve, the current commonly used intelligent driving operating systems are mainly Linux, QNX and Other RTOS (such as free RTOS, VxWorks, etc.). The open-source microkernel seL4 based on the third-generation microkernel technology has attracted more and more attention from domestic automakers and technology companies. Li Auto, NIO, Lotus Cars, Horizon Robotics, Xiaomi and other technology companies have joined the seL4 Foundation to promote the development of seL4 microkernel.
The development of smart cars still faces the dilemma – “lack of chips” and “absence of automotive operating systems”. The former has received enough attention, while the latter still needs to be broken through. At present, foreign vendors QNX (Blackberry), Linux (open source), and Android (Google) are the core players of the narrowly defined automotive operating systems. For example, in terms of smart cockpit systems, the combination of QNX+Android is the mainstream solution chosen by domestic vendors. China-based Huawei HarmonyOS and Powered by AliOS have made certain breakthroughs, but their self-sufficiency rates are still low. In the field of intelligent driving systems, QNX occupies an absolute monopoly position.
Under the background of autonomous controllability, real-time automotive operating systems signalize an important development direction for filling the gap in automotive operating systems. Many domestic technology companies, including Huawei, ZTE, Baidu Apollo and Banma SmartDrive, are seeking breakthroughs in the field of real-time microkernel operating systems.
In a broad sense, an operating system is a bridge for developers to facilitate the development of application algorithms, and it is a development platform including an operating system in the narrow sense, and middleware, etc.
We divide operating systems in a broad sense into several categories:
- General operating system for autonomous driving
- General operating system for smart cockpit
- General operating system for vehicle-cloud integration
Typical general operating systems for autonomous driving include the AUTOSAR CP and AP integrated solutions from iSoft Infrastructure Software, Baidu Apollo AI Open Source Platform, Autoware ROS2.0 Open Source Platform, Huawei AOS, TTTech MotionWise, EMOS from Enjoy Move Technology, ICVOS from AICC, ZF Middleware and so on.
Based on the mass production of AUTOSAR CP, iSoft Infrastructure Software provides AUTOSAR CP+AP integrated solutions for security domain and high-performance computing domain. With its cloud system, it attains intelligent connectivity. The integrated solutions of iSoft Infrastructure Software can be applied to intelligent cockpit domain, vehicle control system domain and ADAS/AD domain. By standardizing the interfaces and architectures of different operating systems, underlying hardware and protocol software, it forges service-oriented software architectures. As for intelligent cockpit domain and ADAS/AD domain, iSoft Infrastructure Software is developing the corresponding operating system kernels to fully lay out automotive basic software platforms.
In addition, autonomous driving SoC chip vendors are not satisfied with just providing hardware, but also seeking a share in the autonomous driving ecosystem to increase barriers to entry. Nvidia has launched DriveWorks open source platform for autonomous driving, and Horizon Robotics has unveiled – TogetherOS?, a real-time automotive operating system with a secure microkernel architecture.
Not to be outdone, OEMs are considering developing their own autonomous driving operating systems, especially the first batch of emerging automakers utilize AUTOSAR Classic Platform +DDS to build autonomous driving operating systems (development platforms). As the technology ecology continues to mature, emerging automakers and OEMs in transition are making efforts to develop autonomous driving operating systems by themselves.
-Tesla.OS (Version) is developed by Tesla itself based on underlying Linux. In terms of functional software, it supports PyTorch, a deep learning programming framework.
-VW.OS, based on Linux+AUTOSAR Adaptive, features decoupling of software and I/O functions as well as SOA.
-Toyota’s Woven Planet Group is integrating Apex.OS SDK into its own vehicle development platform, called Arene. The Apex SDK will handle safety-critical applications and aims to speed up autonomous software development and ultimately bring it to production vehicles.
-Li Auto is developing its own Li OS and plans to create a cross-domain intelligent operating system platform. Li OS targets autonomous driving, and will be connected with intelligent vehicle control and intelligent cockpits in the future.
In terms of automotive middleware (AUTOSAR, ROS2, Cyber RT), different autonomous driving operating system vendors have different options. For example, Baidu Apollo uses the self-developed CyberRT, Autoware adopts ROS2, and other vendors welcome AUTOSAR Classic and AUTOSAR Adaptive. In recent years, Apex.AI OS (compatible with ROS 2 API) has been widely supported by some European OEMs and Tier1 suppliers. Apex.AI has been invested by many leading enterprises in the automotive industry, such as Continental, Toyota, ZF, Jaguar Land Rover, Volvo, Hella and Daimler Truck.
In the field of autonomous driving, the functions of middleware involve communication, module upgrade, task scheduling and actuation management, but its main function lies in communication. The introduction of communication middleware (DDS, SOME/IP, MQTT) can help developers improve efficiency. At present, communication middleware mainly includes SOME/IP, DDS and MQTT. At present, SOME/IP and DDS are two kinds of communication middleware that are most used in autonomous driving.
SOME/IP communication middleware
SOME/IP middleware providers include AUTOSAR toolchain vendors, like foreign companies such as Vector, ETAS, EB, etc., and domestic companies represented by iSoft Infrastructure Software, Jingwei Hirain Technologies, etc. The GENIVI Alliance provides an open source version of SOME/IP.
DDS communication middleware
The commercial closed-source communication middleware is mainly represented by RTI Connext DDS, which accounts for more than 80% of market share. Xpeng is the first enterprise in China that applies Connext DDS to autonomous vehicles. HoloSAR, the autonomous driving middleware of HoloMatic Technology, also integrates RTI Connext DDS.
Other open source communication middleware includes OPEN DDS, FAST DDS, Cyclone DDS, etc. In recent years, a number of communication middleware products have emerged, including iceoryx from Bosch ETAS, Swift from Greenstone, and MotionWise Cyclone DDS. In addition, the new version of AUTOSAR Adaptive bolster DDS in terms of communication management, and the AP products developed by iSoft Infrastructure Software endorse the integration of third-party DDS.
MQTT communication middleware
It can be used in low-bandwidth and unreliable network scenarios to provide data transmission and monitoring of remote devices based on cloud platforms with the IoT communication protocol MQTT.
Autonomous driving toolchain software is becoming an arena
An autonomous driving system consists of perception, planning and decision-making. The key to algorithm research and development mainly lies in data collection, perception model training, simulation tests and real vehicle tests, etc. Perception and planning constitute the main part of data generation, including data collection, data cleaning and data annotation.
Through a series of toolchains, a complete full-chain data closed-loop development process for autonomous driving is formed, featuring a fully closed loop and self-growth. This poses an important technical barrier for major OEMs and Tier1 suppliers in developing autonomous driving systems. The software and data services involved include:
AI deep neural network learning software
The software/framework involved mainly includes TensorFlow/ PyTorch/ OpenCV/ TensorRT, etc. In China, Baidu has launched Paddle Paddle and Huawei has released MindSpore.
Transformer neural network models can better realize modeling in the space-time dimension, and has been applied in production autonomous vehicles on a large scale:
Transformer is one of the core modules in Tesla’s FSD system algorithm. After image features are extracted, the combination of AI neural network algorithms such as Transformer, CNN and 3D convolution completes cross-time image fusion, so as to output 3D information based on 2D image formation.
Domestically, Haomo.AI has proposed to effectively integrate Transformer with massive data. The MANA data intelligence system of Haomo.AI adopts Transformer to fuse vision and LiDAR data at the underlying layer, and then accomplishes deep perception of space, time and sensors.
Xpeng G9 has deployed Transformer: after continuous optimization, the 122% Orin-X computing power required by the dynamic XNet has been reduced to 9% now.
Autonomous driving data collection and automatic annotation system
According to IDC, by 2025, the market size of China’s artificial intelligence data collection and annotation services will hit RMB12.34 billion, mainly driven by the data collection and annotation of autonomous vehicles. Thus, there is demand for data collection, processing, storage, training software and tools.
Xnet, Xpeng’s “next-generation perception architecture”, can generate a “HD map” in real time when it is combined with all sensors in the vehicle. Through the dynamic XNet, the speed and intention of dynamic objects can be recognized more accurately. XNet requires massive data collection, annotation, training and deployment. Xpeng has independently developed an automatic annotation system.
However, many other automakers may cooperate with partners in data collection and annotation. Typical vendors include Speechocean (a global AI training data service provider), Huawei Octopus (data collection, training and simulation services), Vector (CANape, a data collection tool), Appen China (AI data collection and annotation services), ExceedData (data collection and annotation platforms), etc.
Autonomous driving training data set
For autonomous driving with deep learning as the main method, training data sets are the most critical. Algorithms are similar (in particular, many of them are open source), so it is impossible to tell which is the best. Deep learning data sets are related to the final results, so that the former plays a decisive role. The wider the coverage of training data sets, the finer the annotation, the more accurate the classification, the more types, the better the final autonomous driving performance.
Many self-driving companies, including Argoverse of Volkswagen-Ford joint venture Argo, Waymo’s Open, Baidu’s ApolloScape, Nvidia (PilotNet), Honda (H3D), Aptiv(nuScense) have all disclosed some of their training validation datasets, some provides open-source download link. Now the most influential ones are KITTI, Waymo Open and Aptiv nuScenes.
There are few datasets with local characteristics in China, mainly including Huawei “ONCE”, the vehicle-road collaborative autonomous driving dataset “DAIR-V2X”, Jinqiao “JICD” dataset, the large-scale driving behavior dataset DBNet jointly released by Xiamen University and Shanghai Jiaotong University, Xi’an Jiaotong University and Chang’an University jointly constructed and disclosed DADA dataset.
Autonomous driving data storage and computing center (cloud services)
Data storage and management only embody the basic capabilities of cloud services. The demand of automakers for cloud services has shifted from IaaS and PaaS to SaaS (Software as a Service). Cloud service providers are expected to provide or integrate a unified toolchain, open up upstream and downstream links, and help automakers quickly go through the data closed-loop chain.
Xpeng and Alibaba Cloud have jointly built Fuyao, the largest intelligent computing center for autonomous driving in China, which shortens the time for single-machine full-precision training from 276 days to 32 days. If 80 machines are running simultaneously, it only takes 11 hours, with the processing speed accelerated by 602 times.
Large-scale simulation testing and data training
A simulation system includes a simulation scenario library, a simulation test platform and simulation evaluation, which complement each other.
Here, we take the autonomous driving simulation scenario library as an example. In September 2022, Deqing County, Huzhou City teamed up with Alibaba Cloud and Haomo.AI to release “China’s first large-scale autonomous driving scenario library based on CVIS”, which uses real traffic data and meets data compliance requirements. It will further accelerate the maturity of autonomous driving in China and the coordinative development of vehicles, roads and cloud. In addition, CATARC, CAERI, Tencent (TAD Sim), Baidu Apollo, etc. offer autonomous driving scenario libraries.
