The Automotive Digital Thread

There is a lot of hype around digital twin and thread. After 30+ years in virtual product development and product lifecycle management, I prefer to talk about specific requirements and solutions in this field. This article introduces use cases for the digital thread in the automotive industry and SENSEI as our solution for integrating data along this thread.

Automotive Digital Thread

 

Product Development

The digital thread starts in early product development phases. Even before we have well defined requirements structures, there are less structured inputs on strategy, product portfolio, positioning and mandatory usage of common parts. But if we follow the RFLP (Requirements – Functional – Logical – Physical) approach that describes the left wing of the systems engineering V model, our requirements structure is connected with functional and logical architecture structures. In the next step we arrive at physical product structures, e.g. mechanical engineering BoMs (bill of material) with items such as CAD models, electrical and electronical BoMs for wire harnesses and PCBs and software BoMs with modules, packages etc..

Many automotive companies are currently focused on building software capabilities for the software-defined vehicle. Please refer to Automotive Software Application Lifecycle Management for ACES (ACES ALM) for more information on this topic.

I like to think of the product structure as a skeleton of the digital twin. The digital thread connects items and structures with each other. The links are as much a part of the digital twin as the items (nodes), i.e. they need to be managed with the same attention regarding documentation, change management, monitoring etc.. In automotive, we are looking at thousands of requirements, functions and parts.

Engineering structures are not complete with RFLP, we also need the respective integration, validation and verification artifacts on the right wing of the V model as well as production planning and manufacturing BoMs.

Digital Thread connecting product structures

This leads us to the following use cases for the digital thread in product development:

• Horizontal and vertical traceability of engineering structures in the systems engineering V model
• Configuration & change management in the engineering structures with versions, revisions, alternatives, options, branches and baselines
• Provide authoritative data for type approval (homologation)
• PLM – ALM integration: Integrating the ACES software development processes with the overall development of the mobility system

The challenge here is not only to manage the volume of links in the digital thread – graph databases can go a long way in this field – it is also the user experience. To scale in a globally distributed engineering organization with hundreds of external suppliers and service providers, we need a robust method for link management. Business process management has the potential to guide users through the complexity and ensure proper link management.

Product Operations

OEMs have always earned substantial money through after-sales. Data is the new oil: the car as a “smart connected product” has a lot of sensors, generating data which is constantly delivered to the cloud / backend. This availability of data is a strong driver to extend the digital thread into the after-sales phase to enable data-driven engineering and to ensure regulatory compliance.

The Chinese regulation for real-time monitoring (RTM) requires the provision of data on battery status, geo position and about 60 further data points – measured every 10 seconds. Notwithstanding the data privacy issues, this is valuable data for battery development and charging infrastructure planning. The UNECE  regulations for software update and cyber security management require a mapping between the engineering information on compatibility of software and E/E hardware versions with the as-maintained configuration of individual cars in the field.

Data-driven engineering

Use cases for the digital thread in product operations comprise:

• Requirements optimization by understanding customer behavior in different markets
• Product portfolio and variance optimization by understanding actual function usage in the fleet
• After-sales optimization by tracing back error codes to design data and requirements
• Regulatory compliance, e.g. China Real-Time Monitoring and UNECE R155 / 156 SUMS CSMS for software update and cyber-security management

Product Interoperability

We are leaving the pure vehicle business and entering the smart city and automotive mobility systems. Sustainable mobility requires an intelligent combination of multiple modes of transportation such as public transportation, bikes and car-sharing. This goes along with the trend towards battery-electric vehicles and the underlying charging infrastructure. In consequence, we are looking at a digital thread that connects multiple digital twins, e.g. of the city (HD maps), the driver (preferences and payment) and the vehicle (renting location and traffic information).

Use cases for the digital thread in product interoperability are identified in the following picture:

Interoperability use cases for digital twin / thread

 

Standards enable the interoperability between domains such as mobility, telco, banking, insurance and utilities. The Digital Twin Consortium (DTC) brings the relevant players together and drives the development and standardization of digital twin technology.

Other relevant standards are STEP and OSLC. The STandard for the Exchange of Product model data (ISO 10303 STEP) provides detailed data models for physical and functional aspects of a product. In our case of automotive, the STEP application protocol AP 242 for managed model-based 3D engineering is of special importance. The Open Services for Lifecycle Collaboration (OSLC) standardize the use of linked data based on web technologies such as REST APIs, URLs and RDF. Please refer to the OSLC Fest 2022 conference including my presentation on “Integration Architecture for the Automotive Digital Thread”.

OSLC Fest 2022 presentation on integration architecture for the automotive digital thread

 

IT Infrastructure

When looking into the product operations and interoperability use cases, it is obvious that a large part of the digital thread relies on cloud and IoT technology. We also see that the digital thread is in fact a complex enterprise knowledge graph, spanning many applications with different semantics, data models and programming interfaces (APIs).

Our customers are currently designing and building solutions for an integrated digital thread. Some are continuing to build point-to-point integrations, but most are looking for a central link management, either in a PLM backbone or in a dedicated link repository.   

To support our customers, we use an integration architecture blueprint which we call SENSEI (Systems Engineering aNd Scalable Enterprise Integration). It organizes multiple integration components such as API management, streaming, messaging, linked data and business process management into a logical integration layer. Please find more information in our SENSEI whitepaper.

SENSEI - NTT DATA systems engineering integration architecture blueprint

Summary

The automotive digital thread spans product development and operations, it even touches other domains such as telco, banking, insurance and utilities in smart city and mobility scenarios. This requires robust, standards-based solutions with good useability. NTT DATA as a systems integrator with comprehensive experience in automotive and other domains offers business consulting and IT implementation services in this field.

Note: this is a cross-post from my LinkedIn article

Business Capabilities for Automotive Software Application Lifecycle Management (ACES ALM)

Business capabilities are the building blocks of an automotive software development organization. They describe "what" that organization does, not "how" it does it. A comprehensive capability model provides automotive companies with a helpful framework to succeed in a software-defined future, being more robust than organizational structures or process models. 

Business capabilities are “a particular ability or capacity that a business may possess or exchange to achieve a specific purpose or outcome”. This definition is from TOGAF (The Open Group Architecture Framwork), indicating the background of this concept in Enterprise Architecture Management (EAM).

Business capabilities are the building blocks of a business, but they are abstracted from the organizational model to become more stable in times of continuous improvement, reorganizations, and agile working models. Business capabilities, therefore, describe what an organization does to achieve set goals, not how it does it. As an example, the capability “Software Development” can be realized with internal or external resources and can use traditional or agile methods.  

There are several use cases and benefits related to business capabilities. In general, they support the alignment between business and IT by providing a common language and structure for all four views of enterprise architecture, with Business and Information Architecture being closer to the business stakeholders where Application and Technology Architecture views are closer to the IT stakeholders.

Four views of Enterprise Architecture

 

Mapping what needs to be done: the benefits of capability models

Business capability models are typically used in IT to structure the overall application landscape into smaller domains that have a high cohesion internally and loose coupling to other domains. This reduces dependencies and makes complexity more manageable.

Another major usage of business capability models are heatmaps, i.e. visualizations of specific aspects using colors such as red / yellow / green. Example heatmaps include business fit, technological fit or application lifecycle status. We have developed a quick check method to determine the maturity level per capability from a business process and an IT point of view.

For comprehensive transformation programs in enterprises such as establishing a new practice for ACES ALM or post-merger integrations, a capability-based transformation approach is suggested. TOGAF and ARCHIMATE provide a metamodel for strategic planning that allows the prioritization and mapping of business capabilities to elements such as processes, IT and other resources, value streams and courses of action. EAM tools such as LeanIX support this approach with features for roadmap planning and impact analysis of alternative scenarios.

NTT DATA’s ACES ALM business capability model

The NTT DATA ACES ALM business capability model is based on the Automotive SPICE process reference model (mainly VDA scope with some additions) to provide structure and traceability for potential process improvement or analysis activities. It also incorporates the ACES ALM drivers as described in the previous article and best practices from the automotive industry.

The model on the top layer 1 is structured into capabilities for

• Systems & Software Engineering
• Deployment & Operations
• Support
• Management

 

 

  NTT DATA ACES ALM Business Capability Model (example heatmap visualization)

On the second layer, 15 business capabilities are described in detail including a list of potential layer 3 capabilities and typical IT tools (commercial-off-the-shelf and open source software COTS / OSS) for the given capability.

Systems & Software Engineering. The primary lifecycle capabilities in the V model for requirements engineering, functional and logical architecture design, software development and integration & test. Please note that for complex systems such as automotive vehicles, these capabilities need to be implemented for multiple levels of the system, e.g. systems of systems (SoS), product, subsystems, components.

Deployment & Operations. The deployment of ACES software to the vehicles while respecting regulations for software update and cyber security management and while using the data collected during operations for data-driven engineering.

Support. The supporting capabilities for the the primary systems & software engineering. Quality assurance is about the objective, independent assurance of the quality of work products. It is also about the resolution of non-conformances including future prevention. Configuration management ensures the integrity of all work products. It is applied over the lifecycle of work products / configuration items. Reporting and collaboration between globally distributed teams are transverse capabilities. Agile is not just a special project management discipline, but also a transformative business capability on its own. Enterprise-wide adoption requires scaled agile frameworks such as SAFe with an integrated set of processes, methods, roles etc. IT infrastructure for ACES ALM includes provision and management of resources such as compute, storage and networking.

Management. Project management capabilities are based on PMI body of knowledge PMBOK and include layer 3 capabilities such as management of scope, time, cost, quality, HR and risk. ACES ALM governance provides leadership, organizational structure, and processes in order to ensure that the ACES ALM capabilities meet the companies’ strategy and objectives.

 

This concludes the overview on business capabilities for ACES ALM. We will publish a whitepaper on this topic soon. Stay tuned! 

Note: this is a cross-post from my LinkedIn article

Drivers of Automotive Software Application Lifecycle Management (ACES ALM)

 Today's vehicles come with a fair amount of software. Automotive software development and also the respective Application Lifecycle Management (ALM) are subject to certain drivers that carmakers should take into account in an autonomous, connected, electric and shared (ACES) ecosystem. By understanding these drivers, automotive companies can develop the right business capabilities and remain competitive. This second post in my series on "ACES ALM" helps with exactly this understanding.

Traditional ALM focuses on the performance of the software development process and the subsequent lifecycle management. Depending on the environment in which the development processes take place, the balance of speed, cost and quality shifts. Where some Silicon Valley startups seemingly do not have to worry about cost as long as they meet aggressive time-to-market goals, automotive suppliers for safety-critical applications are probably more cost-pressed and quality-aware. Drivers for ACES ALM are not different to traditional ALM. However, there are a couple of factors that have to be considered in addition or with different weight. So let us delve into the drivers, which can be broadly categorized along the four factors of ACES engineering business drivers, processes & methods, regulatory requirements, and IT strategy drivers. 

 

ACES engineering business drivers

From the engineer’s point of view, ACES disrupts major components of the product architecture. Powerful domain controllers allow the execution of multiple parallel functions, which in turn requires an unbundling of function software from dedicated ECU hardware. The onboard computers are working closely together with enterprise IT backends, for example to feed the data-hungry algorithms with sensor data.

Separation of hardware and software. Traditionally, tier-1 suppliers deliver ECU (Electronic Control Unit) hardware combined with software as a single package. This leads to complex distributed systems with 50-100+ ECU’s connected via interfaces such as CAN bus or Ethernet. With OEM’s focusing on software and even operating systems as a core competency, new models are created such as OEM-specific OS or sourcing of software only. As an example, please refer to the Volkswagen Group with CARIAD developing the VW.OS (operating system / embedded) and the VW.AC (automotive cloud / enterprise).

Domain-controller architectures. The 50 to more than 100 ECUs in traditional vehicles pose challenges in cost, weight, space, security and complexity in the vehicle network and the development process. That is why the automotive industry is focusing on domain or zonal controller architectures, where a single high-performance computer can potentially replace many ECUs. For ACES ALM, this trend requires integrated design, development and test of several functions – potentially running concurrently with real-time constraints – with related signals, pins, protocols, sensors and actuators. Domain-controller architectures can also lead to the design of custom chips in order to optimize performance per watt and deal with shortages on standard chips (example: Tesla FSD Chip for “full self-driving”), i.e. require PCB design capabilities.

Integrated enterprise & embedded IT. Modern cars are connected to an OEM backend and potentially several clouds for functions such as infotainment, navigation, emergency services or over-the-air updates. The user experience depends on an integrated system from the human-machine interface (HMI) in the car over the connectivity to the backend implementation of the service in the enterprise IT.

ACES ALM needs to support integrated development of embedded software with related enterprise IT, e.g. by providing development environments with relevant development instances and test data of the enterprise backends.

Data-driven engineering. Modern automotive functions such as Advanced Driver Assistance Systems (ADAS) use machine learning, e.g. for image recognition and the interpretation of scenarios. At the same time, cars are expected to provide sensor data in order to feed the enterprise processes and also various data-driven business models (e.g. real-time traffic information). Tesla’s cars are constantly providing data on usage patterns, issues, preferences etc. Analytics is then applied to direct and prioritize the agile work within the company.

The ACES ALM environment must support this “big loop” of data collection / ingest, storage and management in the Petabyte range as well as the application of AI tools for the data scientists working on analytics, learning, development and re-integration into the car.

Processes and Methods

The growing importance and complexity of ACES software requires disciplined software engineering and configuration management practices as well as their integration into the existing product development process, for example using a model-based systems engineering approach. (Scaled) agile practices might need to be integrated into the traditional phase-gate PDP.

Model-based systems engineering (MBSE). The model-based paradigm (as in model-based systems engineering; MBSE) is becoming increasingly popular as growing complexity requires additional levels of abstraction.

In ACES ALM, this means e.g. code generation from models such as MATLAB Simulink or SysML. These models need to be managed including trace links to requirements and generated code. To realize the full benefits of this approach, the models should be executable for simulation in the verification and validation processes and integrated into the test automation toolchain.

Process and tool baselines. Especially for ADAS functions, the product liability risks for automotive companies are very high. One approach for risk mitigation is the ability to reproduce the used development practices for a given point in time, i.e. the ACES ALM environment should be able to persist baselines of a configuration of processes, methods and tools.

Variant and configuration management and Product Line Engineering. Managing various versions, branches and builds for different OS platforms is a core function of traditional ALM.

ACES ALM also needs to manage code / parameters with respect to the vehicle configurations, including compatibility management as described in the discussion of “software update management”. A vehicle configuration might be determined by model, drive, extras, production site, target country etc.

ISO 26580 focuses on feature-based Systems and Software Product Line Engineering (PLE) as a specialization of general PLE according to ISO 26550. Both documents describe concepts, methods and tools to increase re-use of engineering artifacts across variants.

Scaled Agile. The complexity of automotive applications such as automated driving and connected services requires large teams of a few hundred to a few thousand people. Scaled agile frameworks such as SAFe support the scaling of basic agile principles to this size. Especially the project management and team collaboration modules in ACES ALM must support scaled agile methods.

Regulatory requirements

Many regulations need to be considered for the development of ACES software. These include, for example, Automotive SPICE, UNECE SUMS & CSMS and ISO 26262 Functional Safety. At the same time, new regulatory requirements are also emerging. Meeting these regulatory requirements is critical for the type approval process (homologation), i.e. there are potentially different regulations for each market such as EU, China, US.

The ACES ALM environment needs to ensure the reliable provision of all current, relevant requirements for a given development scope. It further needs to support the traceable implementation and validation of these requirements and maintain an audit trail.

To support assessments or audits, an explicit process model with linked work products for each activity is desirable. Workflow management features make the process model executable, thus ensuring compliance.

Automotive SPICE. ASPICE maturity level 2+ has been required by OEMs for supplier qualification for a long time. By entering the field of software development themselves, ASPICE becomes an important tool for process quality also for OEMs. Explicit modeling of the processes as well as tools that produce and manage the required work products help during an assessment.

The ACES ALM environment also needs to support the vertical and horizontal traceability requirements of ASPICE, i.e. bidirectional traceability and consistency between requirements, design, code, tests etc.

Lifecycle management / Software Update Management. Customers expect state-of-the-art infotainment functions in a car as well as constant fixing of bugs and security improvements. While software can change every month, the hardware in the car has a life cycle of over 10 years and a development process of over 3 years on top of that. OEMs need to manage these different lifecycles, but they also want to enable new business models based on software-defined functions and meet regulatory requirements for type approval such as UNECE R156 SUMS (SW Update Management System) and the upcoming ISO 24089 for SW Update Engineering. For this, OEM’s need the ability to update the embedded software in the after-sales phase – ideally “over the air” (OTA) without having to call drivers into the workshop at great expense.

The ACES ALM environment needs to support the secure distribution and installation of updates in the car. This process needs to deal with compatibility of the SW update with related hardware and software in the car. For UNECE SUMS, the mapping between SW IDs and Regulation IDs is required (e.g. in the form of the so-called RxSWIN ID).

Cybersecurity. Modern connected cars have several new potential security vulnerabilities, e.g. through USB, WIFI, Bluetooth and the OTA update process. The UNECE R155 CSMS regulation requires setting-up a Cybersecurity Management System with risk management to be established throughout development, production and operations (-> (virtual) Security Operations Center SOC). With ASPICE CS, there is an extension of ASPICE for cybersecurity available. Another relevant standard in this area is ISO/SAE 21434 „Road vehicles – Cybersecurity engineering”.

The ACES ALM environment needs to support the risk identification and assessment as well as traceability to related mitigation activities. It also needs to support implementation of secure coding guidelines and security testing incl. penetration tests according to the security concepts defined within the organization.

Functional Safety.  ISO 26262 is a standard for safety-critical electrical / electronic systems in vehicles. It requires a comprehensive lifecycle and quality management. Some of the requirements are similar to Automotive SPICE, but the scope of ISO 26262 also includes production and operations and the rigidity of the requirements depend on the ASIL (Automotive Safety Integrity Level).

The ACES ALM environment needs to support the execution of ISO 26262-specific processes such as HARA (Hazard Analysis and Risk Assessment) and FMEA (Failure Mode and Effects Analysis) and the creation of related information / documents depending on the ASIL. The tools and toolchains also need to be certified regarding “fitness for purpose”.

IT strategy drivers

With embedded software becoming connected and data-driven, general IT trends such as cloud and AI apply in our field of ACES software. The IT strategy transforms these trends not only for enterprise IT, but increasingly also for the ACES IT.

Cloud-native architectures. Cloud-native is not a new or ACES-specific driver for ALM, but it is amplified through connected vehicles which are constantly sending and receiving data. This requires scalability and connectivity that is best provided with cloud-native architectures.

The ACES ALM environment needs to support efficient development in the cloud-native architecture. This might not be the case with traditional toolchains, which are targeted towards platforms such as Linux or Windows.

Developer experience. Embedded software development is tough: developers must deal with Assembler or C code under real-time constraints, limited computing resources and overwhelming regulatory requirements, but at the same time are expected to implement data-driven cloud-native architectures.

Given the scarcity of talent, the ACES ALM environment should guide developers through the processes and provide integrated toolchains with automation applied where possible.

This concludes the overview on drivers of ACES ALM. The next article in this series will identify the business capabilities that are required to deal with these drivers. Stay tuned! 

 

Note: this is a cross-post from my LinkedIn article

 

Automotive Software Application Lifecycle Management for ACES (ACES ALM)

Software eats the car: The four letters “ACES” stand for the major trends in the automotive industry: Vehicles are increasingly Autonomous, Connected, Electric, and Shared. Professional, automotive-grade software engineering is the basis for the software-defined car. At NTT DATA, we provide a comprehensive analysis of the business capabilities required to develop and maintain automotive software over the lifecycle.

Why did software develop such an appetite for our mobile vehicles - and why is it worthwhile for OEMs to serve a solid meal rather than fast food? Leading-edge application lifecycle management is essential in the automotive industry today. Therefore, in this article series, I will define ACES ALM before detailing the challenges in this area and introducing the ACES ALM business capability model in subsequent posts. If you are involved in shaping processes, methods, tools or organization (PMTO) for automotive software, this is for you. If you are in a hurry and don’t want to wait for the article series, please contact me directly.

What is ACES ALM?

Software for autonomous, connected, electric, or shared functionalities is not your typical enterprise application. An accounting system or common office software has no brakes that can fail. On the highway, it is difficult to simply open the task manager to shut down an annoying process. In-Car software is safety-critical, real-time embedded software running on electronic control units (ECU) in the vehicle. The following picture provides an overview of ACES software:

ACES is short for Autonomous / Automated, Connected, Electric, and Shared. The acronym ACES describes four major strategic pillars of automotive companies, especially OEMs. All four elements are part of the overall digitalization trend and require major innovations in IT hardware and software.

ALM is short for Application Lifecycle Management. As a process, ALM is about managing software from its initial conception, through the development and testing phase and ongoing support to its end of life. ALM tools provide support for activities such as software planning, requirements management, architecture design, software development, build, integration, test, deployment, operations and maintenance.

Combine the two and you get ACES ALM: ALM for ACES software.

And why is software eating my car?

More than 10 years ago, entrepreneur and investor Marc Andreessen famously published an essay in  Wall Street Journal on “Why software is eating the world”. The automotive industry has come to realize the truth behind statements such as “Software is also eating much of the value chain of industries that are widely viewed as primarily existing in the physical world.”

As a case in point, Volkswagen Group in 2019 has established CARIAD to increase the workshare of VW in ACES software from currently 10 - 15 % to 60 % – with currently more than 5.000 people working on it. While traditional OEMs are shifting resources to the software world, other players such as Tesla (my article on product development at Tesla) and Alphabet-owned Waymo start on a green field with a Silicon Valley mindset.

The goal of both approaches is to combine a physical product with digital software. The success of this "phygital" convergence thus becomes a decisive competitive factor. Both established automakers and (not so) new players must therefore wisely combine the ingredients for a reliable, functional and safe ACES ALM so that vehicles as a finished product are subsequently desirable for the end customer.

Challenges? Opportunities!

To make ACES ALM a success, automakers must therefore keep an eye on a wide variety of aspects and fields, which can be divided into four clusters.

• First, there are the ACES engineering business drivers, i.e. the fundamental technical interaction between software and hardware and their mutually dependent development including data-driven engineering.
• Second, the right processes and methods are needed to help with such development - from model-based systems engineering (MBSE) to variant and configuration management to scaled agile.
• Third, this technical-methodological complex is encompassed by regulatory requirements for hardware and software in vehicles, be it Automotive SPICE, cybersecurity or functional safety.
• Fourth and finally, there is alignment with the company-wide IT strategy.

I will discuss all four of these areas of challenge - as well as the opportunities they present - in the next article in this series. Stay tuned!

Note: this is a cross-post from my LinkeIn article

Product development at Tesla: Software

 

In December 2021, I published an initial article on Product development at Tesla. In this follow-up, we will look deeper into a domain which is a clear strength of Tesla: software. Although this attribution usually refers to software as part of the product, we will also look into the enterprise software which helps building these products.

Enterprise software @ Tesla

A CleanTechnica article cited Elon Musk as follows:

“We are not dependent on enterprise software. For those who understand, that is a very big deal. My hat off to the great work of the internal applications team. They write the nervous system, the operating system, the Tesla company operating system. Extremely fundamental!”

Most OEM’s use commercial-off-the-shelf software for ERP, CRM / Dealer Management etc. from vendors such as SAP, Oracle or Salesforce. But Tesla decided to develop custom solutions. Why? And how did they do it?

Jay Vijayan was CIO at Tesla from 2012 – 2016. Before that phase, he worked on ERP and CRM at Oracle and VMware. Even with this background, his first task at Tesla still seems like a Mission Impossible: building an ERP in three months. Leaving Tesla in 2016, he founded Tekion Corp. which focuses on enterprise software for the automotive industry, currently an SaaS solution for sales, finance and insurance.

Based on these facts, some speculations on the “Why” and “How” come to my mind: Elon Musk had a strong vision of Tesla as a software-driven company with high vertical integration. He probably also has foreseen the need for flexibility in the enterprise software due to major business model extensions such as charging networks, batteries, CO2 emission credits etc. On the other hand, Jay Vijayan knew how expensive and inflexible traditional enterprise software can be. I can only guess that some of his lessons learned at Tesla found their way into Tekion.

Product software @ Tesla : AI and FSD

On December 24, 2021, Tesla introduced Software V11.0. It brings new features such as the light show, games, adjustable subwoofer punch or blind spot visualization with the live camera view. That is certainly an added value for Tesla owners(read: justifies a higher price), and the over-the-air delivery of these updates is still a benchmark in the industry.

FSD (full self driving) is certainly a controversial feature, but it is essential for the data-driven business model of Tesla. Software plays a major role in improving this feature, Lex Fridman pointed out the importance of AI after the Tesla AI Day in August 2021.

Here are a few highlights from the Tesla AI Day 2021:

• In-house developed toolchain for working in  the “vector space” for debugging of the neural networks, labeling (automatic & manual) and neural net compilation. As Tesla is mainly relying on camera images as sensors, this approach allows to label objects once in 3D and project the labels on many image views.
• In-house data labeling team with 1.000 persons
• The neural network is fully retrained every 1-2 weeks with 10.000 GPU’s , which puts Tesla among the top 5 supercomputer companies in the world
• An AI evaluation infrastructure with 3.000+ FSD computers as hardware-in-the-loop in 3 datacenters + cloud with custom job scheduling & device management software
• The Unreal engine is used for basic physical simulation. Simulation is “a video game with Autopilot as the player“. Again, this is a controversial bet and a more deterministic approach is needed for safety. But this simulated environment with the 3D objects is great for labeling.
• Scalable scenario generation with hand-made, procedural and ML-based scenarios. This also includes a toolchain to replicate scenarios and environments anywhere a Tesla has driven.

Tesla seems to be pretty successful at DevOps with continuous integration and deployment for the full stack from ECU basic software over apps in the car incl. HMI and apps on a phone to backend software. Interestingly, the career website has 15 categories of jobs, but Engineering&IT is combined into one. The job descriptions reveal some details into software development at Tesla:

Excerpt of requirements for a “Full Stack Engineer”:

• Python development experience (2.x, 3.x)
• Experience with web technologies and web frameworks (React/Node/Dash/PHP/CSS, etc.)
• Experience with workflow management platforms (Airflow or similar)
• Basic knowledge of stream processing systems (Kafka, RabbitMQ, or similar)
• Experience with scalable map-reduce data processing preferred (Spark, Hadoop, or similar)
• Experience with at least one of MySQL/Postgres/NoSQL databases
• Experience with containerization (Docker or similar)
• Experience with build systems, automation, continuous integration (Rundeck, Jenkins or similar) 

I hope that this post helps to shed some light on the software development approach of Tesla. Please feel free to comment with any additional insights and links you can share.

Note: I published this article originally on LinkedIn.