Friday, March 18, 2022

Drivers of Automotive Software Application Lifecycle Management (ACES ALM)

 Today's vehicles come with a fair amount of software. Automotive software development and also the respective Application Lifecycle Management (ALM) are subject to certain drivers that carmakers should take into account in an autonomous, connected, electric and shared (ACES) ecosystem. By understanding these drivers, automotive companies can develop the right business capabilities and remain competitive. This second post in my series on "ACES ALM" helps with exactly this understanding.

Traditional ALM focuses on the performance of the software development process and the subsequent lifecycle management. Depending on the environment in which the development processes take place, the balance of speed, cost and quality shifts. Where some Silicon Valley startups seemingly do not have to worry about cost as long as they meet aggressive time-to-market goals, automotive suppliers for safety-critical applications are probably more cost-pressed and quality-aware. Drivers for ACES ALM are not different to traditional ALM. However, there are a couple of factors that have to be considered in addition or with different weight. So let us delve into the drivers, which can be broadly categorized along the four factors of ACES engineering business drivers, processes & methods, regulatory requirements, and IT strategy drivers. 



 

ACES engineering business drivers

From the engineer’s point of view, ACES disrupts major components of the product architecture. Powerful domain controllers allow the execution of multiple parallel functions, which in turn requires an unbundling of function software from dedicated ECU hardware. The onboard computers are working closely together with enterprise IT backends, for example to feed the data-hungry algorithms with sensor data.

Separation of hardware and software. Traditionally, tier-1 suppliers deliver ECU (Electronic Control Unit) hardware combined with software as a single package. This leads to complex distributed systems with 50-100+ ECU’s connected via interfaces such as CAN bus or Ethernet. With OEM’s focusing on software and even operating systems as a core competency, new models are created such as OEM-specific OS or sourcing of software only. As an example, please refer to the Volkswagen Group with CARIAD developing the VW.OS (operating system / embedded) and the VW.AC (automotive cloud / enterprise).

Domain-controller architectures. The 50 to more than 100 ECUs in traditional vehicles pose challenges in cost, weight, space, security and complexity in the vehicle network and the development process. That is why the automotive industry is focusing on domain or zonal controller architectures, where a single high-performance computer can potentially replace many ECUs. For ACES ALM, this trend requires integrated design, development and test of several functions – potentially running concurrently with real-time constraints – with related signals, pins, protocols, sensors and actuators. Domain-controller architectures can also lead to the design of custom chips in order to optimize performance per watt and deal with shortages on standard chips (example: Tesla FSD Chip for “full self-driving”), i.e. require PCB design capabilities.

Integrated enterprise & embedded IT. Modern cars are connected to an OEM backend and potentially several clouds for functions such as infotainment, navigation, emergency services or over-the-air updates. The user experience depends on an integrated system from the human-machine interface (HMI) in the car over the connectivity to the backend implementation of the service in the enterprise IT.

ACES ALM needs to support integrated development of embedded software with related enterprise IT, e.g. by providing development environments with relevant development instances and test data of the enterprise backends.

Data-driven engineering. Modern automotive functions such as Advanced Driver Assistance Systems (ADAS) use machine learning, e.g. for image recognition and the interpretation of scenarios. At the same time, cars are expected to provide sensor data in order to feed the enterprise processes and also various data-driven business models (e.g. real-time traffic information). Tesla’s cars are constantly providing data on usage patterns, issues, preferences etc. Analytics is then applied to direct and prioritize the agile work within the company.

The ACES ALM environment must support this “big loop” of data collection / ingest, storage and management in the Petabyte range as well as the application of AI tools for the data scientists working on analytics, learning, development and re-integration into the car.

Processes and Methods

The growing importance and complexity of ACES software requires disciplined software engineering and configuration management practices as well as their integration into the existing product development process, for example using a model-based systems engineering approach. (Scaled) agile practices might need to be integrated into the traditional phase-gate PDP.

Model-based systems engineering (MBSE). The model-based paradigm (as in model-based systems engineering; MBSE) is becoming increasingly popular as growing complexity requires additional levels of abstraction.

In ACES ALM, this means e.g. code generation from models such as MATLAB Simulink or SysML. These models need to be managed including trace links to requirements and generated code. To realize the full benefits of this approach, the models should be executable for simulation in the verification and validation processes and integrated into the test automation toolchain.

Process and tool baselines. Especially for ADAS functions, the product liability risks for automotive companies are very high. One approach for risk mitigation is the ability to reproduce the used development practices for a given point in time, i.e. the ACES ALM environment should be able to persist baselines of a configuration of processes, methods and tools.

Variant and configuration management and Product Line Engineering. Managing various versions, branches and builds for different OS platforms is a core function of traditional ALM.

ACES ALM also needs to manage code / parameters with respect to the vehicle configurations, including compatibility management as described in the discussion of “software update management”. A vehicle configuration might be determined by model, drive, extras, production site, target country etc.

ISO 26580 focuses on feature-based Systems and Software Product Line Engineering (PLE) as a specialization of general PLE according to ISO 26550. Both documents describe concepts, methods and tools to increase re-use of engineering artifacts across variants.

Scaled Agile. The complexity of automotive applications such as automated driving and connected services requires large teams of a few hundred to a few thousand people. Scaled agile frameworks such as SAFe support the scaling of basic agile principles to this size. Especially the project management and team collaboration modules in ACES ALM must support scaled agile methods.

Regulatory requirements

Many regulations need to be considered for the development of ACES software. These include, for example, Automotive SPICE, UNECE SUMS & CSMS and ISO 26262 Functional Safety. At the same time, new regulatory requirements are also emerging. Meeting these regulatory requirements is critical for the type approval process (homologation), i.e. there are potentially different regulations for each market such as EU, China, US.

The ACES ALM environment needs to ensure the reliable provision of all current, relevant requirements for a given development scope. It further needs to support the traceable implementation and validation of these requirements and maintain an audit trail.

To support assessments or audits, an explicit process model with linked work products for each activity is desirable. Workflow management features make the process model executable, thus ensuring compliance.

Automotive SPICE. ASPICE maturity level 2+ has been required by OEMs for supplier qualification for a long time. By entering the field of software development themselves, ASPICE becomes an important tool for process quality also for OEMs. Explicit modeling of the processes as well as tools that produce and manage the required work products help during an assessment.

The ACES ALM environment also needs to support the vertical and horizontal traceability requirements of ASPICE, i.e. bidirectional traceability and consistency between requirements, design, code, tests etc.

Lifecycle management / Software Update Management. Customers expect state-of-the-art infotainment functions in a car as well as constant fixing of bugs and security improvements. While software can change every month, the hardware in the car has a life cycle of over 10 years and a development process of over 3 years on top of that. OEMs need to manage these different lifecycles, but they also want to enable new business models based on software-defined functions and meet regulatory requirements for type approval such as UNECE R156 SUMS (SW Update Management System) and the upcoming ISO 24089 for SW Update Engineering. For this, OEM’s need the ability to update the embedded software in the after-sales phase – ideally “over the air” (OTA) without having to call drivers into the workshop at great expense.

The ACES ALM environment needs to support the secure distribution and installation of updates in the car. This process needs to deal with compatibility of the SW update with related hardware and software in the car. For UNECE SUMS, the mapping between SW IDs and Regulation IDs is required (e.g. in the form of the so-called RxSWIN ID).

Cybersecurity. Modern connected cars have several new potential security vulnerabilities, e.g. through USB, WIFI, Bluetooth and the OTA update process. The UNECE R155 CSMS regulation requires setting-up a Cybersecurity Management System with risk management to be established throughout development, production and operations (-> (virtual) Security Operations Center SOC). With ASPICE CS, there is an extension of ASPICE for cybersecurity available. Another relevant standard in this area is ISO/SAE 21434 „Road vehicles – Cybersecurity engineering”.

The ACES ALM environment needs to support the risk identification and assessment as well as traceability to related mitigation activities. It also needs to support implementation of secure coding guidelines and security testing incl. penetration tests according to the security concepts defined within the organization.

Functional Safety.  ISO 26262 is a standard for safety-critical electrical / electronic systems in vehicles. It requires a comprehensive lifecycle and quality management. Some of the requirements are similar to Automotive SPICE, but the scope of ISO 26262 also includes production and operations and the rigidity of the requirements depend on the ASIL (Automotive Safety Integrity Level).

The ACES ALM environment needs to support the execution of ISO 26262-specific processes such as HARA (Hazard Analysis and Risk Assessment) and FMEA (Failure Mode and Effects Analysis) and the creation of related information / documents depending on the ASIL. The tools and toolchains also need to be certified regarding “fitness for purpose”.

IT strategy drivers

With embedded software becoming connected and data-driven, general IT trends such as cloud and AI apply in our field of ACES software. The IT strategy transforms these trends not only for enterprise IT, but increasingly also for the ACES IT.

Cloud-native architectures. Cloud-native is not a new or ACES-specific driver for ALM, but it is amplified through connected vehicles which are constantly sending and receiving data. This requires scalability and connectivity that is best provided with cloud-native architectures.

The ACES ALM environment needs to support efficient development in the cloud-native architecture. This might not be the case with traditional toolchains, which are targeted towards platforms such as Linux or Windows.

Developer experience. Embedded software development is tough: developers must deal with Assembler or C code under real-time constraints, limited computing resources and overwhelming regulatory requirements, but at the same time are expected to implement data-driven cloud-native architectures.

Given the scarcity of talent, the ACES ALM environment should guide developers through the processes and provide integrated toolchains with automation applied where possible.


This concludes the overview on drivers of ACES ALM. The next article in this series will identify the business capabilities that are required to deal with these drivers. Stay tuned! 

 

Note: this is a cross-post from my LinkedIn article

 

Thursday, March 3, 2022

Automotive Software Application Lifecycle Management for ACES (ACES ALM)

Software eats the car: The four letters “ACES” stand for the major trends in the automotive industry: Vehicles are increasingly Autonomous, Connected, Electric, and Shared. Professional, automotive-grade software engineering is the basis for the software-defined car. At NTT DATA, we provide a comprehensive analysis of the business capabilities required to develop and maintain automotive software over the lifecycle.

Why did software develop such an appetite for our mobile vehicles - and why is it worthwhile for OEMs to serve a solid meal rather than fast food? Leading-edge application lifecycle management is essential in the automotive industry today. Therefore, in this article series, I will define ACES ALM before detailing the challenges in this area and introducing the ACES ALM business capability model in subsequent posts. If you are involved in shaping processes, methods, tools or organization (PMTO) for automotive software, this is for you. If you are in a hurry and don’t want to wait for the article series, please contact me directly.

What is ACES ALM?

Software for autonomous, connected, electric, or shared functionalities is not your typical enterprise application. An accounting system or common office software has no brakes that can fail. On the highway, it is difficult to simply open the task manager to shut down an annoying process. In-Car software is safety-critical, real-time embedded software running on electronic control units (ECU) in the vehicle. The following picture provides an overview of ACES software:


ACES is short for Autonomous / Automated, Connected, Electric, and Shared. The acronym ACES describes four major strategic pillars of automotive companies, especially OEMs. All four elements are part of the overall digitalization trend and require major innovations in IT hardware and software.

ALM is short for Application Lifecycle Management. As a process, ALM is about managing software from its initial conception, through the development and testing phase and ongoing support to its end of life. ALM tools provide support for activities such as software planning, requirements management, architecture design, software development, build, integration, test, deployment, operations and maintenance.

Combine the two and you get ACES ALM: ALM for ACES software.

And why is software eating my car?

More than 10 years ago, entrepreneur and investor Marc Andreessen famously published an essay in  Wall Street Journal on “Why software is eating the world”. The automotive industry has come to realize the truth behind statements such as “Software is also eating much of the value chain of industries that are widely viewed as primarily existing in the physical world.”

As a case in point, Volkswagen Group in 2019 has established CARIAD to increase the workshare of VW in ACES software from currently 10 - 15 % to 60 % – with currently more than 5.000 people working on it. While traditional OEMs are shifting resources to the software world, other players such as Tesla (my article on product development at Tesla) and Alphabet-owned Waymo start on a green field with a Silicon Valley mindset.

The goal of both approaches is to combine a physical product with digital software. The success of this "phygital" convergence thus becomes a decisive competitive factor. Both established automakers and (not so) new players must therefore wisely combine the ingredients for a reliable, functional and safe ACES ALM so that vehicles as a finished product are subsequently desirable for the end customer.

Challenges? Opportunities!

To make ACES ALM a success, automakers must therefore keep an eye on a wide variety of aspects and fields, which can be divided into four clusters.

  • First, there are the ACES engineering business drivers, i.e. the fundamental technical interaction between software and hardware and their mutually dependent development including data-driven engineering.
  • Second, the right processes and methods are needed to help with such development - from model-based systems engineering (MBSE) to variant and configuration management to scaled agile.
  • Third, this technical-methodological complex is encompassed by regulatory requirements for hardware and software in vehicles, be it Automotive SPICE, cybersecurity or functional safety.
  • Fourth and finally, there is alignment with the company-wide IT strategy.

I will discuss all four of these areas of challenge - as well as the opportunities they present - in the next article in this series. Stay tuned!


Note: this is a cross-post from my LinkeIn article

Sunday, January 9, 2022

Product development at Tesla: Software

 

In December 2021, I published an initial article on Product development at Tesla. In this follow-up, we will look deeper into a domain which is a clear strength of Tesla: software. Although this attribution usually refers to software as part of the product, we will also look into the enterprise software which helps building these products.

Enterprise software @ Tesla

A CleanTechnica article cited Elon Musk as follows:

“We are not dependent on enterprise software. For those who understand, that is a very big deal. My hat off to the great work of the internal applications team. They write the nervous system, the operating system, the Tesla company operating system. Extremely fundamental!”

Most OEM’s use commercial-off-the-shelf software for ERP, CRM / Dealer Management etc. from vendors such as SAP, Oracle or Salesforce. But Tesla decided to develop custom solutions. Why? And how did they do it?

Jay Vijayan was CIO at Tesla from 2012 – 2016. Before that phase, he worked on ERP and CRM at Oracle and VMware. Even with this background, his first task at Tesla still seems like a Mission Impossible: building an ERP in three months. Leaving Tesla in 2016, he founded Tekion Corp. which focuses on enterprise software for the automotive industry, currently an SaaS solution for sales, finance and insurance.

Based on these facts, some speculations on the “Why” and “How” come to my mind: Elon Musk had a strong vision of Tesla as a software-driven company with high vertical integration. He probably also has foreseen the need for flexibility in the enterprise software due to major business model extensions such as charging networks, batteries, CO2 emission credits etc. On the other hand, Jay Vijayan knew how expensive and inflexible traditional enterprise software can be. I can only guess that some of his lessons learned at Tesla found their way into Tekion.

Product software @ Tesla : AI and FSD

On December 24, 2021, Tesla introduced Software V11.0. It brings new features such as the light show, games, adjustable subwoofer punch or blind spot visualization with the live camera view. That is certainly an added value for Tesla owners(read: justifies a higher price), and the over-the-air delivery of these updates is still a benchmark in the industry.

FSD (full self driving) is certainly a controversial feature, but it is essential for the data-driven business model of Tesla. Software plays a major role in improving this feature, Lex Fridman pointed out the importance of AI after the Tesla AI Day in August 2021.

Here are a few highlights from the Tesla AI Day 2021:

  • In-house developed toolchain for working in  the “vector space” for debugging of the neural networks, labeling (automatic & manual) and neural net compilation. As Tesla is mainly relying on camera images as sensors, this approach allows to label objects once in 3D and project the labels on many image views.
  • In-house data labeling team with 1.000 persons
  • The neural network is fully retrained every 1-2 weeks with 10.000 GPU’s , which puts Tesla among the top 5 supercomputer companies in the world
  • An AI evaluation infrastructure with 3.000+ FSD computers as hardware-in-the-loop in 3 datacenters + cloud with custom job scheduling & device management software
  • The Unreal engine is used for basic physical simulation. Simulation is “a video game with Autopilot as the player“. Again, this is a controversial bet and a more deterministic approach is needed for safety. But this simulated environment with the 3D objects is great for labeling.
  • Scalable scenario generation with hand-made, procedural and ML-based scenarios. This also includes a toolchain to replicate scenarios and environments anywhere a Tesla has driven.

Tesla seems to be pretty successful at DevOps with continuous integration and deployment for the full stack from ECU basic software over apps in the car incl. HMI and apps on a phone to backend software. Interestingly, the career website has 15 categories of jobs, but Engineering&IT is combined into one. The job descriptions reveal some details into software development at Tesla:

Excerpt of requirements for a “Full Stack Engineer”:

  • Python development experience (2.x, 3.x)
  • Experience with web technologies and web frameworks (React/Node/Dash/PHP/CSS, etc.)
  • Experience with workflow management platforms (Airflow or similar)
  • Basic knowledge of stream processing systems (Kafka, RabbitMQ, or similar)
  • Experience with scalable map-reduce data processing preferred (Spark, Hadoop, or similar)
  • Experience with at least one of MySQL/Postgres/NoSQL databases
  • Experience with containerization (Docker or similar)
  • Experience with build systems, automation, continuous integration (Rundeck, Jenkins or similar) 

I hope that this post helps to shed some light on the software development approach of Tesla. Please feel free to comment with any additional insights and links you can share.

Note: I published this article originally on LinkedIn.

Monday, December 27, 2021

Product development at Tesla

 

 

In December 2021, I read the special edition of Germany‘s Automobilwoche on „The Tesla Principle“ and was hoping to find some answers on the question "How does product development, systems engineering and PLM work at Tesla?". This article summarizes my findings including feedback I got on the original post. If you have additional insights to share, please feel free to comment here or in the original post.

The articles cover sales, marketing, production, Elon Musk etc. - but not product development. Even a Google search doesn‘t reveal much but some job postings from Tesla asking for skills in Dassault Systemes 3DEXPERIENCE, CATIA, ENOVIA, DELMIA, SIMULIA and Solidworks applications. The related press release from Dassault Systemes on this is from September 2010...

Although Tesla is very secretive about their way of doing things, the articles give some insights into product development at Tesla. Here are my takeaways:

1. Tesla designed their products from the ground up without restictions from existing platforms, production technology etc.. They especially had a Silicon Valley mindset with focus on software and E/E architecture. But they still needed to design and build a mechanical car around the software.

2. They have created an integrated System of Systems – from battery cells over charging infrastructure and powerwalls to actual power production. I am not sure if this is the result of great systems engineering, but it certainly turned out well.

3. Tesla has a culture of innovation rather than cost-cutting

4. Having a tech-savvy CEO and flat hierarchies certainly helps. According to Elon Musk „pace of innovation is all that matters in the long run“


Agile@Tesla

And then, there ist the agile way of working at Tesla that not only applies to product development, but to the whole company including suppliers. As pointed out by Oliver Wörl and documented by Joe Justice in several videos, one cannot underestimate the importance of agile at Tesla:

  • Agile-native company with iterations of 3-6 hours and empowered people
  • An error culture that is required to operate at this speed
  • Data-driven processes - e.g. all cars providing data constantly on usage patterns, issues, preferences etc. Analytics is then applied to direct and prioritize the agile work.
  • Focus on building cars, almost as if product development is not separate from manufacturing but part of the iterative processes. Small iterations are deployed immediately to production (CI / CD = continuous integration / deployment), i.e. a pretty direct integration between product development and manufacturing almost in an "ETO = engineering to order" model as opposed to the mass production model in other OEM's. This can imply individual homologation testing of each car at the end of the production process.
  • No meetings, just online chat. And no Powerpoint presentations.

Note: I published this article originally on LinkedIn.