In December 2021, I published an initial article on Product development at Tesla. In this follow-up, we will look deeper into a domain which is a clear strength of Tesla: software. Although this attribution usually refers to software as part of the product, we will also look into the enterprise software which helps building these products.
Enterprise software @ Tesla
A CleanTechnica article cited Elon Musk as follows:
“We are not dependent on enterprise software. For those who understand, that is a very big deal. My hat off to the great work of the internal applications team. They write the nervous system, the operating system, the Tesla company operating system. Extremely fundamental!”
Most OEM’s use commercial-off-the-shelf software for ERP, CRM / Dealer Management etc. from vendors such as SAP, Oracle or Salesforce. But Tesla decided to develop custom solutions. Why? And how did they do it?
Jay Vijayan was CIO at Tesla from 2012 – 2016. Before that phase, he worked on ERP and CRM at Oracle and VMware. Even with this background, his first task at Tesla still seems like a Mission Impossible: building an ERP in three months. Leaving Tesla in 2016, he founded Tekion Corp. which focuses on enterprise software for the automotive industry, currently an SaaS solution for sales, finance and insurance.
Based on these facts, some speculations on the “Why” and “How” come to my mind: Elon Musk had a strong vision of Tesla as a software-driven company with high vertical integration. He probably also has foreseen the need for flexibility in the enterprise software due to major business model extensions such as charging networks, batteries, CO2 emission credits etc. On the other hand, Jay Vijayan knew how expensive and inflexible traditional enterprise software can be. I can only guess that some of his lessons learned at Tesla found their way into Tekion.
Product software @ Tesla : AI and FSD
On December 24, 2021, Tesla introduced Software V11.0. It brings new features such as the light show, games, adjustable subwoofer punch or blind spot visualization with the live camera view. That is certainly an added value for Tesla owners(read: justifies a higher price), and the over-the-air delivery of these updates is still a benchmark in the industry.
FSD (full self driving) is certainly a controversial feature, but it is essential for the data-driven business model of Tesla. Software plays a major role in improving this feature, Lex Fridman pointed out the importance of AI after the Tesla AI Day in August 2021.
Here are a few highlights from the Tesla AI Day 2021:
- In-house developed toolchain for working in the “vector space” for debugging of the neural networks, labeling (automatic & manual) and neural net compilation. As Tesla is mainly relying on camera images as sensors, this approach allows to label objects once in 3D and project the labels on many image views.
- In-house data labeling team with 1.000 persons
- The neural network is fully retrained every 1-2 weeks with 10.000 GPU’s , which puts Tesla among the top 5 supercomputer companies in the world
- An AI evaluation infrastructure with 3.000+ FSD computers as hardware-in-the-loop in 3 datacenters + cloud with custom job scheduling & device management software
- The Unreal engine is used for basic physical simulation. Simulation is “a video game with Autopilot as the player“. Again, this is a controversial bet and a more deterministic approach is needed for safety. But this simulated environment with the 3D objects is great for labeling.
- Scalable scenario generation with hand-made, procedural and ML-based scenarios. This also includes a toolchain to replicate scenarios and environments anywhere a Tesla has driven.
Tesla seems to be pretty successful at DevOps with continuous integration and deployment for the full stack from ECU basic software over apps in the car incl. HMI and apps on a phone to backend software. Interestingly, the career website has 15 categories of jobs, but Engineering&IT is combined into one. The job descriptions reveal some details into software development at Tesla:
Excerpt of requirements for a “Full Stack Engineer”:
- Python development experience (2.x, 3.x)
- Experience with web technologies and web frameworks (React/Node/Dash/PHP/CSS, etc.)
- Experience with workflow management platforms (Airflow or similar)
- Basic knowledge of stream processing systems (Kafka, RabbitMQ, or similar)
- Experience with scalable map-reduce data processing preferred (Spark, Hadoop, or similar)
- Experience with at least one of MySQL/Postgres/NoSQL databases
- Experience with containerization (Docker or similar)
- Experience with build systems, automation, continuous integration (Rundeck, Jenkins or similar)
I hope that this post helps to shed some light on the software development approach of Tesla. Please feel free to comment with any additional insights and links you can share.
Note: I published this article originally on LinkedIn.
