Self Driving Cars

Books

• Engineering Autonomous Vehicle and Robots, by Shaoshan Liu et al, new edition 2020
• Creating Autonomous Vehicle Systems, by Shaoshan Liu et al, new edition 2020
• Computational Geometry: Algorithms and Applications, Mark de Berg et al (2018)
• Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art, J. Janai et al (2021)
  • Summaries of all papers in survey

Courses

• Cognitive Robotics, 16.412J/6.836J MIT Open Courseware (Spring 2016)
• Self-Driving Cars with ROS and Autoware by APEX.ai, 20 classes YouTube Course (2020). This is a system-level intro to
• Apollo autonomous driving introductory course, requires Chinese phone number for registration. Uses Apollo Auto, the open source project from Baidu for self driving cars.
  • Started with Apollo ROS, now using CyberRT
• Introductions
  • Self Driving Cars An Introduction to Vehicle Autonomy Stack
• Cyrill Stachniss, Uni Bonn: Self Driving Cars (2023)

Workshops

• CVPR 2022
  • Workshop on Autonomous Driving (2022)
• CVPR 2023
  • End-to-End Autonomous Driving (2023), youtube playlist
  • Vision-Centric Autonomous Driving (VCAD) (2023)
  • Workshop on Autonomous Driving (2023)
    • Kodiak: Robust Localization: From Highways to Offroad
    • Chelsea Finn: Benchmarking, Detecting, and Adapting to Distribution Shift
    • Challenge - Argoverse
    • Common Misconceptions in Autonomous Driving
    • Alexandre Alahi: The 7 Foundational Principles Behind Autonomous Mobility
    • Chen Wu, head of Perception at Waymo: Progress Towards Scalable Deployment of Autonomous Driving
    • Challenge - Waymo Open Dataset
    • Ashok Elluswamy: Foundation Models for Autonomy
  • End-to-End Autonomous Driving: Perception, Prediction, Planning and Simulation (2023)

Survey Papers

• Autonomous Agents Modelling Other Agents:A Comprehensive Survey and Open Problems, S.V. Albrecht(2018)
• A Survey of Motion Planning and ControlTechniques for Self-driving Urban Vehicles, M. Cap et al (2016)
• Self-Driving Cars: A Survey, C. Badue et al (2019)
• A Survey of Deep Learning Techniques for Autonomous Driving, S. Grigorescu et al (2019)

Papers - Immitation Learning

• End-to-end Driving via Conditional Imitation Learning, Codevilla et al (2018). Code
• On Offline Evaluation of Vision-based Driving Models, Codevilla et al (2018)
• Deep Imitative Models for Flexible Inference, Planning, and Control, N. Rhinehart et al (2019)
• End-to-end Autonomous Driving Perception with Sequential LatentRepresentation Learning, J. Chen et al (2020)
• Urban Driving with Conditional Imitation Learning, J. Hawke et al (2019). Also see https://wayve.ai/
• Conditional Imitation Learning for AutonomousDriving: Comparing two approaches, Jesper Hagstrom master thesis (2021)

Papers

• A Unifying Formalism for Shortest Path Problems with ExpensiveEdge Evaluations via Lazy Best-First Search over Paths with Edge Selectors, C.M. Dellin and S. Srinivasa (2016)
• Stochastic Functional Gradient for Motion Planning in Continuous Occupancy Maps, G. Francis (2017)
• Autonomous Parking for Articulated Vehicles, R. Kusumakatar et al (2017)
• On a Formal Model of Safe and Scalable Self-driving Cars, S. Shalev-Schwartz et al (2018)
• Planning, Fast and Slow:A Framework for Adaptive Real-Time Safe Trajectory Planning, D. Fridovich-Keil et al (2018)
• Combining Neural Networks and Tree Search for Task and MotionPlanning in Challenging Environments, C. Paxton et al (2017)
• Lazy Receding Horizon A* forEfficient Path Planning in Graphs with Expensive-to-Evaluate Edges, A. Mandalika et al (2018)
• Deep Reinforcement Learning in a Handful of Trialsusing Probabilistic Dynamics Models, K. Chua et al (2018)
• Probabilistically Safe Robot Planning with Confidence-Based Human Predictions, J.F. Fisac et al (2018)
• Multiple Futures Prediction, Y.C. Tang, R. Salakhutdinov (2019)
• Learning to Drive in a Day, Alex Kendall et al (2019)
• A path planning and path-following control frameworkfor a general 2-trailer with a car-like tractor, O. Ljungqvist et al (2019)
• LeTS-Drive: Driving in a Crowd by Learning from Tree Search, P. Cai et al (2019)
• ML-based Fault Injection for Autonomous Vehicles:A Case for Bayesian Fault Injection, S. Jha et al (2019)
• Agent Prioritization for Autonomous Navigation, K. Refaat et al (2019)
• Scalable End-to-End Autonomous Vehicle Testingvia Rare-event Simulation, M. O’Kelley et al (2019)
• VectorNet: Encoding HD Maps and Agent Dynamics fromVectorized Representation, J. Gao et al (2020)
• Autonomous Driving at Intersections:A Critical-Turning-Point Approach for Left Turns, K. Shu et al (2020)
• LiRaNet: End-to-End Trajectory Prediction using Spatio-Temporal Radar Fusion, M. Shah et al (2020)
• Towards Streaming Image Understanding, M. Li, Y.-X. Wang, D. Ramanan (2020)
• Action-based Representation Learning for Autonomous Driving, Y. Xiao et al (2020), code
• Automatic Calibration of Multiple 3D LiDARs in Urban Environments, J. JIao et al (2019)
• On complementing end-to-end human behavior predictors with planning, L. Sun et al (2021)
• Learning to Predict Vehicle Trajectories with Model-based Planning, Haoran Song et al (2021)
• PiP: Planning-informed Trajectory Prediction for Autonomous Driving, H. Song el al (2000)
• Point Cloud Labeling using 3D Convolutional Neural Network, J Huang et al (2016)
• Vehicle Detection from 3D Lidar Using Fully Convolutional Network, Bo Li et al (2016)
• Review: Deep Learning on 3D Point Clouds, S.A.Bello et al (2020)
• Towards safe, data-driven autonomy, Marco Pavone, MIT Robotics talk (2022)
• Differentiable Algorithm Networks for Composable Robot Learning, P. Karkus et al (2019), talk

2007 Darpa Challenge

• Towards Fully Autonomous Driving: Systems and Algorithms, by Jesse Levinson et. al. (2011 IEEE Intelligent Vehicles Symposium (IV), Baden-Baden, Germany, June 5-9, 2011). Describes the Stanford entry at the 2007 DARPA Challenge, and follow-up work.

Interviews, Presentations

• Apex AI: ROS 2 on Autonomous Driving Vehicles, slides (ROSCON 2018)
• Dmitri Dolgov (Waymo CTO): Waymo and the Future of Self-Driving Cars (Dec 2020)
• George Hotz: Hacking the Simulation & Learning to Drive with Neural Nets (Oct 2020)
• Sertac Karaman (MIT, Optimus Ride) on Motion Planning in a Complex World (2018). Describes the MIT entry at the 2007 DARPA Challenge.
• Deva Ramanan (CMU / Argo AI) Embodied Perception In the Wild (June 2020)
• ROS World 2020: Autoware Parallel Session. Documentation: Autoware.Auto
  • Autonomous Valet Parking
  • Localization and State Estimation
  • 3D Perception and Control
  • Behavioral and Motion Planning in Autoware
• Chris Urmson: Delivering the benefits of self-driving technology: Aurora’s approach to development (Apr 2021)
• David Held - 6th BMTT Workshop ICCV 2021
• Mac Schwager: Game Theory for Simultaneous Behavior Prediction and Trajectory Planning in AVs (2022)

Sensors and Sensor Fusion

• F. Rosique et al: A Systematic Review of Perception System and Simulators for Autonomous Vehicles Research (2019)
• De Jong Yang et al: Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review (2021)
• A. Jain et al: High-Fidelity Sensor Calibration for Autonomous Vehicles (2019)
• P. Meruva: Sensor Calibration is Critical to the Future of Automated Vehicles (2020)
• Chen, Xiaozhi, Huimin Ma, Ji Wan, Bo Li, and Tian Xia: Multi-view 3d object detection network for autonomous driving (2016)
• Ku, Jason, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L. Waslander: Joint 3d proposal generation and object detection from view aggregation (2017)
• Qi, Charles R., Wei Liu, Chenxia Wu, Hao Su, Leonidas J. Guibas: Frustum pointnets for 3d object detection from rgb-d data (2018)
• Vora, Sourabh, Alex H. Lang, Bassam Helou, Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection (2019)
• Li, Yingwei, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen et al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection (2022).

Sensor hardware

• Tech Highlight: Sony IMX490 Automotive Sensor Explained

Datasets

• KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D
  • Vision meets Robotics: The KITTI Dataset
  • KITTI Coordinate Transformations
• Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset, S. Ettinger et al (2021)

Point Cloud and Image Labeling

• AWS SageMaker GroundTruth
  • Supports point cloud
• 3D BAT github
  • 3D BAT: A Semi-Automatic, Web-based 3D Annotation Toolbox for Full-Surround, Multi-Modal Data Streams (2019)
• LATTE github
  • LATTE: Accelerating LiDAR Point Cloud Annotation via Sensor Fusion, One-Click Annotation, and Tracking (2019)
• PC-Annotate
  • Annotation Tool and Urban Dataset for 3D Point Cloud Semantic Segmentation (2021)
• Playment
• Scale
• SUSTechPOINTS github
  • SUSTech POINTS: A Portable 3D Point Cloud Interactive Annotation Platform System (2020)
• Papers
  • Single-Click 3D Object Annotation on LiDAR Point Clouds (2021)
• Supervised & Unsupervised Approaches for LiDAR Perception in Urban Environments by C Stachniss (2022)
  • KPConv: Flexible and Deformable Convolution for Point Clouds (2019)
  • KPConv - How Does It Work? (2021)
• Haotian Tang, Shang Yang et al: TorchSparse: Efficient Point Cloud Inference Engine, slides, github
• Y. Liu el al: Segment Any Point Cloud Sequences by Distilling Vision Foundation Models (2023), github

Drivable Area, Occupancy Grid, Occupancy Networks

• Ashok Elluswamy, Tesla: CVPR’22 WAD Keynote.Occupancy networks, NeRFs, implicit networks for collision avoidance

Auto Labeling Data

• Auto4d: Learning to label 4d objects from sequential point clouds, B. Yang et al, Uber (2021)
• Offboard 3D Object Detection from Point Cloud Sequences, Ch. Qi et al, Waymo (2021), video
• X. Chen et al: Automatic Labeling to Generate Training Data for Online LiDAR-based Moving Object Segmentation (2022)
• Y. Liu et al: Segment Any Point Cloud Sequences by Distilling Vision Foundation Models (2023)
• Talks
  • Stanford Seminar: Self-Supervised Pseudo-Lidar Networks, Adrien Gaidon, Toyota Research Institute (2019)
• Motional NuPlan, NuScenes
  • H. Cui: Technically Speaking: Predicting the future in realtime for safer autonomousdriving (2022)
  • H. Cui: Technically Speaking:Auto-labeling With OfflinePerception (2022)
  • H. Caesar: Motional’s nuPlan DatasetWill Advance AV PlanningResearch (2021)

Lidar Calibration

• BALM: Bundle Adjustment for Lidar Mapping, Zheng Liu and Fu Zhang (2010)
• Livox Automatic Calibration Tools github
• Automatic Calibration of Multiple 3D LiDARs in Urban Environments, J. Jiao et al (2019)
• LiDAR and Camera Calibration using Motion Estimated by Sensor Fusion Odometry, Ishikawa et al (2018)
• Automatic Extrinsic Calibration of a Camera and a 3D LiDAR using Line and Plane Correspondences, Zhou et al (2018)

Self-supervised perception

• K. Stocking et al: Linking vision and motion for self-supervised object-centric perception (2023)

Blogs, Videos

• Pixelopolis, a self-driving car demo from Google I/O built with TF-Lite (July 13, 2020)
• Alex Kendall: Now is the Time for Reinforcement Learning on Real Robots
• Serge-Paul Carrasco
  • Waymo (ChauffeurNet) versus Telsa (HydraNet) (2020)
  • Machine Learning Models for Autonomous Vehicles (2020)
• Yu Huang: How to Build a Data Closed-loop Platform for Autonomous Driving? (2021)
• AWS Architecture Blog: J. Tang, D. Phillips: Field Notes: Building an Autonomous Driving and ADAS Data Lake on AWS (2020)
• AWS Autonomous Mobility site
  • AWS re:Invent 2019: Creating a data-driven, cloud-native ecosystem at BMW Group (AUT306)
  • AWS re:Invent 2020: Paving the way toward automated driving with BMW Group
  • All Things Automotive S1 Ep5: Data Lakes with BMW (2021)
• AWS All Things Automotive
  • S1 Ep8: Autonomous & AI-Powered: A Look at Frameworks & Training with Amazon ML Solutions Lab
    • Uses of ML in auto industry; distributed training with Sagemaker and FSx Lustre; federated learning
  • S1 Ep4: IoT & ML Inference with NXP
    • Hardware:
      • Gold box (service-oriented gateway), runs AWS IoT Greengas, Sagemaker Neo DLR, other apps. Uses 2-core Arm Cortex-R52. Runs lambda function to do battery anomaly detection.
        • Runs FreeRTOS, AWS IoT Greengas, uses AWS IoT Decive SDK
      • Blue box (battery management system, dual electric motor drive controller)
      • Green box (powertrain domain controller), controls battery and electric motors. Uses 4-core 2-hyperthread Arm Cortex-R52 at 800MHz. The cores are under a Type 1 hypervisor from OpenSynergy, which allows you to isolate the application on a specific core. Software stack is from NXP.
    • Predict useful life of battery
    • Most difficult part was getting the dataset
    • Two uses of Sagemaker:
      • Calculation of remaining useful life of the engine. Model is LSTM network, and inference is done in the cloud.
      • Detect anomalies in the battery pack. Model uses built-in algorithm XGBoost, inference happens in the Gold Box, using runtime from AWS.
      • This is deployed both in the cloud and on the target using Sagemaker
      • Use Sagemaker Neo to optimize for target hardware. Sagemaker Neo compiles and optimizes the model for the target architecture.
      • To create the datasets used for training, took collection of natural data observations as well as synthetic data
        • Moved them through an RCF (Random Forest Cut) algorithm to create training set for the XGBoost algorithm that can be run on the vehicle.
      • Cloud helps with analysis of data from the fleet, with download of data to the car, vehicle health monitoring
      • Supports remote software updates
    • Integrated with Matlab and Simulink for controller
    • AWS CMS (content management solution) scoops data into the cloud, provides visualization layer and fleet manager.
  • At the Automotive Edge with Arm
    • Ghirish Shirasat, director of ARM Autonomous & IoT Software Strategy/Architecture
    • ARM is an architecture instruction set. ARM also provides a model implementation.
    • AWS supports ARM Graviton cloud ec2 instances
    • SOAFEE framework.
      • Reference implementation developed under MIT license.
    • Virtual target initiative
• Eddie Forson: Teaching Cars To Drive — Highway Path Planning (2018)

Products, Companies

• Aurora
  • xviz open source project
• Autoware.ai project
  • Source code
  • Documentation
  • Open Source Integrated Planner for Autonomous Navigation in Highly Dynamic Environments, H. Darweesh et al (2017)
• Cruise
  • 75 Minutes of Autonomous Driving with Kyle Vogt and Sam Altman (2020)
  • CVPR22: Workshop on Autonomous Driving (2022)
• John Deere
  • John Deere 2023 Tech Summit recap
  • Autonomous vehicle technology deep dive with John Deere and Zoox
• Tesla
  • Tesla AI Day 2021
    • Vincent Feng: An Overview of ResNet and its Variants
    • Jonathan Hui: Understanding Feature Pyramid Networks for object detection (FPN) (2018)
    • Yu Huang: Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day, Medium (2021)
    • J. Zhang: Deep Understanding Tesla FSD (2021)
  • Tesla AI Day 2022
  • Inside Tesla’s Innovative And Homegrown “Dojo” AI Supercomputer (2022)
  • Andrej Karpathy
    • On the visionary AI in Tesla’s autonomous driving (2021)
    • PyTorch at Tesla (2019)
    • Multi-Task Learning in the Wilderness, ICML 2019, about Tesla Autopilot’s hierarchical model that detecs moving objects, static objects, signs, traffic lights. References:
      • Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation, C. Liu et al (2019)
      • Which Tasks Should Be Learned Together in Multi-task Learning?, T. Standley et al (2020)
    • AI for Full-Self Driving at Tesla (ScaledML 2020)
    • Workshop on autonomous driving (CVPR 2021)
    • Workshop on autonomous driving (CVPR 2022)
    • Lex Fridman Podcast #333: Tesla AI, Self-Driving, Optimus, Aliens, and AGI (2022)
• Nuro
  • Albert Meixner, NURO head of Software
    • An Overview of the Nuro Autonomy Stack (2022)
    • Medium: The Nuro Autonomy Stack (2022)
• Uber
  • PnPNet: End-to-End Perception and Prediction with Tracking in the Loop, M. Liang et al (2020)
• Waymo
  • WSJ: The Race for Self-Driving Taxis: Waymo Puts Urban Terrain to the Test (2021)
  • Waymo Team: Riding with Waymo One today (2018)
  • Mayank Bansal:
    • ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst, with Alex Krizhevsky and Abhijit Ogale (2018)
    • Learning to Drive: Beyond Pure Imitation
    • Teaching a Car to Drive Itself by Imitation and Imagination (Google I/O’19)
  • Dragomir Anguelov
    • Machine Learning for Autonomous Driving at Scale (CVPR 2020 invited talk)
    • ML at Waymo: Building a Scalable Autonomous Driving Stack (2021)
    • Behavior Models for Autonomous Driving (2022). Pointers to all detailed papers.
  • Waymo Public Road Safety Performance Data (2020)
  • Simulation City: Introducing Waymo’s most advanced simulation system yet for autonomous driving (2020)
• Parallel Systems
  • ArsTechnica: Autonomous battery-powered rail cars could steal shipments from truckers, by Tim De Chant (2022)
• Locomation (Greater Pittsburgh Area)
• Scale
  • Debugging Model Performance with Scale Nucleus (2021) Features shown:
    • Dataset management
    • Data annotation
    • Scene labeling (i.e., identifying interesting occurrences in a scene), search engine for searching labels (e.g. find me scenes with at least a bus and 3 fire hidrants)
    • Models can be uploaded to Nucleus. It acts as a model registry.
    • Model runs can be committed. Inference results for committed models can be analysed for things like detection failures, false positives, etc.
    • Scenes that failed to be detected correctly can be added to a ‘slice’, sent to the contractor to be re-annotated, before the model can be re-run.
    • Retraining can be done on such a slice of failed scenes, for accelerated learning.
• Zoox
  • How Zoox Autonomous Vehicles ‘Perceive’ Their Surroundings (2022)

Simulation

• Wayve: Neural Rendering (2023)
• F. Mutsch et al: From Model-Based to Data-Driven Simulation: Challenges and Trends in Autonomous Driving (2023)
• Uber simulation
  • Inside Uber ATG’s Data Mining Operation: Identifying Real Road Scenarios at Scale for Machine Learning
  • Engineering Uber’s Self-Driving Car Visualization Platform for the Web (2017)
• Waabi
  • Waabi CVPR 2021
  • Waabi blog: How Waabi World works (2022)
  • The Verge: Welcome to Waabi World, the ‘ultimate simulator’ for autonomous vehicles (2022)
  • S. Manivasagam et al: LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World (2020)
  • A. Corso et al: Adaptive Stress Testing with Reward Augmentation for Autonomous Vehicle Validation (2019)

Market Reports

• McKinsey: Distraction or disruption? Autonomous trucks gain ground in US logistics, by A. Chottani et al (2018)
• A. Engholm et al: Cost Analysis of Driverless Truck Operations (2020)
• A. Engholm et al: Impacts of large-scale driverless truck adoption on the freight transport system (2021)

Frameworks

• Lidar
  • mmdetection3d
  • openpcdet

Development platforms

• AVA AP1 (ARM)
• Ambarella

People

• Felipe Codevilla
• Marco Pavone

Libraries

• COiLTRAiNE: Conditional Imitation Learning Training Framework

Other

• Computer Vision
• Machine Learning
• MLOps
• Probabilities and Statistics
• Reinforcement Learning
• Robotics