Databricks Software Stack
Conferences
- Data+AI Summit 2022
- Spark AI Archives
- Introducing MLflow for End-to-End Machine Learning on Databricks, S. Owen (2020)
- A Journey to Building an Autonomous Streaming Data Platform—Scaling to Trillion Events Monthly at Nvidia, Satish Dandu (2020)
- Lessons Learned from Using Spark for Evaluating Road Detection at BMW Autonomous Driving, Gheorghe Pucea (2019)
- Apple: Threat Detection and Response at Scale, D. Brezinski (2018)
- MLflow Pipelines: Accelerating MLOps from Development to Production (2022)
Papers
- Matei Zaharia et al: Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing (2012)
Books
- Bill Chambers, Matei Zaharia: Spark: The Definitive Guide
- The Delta Lake Series Complete Collection (2021)
Posts
- O. Anomnachi: Comparing Databricks and Apache Spark (2020)
- P. Borges et al: Enabling Computer Vision Applications With the Data Lakehouse (2021)
- Databricks & Embarq: Ensuring long-haul truck safety with data and AI
- I. G. Arnedo: Building our Data Platform: Why we have chosen Databricks over Snowflake
- Forrester: Databricks delivers nearly $29 million in economic benefits and pays for itself in less than six months (2020) (commissioned by Databricks)
- Accelerating Data Ingestion with Databricks Autoloader (2021)
- Notebook gallery
- Diving Into Delta Lake: Schema Enforcement & Evolution
- 5 things I wish I knew about Databricks … before I started (2022)
- Databricks Access Control – The 3 Most Important Steps (2022)
- Towards Data Science: Unleash The Power of Databricks CLI (2021)
- ML Solutions Accelerators
- N. Karpov: Introducing Support for Delta Lake Tables in AWS Lambda (2023)
Delta Tables
- V. Teixeira: Delta Lake: Keeping It Fast and Clean (2023)
Containers
- Conde Nast: Bring Your Own Container: Using Docker Images In Production (2021)
- Docs
Terraform for Databricks
- Giving Away The Keys To The Kingdom: Using Terraform To Automate Databricks (2021). (blog post)
- Databricks on Azure with Terraform (2020)
Gitlab for Databricks
Metastore
Unity Catalog
- Matei Zaharia: Announcing the Unity Catalog (2021)
- A Practitioner’s Guide to Unity Catalog—A Technical Deep Dive (2022)
MLOps for Databricks
- Designing Better MLOps Systems (2022), slides
- Survey of Production ML Tech Stacks, slides, AI Landscape
- MLOps on Databricks: A How-To Guide (2022)
- databricks/mlops-stack
ML
Data Loading
- DeltaTorch. Example: DeltaTorch with PyTorch Lightning for image classification This example also uses SparkTorchDistributor, which you can run either locally (if you have one big machine with multiple GPUs) or in a distributed way with multiple machines. Most efficient to use DeltaTorch and load the data from S3 using DBFS.
MLFlow
- Workshop: Managing the Complete Machine Learning Lifecycle with MLflow 1 2 3 [2020]
- Vaibhav Satpathy
- Setup MLflow on AWS EC2 (2021)
- MLOps deployment on AWS Fargate: I II (2021)
Feature Store
- Enable Production ML with Databricks Feature Store (2022)
- Prosenjit Chakraborty: Databricks Feature Store (2021)
Computer Vision
- Claudiu Barbura: Video Analytics At Scale: DL, CV, ML On Databricks Platform (2021)
Distributed Training
- Horovod
- Databricks docs: Introducing HorovodRunner for Distributed Deep Learning Training
- Databricks docs: HorovodRunner: distributed deep learning with Horovod
- T. Addair: End-to-End Deep Learning with Horovod on Apache Spark (2020)
- Project Hydrogen