--- name: databricks-local-dev-loop-5c6f660f855d description: | Configure Databricks local development with dbx, Databricks Connect, and IDE. Use when setting up a local dev environment, configuring test workflows, or establishing a fast iteration cycle with Databricks. Trigger with phrases like "databricks dev setup", "databricks local", "databricks IDE", "develop with databricks", "databricks connect". allowed-tools: Read, Write, Edit, Bash(pip:*), Bash(databricks:*), Bash(dbx:*), Grep version: 1.0.0 license: MIT author: Jeremy Longshore --- # Databricks Local Dev Loop ## Overview Set up a fast, reproducible local development workflow for Databricks. ## Prerequisites - Completed `databricks-install-auth` setup - Python 3.8+ with pip - VS Code or PyCharm IDE - Access to a running cluster ## Instructions ### Step 1: Project Structure ``` my-databricks-project/ ├── src/ │ ├── __init__.py │ ├── pipelines/ │ │ ├── __init__.py │ │ ├── bronze.py # Raw data ingestion │ │ ├── silver.py # Data cleansing │ │ └── gold.py # Business aggregations │ └── utils/ │ ├── __init__.py │ └── helpers.py ├── tests/ │ ├── __init__.py │ ├── unit/ │ │ └── test_helpers.py │ └── integration/ │ └── test_pipelines.py ├── notebooks/ # Databricks notebooks │ └── exploration.py ├── resources/ # Asset Bundle configs │ └── jobs.yml ├── databricks.yml # Asset Bundle project config ├── .env.local # Local secrets (git-ignored) ├── .env.example # Template for team ├── pyproject.toml └── requirements.txt ``` ### Step 2: Install Development Tools ```bash # Install Databricks SDK and CLI pip install databricks-sdk databricks-cli # Install dbx for deployment pip install dbx # Install Databricks Connect v2 (for local Spark) pip install databricks-connect==14.3.* # Install testing tools pip install pytest pytest-cov ``` ### Step 3: Configure Databricks Connect ```bash # Configure Databricks Connect for local development databricks-connect configure # Or set environment variables export DATABRICKS_HOST="https://adb-1234567890.1.azuredatabricks.net" export DATABRICKS_TOKEN="dapi..." export DATABRICKS_CLUSTER_ID="1234-567890-abcde123" ``` ### Step 4: Create databricks.yml (Asset Bundle) ```yaml # databricks.yml bundle: name: my-databricks-project workspace: host: ${DATABRICKS_HOST} variables: catalog: description: Unity Catalog name default: main schema: description: Schema name default: default targets: dev: default: true mode: development workspace: root_path: /Users/${workspace.current_user.userName}/.bundle/${bundle.name}/dev staging: mode: development workspace: root_path: /Shared/.bundle/${bundle.name}/staging prod: mode: production workspace: root_path: /Shared/.bundle/${bundle.name}/prod ``` ### Step 5: Local Testing Setup ```python # tests/conftest.py import pytest from pyspark.sql import SparkSession @pytest.fixture(scope="session") def spark(): """Create local SparkSession for unit tests.""" return SparkSession.builder \ .master("local[*]") \ .appName("unit-tests") \ .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \ .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \ .getOrCreate() @pytest.fixture(scope="session") def dbx_spark(): """Connect to Databricks cluster for integration tests.""" from databricks.connect import DatabricksSession return DatabricksSession.builder.getOrCreate() ``` ### Step 6: VS Code Configuration ```json // .vscode/settings.json { "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python", "python.testing.pytestEnabled": true, "python.testing.pytestArgs": ["tests"], "python.linting.enabled": true, "python.linting.pylintEnabled": true, "editor.formatOnSave": true, "[python]": { "editor.defaultFormatter": "ms-python.black-formatter" }, "databricks.python.envFile": "${workspaceFolder}/.env.local" } ``` ```json // .vscode/launch.json { "version": "0.2.0", "configurations": [ { "name": "Python: Current File (Databricks Connect)", "type": "python", "request": "launch", "program": "${file}", "console": "integratedTerminal", "env": { "DATABRICKS_HOST": "${env:DATABRICKS_HOST}", "DATABRICKS_TOKEN": "${env:DATABRICKS_TOKEN}", "DATABRICKS_CLUSTER_ID": "${env:DATABRICKS_CLUSTER_ID}" } } ] } ``` ## Output - Working local development environment - Databricks Connect configured for remote execution - Unit and integration test setup - VS Code/PyCharm integration ready ## Error Handling | Error | Cause | Solution | |-------|-------|----------| | `Cluster not running` | Auto-terminated | Start cluster first | | `Version mismatch` | DBR vs Connect version | Match databricks-connect version to DBR | | `Module not found` | Missing local install | Run `pip install -e .` | | `Connection timeout` | Network/firewall | Check VPN and firewall rules | | `SparkSession already exists` | Multiple sessions | Use `getOrCreate()` pattern | ## Examples ### Run Tests Locally ```bash # Unit tests (local Spark) pytest tests/unit/ -v # Integration tests (Databricks Connect) pytest tests/integration/ -v --tb=short # With coverage pytest tests/ --cov=src --cov-report=html ``` ### Deploy with Asset Bundles ```bash # Validate bundle databricks bundle validate # Deploy to dev databricks bundle deploy -t dev # Run job databricks bundle run -t dev my-job ``` ### Interactive Development ```python # src/pipelines/bronze.py from pyspark.sql import SparkSession, DataFrame def ingest_raw_data(spark: SparkSession, source_path: str) -> DataFrame: """Ingest raw data from source.""" return spark.read.format("json").load(source_path) if __name__ == "__main__": # Works locally with Databricks Connect from databricks.connect import DatabricksSession spark = DatabricksSession.builder.getOrCreate() df = ingest_raw_data(spark, "/mnt/raw/events") df.show() ``` ### Hot Reload with dbx ```bash # Watch for changes and sync dbx sync --watch # Or use Asset Bundles databricks bundle sync -t dev --watch ``` ## Resources - [Databricks Connect](https://docs.databricks.com/dev-tools/databricks-connect.html) - [Asset Bundles](https://docs.databricks.com/dev-tools/bundles/index.html) - [VS Code Extension](https://docs.databricks.com/dev-tools/vscode-ext.html) - [Testing Notebooks](https://docs.databricks.com/notebooks/testing.html) ## Next Steps See `databricks-sdk-patterns` for production-ready code patterns. --- Source attribution: Dicklesworthstone Original project: https://github.com/Dicklesworthstone/pi_agent_rust/blob/f28fc46ee0db8e710d95b8b55592cd38d60f2b26/tests/ext_conformance/artifacts/plugins-community/plugins/saas-packs/databricks-pack/skills/databricks-local-dev-loop/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Dave AI. ## Upstream license notice MIT License (with OpenAI/Anthropic Rider) Copyright (c) 2026 Jeffrey Emanuel Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: ADDITIONAL RIDER / RESTRICTION (OpenAI / Anthropic): This rider is part of the "conditions" of this License. In the event of any conflict between this rider and any other portion of this License, this rider controls. "Restricted Parties" means OpenAI, L.L.C.; Anthropic, PBC; any of their respective Affiliates; and any person or entity acting directly or indirectly on behalf of, for the benefit of, or under the direction of any of the foregoing (including any officer, director, employee, contractor, agent, consultant, service provider, or representative). Notwithstanding any other provision of this License, no rights are granted to any Restricted Party. Any purported license, sublicense, assignment, transfer, or other permission to any Restricted Party is null and void absent the express prior written permission of Jeffrey Emanuel. You may not provide, disclose, distribute, sublicense, sell, lease, lend, host, make available, or otherwise permit access to the Software or any derivative work of the Software (as defined in applicable copyright law) (collectively, "Derivative Works") to or for any Restricted Party. For purposes of this rider, "use" includes, without limitation: copying, modifying, merging, publishing, distributing, sublicensing, selling, transferring, making available, hosting, deploying, executing, benchmarking, testing, analyzing, indexing, or incorporating the Software or any Derivative Works into any dataset, training corpus, evaluation harness, or pipeline for machine learning or other automated systems. This rider applies to the Software and all Derivative Works. As a condition of use, you agree that this rider is a precondition to exercising any rights under this License, and you agree that any distribution of the Software or any Derivative Works must include this rider provision unmodified. Any breach of this rider automatically and immediately terminates the permissions granted by this License. Upon termination, you must immediately cease all use and distribution of the Software and any Derivative Works and destroy all copies under your control. You agree that a breach of this rider would cause irreparable harm and that Jeffrey Emanuel may seek injunctive or other equitable relief to enforce this rider, in addition to any other remedies available at law. To the maximum extent permitted by applicable law, the prevailing party in any action to enforce this rider shall be entitled to recover reasonable attorneys' fees and costs. For purposes of this rider, "Affiliate" means any entity that directly or indirectly controls, is controlled by, or is under common control with the specified party. "Control" means ownership of more than 50% of the voting securities or other ownership interest, or the power to direct management or policies by contract or otherwise. The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Dave AI resource metadata Creator / publisher: Dicklesworthstone Original project: https://github.com/Dicklesworthstone/pi_agent_rust/blob/f28fc46ee0db8e710d95b8b55592cd38d60f2b26/tests/ext_conformance/artifacts/plugins-community/plugins/saas-packs/databricks-pack/skills/databricks-local-dev-loop/SKILL.md Attribution basis: Source repository record Fields of work: Development & Engineering Roles: Backend Developer, Data Engineer, ML / AI Engineer Specialties: API, Backend Tools: Databricks, Pyspark, Pytest Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: c430d8a1dd41fc0dde231d6a7605ab1d41bbde8f7eb04c9924ef892b1278f768