data-engineer
Build scalable data pipelines, modern data warehouses, and
4,964,815 AI resources indexedWorking toward the world’s largest AI resource index.
Reusable instructions for compatible AI agents.
Field counts come from the latest full-catalog snapshot. A resource can belong to several fields and platforms.
Build scalable data pipelines, modern data warehouses, and
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
Apache Spark distributed computing. Use for big data processing.
Troubleshoot and optimize Delta Lake and Apache Spark performance in Microsoft Fabric. Use when diagnosing slow Spark jobs, small file problems, data skew, shuffle bottlenecks, out-of-memory errors, V-Order tuning, OPTIMIZE/VACUUM operations, partition strategy, resource profile selection (writeH...
Use when building Apache Spark applications, distributed data processing pipelines, or optimizing big data workloads. Invoke for DataFrame API, Spark SQL, RDD operations, performance tuning, streaming analytics.
Master enterprise-grade Scala development with functional programming, distributed systems, and big data processing. Expert in Apache Pekko, Akka, Spark, ZIO/Cats Effect, and reactive architectures.
Analyzes and optimizes Apache Spark jobs for performance, cost, and resource utilization
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
Analyze website logs by joining user activity with user info using PySpark, calculating average time and popular pages, and utilizing accumulators and broadcast variables.
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle oper...
Diagnose and resolve performance issues in Microsoft Fabric notebooks running Apache Spark. Use when notebooks are slow, Spark jobs are timing out, sessions hit HTTP 430 throttling errors, data skew causes straggling tasks, shuffle operations are inefficient, Delta Lake tables need optimization, ...
Try a broader search or clear some filters.
Platform labels describe intended use. Read each resource's author credits, requirements and available access methods.