--- name: nightly-sync-cfea603fe8a2 description: Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues. when_to_use: Working on the nightly sync PR; investigating a nightly sync failure; resolving merge conflicts between main and dev; 'nightly sync failed', 'main-to-dev merge', 'sync bot'. --- # Nightly Sync: Main to Dev This skill is read by the automated sync bot during the nightly-sync-main-to-dev workflow. It contains all domain knowledge for merging main into dev, resolving conflicts, iterating on CI, and shipping the PR. --- ## Phase 1: Create the Sync Branch and Merge ### Branch Setup 1. Create branch `$BRANCH` from `origin/dev` 2. Merge: `git merge origin/main -X theirs --no-edit` 3. If conflicts remain (e.g. add/add), resolve by favoring main ### Preserving Dev-Only Additions Do NOT blanket-override all shared files with main's version. Dev has features not yet in main (new classes, new modules, new tests). The merge preserves both sides' non-conflicting additions — only intervene where there is an actual conflict. ### Squash-Merge Chain Detection Dev often develops features as a chain of PRs (PR1 → PR2 → PR3) where each builds on the last. When PR1 is squash-merged to main, git sees main's squashed version and dev's original commits as unrelated changes. `-X theirs` will pick main's PR1 code and silently discard PR2/PR3's improvements on dev. After the merge, check for this pattern: 1. For each file where `-X theirs` resolved a conflict, run `git log --oneline origin/dev -- ` to see if dev has commits that came AFTER the code main is bringing in. 2. If dev has follow-up commits (bug fixes, refactors, extensions), **favor dev's version** for those sections. 3. If the conflict is just main bringing in a clean copy of what dev already has (no follow-ups), main's version is fine. Practical check: run `git diff origin/dev -- ` on conflicted files. If dev's code was removed or reverted, investigate whether dev's version is the more evolved one. Real examples from PR #4291: - `emerging_optimizers.py`: Main's version was MORE complete — it squash-merged dev's PRs plus added more. `-X theirs` was correct. - `distrib_optimizer.py`: Main overwrote dev's `GroupedQuantizedTensor` support. Had to restore `_is_distopt_quantized_param` and the expanded `_expand_quantized_param_shard_for_cast` loop while keeping main's NVFP4 additions. This required a surgical merge combining sections from both. Key insight: squash-merge chains can go in EITHER direction. Sometimes main is ahead (it squash-merged dev's work + more), sometimes dev is ahead (it has follow-up PRs). Always diff both ways before deciding which version to favor. ### Files to Override from Main These files have known semantic conflicts where dev's versions reference args or APIs that main removed or renamed. Take main's version with `git checkout origin/main -- `: - `megatron/training/training.py` — references dev-only args - `megatron/training/initialize.py` — references dev-only args - `megatron/training/utils.py` — references dev-only args - `megatron/training/datasets/data_samplers.py` — references dev-only args - `megatron/core/optimizer/layer_wise_optimizer.py` — constructor signature **Caveat for ALL overrides:** After taking main's version of any file, you MUST run the API Mismatch Detection procedure (see below) on that file. Taking main's caller code while keeping dev's callee implementations is the #1 source of sync bugs. **IMPORTANT: Do NOT take main's `pyproject.toml`, `uv.lock`, or `docker/Dockerfile.ci.dev`.** These three files are a tightly coupled triple — the Dockerfile's `uv sync` command must match the dependency groups in `pyproject.toml`, and `uv.lock` must be consistent with both. Main's versions are missing dev-only dependencies (e.g. `fast-hadamard-transform`, correct TransformerEngine revision) and the `--group no_pypi_wheels` flag needed to install them. Keep dev's versions of all three files. **IMPORTANT: `.github/CODEOWNERS` must NEVER be modified by the sync bot under any circumstances.** Dev's CODEOWNERS is intentionally different from main's — do not take main's version, do not merge them, do not touch the file. If the merge produces a conflict or a non-zero diff against `origin/dev` on this path, restore dev's version verbatim: ``` git checkout origin/dev -- .github/CODEOWNERS ``` Then verify with `git diff origin/dev -- .github/CODEOWNERS` — output must be empty. Modifying CODEOWNERS triggers spurious reviewer requests and conflicts with the dev team's governance; rolling back a CODEOWNERS change after the PR lands is painful. **NEVER manually edit `uv.lock`.** It is a machine-generated lockfile. If it needs to change, it must be regenerated with `uv lock` inside a CUDA container (see `.claude/skills/build-and-test/SKILL.md`). ### Git Source Reconciliation (pyproject.toml) After keeping dev's `pyproject.toml`, check whether main has added NEW git sources to `[tool.uv.sources]` that don't exist in dev's version. Main's merged code may import from packages only available at specific git revisions. 1. Diff the `[tool.uv.sources]` sections: `git show origin/main:pyproject.toml` vs `git show origin/dev:pyproject.toml` 2. For each git source in main but not dev, add it to dev's `pyproject.toml` 3. For sources in both but at different revisions, check whether dev's revision works. If dev's revision is broken (TOML parse errors, missing classes main's code imports), take main's revision instead. Real examples from PR #4291: - `nvidia-resiliency-ext`: Main's `torch.py` imports `get_write_results_queue` which only existed in main's pinned git revision, not on PyPI. Had to add main's git source to dev's pyproject.toml. - `nemo-run`: Dev's pinned revision had a TOML parse error with uv 0.7.2. Had to swap to main's revision. After any changes to `pyproject.toml`, regenerate `uv.lock` inside a CUDA container: ```bash docker run --rm -v $(pwd):/workspace nvcr.io/nvidia/pytorch:26.02-py3 \ bash -c "pip install uv==0.7.2 && cd /workspace && \ uv venv .venv --system-site-packages && uv sync --only-group build && uv lock" # Clean up root-owned .venv: docker run --rm -v $(pwd):/workspace nvcr.io/nvidia/pytorch:26.02-py3 \ bash -c "rm -rf /workspace/.venv" ``` ### API Mismatch Detection (Post-Merge Audit) The merge can create "Frankenstein" code where main's callers use dev's implementations (or vice versa) with different method signatures. This compiles fine but fails at runtime. After the merge, audit cross-boundary call sites: 1. Identify files where main's version was taken (`-X theirs` or explicit `git checkout origin/main`) 2. For each, find all external call sites: classes it instantiates, methods it calls on imported objects, functions from other modules it invokes 3. Verify method names, parameter counts, and signatures match between the caller and the implementation in the merged tree 4. Pay special attention to "interface" modules (files defining base classes) — if main and dev evolved the interface differently, every caller and implementer must agree Real examples from PR #4291: - `multi_latent_attention.py` (main) called `off_interface.group_commit()` but dev's interface only had `group_offload()` — method renamed - `mamba_model.py` (main) called `init_chunk_handler(3 params)` but dev's interface required 6 params — signature expanded on dev - `mamba_model.py` called `mark_not_offloadable()` but dev had `mark_not_offload()` — method renamed - `bulk_offload()` did `.remove()` after `bulk_offload_group()` already `.pop()`d the same item — double-removal from a list Practical detection: ```bash # For each file taken from main, find what it imports and calls grep -rn "from import\|\." megatron/ # Cross-reference with the actual implementations in the merged tree ``` ### File-Specific Merge Lessons These lessons were learned from PR #4291. They may recur if the same files continue to diverge: - `gated_delta_net.py`: If the merge creates code calling non-existent helper methods (e.g. `_resolve_cu_seqlens`), take dev's version wholesale. - `model_chunk_schedule_plan.py`: Watch for missing imports (e.g. `CudaGraphScope`) silently dropped during conflict resolution. - `fine_grained_activation_offload.py`: Critical interface file used by many callers. If main and dev have divergent method names/signatures, prefer dev's implementation and patch main-originated callers to match. - `distrib_optimizer.py`: Dev may have broader type abstractions (e.g. `_is_distopt_quantized_param` covering both FP8 and GroupedQuantizedTensor). Main may simplify to explicit type checks. Restore dev's abstractions. ### Special Handling: data_schedule.py Main and dev have completely different classes in this file: - Main: `HybridCPDataLoaderWrapper` (imported by main's `training.py`) - Dev: `BasePackingScheduler`, `DpBalancedScheduler`, `DefaultDynamicCPScheduler`, `wrap_data_iterator`, `get_batch_on_this_rank_for_sequence_packing` (imported by `pretrain_gpt.py` and tests) **Do NOT take either version wholesale.** Keep dev's file and append main's `HybridCPDataLoaderWrapper` class (plus any missing imports like `BalancedCPScheduler`, `Any`, `List`) at the end. ### Restore Deleted Files Compare `git ls-tree` between `origin/main` and HEAD to find files in main that are missing from the merged tree. For each: - **Restore** if main's code imports/references it and would break without it (e.g. `hybrid_cp_schedule.py` if `data_schedule.py` imports from it) - **Do NOT restore** if dev intentionally deleted it — check `git log origin/dev -- ` for the deletion commit to understand intent - When in doubt, check whether any file in the merged tree imports from the missing file. If nothing imports it, skip it. ### Formatting Run on ALL changed Python files (relative to `origin/dev`), in this order: 1. `black` (version 24, `--config pyproject.toml`) 2. `isort` 3. Order matters: black first, then isort — reverse order can undo isort's work 4. `pylint` on changed `megatron/core/` files — fix missing-docstring and line-too-long violations before pushing ### Pre-push invariant checks Before every `git push` in this workflow (the initial push in Phase 1 AND every fix-push in Phase 3), run these bash checks. If any fails, fix the condition and re-check before pushing: ```bash # 1. CODEOWNERS must be identical to dev's. if ! git diff --quiet origin/dev -- .github/CODEOWNERS; then echo "ABORT: .github/CODEOWNERS differs from origin/dev. Restore with:" echo " git checkout origin/dev -- .github/CODEOWNERS" exit 1 fi # 2. Dependency-management triple must be identical to dev's. for f in pyproject.toml uv.lock docker/Dockerfile.ci.dev; do if ! git diff --quiet origin/dev -- "$f"; then # pyproject.toml is allowed to differ ONLY for git source reconciliation # (new [tool.uv.sources] entries from main). If you intentionally edited # it for that reason, bypass this check by re-running with $f skipped. echo "WARNING: $f differs from origin/dev" fi done ``` The CODEOWNERS check is a HARD abort — never push if it fails. ### Commit and Push Phase 1 produces a single commit on the sync branch. The merge itself creates the merge commit; fold any post-merge work (formatting, conflict surgery, restored files, regenerated `uv.lock`) into it rather than stacking a second commit: ```bash git add -A git commit --amend --no-edit # rewrites the merge commit's tree; # parents are preserved. git push -u origin "$BRANCH" # only non-force push of the run. ``` Once pushed, this commit is immutable for the rest of the run. Phase 3 fixes go into a separate rolling fix commit on top (see Phase 3 step 4 and the two-commit policy in Rules). --- ## Phase 2: Create the Draft PR - Title: `chore: nightly sync main into dev ($DATE)` - Create as **draft**: `gh pr create --draft` - Body should include: 1. Summary of what was synced (number of commits from main) 2. **Python-only line-change stats**, so reviewers can gauge the real code surface (excluding golden-value JSON, uv.lock, etc.). Compute with: ```bash git diff --numstat origin/dev...HEAD -- '*.py' \ | awk 'BEGIN{a=0;d=0} {a+=$1; d+=$2} END{ printf "Python lines: +%d / -%d across %d files\n", a, d, NR }' ``` Include the exact line (e.g. `Python lines: +1234 / -567 across 42 files`) in the PR body so reviewers see it at a glance. 3. List of files where main's version was taken over the merge 4. List of files that were deleted in dev but restored (and why) 5. The remerge-diff output (`git show --remerge-diff HEAD` on the merge commit) so reviewers can inspect ONLY the conflict resolutions. If the output is very long, summarize conflicts by file and put the full diff in a collapsed `
` block. If git is too old for `--remerge-diff`, note the git version and describe the merge strategy used instead. - Save the PR number for later phases - **Add the `Run functional tests` and `Run MBridge tests` labels** to the PR immediately after creation. The `Run functional tests` label ensures `/ok to test` triggers the full CI suite (unit tests + functional/ integration tests with 100-step training and golden value comparison). The `Run MBridge tests` label triggers the MBridge test suite. Without these labels, only a lightweight subset runs. ```bash gh pr edit --repo $REPO \ --add-label "Run functional tests" \ --add-label "Run MBridge tests" ``` --- ## Phase 3: CI Iteration ### CI Architecture - **`Nemo_CICD_Test`** is a downstream gate job aggregating unit test, integration test, and other results. If it fails, investigate the upstream jobs it depends on — do NOT debug the gate itself. - **Integration tests** (H100, GB200) may be skipped for non-maintainer PRs. This is expected; the `Nemo_CICD_Test` gate will fail as a result. - **`tests/unit_tests/conftest.py`** imports from `megatron.training.training`, so a broken import in `training.py` (or anything it transitively imports) cascades to fail ALL test suites. If every test job fails with ImportError, check the training.py import chain first. ### Execution model: one step, no background You run inside ONE GitHub Actions step. The moment you stop emitting tool calls, the step ends and the runner container is destroyed. Any background process you started dies with it. There is NO persistent session and NO future wakeup. See the workflow prompt's "NO background tasks" block for the full ban list. Practical rule: every wait for CI to resolve is a SINGLE foreground Bash tool call that blocks inline until the wait is resolved. ### The Fix-Then-Retrigger Loop Two nested loops. Do NOT conflate them: - The **outer loop** is YOUR sequence of tool calls (each iteration: one `/ok to test`, one blocking poll, maybe one fix-and-push). It is NOT a Bash loop. It advances because you make new tool calls. - The **inner loop** is a single blocking Bash tool call using `while true; do ... sleep 120; done`. It runs during one iteration of the outer loop and ends when CI reaches a terminal state for that iteration. The outer loop terminates ONLY when Phase 4's gate is satisfied. **Source of truth:** `gh pr view --repo $REPO --json statusCheckRollup`. This lists every required check, including external status contexts (GitLab CI, `copy-pr-bot`, etc.) that `gh api .../actions/runs/.../jobs` does NOT show. **Outer-loop iteration (each iteration is a few tool calls):** 1. `latest_sha=$(git rev-parse HEAD)` (one Bash call). 2. Post `/ok to test $latest_sha` on the PR: `gh pr comment --repo $REPO --body "/ok to test $latest_sha"` 3. ONE blocking Bash tool call. This is the inner loop. Copy this template verbatim, only changing `REPO` and `PR`: ```bash REPO='NVIDIA/Megatron-LM' PR='' # Names matched case-insensitively, anchored to the START of the name. EXEMPT='copy-pr-bot|is-not-external-contributor|greptile|coderabbit|codeowners|.*review|.*approval|codecov|coverage|build-docs|doc-build|readthedocs|sphinx' # Sentinel check that tells us CI has fully run. Update this if the # aggregate gate job is renamed. SENTINEL='Nemo_CICD_Test' while true; do # Normalize both CheckRun (.status / .conclusion) and StatusContext # (.state) entries into the same {name, status, conclusion} shape. rollup=$(gh pr view "$PR" --repo "$REPO" --json statusCheckRollup --jq ' .statusCheckRollup[] | [ (.name // .context // "?"), (if .__typename == "StatusContext" then (if (.state == "PENDING" or .state == "EXPECTED") then "IN_PROGRESS" else "COMPLETED" end) else (.status // "UNKNOWN") end), (if .__typename == "StatusContext" then (if .state == "SUCCESS" then "SUCCESS" elif (.state == "FAILURE" or .state == "ERROR") then "FAILURE" else "NEUTRAL" end) else (.conclusion // "UNKNOWN") end) ] | @tsv') # Sentinel: do NOT declare green until the CI aggregate gate has # reached a terminal state. Before /ok to test triggers the run, # the sentinel is absent; while CI is running, it's IN_PROGRESS. sentinel_line=$(printf '%s\n' "$rollup" | awk -F'\t' -v s="$SENTINEL" '$1 == s') sentinel_status=$(printf '%s\n' "$sentinel_line" | awk -F'\t' 'NR==1 {print $2}') if [ "$sentinel_status" != "COMPLETED" ]; then echo "=== $(date -u) waiting for $SENTINEL (status: ${sentinel_status:-absent}) ===" sleep 120 continue fi # Classify non-exempt checks (exempt list applied to the NAME only). non_exempt=$(printf '%s\n' "$rollup" | awk -F'\t' -v p="^($EXEMPT)" 'tolower($1) !~ tolower(p)') failed=$(printf '%s\n' "$non_exempt" | awk -F'\t' '$2 == "COMPLETED" && $3 !~ /^(SUCCESS|SKIPPED|NEUTRAL)$/') pending=$(printf '%s\n' "$non_exempt" | awk -F'\t' '$2 != "COMPLETED"') if [ -n "$failed" ]; then echo "=== NON-EXEMPT FAILURES ===" printf '%s\n' "$failed" echo "RESULT=FAILURE" exit 0 fi if [ -n "$pending" ]; then # Sentinel is COMPLETED but a non-exempt check is still pending — # rare but possible. Keep waiting; do NOT ship. echo "=== $(date -u) sentinel done but non-exempt checks still pending ===" printf '%s\n' "$pending" sleep 120 continue fi echo "=== ALL NON-EXEMPT CHECKS COMPLETED GREEN ===" printf '%s\n' "$non_exempt" echo "RESULT=GREEN" exit 0 done ``` This Bash call blocks for as long as CI takes (minutes to hours). Do NOT split it into many short polls interleaved with other tool calls — that wastes `--max-turns` and creates windows where you could lose track of the loop state. 4. Read the tool output: - If `RESULT=FAILURE`: diagnose via `gh api repos/$REPO/actions/jobs//logs` (or the external-context equivalent) and fix the code. The Phase 1 commit is immutable; fixes accumulate in a single rolling fix commit on top of it: ```bash git add -A if git rev-parse --verify HEAD^2 >/dev/null 2>&1; then # HEAD has two parents → still the Phase 1 merge commit. # First failure of this run: create the fix commit. git commit -m "fix: post-CI corrections" git push origin "$BRANCH" else # HEAD is the existing fix commit → amend it. git commit --amend --no-edit git push --force-with-lease origin "$BRANCH" fi ``` `--force-with-lease` (not `--force`): if a human pushed onto the branch since the bot last fetched, the lease aborts the push instead of clobbering them — fetch and decide what to do. Start a new outer-loop iteration at step 1 with the new HEAD SHA. - If `RESULT=GREEN`: outer loop is done. Proceed to Phase 4. **Why not wait-for-run-to-register first?** `gh pr comment` with `/ok to test ` is handled by `copy-pr-bot`, which takes a few seconds to trigger the CI run. The `statusCheckRollup` poll in step 3 will initially show checks in `PENDING` / `QUEUED`; that's fine — the inner loop treats those as "keep waiting" and will see them advance as CI progresses. No separate registration poll needed. ### Anti-Patterns (what went wrong on run 24800621116) - **Do NOT classify a queued/in-progress job as "infrastructure- blocked" and ship.** A stuck queue drains eventually — wait. If the job eventually passes, great; if it fails, go fix it. - **Do NOT mark ready while any required check is `PENDING` / `QUEUED` / `IN_PROGRESS` on the HEAD SHA.** A push is not a pass; only a `COMPLETED` + green status is. - **Do NOT declare an untested job "pre-existing."** Pre-existing means the test ran to completion and failed the same way on recent dev CI. A job that never ran on your PR cannot be pre-existing. - **Do NOT use `gh api .../actions/runs/.../jobs` alone** as the gate signal. External status contexts (GitLab CI pipelines, copy-pr-bot status, etc.) do NOT appear there. Use `statusCheckRollup`. - **Do NOT start any background process.** No `&`, no `nohup`, no `run_in_background: true`, no `ScheduleWakeup`. The GitHub Actions step owns your shell; when the step ends, every background process is killed and cannot resume. - **Do NOT push directly to `pull-request/` branches.** The community bot manages those branches when it processes `/ok to test`. Pushing to them directly breaks the CI trigger mechanism. Always push to your own sync branch (e.g. `main2dev/`) instead. - **Do NOT forget the `Run functional tests` and `Run MBridge tests` labels.** Without `Run functional tests`, the internal GitLab functional tests do not run; without `Run MBridge tests`, the MBridge test suite does not run. ### Failure Investigation 1. Fetch logs: `gh api repos/$REPO/actions/jobs//logs` 2. Grep for: `ImportError`, `ModuleNotFoundError`, `FAILED`, `would reformat`, `line-too-long`, `Traceback` 3. Read the error, understand root cause, fix the code ### Common Issues - **ImportError for a class/module:** Dev test imports a class from a file where we took main's version. Restore only the missing class/function — not the entire file. If a file's classes are completely different between main and dev, keep both sets of code. - **Formatting failures (black/pylint):** Run `black --config pyproject.toml` on offending files. For pylint long-line or missing-docstring, edit directly. - **Circular imports:** `isort` can reorder imports in a way that introduces circular dependencies (e.g. `megatron/legacy/model/__init__.py`). Check `git diff` on `__init__.py` files to see if import order changed. - **Dependency version mismatches:** Taking main's `pyproject.toml`/`uv.lock` can change library versions in the CI container. Dev-only code may depend on newer versions (e.g. TransformerEngine's `single_grouped_weight`). If failures trace to missing kwargs or changed APIs in third-party libs, this is the cause. - **API mismatch (AttributeError / TypeError at runtime):** Main's callers reference methods that don't exist (or have different signatures) in dev's implementations. See "API Mismatch Detection" in Phase 1. Fix by adding shims, renaming methods, or adjusting call signatures. - **Infrastructure / network failures (apt-get, pip download):** Errors like `archive.ubuntu.com unreachable` or `Connection timed out` during package installation are transient CI infrastructure issues, not code problems. Retry CI with the same SHA. Do not investigate as code failures. ### Pre-Existing Failure Verification **You MUST empirically verify before classifying any failure as pre-existing.** 1. `gh pr list --repo $REPO --base dev --state merged --limit 3` 2. `gh pr checks --repo $REPO` on a recently merged dev PR 3. If the same test bucket **passes on recent dev CI** → the failure is sync-caused. You must fix it. 4. Only if the test **also fails on recent dev CI** can you classify it as pre-existing. Document with the dev PR number and CI run as evidence. ### Internal GitLab Functional Tests GitHub CI covers unit tests and some integration tests. Internal GitLab (`gitlab-master.nvidia.com`) runs additional functional tests on H100/GB200 hardware that may reveal issues GitHub CI does not catch. These surface in `statusCheckRollup` as external status contexts (the bash template already handles them via the `__typename == "StatusContext"` branch). - Fine-grained activation offloading failures, for example, only showed up in GitLab functional tests during PR #4291 - If GitHub CI passes but a reviewer reports GitLab failures, investigate with the same rigor as GitHub CI failures - The sync PR should ideally pass both GitHub and GitLab CI before merge, but GitHub CI passing (i.e. the Phase 4 gate above) is the minimum before `gh pr ready` --- ## Phase 4: Mark PR Ready — Strict Gate Run `gh pr ready` ONLY when every non-exempt required check on the latest CI run (against the current HEAD SHA) satisfies BOTH: 1. `status == "completed"` — NOT `queued`, `in_progress`, `pending`, `waiting`, or `requested`. 2. `conclusion ∈ {"success", "skipped", "neutral"}`. If a non-exempt check is pending/queued/in-progress: keep polling; do not run `gh pr ready`. If it fails: go back to Phase 3's loop. The exempt list (approval/coverage/docs) is defined in Phase 3; only those checks may be ignored. A pre-existing failure (same test failing identically on recent dev CI) may be accepted, but ONLY after it has fully run, been empirically verified against dev, and documented in the PR body with evidence (dev PR number + CI run URL). ``` gh pr ready --repo $REPO ``` Then comment on the PR confirming it is ready for human review. The comment should include: - Which non-exempt checks passed (summary from the bash template's final `ALL NON-EXEMPT CHECKS COMPLETED GREEN` output) - Any documented pre-existing failures with evidence (dev PR number + CI run URL showing the same failure on recent dev CI) - Which files were taken from main vs. merged manually - Any API mismatches detected and fixed - Any `pyproject.toml` git source reconciliation performed - Links to the CI runs that validated the fixes --- ## Rules - Prioritize main over dev on genuine conflicts. Preserve dev-only additions that do not conflict. - **Two-commit policy:** the PR contains at most two bot-authored commits — the Phase 1 merge commit (immutable once pushed) and a single rolling fix commit on top. The fix commit is created on the first Phase 3 failure (normal push) and amended on every subsequent failure (`git commit --amend --no-edit` + `git push --force-with-lease`). Never modify the Phase 1 commit after pushing it; never let the fix-commit count exceed one. - CI triggers via comment: `/ok to test ` - CI runs appear on branch `pull-request/` - Git committer identity: `svcnvidia-nemo-ci` - After editing imports, run `isort` on those files - **Push directly to NVIDIA/Megatron-LM** (not a fork). The bot uses a PAT with write access. CLAUDE.md says "never push directly" but that rule is for human contributors — the sync bot is an exception. --- Source attribution: yo-steven Original project: https://github.com/yo-steven/skills-exploration-20260522/blob/6e1127ffb7690aa8038b172d0ae7cc720754870d/skills/Megatron-Core/nightly-sync/SKILL.md License: Apache-2.0 License information: https://www.apache.org/licenses/LICENSE-2.0 Original upstream instructions; downloaded from Dave AI. ## Upstream license notice SPDX-License-Identifier: Apache-2.0 AND CC-BY-4.0 Copyright (c) 2026 NVIDIA Corporation This project is licensed under a dual license: - Apache License 2.0 (for code and configuration) - Creative Commons Attribution 4.0 International (CC BY 4.0) (for documentation) You may use the contents of this repository under the terms of either license, or both, as applicable to the type of content. ================================================================================ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to the Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by the Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding any notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS ================================================================================ Creative Commons Attribution 4.0 International License (CC BY 4.0) https://creativecommons.org/licenses/by/4.0/ You are free to: Share — copy and redistribute the material in any medium or format for any purpose, even commercially. Adapt — remix, transform, and build upon the material for any purpose, even commercially. The licensor cannot revoke these freedoms as long as you follow the license terms. Under the following terms: Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits. Notices: You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation. No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material. For the full legal text, see: https://creativecommons.org/licenses/by/4.0/legalcode # Dave AI resource metadata Creator / publisher: yo-steven Original project: https://github.com/yo-steven/skills-exploration-20260522/blob/6e1127ffb7690aa8038b172d0ae7cc720754870d/skills/Megatron-Core/nightly-sync/SKILL.md Attribution basis: Source repository record Fields of work: Development & Engineering Roles: Backend Developer, DevOps / SRE, Full-Stack Developer Specialties: Backend Tools: Tool not specified Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: 8bba0e93b3a1595a3064fd12cece999007fb076c0baea59b239367271c4136d1