perf(ssh,modal): bulk file sync via tar pipe and tar/base64 archive

SSH: symlink-staging + tar -ch piped over SSH in a single TCP stream.
Eliminates per-file scp round-trips. Handles timeout (kills both
processes), SSH Popen failure (kills tar), and tar create failure.

Modal: in-memory gzipped tar archive, base64-encoded, decoded+extracted
in one exec call. Checks exit code and raises on failure.

Both backends use shared helpers extracted into file_sync.py:
- quoted_mkdir_command() — mirrors existing quoted_rm_command()
- unique_parent_dirs() — deduplicates parent dirs from file pairs

Migrates _ensure_remote_dirs to use the new helpers.

28 new tests (21 SSH + 7 Modal), all passing.

Closes #7465
Closes #7467
This commit is contained in:
kshitijk4poor 2026-04-11 11:17:17 +05:30 committed by alt-glitch
parent 723b5bec85
commit 04d4f41e77
5 changed files with 897 additions and 5 deletions

View file

@ -10,6 +10,7 @@ import logging
import os
import shlex
import time
from pathlib import Path
from typing import Callable
from tools.environments.base import _file_mtime_key
@ -60,6 +61,16 @@ def quoted_rm_command(remote_paths: list[str]) -> str:
return "rm -f " + " ".join(shlex.quote(p) for p in remote_paths)
def quoted_mkdir_command(dirs: list[str]) -> str:
"""Build a shell ``mkdir -p`` command for a batch of directories."""
return "mkdir -p " + " ".join(shlex.quote(d) for d in dirs)
def unique_parent_dirs(files: list[tuple[str, str]]) -> list[str]:
"""Extract sorted unique parent directories from (host, remote) pairs."""
return sorted({str(Path(remote).parent) for _, remote in files})
class FileSyncManager:
"""Tracks local file changes and syncs to a remote environment.