Documentation

Instances

An instance is a containerized GPU sandbox running on a DGX Spark host. Each instance has a name, one attached SSH key, and a selected GPU profile.

GPU Types

• NVIDIA DGX Spark GB10 - single node, 128 GB unified memory. $0.65 / hour
• 2x NVIDIA DGX Spark GB10 - dual node interconnect, 256 GB unified memory. $1.50 / hour If this profile has no capacity, launch falls back to queueing. See 2x DGX Spark for how the cluster works once you are in.

Lifecycle

Once launched, an instance remains available until you delete it. Dashboard status badges reflect live container state (`running`, `exited`, `unknown`).

• Restart cycles the container without ending the billing session.
• Delete removes the container and ends billing for that session.

SSH Access

Running instances expose an SSH snippet in the format ssh user@<hostname>.ssh.enverge.dev. First connect after create/restart can take up to a minute while relay state settles.

Docker

Docker is pre-installed and already running inside your instance — run your own containers with no setup. Plain builds and runs work normally.

To give a container the GPU, use the nvidia runtime plus the NVIDIA_VISIBLE_DEVICES environment variable:

docker run --rm --runtime nvidia -e NVIDIA_VISIBLE_DEVICES=all \
  nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi

2x DGX Spark cluster

A 2x DGX Spark instance is a two-node GPU cluster behind a single SSH endpoint. You land on node0 (the head). A second Spark (node1, the worker) is attached over a ~200 Gb/s RoCE fabric. You do not SSH to the worker directly — you launch a distributed job from node0 and it uses both GPUs.

Memory model

The tier advertises 256 GB unified memory — that is aggregate across two 128 GB pools joined by the fabric, not one flat address space. No single process or tensor can exceed ~128 GB. Use model parallelism or sharding to spread work across both nodes.

Pre-configured on your instance

• NCCL over RoCE — both ConnectX-7 rails are wired for RDMA (/etc/nccl.conf). NCCL auto-discovers the fabric; no manual NCCL_IB_* exports needed.
• VLLM_HOST_IP — preset to this node’s rail IP (from /etc/profile.d/enverge-cluster.sh) so vLLM’s multi-node control plane advertises the correct address instead of the default-route interface.
• Worker SSH — passwordless SSH from this container to the worker’s container over the rail is already set up for launchers that spawn remote ranks (check /etc/enverge/cluster.md on the instance for this VM’s rail IPs and worker address).

Running a multi-node job

Launch rank 0 here and rank 1 on the worker over the rail. Any NCCL-based stack works once the workload is sharded. The shape is:

torchrun --nnodes=2 --nproc_per_node=1 --node_rank=0 \
         --rdzv_endpoint=<node0-rail-ip>:29400 your_job.py

your_job.py is your distributed script — nothing is pre-installed to run for you. Set --node_rank=1 on the worker, or let Ray / mpirun spawn the remote rank via the pre-configured rail SSH. Replace <node0-rail-ip> with this instance’s rail-2 address (see /etc/enverge/cluster.md for the filled-in template, or ip -4 addr show enP2p1s0f0np0).

Unified memory on GB10

Shared filesystem

There is no cluster-wide shared filesystem yet. Model weights, datasets, and code must be present on each node (or downloaded separately). Plan for N× download size when pulling large checkpoints.

Queue

When capacity is unavailable, launch requests can enter a queue instead of failing. Queue order is first-in, first-out per GPU type.

• You can have one active queue entry per GPU type.
• Queue entries can be cancelled at any time.
• When capacity opens, the oldest matching entry launches automatically.

SSH Keys

SSH keys are managed during instance creation through a named key picker.

• Names are unique per account.
• Only paste public keys, never private keys.
• Keys attached to active instances cannot be deleted.
• Removing an account key does not retroactively remove it from existing instances.

Billing

Billing is metered by usage and settled through Stripe. Session billing spans from create to delete.

Until a card is on file, launch and restart are gated. Set one up from the dashboard banner or the billing section in your user menu.

On a team, only the owner adds or manages the card. Members see read-only “billing managed by {team}” copy instead.

Queueing itself is free. Payment method checks occur when an instance is launched.

Teams

A team lets several people share one payment method. Everyone launches their own instances; usage is tracked per person and billed to the team. Open team from the user menu in the app.

Roles

• Owner - creates the team, manages billing, sends invites, sees each member's usage, and can remove members or revoke pending invites.
• Member - launches instances billed to the team. Cannot add a personal payment method. Sees their own usage in view usage; the owner sees everyone's totals on the team page.

Create a Team

Any signed-in user without an active team can create one from team in the user menu. You become the owner. If you already had solo billing, it moves to the team - you are not billed twice.

Invite a Member

• Owner enters an email on the team page and clicks send invite.
• Invitee receives an email with an accept link.
• New users sign up first (the invite authorizes that email), then accept. Existing users sign in, then accept.
• Once active, the member can launch instances as soon as the team has a payment method on file.

Limits

• One active team per user.
• One payment method per team.
• Removing a member (or revoking a pending invite) takes effect immediately. Removed members fall back to solo billing if they set up their own card.

Usage History

Usage pages show both runtime sessions and queue history. Team members see only their own sessions and queue history; the team owner sees per-member totals on the team page.

• Sessions table: one row per create-to-delete run.
• Queue history: wait duration and final outcome (launched/cancelled).
• Both views are paginated and refreshable from the UI.

Account

• Sign in with email/password or magic link.
• Password reset flow is available from sign-in.
• Sign out is available from the user menu in the app.