Standard W&B Setup via Environment Variables (API Key, Project, Watch)

General Python For Data Science & Machine Learning
, , , ,
1

Summary

This snippet configures Weights & Biases (W&B) by setting environment variables for:

  • authentication (WANDB_API_KEY)

  • default project assignment (WANDB_PROJECT)

  • model/gradient watching behavior (WANDB_WATCH)

This is a lightweight way to standardize experiment tracking across notebooks, scripts, and training pipelines.

Security Requirement (Mandatory)

Never hardcode or paste WANDB_API_KEY into notebooks, Discourse, README files, GitHub, or shared chat logs. Treat it like a password.

Use one of these approaches instead:

  • local .env file (excluded via .gitignore)

  • CI/CD secrets (GitHub Actions, Azure DevOps, etc.)

  • OS-level environment variables (preferred for production)

Recommended Snippet

Option A: Set in code (acceptable for local-only use if the key is sourced securely)

import os

# Do NOT hardcode the key here. Pull it from a secure source.
# Example: os.environ["WANDB_API_KEY"] = os.getenv("WANDB_API_KEY")

os.environ["WANDB_PROJECT"] = "<Dataset>_<Model>"
os.environ["WANDB_WATCH"] = "all"

Option B: Set in shell (preferred)

Linux/macOS

export WANDB_API_KEY="***"
export WANDB_PROJECT="<Dataset>_<Model>"
export WANDB_WATCH="all"

What Each Variable Does

WANDB_API_KEY

Authenticates the machine/user to your W&B account/workspace.

WANDB_PROJECT

Sets the default project name (e.g., MyDataset_GPT) so runs automatically group correctly without requiring code changes.

WANDB_WATCH

Controls how W&B watches the model (parameters/gradients) when wandb.watch() is used by your framework/integration.

  • "all" is the most verbose (can be expensive/noisy for large models).

  • Alternatives often used: "gradients", "parameters", "false" (exact behavior can depend on integration).

Recommendation: default to "gradients" or "false" for large-scale training unless you explicitly need full parameter/gradient logging.

Typical Minimal W&B Init Pattern

import wandb

wandb.init(project=os.environ.get("WANDB_PROJECT", "default-project"))

Many training frameworks (e.g., Lightning, Hugging Face Trainer) can pick up WANDB_PROJECT automatically when W&B logging is enabled.

Common Pitfalls

  • Accidentally committing secrets: add .env to .gitignore, use secret scanners, and keep keys out of notebooks.

  • Too much logging overhead: "all" can significantly slow training or bloat run sizes on big models.

  • Wrong project naming: ensure consistent naming across teams to avoid fragmented dashboards.

Metadata

Category: Engineering → MLOps → Experiment Tracking
Tags: wandb, experiment-tracking, secrets, mlops, python