Graphistry Core

• was: Single GPU configuration with limited multi-GPU support, GPU assignment conflicts, suboptimal worker distribution.
• now: Per-service GPU assignment (Docker Compose) and multi-GPU pod support (Kubernetes) with round-robin load distribution, fail-fast validation, structured observability.
• status: released | action: Docker Compose: configure using service-specific env vars in custom.env; Kubernetes: set pod resource limits + worker count env vars.
• impact: All multi-GPU deployments - prevents GPU hotspots, silent failures, and enables per-GPU performance monitoring.
• note: Compatible with both Docker Compose and Kubernetes orchestrators.
  • Docker Compose: Added 7 service-specific CUDA_VISIBLE_DEVICES env vars for per-service GPU isolation: GAK_PUBLIC_CUDA_VISIBLE_DEVICES, GAK_PRIVATE_CUDA_VISIBLE_DEVICES, STREAMGL_CUDA_VISIBLE_DEVICES, FORGE_CUDA_VISIBLE_DEVICES, DASK_SCHEDULER_CUDA_VISIBLE_DEVICES, DCW_CUDA_VISIBLE_DEVICES, NOTEBOOK_CUDA_VISIBLE_DEVICES.
  • Zero-config default: All GPU services default to GPU 0 when unconfigured. Release script defaults global CUDA_VISIBLE_DEVICES to 0, compose files use :-0 fallback. Single-GPU deployments work without the GPU Configuration Wizard or any env vars.
  • Kubernetes: Fully compatible with NVIDIA device plugin GPU assignment, respects pod-level CUDA_VISIBLE_DEVICES.
  • Round-robin distribution (worker_index % num_gpus) for forge-etl-python, dask-cuda-worker, streamgl-gpu prevents hotspots (e.g., 2 GPUs + 5 workers = 3:2 ratio instead of 1:4).
  • Logs warning when workers < GPUs for intentional underutilization (matches PyTorch/dask-cuda behavior). Round-robin assignment to available GPUs. Supports all-integer (0,1,2) and all-UUID (GPU-xxx,GPU-yyy) formats for VMware/Nutanix/MIG compatibility.
  • Standardized GPU init logging: [SERVICE_NAME] GPU initialized - logical_device=X/Y physical_gpu=Z memory=A.BGB/C.DGB.
  • OpenTelemetry metrics enhanced with GPU assignment attributes for per-GPU performance analysis.
  • GPU watcher: each FEP worker monitors exactly one GPU assigned via init scripts, using CUDA_VISIBLE_DEVICES value (integer index or UUID) directly in logs.
  • Init scripts validate CUDA_VISIBLE_DEVICES at startup: accepts integer indices (0, 1, 2) or UUIDs (GPU-xxx, MIG-xxx), rejects invalid formats with clear errors.
  • OpenCL validation: integer-only CUDA_VISIBLE_DEVICES required for streamgl-gpu (OpenCL limitation), fail-fast prevents silent failures where workers compete for same GPU.

• was: Manual GPU configuration for multi-GPU deployments, often misconfigured leading to underutilization or contention.
• now: Auto-detects GPU hardware or uses 100+ presets (DGX, HGX, cloud instances) on cloud VM first boot (AWS, Azure, GCP), generates optimal worker settings with round-robin distribution.
• status: released | action: Cloud: auto-configured on first boot. Air-gapped/manual: Run ./etc/scripts/gpu-config-wizard.sh -E ./data/config/custom.env or -p dgx-a100 for planning.
• impact: All multi-GPU deployments - eliminates manual config errors, optimal worker distribution, idempotent and safe for reboots.
• note: Replication model: 4 forge, 4 streamgl, 1 dask per GPU by default. Idempotent -- preserves user customizations on reboots. Included in cloud images and air-gapped tarballs.

• was: Deployment script named release was confusing (sounds like creating a software release, not running services).
• now: Renamed to graphistry. Primary command: ./graphistry up -d. All wrappers (./dc, ./dc.dev, ./nc) updated to call ./graphistry. Bundler includes graphistry in tarballs. Systemd services use ./graphistry. Deleted unused dc.release.
• status: released | action: Use ./graphistry instead of ./release for deployments.
• impact: All deployments - more intuitive command. Configure via data/config/custom.env (log levels, GPU) and data/config/telemetry.env (monitoring).

• was: Limited visibility into encodingsApplied flag in OpenTelemetry traces, difficult to debug multi-window encoding behavior
• now: encodingsApplied flag included as span attribute throughout encoding flow (applyEncodings, applyEncodingsService, setDefaultEncoding, applyCollectionsOnNBody) and in critical logs
• status: released | action: none required
• impact: Operators debugging multi-window visualization issues - full visibility into encoding guard decisions

• was: PM2 CPU memory restart threshold hardcoded, limiting operator control over worker memory management.
• now: MAX_MEMORY_RESTART env var controls when PM2 restarts the visualization worker due to CPU memory usage (default: 5GB).
• status: released | action: Only needed for very large graphs. Set MAX_MEMORY_RESTART in custom.env (e.g., MAX_MEMORY_RESTART=8G) to increase threshold.
• impact: Operators managing high-capacity deployments with large graphs - prevents premature worker restarts during memory-intensive visualizations.

• was: Docker compose wrappers (./graphistry, ./dc, ./dc.dev, ./nc) always ran with verbose trace (set -x), flooding logs. Scripts continued after errors, risking partial deployments. No verbosity control.
• now: Production (./graphistry) runs quietly with fail-fast (set -e). Dev scripts (./dc, ./dc.dev, ./nc) show trace by default. New GRAPHISTRY_DC_DEBUG env var controls verbosity across all scripts.
• status: released | action: Production - no action needed. Dev - set GRAPHISTRY_DC_DEBUG=false for quieter output. Debug production - set GRAPHISTRY_DC_DEBUG=true for verbose trace.
• impact: All deployments - cleaner production logs, fail-fast prevents partial deployments, configurable verbosity.

• was: Nexus container lacked access to the shared data volume for uploaded file artifacts. Disk cleanup during deletion silently failed, leaving orphaned files on disk indefinitely.
• now: Shared data volume mounted in Nexus container in release.yml. Deletion now correctly removes both database records and on-disk artifacts.
• status: fixed | action: After upgrading, restart Nexus: ./graphistry up --no-build --no-deps --force-recreate -d nexus
• impact: All Docker Compose deployments - prevents disk space accumulation from orphaned files never cleaned up on delete.
• note: Kubernetes (Nexus Helm chart) deployments were not affected - Nexus already had the correct data volume mount.

• was: Cloud VM images (Azure, AWS, GCP) and S3 tarballs could ship without core containers (caddy, graph-app-kit-st, autoheal, redis), pulled on first boot instead of pre-baked, breaking airgapped deployments and adding startup latency.
• now: Docker packages pinned to versions matching Docker Engine 23.0.1 (docker-compose-plugin=2.16.0, docker-buildx-plugin=0.10.2), fixing API version mismatch that caused silent pull failures. Removed --ignore-pull-failures from cloud build scripts so missing containers fail the build immediately.
• status: fixed | action: No action required for new deployments. Existing VMs from affected images will pull missing containers on boot if internet is available.
• impact: All cloud and airgapped deployments - VM images and S3 tarballs now ship with all required containers pre-baked, enabling fully offline operation.

• was: Cloud build scripts (S3 bundler, Azure, AWS, GCP Packer) used blanket docker compose pull pulling every service including retagged images whose tags (e.g., graphistry/streamgl-gpu:VERSION) don't exist remotely. Fatal in S3 bundler; silently ignored in Packer.
• now: Replaced with targeted pulls by service name for images that need pulling (redis, autoheal, graph-app-kit, telemetry). Fixed S3 bundler ordering to run after explicit pulls and retags.
• status: fixed | action: No action required for new deployments.
• impact: All cloud and airgapped deployments - build scripts now pull all required images reliably without silent failures.

• was: GCP images failed on first boot with Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument. All pre-baked containers unusable, requiring manual recovery. Both CUDA 11.8 and 12.8 affected.
• now: Docker daemon and containerd gracefully stopped before GCP image capture, ensuring overlay2 metadata is flushed to disk. GCP Packer hard-terminates instances (unlike AWS), so explicit shutdown is required.
• status: fixed | action: No action required for new deployments. Existing GCP images must be rebuilt.
• impact: All GCP deployments - images now boot and start all Docker services correctly on first launch.

• was: Outdated docs with unclear script relationships and stale references.
• now: Documents ./graphistry as primary command (wraps docker compose with GPU, telemetry, cluster context). Explains script hierarchy (./dc, ./dc.dev, ./nc use ./graphistry internally). Debugging guidance via log settings and telemetry.
• status: released | action: none required
• impact: All users - clearer deployment docs

• was: Deleting a file via API removed the database record but provided no way to check if the file was still referenced by datasets. No visibility into file dependencies before deletion.
• now: New ?hard=true parameter on DELETE /api/v1/files/<id>/ checks dataset references before deleting. Returns 409 listing referencing datasets if dependencies exist. Use ?force=true to override, ?dry_run=true to preview. Staff-only.
• status: released | action: none required - default behavior unchanged. Use new parameters for dependency-aware deletion.
• impact: Administrators managing file storage and compliance audits - safely identify and remove unused files without breaking datasets.

• was: Collections panel had a basic interface with limited set creation and simple operators (equals, greater than, less than). Creating sets required navigating away from the panel. Set editor was a single monolithic component with limited flexibility.
• now: Redesigned collections panel with modernized layout, updated color palette, dynamic positioning, and set description display. Create sets directly via "Add Collection" button. Intersection members shown as named pill badges. Set editor supports advanced query building with independent operations and conditions, new string operators (Contains, Starts With, Ends With, In List, Is Null, Is Not Null, Not Equal), node/edge mode selection with attribute filtering per type, edge traversal config (direction, hops, to-fixed-point), and per-condition case sensitivity.
• status: released | action: none required
• impact: All users working with collections - improved workflow for creating, editing, and managing graph filter sets with more powerful query capabilities.

• was: SSO login authenticated users but did not use IdP claims for role or organization assignment. Administrators had to manually assign permissions and org memberships after onboarding.
• now: Configure claim field mappings in SSO provider settings to auto-assign roles (viewer/editor) and org memberships from IdP claims at login. Supports separate claims or combined claims via format templates (e.g., {role}-{org}). Multi-org from single login. Org name resolution uses 3-tier matching: explicit mapping dictionary, case-insensitive slug, normalized slug. Compatible with ADFS, Okta, Azure Entra, Auth0, Keycloak, and custom OIDC. Legacy key formats handled automatically.
• status: released | action: Configure claim field mappings in the SSO provider admin form. Add a Role mapping with your IdP's claim name (e.g., roles, groups, <http://schemas.microsoft.com/.../claims/role>). Optionally add Organization mapping and value mapping dictionaries.
• impact: Enterprise SSO administrators - eliminates manual role and org assignment, ensuring permissions are consistent with IdP policy.

• was: No limit on failed login attempts, allowing unlimited password guessing attacks.
• now: Rate-limited per user and per IP. After 10 failed attempts from same IP, or 20 total from any IP, blocked for 5 minutes.
• status: released | action: none required (enabled by default). Configure via RATE_LIMIT_MAX_LOGIN_ATTEMPTS, RATE_LIMIT_MAX_LOGIN_ATTEMPTS_PER_IP, RATE_LIMIT_WINDOW_SECONDS env vars.
• impact: All users - improved account security against password guessing and credential stuffing.
• note: Applies to username/password form login. SSO, LDAP, and API token auth use their provider's rate limiting. Admins can clear rate limits for locked-out users via Django admin.

• was: URL parameters for collections could have casing altered during parsing, corrupting JSON-encoded values and causing errors when loading saved states.
• now: URL parameter parsing preserves original casing, ensuring JSON values and case-sensitive data remain intact.
• status: fixed | action: none required
• impact: Users sharing or bookmarking visualization URLs with collections - saved states load reliably without parsing errors.

• was: Authentication cookies with additional user attributes (staff status, superuser flag) caused JSON parsing errors and log noise.
• now: Cookie parser handles all Python-style values in JWT tokens, eliminating parsing errors.
• status: fixed | action: none required
• impact: All users - reduced error log noise and improved authentication reliability.

• was: Encoding calculations (colors, sizes, icons) computed all statistics upfront, slowing graph load and interactive encoding changes.
• now: Optimized pipeline computes statistics on-demand, improving responsiveness.
• status: fixed | action: none required
• impact: All users - faster graph loading and encoding changes, especially with large datasets.

• was: Graphs with standalone nodes (no edges) failed to display when play mode was enabled.
• now: Standalone nodes render properly with play mode active.
• status: fixed | action: none required
• impact: Users visualizing graphs with isolated nodes in play mode.

• was: Changing histogram color encoding attribute (e.g., from "type" to "degree") via dropdown could crash with an internal error.
• now: Encoding changes work reliably. Users can freely switch between attributes in the histogram dropdown without errors.
• status: fixed | action: none required
• impact: Users changing histogram color/size encodings via the dropdown selector

• was: Opening the same visualization in multiple windows could result in histogram colors not appearing in subsequent windows, even though graph colors rendered correctly.
• now: Histogram legend data computed fresh for each window while skipping redundant GPU operations for already-applied encodings.
• scope: Fix applies to initial graph load only; user-initiated encoding changes and collections work as before.
• status: fixed | action: none required
• impact: Users opening same visualization in multiple tabs/windows - histogram colors now display consistently.
• technical: streamgl-gpu is now single source of truth for encodingsApplied flag, passed via Arrow metadata to frontend and through Falcor to streamgl-viz.

• was: Applying filters that hide all nodes could cause the GPU worker to crash and restart, leading to visualization failures
• now: GPU worker handles zero-node filter results gracefully without crashing
• status: fixed | action: none required
• impact: Users applying aggressive filters that exclude all nodes from the visible graph

• was: In multi-tab scenarios, removing a filter that was created in a different browser tab could crash the visualization
• now: Filter removal handles multi-tab sync gracefully, logging the issue instead of crashing
• status: fixed | action: none required
• impact: Users working with filters across multiple tabs on the same visualization

• was: GPU watcher restarts due to high memory usage did not fully release GPU memory. Individual workers were terminated but others retained stale GPU contexts, causing cumulative memory leaks.
• now: Watcher triggers full container restart for complete GPU context cleanup and fresh state. Telemetry flushed before restart and during graceful shutdown. Docker Compose: container exits, Docker restarts per restart policy. Kubernetes: container exits, kubelet restarts within same pod (DaemonSet).
• status: fixed | action: none required
• impact: All GPU deployments using the GPU watcher - GPU memory properly released on watcher-triggered restarts, preventing cumulative leaks that previously required manual container restarts.

• was: v2 endpoints (e.g., /api/v2/my/organizations/) ignored query parameters like ?limit=20 or ?offset=10, breaking pagination and filtering. Default page sizes applied regardless of requested limits.
• now: Query parameters correctly forwarded to backend. Pagination, filtering, and search work as expected across all v2 endpoints.
• status: fixed | action: none required
• impact: All users and integrations using v2 API endpoints - dropdown lists, data tables, and API clients now correctly respect limit/offset parameters.

• was: Loading large datasets could fail with connection reset errors, especially under concurrent load.
• now: Improved async event loop handling keeps TCP connections stable during heavy GPU processing.
• status: fixed | action: none required
• impact: Users loading large datasets - fewer connection failures.

• was: GPU initialization errors in the visualization worker were silently swallowed, making debugging difficult.
• now: OpenCL and GPU-related errors properly logged for diagnostics.
• status: fixed | action: none required
• impact: Administrators troubleshooting GPU issues - clearer error messages.

• was: Opening the same dataset in multiple windows could cause incorrect colors in some windows.
• now: Concurrent session handling serializes encoding operations to prevent cross-session interference.
• status: fixed | action: none required
• impact: Users opening the same visualization in multiple tabs - consistent color encodings.

• was: Data table took several seconds to load rows on large graphs (63k+ nodes), causing UI lag during scrolling and pagination.
• now: Optimized row loading via Apache Arrow's native table operations, reducing load times from seconds to milliseconds.
• status: fixed | action: none required
• impact: All users viewing data tables on medium-to-large graphs - significantly faster responsiveness.

• was: Data table search used an asynchronous algorithm that could be slow on large datasets.
• now: New synchronous algorithm provides faster, more responsive search results.
• status: fixed | action: none required
• impact: All users searching within data tables - improved responsiveness.

• was: Sorting columns could crash due to comparator transitivity edge cases with certain data values.
• now: Sort comparator handles all edge cases correctly, preventing crashes.
• status: fixed | action: none required
• impact: Users sorting data table columns with mixed or edge-case values.

• was: During GPU crashes, trace spans were never ended before process exit, losing crash telemetry. Error handlers used span methods after the span should have been closed.
• now: Span properly ended after recording crash metrics and before flushing telemetry. Error handlers use logger directly since span is closed.
• status: fixed | action: none required
• impact: Operators debugging GPU crashes - crash traces now properly recorded in OpenTelemetry.

• was: After a crash or restart, the router could route sessions to a worker before its GPU context was initialized. Workers marked available on process startup, before OpenCL initialization completed.
• now: Workers must be both online (process running) AND ready (GPU context initialized) before receiving sessions. Router waits for explicit readiness signal after GPU initialization.
• status: fixed | action: none required
• impact: Users starting GPU sessions after worker restarts - fewer startup failures and more reliable recovery.

• was: GPU worker crashes and OpenCL errors only partially captured in traces, making it hard to understand restart causes.
• now: Traces include clearer error details, properly recording OpenCL failures and worker crashes.
• status: fixed | action: none required
• impact: Operators and developers diagnosing GPU crashes via OpenTelemetry in production.

• was: Visualizations could crash with "Invalid command queue" errors during initial layout, especially with clustered positions. Worker crashes caused all active sessions on that worker to be lost, leading to unexpected page refreshes. Teams on shared session links would lose analysis work.
• now: Fixed GPU thread synchronization in the force-directed layout engine. Root cause: deadlock where GPU threads could exit early while others waited indefinitely during force calculations. Fix ensures proper coordination during tree construction, force computation, and position updates, with safeguards against invalid numerical values propagating through the layout.
• status: fixed | action: none required
• impact: All users - graphs render reliably without unexpected refreshes. Teams on shared graphs no longer risk losing analysis work from GPU crashes. Users with clustered positions can analyze dense datasets without crashes affecting their work or other users on the same worker.
• note: Optional debug infrastructure via FA_JS_DEBUG=true env var.

• was: Graphs with coordinates near the poles (latitude approaching +/-90 degrees) would briefly appear then disappear. Blank visualization with no error message, difficult to diagnose.
• now: Coordinates automatically clamped to Web Mercator bounds (+/-85.051129 degrees) before projection, matching Google Maps, Mapbox, and Kepler.gl standards.
• status: fixed | action: none required
• impact: Users with geolocated datasets containing polar or near-polar coordinates - graphs now render correctly. Full Kepler.gl dual-view map compatibility.

• was: Updating a profile with an email already taken by another user produced unclear or unexpected errors.
• now: Form-level validation displays a clear message next to the email field explaining it is already in use.
• status: fixed | action: none required
• impact: Users and administrators updating email addresses in profile settings or Django admin.

• was: Adding a user with DISABLE_EMAIL=True and GRAPHISTRY_CLOUD=False incorrectly showed invite link messages even when required_invite=False.
• now: Invite link message only displays when required_invite=True.
• status: fixed | action: none required
• impact: Self-hosted administrators adding users to organizations with open joining enabled.

• was: Deleting a file or dataset returned success (HTTP 204) but left data files on disk, causing silent disk space accumulation with no indication to administrators.
• now: Deletion removes both database record and on-disk artifacts. Signal handlers use granular exception handling: missing folders handled gracefully, real I/O errors prevent record removal to ensure data integrity.
• status: fixed | action: none required
• impact: All deployments - disk space now properly reclaimed on file and dataset deletion.

• was: Failed deletes (files, datasets, named endpoints) dumped raw HTML into the gallery cell, making the page unusable until refresh.
• now: Clean, human-readable error messages extracted from the server response without disrupting page layout.
• status: fixed | action: none required
• impact: All dashboard users - clearer error feedback when delete operations fail.

• was: Legacy workaround retried every failed delete, causing double-delete attempts and confusing error messages.
• now: Removed redundant retry logic. Deletes execute once with clear success or failure feedback.
• status: fixed | action: none required
• impact: All dashboard users - predictable delete behavior with accurate error reporting.

• was: Changing size encodings could incorrectly redirect to color encoding resets when no collections were present.
• now: Size encodings apply correctly regardless of whether collections exist.
• status: fixed | action: none required
• impact: Users applying size encodings to visualizations without collections

• was: Point size encoding failed when data contained null values, due to out-of-bounds categorical codes in the encoding pipeline.
• now: Out-of-bounds categorical codes detected and converted to NaN, allowing size encodings with null-containing columns.
• status: fixed | action: none required
• impact: Users applying point size encodings to datasets with missing or null values.

• was: Editing an SSO provider with role_mapping or organization_mapping value dictionaries silently dropped those mappings on save, requiring re-entry.
• now: Value mapping dictionaries preserved across form edits. SSO provider form correctly round-trips all mapping data including role and org value dictionaries.
• status: fixed | action: none required
• impact: Admins editing SSO providers with value mapping dictionaries - mappings no longer lost on save.

Legacy VGraph Infrastructure Removal

• was: Platform supported VGraph/protobuf format via streamgl-vgraph-etl (legacy ETL API v1/v2) and forge-etl (TypeScript encoding extraction) services, with @graphistry/vgraph and @graphistry/vgraph-stream libraries.
• now: All VGraph services and libraries removed. Legacy ETL endpoints (/etl, /api/check, /api/encrypt, /api/decrypt, /api/v1/etl/vgraph/) return HTTP 410 Gone. VGraph encoding extraction ported to forge-etl-python. Existing VGraph datasets from previous deployments remain viewable via forge-etl-python.
• status: breaking | action: Clients using /etl endpoint or API v1/v2 must migrate to /api/v2/upload or /api/v1/etl/arrow/
• impact: Legacy integrations using /etl or VGraph format - must migrate to Arrow upload (e.g., PyGraphistry register(api=3), REST /api/v2/upload). Existing VGraph datasets continue to work.

Demo Datasets Storage Format

• was: Demo datasets (Miserables, Facebook, Twitter, etc.) stored as legacy VGraph format in data/datasets_demo/{name}/
• now: Demo datasets stored as Parquet format in data/datasets_demo/parquet/{name}/ with nodes.parquet, edges.parquet, and metadata.json files
• status: breaking | action: Users with scripts accessing demo files must update paths from data/datasets_demo/{name}/ to data/datasets_demo/parquet/{name}/
• impact: Users with scripts that directly access demo dataset files - update paths to new parquet location