Operations depend on the target layout. Not every conversion needs every step.
Cloud-Native Geo in Practice
Beyond COGs and Zarr:
Building Truly Cloud-Native
Raster Workflows at NASA
with Virtual Zarr
Do we always need to physically transform an existing Earth science archive to make it cloud-native?
COG
Materialized
Zarr
Virtual Zarr
Presented by
Siddharth Chaudhary
Data System Evolution (DSE)
NASA Office of Data Science and Informatics (ODSI)
Team
Aimee Barciauskas, Max Jones, Sean Harkins
Brianna Corremonte, Brian Freitag
01 / 15
01 / The architectural choice.
The Cloud-Native Raster Problem.
Cloud-native access can introduce a second physical representation.
The decision.Is a complete rewrite necessary for this workload?
02 / Three useful representations.
COG vs Zarr vs Virtual Zarr.
COG.
Physical raster layout.
Arrange pixels and overviews for raster access.
Physical Zarr.
Physical array layout.
Arrange chunks across the dimensions you need.
Virtual Zarr.
Logical array layout.
Reference compatible chunks already in the archive.
No universal winner.These are architectural choices, not mutually exclusive ecosystems.
03 / The conceptual pivot.
Virtual Zarr.
“Make the data look like Zarr without necessarily
making the data become Zarr.”
Payload / request path Reference lookup
Still real I/O.References describe where the science payload lives.
04 / Construction is a separate stage.
From Existing NetCDF to Virtual Zarr.
● Documented capabilityVersion-matched NLDAS-3 recipe, not a reproduced production build.
1
Inspect.
Time, grid and units
Chunk layout + codecs
→
2
Generate.
Parse source files
Create references
→
3
Combine.
Order daily records
Check compatibility
→
4
Persist.
Write a repository
Commit + validate
→
5
Reopen.
Compatible store
Read with xarray
NEW ARTIFACTS.
References, array metadata and selected coordinates.
UNCHANGED SOURCE LAYOUT.
Original files, source chunks and their compression.
The handoff.Serialize the virtual object, then reopen it for analysis.
05 / One worked example throughout.
NASA Use Case. NLDAS-3.
NLDAS_FOR0010_D · 030.beta
Daily meteorological forcing.
Not a land-surface model-output demonstration.
2001–2023Beta forcing coverage
reported by the provider
reported by the provider
1 kmNominal spatial resolution
on a 0.01° grid
on a 0.01° grid
SOURCE FILE ORGANIZATION.
daily/{yyyymm}/
NLDAS_FOR0010_D.A{yyyymmdd}.030.beta.ncSchematic layout. Not a scientific map.
The inherited constraint.A logical view does not change the source chunk geometry.
06 / Demo recipe.
Demo. Building the Virtual Store.
◇ Adapted code. Not executed here.Three-file local subset. Full setup and validation are in notes.
Python 3.12 / VirtualiZarr 2.0.1 / Icechunk 1.1.1
vds = open_virtual_mfdataset(
urls, registry=registry, parser=HDFParser(),
combine="nested", concat_dim="time",
coords="minimal", compat="equals",
loadable_variables=["time", "lat", "lon"],
decode_times=True,
)
session = build_repo.writable_session("main")
vds.vz.to_icechunk(session.store)
snapshot_id = session.commit("NLDAS-3 daily subset")Created.
A committed store, metadata and coordinates.
Not rewritten.
Source files and encoded data chunks.
Before publishing.
Validate values, coordinates, units and masks.
07 / Demo recipe
Demo xarray to Analysis
◇ Adapted code. Not executed here.Published Icechunk path. Source and store credentials are configured separately.
Read-only session / xarray 2025.6.0 / Zarr format 3
session = repo.readonly_session("main")
ds = xr.open_zarr(session.store, consolidated=False,
zarr_format=3, chunks={})
point = ds["Tair"].sel(
lat=40.0, lon=-100.0, method="nearest")
series = point.sel(time=slice("2020-01-01", "2020-01-31"))
assert series.attrs.get("units") == "K"
series = series.compute()
series.plot()Lazy selection is not retrieval.
.compute() is the execution boundary.DEMONSTRATION PLAN.One point.
One point.
One month.
Tair in Kelvin
January 2020 time labels
Nearest cell to
40° N, 100° W
No scientific output was computed for this deck.
08 / A separate delivery layer
Demo Virtual Zarr to Tiles
Custom opener.
Connect Icechunk sessions to the reader.
Correct pixels.
Validate masks, georeferencing and resampling.
Measure the path.
Measure chunk reads, requests and tile latency.
The distinction.A logical array is an input to a service, not a tile service by itself.
Real-world use cases / NetCDF baselines
Benchmark. NLDAS-3 and TEMPO.
NLDAS-3 / REPORTED SUMMARY
Time-series extraction.
Virtual Zarr versus NetCDF
≈200×
End-to-end
≈400×
Dataset opening
≈20×
Query / 1 point
≈30×
Query / 5 points
Dataset of interest for the
Water Insights portal.
TEMPO / MARCH 14, 2025
Virtual Zarr versus Remote NetCDF
≈140×
≈12–14×
250 points
50–500 km AOI side
Median paired speedup. 1× = equal. Headline values rounded from plot.
Workload-specific.Separate cases, not a COG or materialized-Zarr comparison.
10 / Cost follows the lifecycle.
What Actually Costs Money?
COMPARISON BASELINE.
The original archive is retained under both physical and virtual designs.
1
One time.
Inspect and engineer.
Rewrite or build references.
Validate equivalence.
PREPARATION.
2
Recurring.
Retain source storage.
Keep copies or references.
Refresh, validate and operate.
STEWARDSHIP.
3
Per query.
Resolve and request bytes.
Decode and compute.
Transfer and serve results.
CONSUMPTION.
The principle.Virtualization reduces costs, but doesn’t eliminate them
11 / KNOW THE TRADE-OFFS
Where Virtual Zarr Can Become Challenging
◇ General engineering risks. Not observed failures here.
01
Chunk geometry mismatch
Long point histories can intersect many spatial chunks, increasing the amount of data that must be accessed.
Test a different physical layout for the hot workload.
02
Request amplification
Many small objects or fragmented reads can increase the number of requests and access overhead.
Measure requests, useful bytes and concurrency.
03
Encoding or schema mismatch
Codecs, fill values, scales, coordinates or array metadata may not align with the intended representation.
Validate the resulting dataset rather than assuming the mapping is correct.
04
Source identity changes
Replaced, deleted or inaccessible source objects can invalidate the virtual mapping.
Pin source identities, maintain access and monitor for changes.
LIMITS ARE PART OF THE DESIGN.Virtual Zarr exposes the source layout through a virtual multidimensional view; it does not make that layout disappear.
12 / FROM LESSONS TO PRACTICE
Design Patterns for Production
01
Start from the query
Inspect source layout and benchmark the real workload
02
Make equivalence measurable
Compare decoded values, coordinates, units and masks
03
Version both layers
Record source identity and reference snapshot. Name an owner
04
Monitor the read path
Control access, observe requests and detect broken references
A PRACTICAL HYBRID
Materialize where repeated demand justifies it.
KEEP DISCOVERY SEPARATE.An access URI points to the store; it doesn't define the virtual view.
13 / Takeaways
Takeaways
01
Cloud-native access does not always require a full rewrite.
02
Each representation puts costs in different places.
03
Source layout must fit the access pattern.
04
Less duplication still requires operational care.
Choose the workflow.
Then choose the representation.
Cloud-Native Geo in Practice Beyond COGs and Zarr: Building Truly Cloud-Native Raster Workflows at NASA with Virtual Zarr Do we always need to physically transform an existing Earth science archive to make it cloud-native? TILED RASTER overviews COG x y time Materialized Zarr refs x / y / t source bytes Virtual Zarr Presented by Siddharth Chaudhary Data System Evolution NASA Office of Data Science and Informatics Team Aimee Barciauskas, Max Jones, Sean Harkins Brianna Corremonte, Brian Freitag 01 / 15