Cloud-Native Geo in Practice
Choosing Between COG,
Zarr, and Virtual Zarr:
Lessons from NASA on
Cloud-Native Format Decisions
COG
Materialized
Zarr
Virtual Zarr
Presented by
Siddharth Chaudhary
Data System Evolution (DSE)
NASA Office of Data Science and Informatics (ODSI)
Team
Aimee Barciauskas, Max Jones, Sean Harkins
Brianna Corremonte, Brian Freitag
01 / 13
REFRAME THE QUESTION
The Format Is Not the Decision
Proposed design guidance
Format-first.
Workload-first.
Keep the original science intact.
Choose the representation based on how the data will be used.
KNOW WHAT IS ACTUALLY STORED
Three Formats, Three Physical Models
Two physical layouts and a virtual reference layer.
COG
Materialized Zarr
Virtual Zarr
Raster blocks + overviews.
Spatial layout chosen at build.
Stored array chunks.
Choose chunks + shards.
References + retained sources.
Physical layout is inherited.
DESCRIBE THE QUERY
Start With How You Use the Data
I need a map window
Small area, one time
I need a time series
Same place, many times
I need several dimensions at once
Time, space, variables, levels.
PERFORMANCE
Latency / throughput / concurrency
DATA / FORMAT
Layout / codecs / client support
MAINTENANCE
Updates / cost / accountable owner
FROM WORKLOAD TO A SHORTLIST
Decision Tree
Simplified view of the supplied workload advisor.
All matching checks apply.
Cautions are recorded, not subtracted.
Up to 3 compatible options
within 6 points of the leader.
PROPOSED LOOP
Benchmark equivalent work.
Record the decision and revisit triggers.
Retain the existing path
or combine representations.
GHG CENTER / MICASA
Case Study 1: When COG Makes Sense
WORKLOAD monthly map / CONTEXT published raster assets / CANDIDATE COG
Browse one month.
Inspect a spatial pattern.
Documented catalog snapshot
MiCASA monthly carbon flux v1
Example layer Rh, December 2010
Asset grid 0.1°
Published asset g C m⁻² day⁻¹
Presenter window 2001-2025
Saved catalog 2001-2024; verify 2025
PUBLISHED ACCESS PATTERN
WHAT WOULD CHANGE THE CHOICE?
Frequent multi-year point series.
Test a time-oriented derivative.
PROVISIONAL CASE / MUR SEA-SURFACE TEMPERATURE
Case Study 2: When Zarr Makes Sense
WORKLOAD regional multi-day analysis / CONTEXT layout control / CANDIDATE materialized Zarr
Design the array layout
for repeated analysis.
External NASA-data example; replaceable with a DSE case.
NASA JPL MUR Level 4 SST v4.1
Public store managed by Farallon Institute.
Listed extent June 2002-January 2020
Documented chunks (5, 1799, 3600)
Dimensions time, lat, lon
Documented layout, not a universal optimum.
CHOOSE CHUNKS FOR THE QUERY
Compatible xarray / Dask analysis.
WHAT WOULD CHANGE THE CHOICE?
An adequate source layout or rare queries.
A rewrite may not repay its lifecycle cost.
AIR4US / PRESENTER-REPORTED ODSI DSE WORK
Case Study 3: When Virtual Zarr Makes Sense
WORKLOAD compatible array access / CONTEXT retain existing sources / CANDIDATE Virtual Zarr
Presenter-reported implementation
Conceptual access architecture
Retain the archive. Expose a logical array.
CHANGE THE CHOICE Source geometry or reader limits fail the workload. GRIB message boundaries can still govern reads.
Real-world use cases / NetCDF baselines
Benchmark. NLDAS-3 and TEMPO.
NLDAS-3 / REPORTED SUMMARY
Time-series extraction.
Virtual Zarr versus NetCDF
≈200×
End-to-end
≈400×
Dataset opening
≈20×
Query / 1 point
≈30×
Query / 5 points
Dataset of interest for the
Water Insights portal.
TEMPO / MARCH 14, 2025
Virtual Zarr versus Remote NetCDF
≈140×
≈12–14×
250 points
50–500 km AOI side
Median paired speedup. 1× = equal. Headline values rounded from plot.
GENERAL GUIDANCE / DSE INCIDENTS PENDING
Common Anti-Patterns
THE MISTAKE
THE CORRECTION
01
Convert the whole archive first.
→
Test a representative sample first.
02
Use one layout for every query.
→
Separate map and analytical needs.
03
Expect references to fix layout.
→
Inspect chunks, codecs and sources.
04
Time a lazy open, then stop.
→
Execute equivalent work; count bytes.
DSE examples remain editable in the notes. No incident is invented or attributed to the team.
A PROPOSED SHARED-RESPONSIBILITY MODEL
Operational Decision: Who Decides?
Not a NASA-wide mandate
Data producers
Scientific meaning
Data quality + metadata
Acceptance tests
Data archivers
Preserve data + versions
Access + provenance
Enable reliable access
Data users
Representative queries
Analysis + derivatives
Feedback + reuse
ONE DECISION RECORD
Workload + source + alternatives + acceptance criteria
Accountable owner + update strategy + revisit trigger
A PROPOSED SYNTHESIS FOR THIS TALK
Mental Model (recap)
01
What are the functional user requirements?
02
What limitations do different solutions introduce?
03
What are the performance and financial costs of different solutions?
04
Who owns validation, updates and source stability?
05
What evidence would confirm or change this choice?
A HYBRID CAN BE VALID.
Retained archive
+ optional virtual access
Materialized high-value
analytical subsets
Map-oriented
derivatives
Only where justified.
CHOOSE DELIBERATELY
Takeaways
Choose the workload.
Then choose the layout.
01
Start with users, queries and constraints.
02
Know where layout control and costs sit.
03
Benchmark equivalent work before scaling.
04
Plan validation, ownership and updates.
Write one query. Test candidate layouts. Record the trade-off.