S3

Lakehouse resolves the contradiction between the lack of ACID transactions in data lakes and vendor lock-in by data warehouse vendors by building a transactional metadata layer (such as Delta Lake, Apache Iceberg, and Apache Hudi) on top of object storage.

Milvus将数据持久化到对象存储（S3/MinIO），计算节点无状态化，可以独立弹性伸缩

restic supports pushing snapshots to S3-compatible object storage or an SFTP remote.

Hoistbay currently covers only S3 and CloudFront and does not support broader cloud storage or enterprise-grade collaboration/permission management features.

restic supports multiple backends such as local directories, SFTP, and S3-compatible object storage, and B2 (Backblaze B2) is a common one among them.

Production-grade bulk ingestion systems should enforce explicit encoding metadata at the ingestion boundary (HTTP Content-Type headers, S3 object metadata tags, or schema contracts), treating automatic heuristic detection only as a fallback layer.

Storage-compute separation means deploying the data persistence layer and the compute layer independently and scaling them independently; in a search system, index building, merging, and query serving can run on different node pools while sharing the same S3 data.

Putting the search index on S3 is better suited to systems that are read-heavy and write-light, not extremely sensitive to P99 latency, and whose index scale is large enough that storage cost becomes the main pain point; log retrieval, archival search, and cold data analytics are typical beneficiary scenarios.

Local SSDs have microsecond-level random read latency and a fixed hardware cost model, which is entirely different from S3's billing model.

After moving the index to S3, common ways to cope with latency include tiering (keeping hot data in local cache/memory and cold segments on S3), moving prefetching and caching strategies forward, and changing the index structure to reduce the number of objects touched by a single query.

After moving the index to S3, the query path gains an extra hop of network I/O, and a disk read that originally took tens of microseconds may become a millisecond-level object storage request, so the latency budget must be reallocated.

The typical first-byte latency of S3 is on the order of tens of milliseconds, and billing is based on the number of requests (GET/PUT/LIST) and the amount of data transferred.

Tiered caching brings consistency complexity, prefetching brings memory pressure, and coarse-grained objects bring read amplification—these are the respective costs of the means for coping with S3 index latency.

Real-time search with a hard requirement for sub-millisecond latency and frequently updated indexes are still better suited to local disks or in-memory solutions rather than an S3 storage-compute separation solution.

Cloudflare R2 does not charge separately for egress traffic, while S3's download traffic is usually a relatively large part of the bill, so for videos that are frequently shared and viewed by many people, R2 is often more cost-effective.

Reelbird does not rely on an official cloud storage service, but instead allows users to connect their own S3 or Cloudflare R2 storage buckets, and Reelbird itself does not host any content.

The performance data provided by Logic42 shows that when the number of files per partition increases from 4 (512 MB/file) to 18,500 (120 KB/file), Trino planning latency rises from 45 ms to 38,000 ms, and end-to-end query time goes from 480 ms to timeout (OOM).

S3 and Google Cloud Storage are not hierarchical POSIX file systems, and when the engine lists 500,000 files in a single partition it must page through hundreds of metadata pagination markers.

In this case, the real bottleneck of H100 training was non-GPU overhead in the data pipeline, including repeatedly downloading the dataset from S3, the base model not being cached, and single-threaded preprocessing with num_workers set too low.

The dataset is about 25GB and stored in an S3 bucket, and every training start re-pulls it in full, causing cumulative network transfer and startup waiting costs.