Resource hierarchy
Your data in Pinecone is organized as a hierarchy of resources. An organization contains projects, a project contains indexes, and an index is divided into namespaces that hold your documents or records. For a definition of each object, see Key terms. This diagram shows an organization with two projects. The first project holds two indexes, and the first index is divided into three namespaces:Serverless infrastructure
Pinecone runs as a managed service on AWS, GCP, and Azure. When you send a request to Pinecone, it goes through an API gateway that routes it to either a global control plane or a regional data plane. All your data is stored in distributed object storage. This diagram shows how the API gateway routes your application’s requests to the control plane or to the data plane in your index’s cloud region:
API gateway
Every request to Pinecone includes an API key that’s assigned to a specific project. The API gateway first checks that your API key has access to the project. Then it routes your request to the global control plane (for managing projects and indexes) or a regional data plane (for reading and writing data).Control plane
The global control plane manages resources like projects and indexes and tracks them in a dedicated database. It also handles billing and user management, and it coordinates operations across regions.Data plane
The data plane handles all requests to write and read data in indexes within a specific cloud region. Each index is divided into one or more logical namespaces, and every read and write targets one namespace. Pinecone separates writes and reads into different paths, and each path scales independently with demand, so queries never slow down writes, and writes never slow down queries.Data-plane APIs
Each index is created for, and served by, one data-plane API. The Documents API serves document indexes and is the API to build on. The legacy Vectors API and legacy Records API serve legacy vector indexes and legacy integrated indexes, and they keep working.
To move data from a legacy index to a document index, see Move to the Documents API.
Object storage
Pinecone stores each namespace’s data in distributed object storage as immutable files called slabs. The index builder creates slabs and optimizes them for querying.Write path
This diagram shows the log writer recording a write request in the request log and passing the data to the index builder, which moves it from the memtable into a namespace’s slabs:
Request log
When you send a write request (to add, update, or delete data), the data plane first logs the request with a unique sequence number (LSN). The sequence number keeps operations in order and tracks the state of the index. Pinecone then returns200 OK. Your write is durable at this point, and Pinecone processes it in the background.
Index builder
The index builder writes your data and its sequence number to an in-memory structure called a memtable. For an update or delete, it also tracks how queries should handle the old version. Periodically, the index builder moves data from the memtable into slabs in object storage. Small slabs use lightweight indexing that needs few resources. As slabs grow, the index builder merges them into larger slabs with indexing that performs better at scale. Each slab gets indexing suited to its size, and the cost of heavier indexing is spread over the life of the namespace.All reads check the memtable, so you can search data immediately after you write it, before it reaches object storage. For more details, see Query executors.
Read path
This diagram shows the query router merging recent data from the memtable with ranked results from the query executors, which fetch slabs from object storage when they aren’t cached:
