gRPC and Protocol Buffers

01The idea: remote procedure calls with a contract

With REST, you think in resources and HTTP verbs: GET /orders/42. With gRPC, you think in functions: OrderService.GetOrder(id). This is called RPC, remote procedure call. The difference that matters is the contract. In gRPC the API is written in a .proto file using Protocol Buffers (Protobuf), an interface definition language. A compiler (protoc or the Buf CLI) turns that file into code: message classes, a client stub you call like a normal method, and a server interface you implement. If the server changes a field type, clients that regenerate get a compile error instead of a production surprise. Think of the .proto file as a legally binding menu both kitchen and customers must follow.

• Protobuf: the schema language and the binary encoding.
• Stub: generated client object whose methods make network calls.
• Channel: a long-lived connection to a server address.

02Writing a .proto file

A .proto file declares a package, messages (data shapes) and services (groups of RPC methods). Each field has a type and a field number. The number, not the name, is what goes on the wire, so it must never change or be reused once published. proto3 syntax is the common choice; newer Protobuf 'editions' unify proto2 and proto3 behaviour with feature flags, but proto3 files remain fully supported. Use well-known types such as google.protobuf.Timestamp for dates and keep one request and one response message per method so you can add fields later without breaking signatures.

1syntax = "proto3";
2 
3package shop.orders.v1;
4 
5import "google/protobuf/timestamp.proto";
6 
7service OrderService {
8  rpc GetOrder(GetOrderRequest) returns (GetOrderResponse);
9  rpc WatchOrders(WatchOrdersRequest) returns (stream OrderEvent);
10}
11 
12message GetOrderRequest { string order_id = 1; }
13message GetOrderResponse { Order order = 1; }
14message WatchOrdersRequest { string customer_id = 1; }
15message OrderEvent { Order order = 1; string type = 2; }
16 
17message Order {
18  string id = 1;
19  int64 total_cents = 2;
20  google.protobuf.Timestamp created_at = 3;
21}

03Under the hood: HTTP/2, framing and status codes

gRPC runs over HTTP/2. Each call is an HTTP/2 stream on a shared TCP connection, so many calls are multiplexed without head-of-line blocking at the HTTP level. The request is a POST to /package.Service/Method with content-type application/grpc. Each message is length-prefixed binary Protobuf. Metadata (like headers) carries auth tokens and tracing IDs. Results end with trailers that contain grpc-status and grpc-message. gRPC has its own status codes, separate from HTTP: OK, INVALID_ARGUMENT, NOT_FOUND, ALREADY_EXISTS, PERMISSION_DENIED, UNAUTHENTICATED, RESOURCE_EXHAUSTED, UNAVAILABLE, DEADLINE_EXCEEDED and others. Clients should retry only safe codes like UNAVAILABLE, and only for idempotent methods.

1POST /shop.orders.v1.OrderService/GetOrder HTTP/2
2content-type: application/grpc
3te: trailers
4authorization: Bearer <token>
5grpc-timeout: 2S
6 
7<5-byte prefix><binary GetOrderRequest>
8 
9HTTP/2 200
10content-type: application/grpc
11<5-byte prefix><binary GetOrderResponse>
12grpc-status: 0

04The four kinds of RPC

gRPC supports four call shapes. Unary is one request and one response, like a normal function. Server streaming sends one request and receives a stream of responses, useful for live order updates or large result sets. Client streaming sends a stream of requests and gets one response, useful for uploading telemetry or chunks. Bidirectional streaming lets both sides send independently on the same call, used for chat, collaborative editing or real-time control. Streams are ordered within one call, and flow control is handled by HTTP/2. Long-lived streams need keepalives and reconnection logic, because proxies and load balancers close idle connections.

• Unary: rpc Get(Req) returns (Res)
• Server streaming: rpc Watch(Req) returns (stream Event)
• Client streaming: rpc Upload(stream Chunk) returns (Summary)
• Bidirectional: rpc Chat(stream Msg) returns (stream Msg)

05A Node.js server and client in practice

In Node.js, the official implementation is @grpc/grpc-js. You can load the .proto at runtime with @grpc/proto-loader, as below, or generate TypeScript code ahead of time with Buf and plugins such as ts-proto or protobuf-es for full type safety. Every client call should pass a deadline; without one, a hung server can hold resources forever. Production servers use TLS (createSsl), or run plaintext only inside a mesh that provides mTLS.

1import * as grpc from "@grpc/grpc-js";
2import * as loader from "@grpc/proto-loader";
3 
4const def = loader.loadSync("orders.proto", { keepCase: false, longs: String });
5const pkg = grpc.loadPackageDefinition(def) as any;
6const Svc = pkg.shop.orders.v1.OrderService;
7 
8const server = new grpc.Server();
9server.addService(Svc.service, {
10  getOrder: (call: any, cb: any) => {
11    if (!call.request.orderId) return cb({ code: grpc.status.INVALID_ARGUMENT, details: "orderId required" });
12    cb(null, { order: { id: call.request.orderId, totalCents: "4200" } });
13  },
14});
15server.bindAsync("0.0.0.0:50051", grpc.ServerCredentials.createInsecure(), () => {});
16 
17const client = new Svc("localhost:50051", grpc.credentials.createInsecure());
18client.getOrder({ orderId: "42" }, { deadline: Date.now() + 2000 }, console.log);

06gRPC in the browser and at the edge

Browsers cannot fully control HTTP/2 frames or read trailers from fetch, so they cannot speak native gRPC. There are two common solutions. gRPC-Web is a slightly different wire format that a proxy (Envoy) translates to real gRPC; it supports unary and server streaming only. Connect is a newer protocol family from Buf whose servers speak gRPC, gRPC-Web and a simple HTTP/JSON-friendly Connect protocol at the same time, so a browser can call the same service with plain fetch. Many teams instead keep REST or GraphQL at the public edge and use gRPC only behind it, with a gateway such as grpc-gateway or Envoy transcoding JSON to gRPC.

07Tooling and production checklist

Use the Buf CLI to lint .proto files, detect breaking changes in CI and generate code reproducibly from a buf.gen.yaml. Version packages in the name (orders.v1, orders.v2). Enable the standard health checking service so load balancers and Kubernetes probes can check readiness, and server reflection in non-production so tools like grpcurl can explore the API. Propagate deadlines and tracing metadata (OpenTelemetry) across calls. Set message size limits deliberately; the default maximum receive size is 4 MB in most implementations.

1# Lint, check for breaking changes vs main, then generate code
2buf lint
3buf breaking --against ".git#branch=main"
4buf generate
5 
6# Call a running server (needs reflection or the .proto)
7grpcurl -plaintext -d '{"order_id":"42"}' localhost:50051 shop.orders.v1.OrderService/GetOrder
8grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check

08Protobuf wire format and why it is small

Each field is encoded as a tag (field number shifted left three bits, plus a wire type) followed by the value. Integers use varints: small numbers take one byte, larger ones more. That is why field numbers 1 to 15 are precious: their tag fits in one byte, so reserve them for frequent fields. Negative numbers in int32/int64 always take ten bytes; use sint32/sint64 (zigzag encoding) for values that are often negative. Strings, bytes and nested messages are length-delimited. Default values (0, empty string, false) are not written at all in proto3 unless the field is marked optional, which is why you cannot tell 'unset' from 'zero' without optional or wrapper types. Repeated scalars are packed by default in proto3.

1message Reading {
2  int32 sensor_id = 1;        // tag 0x08, varint
3  sint32 delta = 2;           // zigzag: -1 encodes as 1 byte
4  optional double value = 3;  // presence tracked: unset != 0.0
5  repeated int32 samples = 4; // packed into one length-delimited block
6  reserved 5, 6;
7  reserved "legacy_name";
8}

09Load balancing, name resolution and xDS

A gRPC channel is not a single connection; it is a manager that resolves a target name (dns:///orders.internal:50051) into addresses, opens subchannels to each, and picks one per call according to a load-balancing policy. The default pick_first sends everything to the first healthy address; round_robin spreads calls across all. Service config, delivered by DNS or set in code, configures policies, retries and per-method timeouts. At larger scale, gRPC clients can speak xDS, the Envoy control-plane protocol, to receive endpoints, weights and routing rules directly from a control plane, giving proxyless service mesh behaviour. This avoids a sidecar hop but couples clients to the mesh's control plane.

1// Node.js: client-side round robin + retry policy via service config
2const serviceConfig = JSON.stringify({
3  loadBalancingConfig: [{ round_robin: {} }],
4  methodConfig: [{
5    name: [{ service: "shop.orders.v1.OrderService", method: "GetOrder" }],
6    timeout: "2s",
7    retryPolicy: { maxAttempts: 3, initialBackoff: "0.1s", maxBackoff: "1s",
8      backoffMultiplier: 2, retryableStatusCodes: ["UNAVAILABLE"] },
9  }],
10});
11const client = new Svc("dns:///orders.internal:50051", creds,
12  { "grpc.service_config": serviceConfig });

10Streaming at scale: flow control and backpressure

HTTP/2 flow control gives each stream and connection a window of bytes the receiver is willing to accept. If a client reads a server stream slowly, the window fills and the server's writes block or buffer, which is backpressure working as designed. Problems appear when server code ignores it and buffers unbounded messages in memory. In Node.js, respect the return value of call.write() and wait for 'drain'. Long-lived streams also need keepalive pings so NAT gateways and proxies do not silently drop idle connections, but aggressive keepalives get the client rejected with GOAWAY 'too_many_pings'. Design streams to be resumable: include a cursor or sequence number so a reconnecting client can continue where it left off.

11Security: mTLS, auth and interceptors

Between services, mutual TLS gives both encryption and identity: each workload has a certificate, often issued automatically by a mesh or SPIFFE/SPIRE, and servers authorise callers by their certificate identity. User identity travels separately as a bearer token in metadata, validated in a server interceptor, which is gRPC's middleware. Interceptors are also where you add logging, metrics, tracing, rate limiting and panic recovery, consistently for every method. Disable server reflection in production or protect it, cap max message sizes to limit memory abuse, and validate input explicitly; protovalidate lets you declare rules such as string length or email format in the .proto and enforce them in generated code.