Published on

A Dive Into gRPC and Protocol Buffers

Authors

As I continue to explore and deepen my understanding of modern web development, one technology that has particularly captured my interest is gRPC. It’s a high-performance, open-source framework developed by Google, designed to simplify and optimize the way our distributed systems communicate.

The What and Why of gRPC ​

If you've ever wondered how systems can efficiently communicate across different languages, platforms, or environments, gRPC (Google Remote Procedure Call) might be the answer. Designed to work over HTTP/2, gRPC brings advanced features such as bi-directional streaming, flow control, and header compression. This enables efficient, real-time communication between client and server applications.

Key Concepts of gRPC ​

gRPC isn't just another protocol; it's a whole ecosystem designed to handle modern communication needs. Here are some key features:

  1. Language Agnostic: gRPC supports a variety of programming languages, from Java and C++ to Go, Python, and Ruby, making it highly versatile for polyglot systems.
  2. Bi-directional Streaming: With built-in support for streaming, gRPC can handle real-time communication smoothly.
  3. Pluggable Architecture: Customize authentication, load balancing, or even message encoding as needed.
  4. Strongly Typed APIs: Using Protocol Buffers (protobufs), gRPC generates strongly typed APIs, helping to prevent errors and keep code clean.
  5. High Performance: Leveraging HTTP/2 and binary serialization with protobufs, gRPC offers reduced network overhead compared to traditional JSON over HTTP/1.1.

Why Use Protocol Buffers? ​

At the core of gRPC’s performance is its use of Protocol Buffers as the Interface Definition Language (IDL) and serialization format. Protocol Buffers are both efficient and language-neutral, making them ideal for use in modern microservices and performance-critical applications.

For those not familiar, Protocol Buffers are Google's data interchange format. Compared to JSON or XML, they’re more efficient in both size and speed. The result? Faster data exchange and lower network usage, especially for large-scale systems.

How to Implement gRPC in Your Projects ​

If you’re ready to jump in, let’s look at two potential use cases: Nest.js (for the backend) and Vue.js (for the frontend).

Using gRPC with Nest.js ​

Nest.js is a powerful framework for building scalable server-side applications. Integrating gRPC into a Nest.js project can be straightforward with the following steps:

Step 1: Install the necessary packages ​

bash
npm install --save @nestjs/microservices @grpc/proto-loader grpc

Step 2: Define your service and messages in a .proto file ​

proto
syntax = "proto3";

package example;

service ExampleService {
	rpc GetExampleData (ExampleRequest) returns (ExampleResponse);
}

message ExampleRequest {
	int32 id = 1;
}

message ExampleResponse {
	string message = 1;
}

Step 3: Set up the gRPC client and server in your Nest.js project ​

Configure the gRPC server in main.ts:

typescript
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { grpcClientOptions } from './grpc-client.options';

async function bootstrap() {
	const app = await NestFactory.createMicroservice(AppModule, grpcClientOptions);
	await app.listen();
}
bootstrap();

Define the gRPC client options in grpc-client.options.ts:

typescript
import { join } from 'path';
import { Transport } from '@nestjs/microservices';

export const grpcClientOptions = {
	transport: Transport.GRPC,
	options: {
		package: 'example',
		protoPath: join(__dirname, 'path/to/your/proto/file/example.proto'),
		url: 'localhost:5000',
	},
};

Step 4: Implement the service ​

typescript
import { Injectable } from '@nestjs/common';

@Injectable()
export class ExampleService {
	getExampleData(request: { id: number }): { message: string } {
		return { message: `Your request ID is: ${request.id}` };
	}
}

Step 5: Create a controller to handle the gRPC methods ​

typescript
import { Controller } from '@nestjs/common';
import { GrpcMethod } from '@nestjs/microservices';
import { ExampleService } from './example.service';

@Controller()
export class ExampleController {
	constructor(private readonly exampleService: ExampleService) {}

	@GrpcMethod('ExampleService', 'GetExampleData')
	getExampleData(request: { id: number }): { message: string } {
		return this.exampleService.getExampleData(request);
	}
}

Using gRPC with Vue.js ​

For browser-based gRPC communication, you can use the nice-grpc-web library. This library is a more modern and user-friendly gRPC-web client for TypeScript that works well with Vue.js.

Step 1: Install Dependencies ​

First, install nice-grpc-web and google-protobuf:

bash
npm install --save @nice-grpc/web google-protobuf

Step 2: Generate TypeScript Code from Your .proto File ​

Use the protoc compiler with the grpc-web plugin to generate TypeScript code:

bash
protoc -I=. example.proto \
  --js_out=import_style=commonjs:./generated \
  --grpc-web_out=import_style=typescript,mode=grpcwebtext:./generated

This command generates the necessary JavaScript and TypeScript files in the ./generated folder.

Step 3: Configure the gRPC Client with nice-grpc-web ​

Set up your gRPC client using nice-grpc-web:

typescript
// src/grpc-client.ts
import { createChannel, createClient } from '@nice-grpc/web';
import { ExampleServiceDefinition } from './generated/ExampleService_pb_service';

const channel = createChannel('http://localhost:5000'); // Replace with your gRPC server URL
export const exampleServiceClient = createClient(ExampleServiceDefinition, channel);

Step 4: Use the Client in a Vue Component ​

Here's how to use the client in a Vue 3 component using the Composition API:

vue
<template>
  <div>
    <button @click="fetchExampleData">Fetch Data</button>
    <p>{{ message }}</p>
  </div>
</template>

<script setup lang="ts">
import { ref } from 'vue';
import { exampleServiceClient } from '../grpc-client';
import { ExampleRequest } from './generated/example_pb';

const message = ref('');

const fetchExampleData = async () => {
  try {
    const request = new ExampleRequest();
    request.setId(1); // Set your request parameters

    const response = await exampleServiceClient.getExampleData(request);
    message.value = response.getMessage();
  } catch (error) {
    console.error('Error fetching data:', error);
  }
};
</script>

Using gRPC with Envoy ​

When working with gRPC in browser-based web applications, you face a significant limitation: browsers can’t directly communicate with gRPC servers. This is because gRPC uses the HTTP/2 protocol in a way that browsers don’t natively support. Specifically, browser clients can’t establish HTTP/2 connections in the manner required by gRPC.

To bridge this gap, we use a proxy like Envoy with the grpc_web filter. The proxy translates browser-compatible HTTP/1.1 or HTTP/2 requests into standard gRPC requests, enabling seamless communication between browser-based clients and gRPC servers.

Setting Up Envoy Proxy ​

To make gRPC work from a browser client, you need to set up a proxy like Envoy. This ensures that the nice-grpc-web client can interact with your backend gRPC service by translating gRPC-Web requests into standard gRPC requests.

Reference nice-grpc documentation.

Step 1: Create an Envoy Configuration File ​

First, create a file named envoy.yaml with the following content:

yaml
static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address: { address: 0.0.0.0, port_value: 8080 }
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                codec_type: auto
                stat_prefix: ingress_http
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains: ['*']
                      routes:
                        - match: { prefix: '/' }
                          route:
                            cluster: service
                            timeout: 10s
                            max_stream_duration:
                              grpc_timeout_header_max: 10s
                      cors:
                        allow_origin_string_match:
                          - prefix: '*'
                        allow_methods: GET, PUT, DELETE, POST, OPTIONS
                        allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
                        max_age: '1728000'
                        expose_headers: grpc-status,grpc-message
                http_filters:
                  - name: envoy.filters.http.grpc_web
                    typed_config:
                      '@type': type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb
                  - name: envoy.filters.http.cors
                    typed_config:
                      '@type': type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors
                  - name: envoy.filters.http.router
                    typed_config:
                      '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
    - name: service
      connect_timeout: 10.25s
      type: logical_dns
      http2_protocol_options: {}
      lb_policy: round_robin
      load_assignment:
        cluster_name: service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: backend-service
                      port_value: 50051
  • Listener: Envoy listens on port 8080 and handles incoming HTTP/1.1 or HTTP/2 requests.
  • Filters:
    • grpc_web: Translates gRPC-Web requests from the browser into standard gRPC requests.
    • cors: Manages cross-origin requests, essential for browser security.
    • router: Routes requests to the appropriate backend service.
  • Cluster: Configures the backend gRPC service, which Envoy forwards requests to. In this example, the backend service is running on port 50051.

Step 2: Set Up Docker Compose ​

To run Envoy alongside your gRPC backend service, use a docker-compose.yml file:

yaml
version: '3.9'

services:
  envoy:
    image: envoyproxy/envoy:v1.28-latest
    ports:
      - '8080:8080'
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml
  backend-service:
    image: your-backend-service-image
    ports:
      - '50051:50051'
  • Envoy Service: Runs Envoy using the envoyproxy/envoy Docker image and exposes port 8080.
  • Backend Service: The gRPC backend service that Envoy forwards requests to. Replace your-backend-service-image with the actual image name of your backend service.

Wrapping Up ​

Using gRPC for web applications brings substantial performance and scalability benefits. However, due to browser limitations, a proxy like Envoy is essential for making gRPC-Web work seamlessly. By setting up an Envoy proxy, you ensure that your nice-grpc-web client can communicate effectively with your backend gRPC service.

This setup enables you to build high-performance, type-safe web applications using Vue 3’s Composition API and modern gRPC technology. Remember, the proxy configuration not only handles request translation but also manages CORS policies, making it a robust solution for web-based gRPC communication.

Check out this video on youtube for a visual intro to gRPC:

I hope this guide has provided clarity on how to set up a complete gRPC-Web architecture. If you have any questions or need further assistance, feel free to reach out. Happy coding!