The Ops Community ⚙️: Kevin Wan

Implementing service discovery for microservices

Kevin Wan — Tue, 28 Jun 2022 04:21:23 +0000

What is service registration discovery?

For developers who work with microservices, the concepts of service discovery should be familiar.

For example, if service A depends on service B, we need to tell service A where to invoke service B. This is the problem that service discovery has to solve.

Service B registers itself to the Service Registry called Service Registration
Service A discovering node information of Service B from Service Registry is called Service Discovery

Service Registration

Service registration is for the server side and is required after the service is started and is divided into several parts.

Startup registration
Timed Renewal
Withdrawal

Startup Registration

When a service node is up, it needs to register itself to the Service Registry so that other nodes can easily discover itself. The registration needs to be done when the service is up and ready to accept requests, and a validity period will be set to prevent the process from being accessed even after an abnormal exit.

Timed Renewal

The equivalent of keep alive is to periodically tell Service Registry that it is still alive and can continue to serve.

Withdrawal

When a process exits, we should actively revoke the registration information so that the caller can distribute the request to another node in time. At the same time, go-zero ensures that even if a node exits without active deregistration, the node can be taken off in time by adaptive load balancing.

Service Discovery

Service discovery is for the caller side and is generally divided into two types of problems.

Inventory fetch
Incremental watch

There is also a common engineering problem of

Coping with service discovery failures

When a service discovery service (such as etcd, consul, nacos, etc.) goes down, we don't modify the list of endpoints we've already fetched, so we can better ensure that the services we depend on can still interact properly after etcd, etc. goes down.

Inventory fetch

When Service A starts, it needs to get the list of existing nodes of Service B from Service Registry: Service B1, Service B2, Service B3, and then select the appropriate nodes to send requests based on its own load balancing algorithm.

Incremental watch

The above diagram already has Service B1, Service B2, Service B3, if Service B4 is started, then we need to notify Service A that there is an additional node. As shown in the figure.

Service discovery failures

For service callers, we all cache a list of available nodes in memory. Whether we use etcd, consul or nacos, we may face a service discovery cluster failure, take etcd as an example, when we encounter etcd failure, we need to freeze the node information of Service B without changing it, we must not empty the node information at this time, once it is empty, we can't get it, while the nodes of Service B nodes are probably normal, and go-zero will automatically isolate and restore the failed node.

This is the basic principle of service registration and service discovery, but of course it is still complicated to implement, so let's take a look at what service discovery methods are supported in go-zero.

go-zero's built-in service discovery

go-zero supports three service discovery methods by default.

direct connection
direct connection * etcd-based service discovery
kubernetes endpoints-based service discovery

Direct connection

Direct connection is the simplest way, when our service is simple enough, such as a single machine can host our business, we can just use this way.

Specify endpoints directly in the rpc configuration file, e.g.

Rpc:
  Endpoints:
  - 192.168.0.111:3456
  - 192.168.0.112:3456

The zrpc caller will allocate the load to these two nodes, one of the nodes has a problem zrpc will automatically take off, and when the node is restored will again allocate the load.

The disadvantage of this approach is that the nodes cannot be added dynamically, and each new node requires a change in the caller's configuration and a restart.

etcd-based service discovery

Once our service has a certain size, because a service may be dependent on many services, we need to be able to dynamically add and remove nodes without having to modify many caller configurations and restarts.

Common service discovery solutions are etcd, consul, nacos, etc.

go-zero has a built-in etcd-based service discovery scheme, which is used as follows.

Rpc:
  Etcd:
     Hosts:
     - 192.168.0.111:2379
     - 192.168.0.112:2379
     - 192.168.0.113:2379
     Key: user.rpc

Hosts is the etcd cluster address
Key is the key on which the service is registered

Service discovery based on Kubernetes Endpoints

If our services are deployed on a Kubernetes cluster, Kubernetes itself manages the cluster state through its own etcd, and all services register their node information to Endpoints objects, so we can directly give deployment permissions to read the cluster's Endpoints object to get the node information.

Each Pod of Service B will register itself to the Endpoints of the cluster when it starts
When each Pod of Service A starts, it can get the node information of Service B from the Endpoints of the cluster
When the node of Service B changes, Service A can sense it through Endpoints of the watch cluster

Before this mechanism can work, we need to configure the pod in the current namespace to have access to the cluster Endpoints, where there are three concepts.

ClusterRole
- ClusterRole defines a cluster-wide permission role that is not controlled by namespace.
ServiceAccount
- Defines the service account within the namespace scope
ClusterRoleBinding
- Bind a defined ClusterRole to a ServiceAccount in a different namespace

The specific Kubernetes configuration file can be found here, where namespace is modified as needed.

Note: Remember to check if these configurations are in place when you start up and don't have access to Endpoints :)

zrpc's Kubernetes Endpoints based service discovery is used as follows.

Rpc:
  Target: k8s://mynamespace/myservice:3456

where.

mynamespace: the namespace where the invoked rpc service is located
myservice: the name of the invoked rpc service
3456: the port of the called rpc service

Be sure to add serviceAccountName to specify which ServiceAccount to use when creating the deployment configuration file, for example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: alpine-deployment
  labels:
    app: alpine
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alpine
  template:
    metadata:
      labels:
        app: alpine
    spec:
      serviceAccountName: endpoints-reader
      containers:
      - name: alpine
        image: alpine
        command:
        - sleep
        - infinity

Note that serviceAccountName specifies which ServiceAccount is used for the pod created by the deployment.

After both server and client are deployed to the Kubernetes cluster, you can restart all server nodes on a rolling basis with the following command

kubectl rollout restart deploy -n adhoc server-deployment

Check the client node log with the following command.

kubectl -n adhoc logs -f deploy/client-deployment --all-containers=true

You can see that our service discovery mechanism follows the changes to the server node perfectly, and there are no exception requests during service updates.

The full code example is available at https://github.com/zeromicro/zero-examples/tree/main/discovery/k8s

In the next article I will explain how to implement service registration discovery based on consul, nacos, etc. in go-zero, so stay tuned!

Project address

https://github.com/zeromicro/go-zero

Welcome to use go-zero and star to support us!

Sending type-safe HTTP requests with Go

Kevin Wan — Mon, 30 May 2022 15:47:26 +0000

Scenarios

For Gophers, we basically write code for clients to request. Sometimes we need to request RESTful APIs provided by third parties. At this time, we feel that it's hard to assemble the requests, not difficult, but error-prone.

For example, if we want to send a request like this, it looks very simple, but it's still tedious to actually write it.

POST /articles/5/update?device=ios HTTP/1.1
Host: go-zero.dev
Authorization: Bearer <jwt-token>

{"author": "kevin", "body": "this is not important!", "title": "my title", "type":6}

Go native way

This API is actually quite simple and we can write it directly from scratch.

func main() {
    var buf bytes.
    encoder := json.NewEncoder(&buf)
    params := map[string]interface{}{
        "title":  "my title",
        "body":   "this is not important!",
        "author": "kevin",
        "type":   6,
    }
    if err := encoder.Encode(params); err ! = nil {
        fmt.Fprintln(os.Stderr, err)
        return
    }

    url := fmt.Sprintf("http://go-zero.dev/articles/%d/update?device=%s", 5, "ios")
    req, err := http.NewRequest(http.MethodPost, url, &buf)
    if err ! = nil {
        fmt.Fprintln(os.Stderr, err)
        return
    }

    req.Header.Add("Authorization", "Bearer <jwt-token>")
    cli := http.Client{}
    resp, err := cli.Do(req)
    if err ! = nil {
        fmt.Fprintln(os.Stderr, err)
        return
    }

    io.Copy(os.Stdout, resp.Body)
}

We ran a test and found that we didn't get 200 OK, dumped the packet and the request looks as follows. Can you think of the reason for the failure?

POST /articles/5/update?device=ios HTTP/1.1
Host: go-zero.dev
User-Agent: Go-http-client/1.1
Content-Length: 79
Authorization: Bearer <jwt-token>
Accept-Encoding: gzip

{"author": "kevin", "body": "this is not important!", "title": "my title", "type":6}

The specific reasons for the failure will be discussed below, so let's explain this code first. You can see that the map[string]interface{} is used for the splicing parameters, for each field we can't check if the type matches. Only when we send it out and receive 200 OK from the server, we can confirm that it is passed correctly. For example, the type parameter is used here as int type, we may write it as string type by mistake. But it is still hard to find out that this parameter is written incorrectly without requesting it.

So let's see how the httpc package in go-zero is used for type safety.

`httpc` implementation

Let's see how the code for requesting with the httpc package is written.

const url = "http://go-zero.dev/articles/:id/update"

type UpdateArticle struct {
    ID            int    `path: "id"`
    Device        string `form: "device,options=ios,android,web,desktop"`
    Authorization string `header: "Authorization"`
    Title         string `json: "title"`
    Body          string `json: "body"`
    Author        string `json: "author"`
    Type          int    `json: "type"`
}

func main() {
    data := &UpdateArticle{
        ID:            5,
        Device:        "ios",
        Authorization: "Bearer <jwt-token>",
        Title:         "my title",
        Body:          "this is not important!",
        Author:        "kevin",
        Type:          6,
    }

    resp, err := httpc.Do(context.Background(), http.MethodPost, url, data)
    if err ! = nil {
        fmt.Fprintln(os.Stderr, err)
        return
    }

    io.Copy(os.Stdout, resp.Body)
}

Let's verify the code by sending a request, the result is as expected.

POST /articles/5/update?device=ios HTTP/1.1
Host: go-zero.dev
User-Agent: Go-http-client/1.1
Content-Length: 79
Content-Type: application/json; charset=utf-8
Authorization: Bearer <jwt-token>
Accept-Encoding: gzip

{"author": "kevin", "body": "this is not important!", "title": "my title", "type":6}

Did you find out that, in contrast to the previous one, there is one more header was set, Content-Type: application/json; charset=utf-8, and we forgot to set Content-Type in the previous code.

httpc implementation is very easy to understand by defining the type of the request and sending with Do. With support for path, form, header and json as shown in our code, it is very easy and type-safe to send HTTP requests.

More Capabilities

In addition to the ease of use and type-safety shown above, the httpc package has the following features.

timeout control with context. You can pass ctx for the requests.
automatic integration of OpenTelemetry. The trace-id, span-id returned by the server will be automatically written in the log, so that the client and server can work together to investigate the problems.
use httpc.Service to get the circuit breaker ability. When the server side has problems, it will automatically stop sending the requests, to avoid useless requests and reduce the pressure on the server side.

Project address

https://github.com/zeromicro/go-zero

Welcome to use go-zero and star to support us!

Developing a RESTful API with Go

Kevin Wan — Thu, 26 May 2022 04:46:53 +0000

When to use RESTful API

For most startups, we should focus more on delivering the products in the early stage of business. The monolithic services have the advantages of simple architecture, easy deployment, and better development productivity, which can help us achieve the product requirements quickly. While we use monolithic services to deliver products quickly, we also need to reserve the possibility for business incresement, so we usually split different business modules clearly in monolithic services.

Shopping mall monolithic service architecture

We take the mall as an example to build a monolithic service. The mall service is generally relatively complex and consists of multiple modules, the more important modules include account, product and order modules, etc. Each module will have its own independent business logic, and each module will also depend on some others. For example, the order module and the product module will depend on the account module. In the monolithic application this kind of dependency is usually accomplished by method calls between modules. Monolithic services generally share storage resources, such as MySQL and Redis.

The overall architecture of monolithic services is relatively simple, which is also the advantage of monolithic services. Customer requests are parsed through DNS and forwarded to the mall's backend services through Nginx. Mall services are deployed on cloud hosts. In order to achieve greater throughput and high availability, the service will generally deployed with multiple copies. This simple architecture can carry high throughput if well optimized.

For example, a request for order details interface /order/detail is routed to the order module, which relies on the account module and the product module to compose the complete order details back to the user, and multiple modules in a single service generally share the database and cache.

Monolithic Service

The next section describes how to quickly implement a mall monolithic service based on go-zero. Devs who have used go-zero know that we provide an API format file to describe the Restful API, and then we can generate the corresponding code by goctl with one command, we just need to fill in the corresponding business logic in the logic files. The mall service contains several modules, and in order to make the modules independent from each other, different modules are defined by separate APIs, but all the APIs are defined for the same service (mall-api).

Create user.api, order.api, product.api and mall.api in the api directory, where mall.api is the aggregated api file. Other api files are imported via import directives.

api
|-- mall.api
|-- order.api
|-- product.api
|-- user.api

Mall API Definition

mall.api is defined as follows, where syntax = "v1" means that this is the v1 syntax of zero-api.

syntax = "v1"

import "user.api"
import "order.api"
import "product.api"

Account module API definition

View user details
Get all orders for a user

user.api is defined as follows.

syntax = "v1"

type (
    UserRequest {
        ID int64 `path:"id"`
    }

    UserReply {
        ID      int64   `json:"id"`
        Name    string  `json:"name"`
        Balance float64 `json:"balance"`
    }

    UserOrdersRequest {
        ID int64 `path:"id"`
    }

    UserOrdersReply {
        ID       string `json:"id"`
        State    uint32 `json:"state"`
        CreateAt string `json:"create_at"`
    }
)

service mall-api {
    @handler UserHandler
    get /user/:id (UserRequest) returns (UserReply)

    @handler UserOrdersHandler
    get /user/:id/orders (UserOrdersRequest) returns (UserOrdersReply)
}

Order module API definition

Get order details
Generate orders

order.api is defined as follows.

syntax = "v1"

type (
    OrderRequest {
        ID string `path:"id"`
    }

    OrderReply {
        ID       string `json:"id"`
        State    uint32 `json:"state"`
        CreateAt string `json:"create_at"`
    }

    OrderCreateRequest {
        ProductID int64 `json:"product_id"`
    }

    OrderCreateReply {
        Code int `json:"code"`
    }
)

service mall-api {
    @handler OrderHandler
    get /order/:id (OrderRequest) returns (OrderReply)

    @handler OrderCreateHandler
    post /order/create (OrderCreateRequest) returns (OrderCreateReply)
}

Product module API definition

View product details

product.api is defined as follows.

syntax = "v1"

type ProductRequest {
    ID int64 `path:"id"`
}

type ProductReply {
    ID    int64   `json:"id"`
    Name  string  `json:"name"`
    Price float64 `json:"price"`
    Count int64   `json:"count"`
}

service mall-api {
    @handler ProductHandler
    get /product/:id (ProductRequest) returns (ProductReply)
}

Generating the monolithic service

With the API already defined, generating a service with the API becomes very simple, we use goctl to generate the monolithic service code.

$ goctl api go -api api/mall.api -dir .

The generated code is structured as follows.

.
├── api
│ ├── mall.api
│ ├── order.api
│ ├── product.api
│ └── user.api
├── etc
│ └── mall-api.yaml
├─ internal
│ ├── config
│ │ └── config.go
│ ├── handler
│ │ ├── ordercreatehandler.go
│ │ ├── orderhandler.go
│ │ ├── producthandler.go
│ │ ├── routes.go
│ │ ├── userhandler.go
│ │ └─ userordershandler.go
│ ├─ logic
│ │ ├─ ordercreatelogic.go
│ │ ├── orderlogic.go
│ │ ├── productlogic.go
│ │ ├── userlogic.go
│ │ └── userorderslogic.go
│ ├── svc
│ │ └── servicecontext.go
│ └── types
│ └── types.go
└── mall.go

Let's explain the generated files.

api: holds the API description file
etc: used to define the project configuration, all configuration items can be written in mall-api.yaml
internal/config: the configuration definition of the service
internal/handler: the implementation of the handler corresponding to the routes defined in the API file
internal/logic: used to put the business logic corresponding to each route, the reason for the distinction between handler and logic is to make the business processing part as less dependent as possible, to separate HTTP requests from the logic processing code, and to facilitate the subsequent splitting into RPC service
internal/svc: used to define the dependencies of the business logic processing, we can create the dependent resources in the main function and pass them to handler and logic via ServiceContext
internal/types: defines the API request and response data structures
mall.go: the file where the main function is located, with the same name as the service in the API definition, minus the -api suffix

The generated service can be run without any modification:

$ go run mall.go
Starting server at 0.0.0.0:8888...

Implementing the business logic

Next, let's implement the business logic. The logic will be simple for demonstration purposes, not real business logic.

First, let's implement the logic of getting all orders for users. Since there is no order-related information in the user module, we need to rely on the order module to query the orders of users, so we add a dependency on OrderLogic in UserOrdersLogic.

type UserOrdersLogic struct {
    logx.Logger
    ctx        context.Context
    svcCtx     *svc.ServiceContext
    orderLogic *OrderLogic
}

func NewUserOrdersLogic(ctx context.Context, svcCtx *svc.ServiceContext) *UserOrdersLogic {
    return &UserOrdersLogic{
        Logger:     logx.WithContext(ctx),
        ctx:        ctx,
        svcCtx:     svcCtx,
        orderLogic: NewOrderLogic(ctx, svcCtx),
    }
}

Implement a method in OrderLogic to query all orders based on user id

func (l *OrderLogic) ordersByUser(uid int64) ([]*types.OrderReply, error) {
    if uid == 123 {
        // It should actually be queried from database or cache
        return []*types.OrderReply{
            {
                ID:       "236802838635",
                State:    1,
                CreateAt: "2022-5-12 22:59:59",
            },
            {
                ID:       "236802838636",
                State:    1,
                CreateAt: "2022-5-10 20:59:59",
            },
        }, nil
    }

    return nil, nil
}

Call the ordersByUser method in the UserOrders method of UserOrdersLogic.

func (l *UserOrdersLogic) UserOrders(req *types.UserOrdersRequest) (*types.UserOrdersReply, error) {
    orders, err := l.orderLogic.ordersByUser(req.ID)
    if err ! = nil {
        return nil, err
    }

    return &types.UserOrdersReply{
        Orders: orders,
    }, nil
}

At this point we restart the mall-api service and request all the user's orders in the browser.

http://localhost:8888/user/123/orders

The return result is as follows, as we expected

{
    "orders": [
        {
            "id": "236802838635",
            "state": 1,
            "create_at": "2022-5-12 22:59:59"
        },
        {
            "id": "236802838636",
            "state": 1,
            "create_at": "2022-5-10 20:59:59"
        }
    ]
}

Next we'll implement the logic for creating an order. To create an order we first need to see if the item in stock is enough, so we need to rely on the item module in the order module.

type OrderCreateLogic struct {
    logx.Logger
    ctx          context.Context
    svcCtx       *svc.ServiceContext
    productLogic *ProductLogic
    productLogic *ProductLogic
}

func NewOrderCreateLogic(ctx context.Context, svcCtx *svc.ServiceContext) *OrderCreateLogic {
    return &OrderCreateLogic{
        Logger:       logx.WithContext(ctx),
        ctx:          ctx,
        svcCtx:       svcCtx,
        productLogic: NewProductLogic(ctx, svcCtx),
    }
}

The logic for creating an order is as follows.

const (
    success = 0
    failure = -1
)

func (l *OrderCreateLogic) OrderCreate(req *types.OrderCreateRequest) (*types.OrderCreateReply, error) {
    product, err := l.productLogic.productByID(req.ProductID)
    if err ! = nil {
        return nil, err
    }

    if product.Count > 0 {
        return &types.OrderCreateReply{Code: success}, nil
    }

    return &types.OrderCreateReply{Code: failure}, nil
}

The logic of the dependent product module is as follows.

func (l *ProductLogic) Product(req *types.ProductRequest) (*types.ProductReply, error) {
    return l.productByID(req.ID)
}

func (l *ProductLogic) productByID(id int64) (*types.ProductReply, error) {
    return &types.ProductReply{
        ID: id,
        Name: "apple watch 3",
        Price: 3333.33,
        Count: 99,
    }, nil
}

The above shows that using go-zero to develop a monolithic service is very simple, which helps us to develop quickly. And we also separated modules, which also provides the possibility of changing to microservices later.

Summary

The above example shows that it is very simple to use go-zero to develop monolithic services. You only need to define the api file, and then the goctl tool can automatically generate the project code. We only need to fill in the business logic code in the logic package. In this article we just demonstrated how to quickly develop monolithic services based on go-zero, which does not involve databases. In fact, goctl can also generate CRUD and cache code with one command.

And for different business scenarios, customization can also be achieved through customizing templates. And customized templates can be shared within the team through remote git repositories, which can be very efficient for team collaboration.

Project address

https://github.com/zeromicro/go-zero

Welcome to use go-zero and star to support us!

The Ops Community ⚙️: Kevin Wan

Implementing service discovery for microservices

What is service registration discovery?

Service Registration

Startup Registration

Timed Renewal

Withdrawal

Service Discovery

Inventory fetch

Incremental watch

Service discovery failures

go-zero's built-in service discovery

Direct connection

etcd-based service discovery

Service discovery based on Kubernetes Endpoints

Project address

Sending type-safe HTTP requests with Go

Scenarios

Go native way

httpc implementation

More Capabilities

Project address

Developing a RESTful API with Go

When to use RESTful API

Shopping mall monolithic service architecture

Monolithic Service

Mall API Definition

Account module API definition

Order module API definition

Product module API definition

Generating the monolithic service

Implementing the business logic

Summary

Project address

`httpc` implementation