Open Match – Guides

Docs: Matchmaking using Open Match

Mon, 01 Jan 0001 00:00:00 +0000

Objective

The guide presents a high-level Online Game Services architecture that leverages Open Match for matchmaking. It explains in details the core customizations that need to be added to Open Match to build your Matchmaker. It also takes a deep dive into the external components that interact with Open Match. Along the way, it covers the Open Match concepts needed for building these components.

This guide is a pre-reading for the tutorial for building a basic Matchmaker. You can use this Tutorial Solution as a reference when going through each component in this guide.

Game Services Architecture

Open Match Core

Open Match comprises of a set of services hosted in a Kubernetes cluster. These services expose the core functionality of Open Match over gRPC and HTTP

Open Match FrontendService

The Open Match FrontendService is used to create, delete and get details of the current state of a Ticket.

Open Match BackendService

The Open Match BackendService is used to generate Matches and create Assignments for the Matches when invoked by the Director, and returning Assignments to the Open Match Frontend when requested.

Open Match QueryService

The QueryService enables querying the Tickets that meet certain constraints from the current Ticket pool.

Customized Components

To build a custom Matchmaker using Open Match, the user needs to provide the following customizations. These can be authored as services exposed over gRPC or HTTP endpoints.

Match Function

The core matchmaking logic is implemented as a Match Function service. Open Match Backend triggers the Match Function service when it receives a request to generate Matches. The Match Function execution receives MatchProfiles, fetches Tickets that match the profile from QueryService and returns Matches.

Evaluator (Optional)

Open Match allows for concurrent Match Functions to execute on the same player pool. The Matches generated by such concurrent Match Function executions may have overlapping Tickets. Given that one Ticket should only be used by one resulting Match, Open Match allows the user to plug in an Evaluator component to deduplicate Match proposals before returning results. Open Match provides a default Evaluator which suffices for most common use cases. See Evaluator Guide for details on proposal evaluation.

External Components

Although Open Match based Matchmaker offers core matchmaking functionality, other Game Services will be needed to handle key features such as handling player connections, fetching DGS (Dedicated Game Server) allocations etc. These services interact with Open Match to enable game specific matchmaking scenarios. Here is an overview of these services that are external to an Open Match based Matchmaker but are a part of an end to end setup:

Game Frontend

This is the component that receives the matchmaking request from the game client and eventually provides the game client with the Assignment. The Game Frontend typically authenticates the player, fetches player data and creates a matchmaking request in Open Match and fetches the Assignment for the Ticket.

Director

This is the component that understands the types of matches (MatchProfiles) that can be served and fetches matches from the Open Match Backend. The Director also interfaces with the DGS allocation system to fetch Game Servers for Matches and creating Game Server Assignments from these details in Open Match, via communicating with the Open Match Backend.

Matchmaking Flow

Here is an overview of how a Game Client gets assigned to a Game Server in this architecture.

Creating a Matchmaking request

A Game Client (for the Player) intending to find a match connects to the Game Frontend requesting for an Assignment.
The Game Frontend validates the player, fetches its properties from the platform services and calls Open Match Frontend to create a Ticket for this player. Open Match creates a Ticket and indexes its SearchFields.

Now the Ticket representing this player is a part of a Ticket Pool actively being considered for matchmaking by Open Match.

Creating a Match

The Director periodically calls FetchMatches on Open Match Backend to generate Matches for a MatchProfile.
Open Match Backend triggers MatchFunction execution for this MatchProfile.
The MatchFunction fetches all the Tickets from the active matchmaking Pool that match the filters in the MatchProfile from Open Match QueryService API.
The MatchFunction generates a Match using a subset of the queried Tickets.
The Open Match Backend returns this Match to the Director.

At this point, the Ticket representing the Player is a part of the Match but does not have an assignment yet.

Setting an Assignment

With a fetched Match needing an allocation, the Director requests the DGS allocation system for a Game Server for the Match.
The Director then creates an Assignment containing the details of connecting to the allocated Game Server.
The Director calls SetAssignment to set this Assignment on all the Tickets in the Match. This stores the Assignment on the Ticket in Open Match State Store.
The Game Frontend fetches the Assignment from the Open Match Backend and returns it back to the Game Client.
The Game Client connects to the Game Server and starts the game.

Next Steps

Now that you are familiar with building an Open Match based Matchmaker, its time to build one!

Please proceed with the step-by-step tutorial for building a basic Matchmaker

Components Deep Dive

The Below sections take a deepdive into how each of these components interact with Open Match and explains the core Open Match concepts needed to build these components:

Docs: Invoking Open Match APIs

Mon, 01 Jan 0001 00:00:00 +0000

Open Match has a resource-based REST API that is served from HTTP and gRPC. It also complies with the OpenAPI (fka Swagger) API specification which means it’s easy to download the schema and generate clients in many different languages.

Open Match default hostnames and endpoints

The following defines the in-cluster hostnames and endpoints of Open Match’s external services. The corresponding Helm configurations used to generate these configs could be found here.

swaggerui:
  hostName: om-swaggerui
  httpPort: 51500
query:
  hostName: om-query
  grpcPort: 50503
  httpPort: 51503
frontend:
  hostName: om-frontend
  grpcPort: 50504
  httpPort: 51504
backend:
  hostName: om-backend
  grpcPort: 50505
  httpPort: 51505

Invoking Open Match gRPC APIs

We recommend using gRPC to talk to Open Match. The following is an example of an in-cluster program that creates a vanilla gRPC client to talk to Open Match Frontend and then calls frontend.CreateTicket method.

import (
  "fmt"
  "log"

  "google.golang.org/grpc"
  "open-match.dev/open-match/pkg/pb"
  "open-match.dev/open-match/pkg/structs"
)

func main() {
  // Create a gRPC frontend client
  conn, err := grpc.Dial("om-frontend:50504"), grpc.WithInsecure())
  if err != nil {
    log.Fatalf("grpc.Dial failed with %v", err)
  }
  defer conn.Close()
  feClient := pb.NewFrontendClient(conn)

  // Create an Open Match CreateTicketRequest with Open Match's public package
  sent := &pb.CreateTicketRequest{
    Ticket: &pb.Ticket{
      SearchFields: &pb.SearchFields{
        Tags: []string{"beta-gameplay"},
      },
    },
  }
  ticket, err := feClient.CreateTicket(sent)
  if err != nil {
    log.Errorf("feClient.CreateTicket failed with %v", err)
  }

  fmt.Println("Open Match assigned id %s to the ticket", ticket.GetId())
}

Invoking Open Match HTTP APIs

Open Match provides REST API for all of its external components using the grpc-gateway plugin. Below is a vanilla in-cluster Go program to interact with Open Match via its HTTP endpoint. Please see the Open Match’s API definitions for the latest HTTP endpoints each service is using.

import (
  "fmt"
  "net/http"
  "io/ioutil"

  "github.com/golang/protobuf/jsonpb"
  "open-match.dev/open-match/pkg/pb"
  "open-match.dev/open-match/pkg/structs"
)

func main() {
  var m jsonpb.Marshaler
  // The HTTP endpoint of frontend.CreateTicket API
  apiURL := fmt.Sprintf("http://om-frontend:%d/v1/frontendservice/tickets", 51504)
  // Create an Open Match CreateTicketRequest with Open Match's public package
  sent := &pb.CreateTicketRequest{
    Ticket: &pb.Ticket{
      SearchFields: &pb.SearchFields{
        Tags: []string{"beta-gameplay"},
      },
    },
  }
  payload, err := m.MarshalToString(&sent)
  if err != nil {
    log.Fatalf("m.MarshalToString(%#v) failed with %v; want success", payload, err)
  }
  // Send the Request to frontend.CreateTicket endpoint
  resp, err := http.Post(apiURL, "application/json", strings.NewReader(payload))
  if err != nil {
    log.Fatalf("http.Post(%q) failed with %v, want success", apiURL, err)
  }
  defer resp.Close()
  buf, err := ioutil.ReadAll(resp.Body)
  if err != nil {
    log.Errorf("ioutil.ReadAll(resp.Body) failed with %v; want success", err)
  }
  // Unmarshal the response to a Go struct
  var received *pb.Ticket
  if err := jsonpb.UnmarshalString(string(buf), received); err != nil {
    log.Errorf("jsonpb.UnmarshalString(%s, &msg) failed with %v; want success", buf, err)
  }

  fmt.Println("Open Match assigned id %s to the ticket", received.GetId())
}

Invoking Open Match APIs via the Swagger UI

You can view and talk to this API via the Swagger UI application that is deployed with your Open Match cluster.

Google Kubernetes Engine

If your cluster runs on GKE you can access the tool from your cluster using a Public IP address. Go to Cloud Console > Kubernetes Engine > Services & Ingress and look for om-swaggerui. In that row, there’s a link to view the API browser.

Locally

The Swagger UI is accessible from your cluster via port 51500. Kubernetes’s virtual network is by default private so you’ll need to add a proxy to communicate with it.

# Open the port to the pod so that it can be accessed locally.
kubectl port-forward --namespace open-match service/om-swaggerui 51500:51500

From there you can access the proxy from http://localhost:51500.

Using the Swagger UI

Swagger UI is a generic tool for viewing APIs and interacting with them. Open Match has many APIs but the default one is the Frontend.

By clicking on a function you can see the schema of the API. To call an API click Try it out and then fill in body and then select Execute. You’ll then see the HTTP code and response.

Disable Swagger UI

For security purposes you will want to disable the Swagger UI application by setting --set open-match-core.swaggerui.enabled=false in the helm command or add the following YAML to your values.yaml.

open-match-core:
  swaggerui:
    enabled: false

Docs: Instrument Open Match with Telemetry Supports

Mon, 01 Jan 0001 00:00:00 +0000

Open Match is instrumented with OpenCensus, a telemetry library. The metrics that Open Match provides give you insight into the performance and health of your Open Match cluster. Currently, Open Match supports the following telemetry backends:

Prometheus
Grafana
Jaeger
Stackdriver

Install telemetry backend

At this point we provides two ways to install supported telemetry backend, either via K8S yaml files or via helm.s

Install via K8S yaml

# Install Prometheus
kubectl apply -n open-match -f https://open-match.dev/install/v1.8.0/yaml/03-prometheus-chart.yaml
# Install Grafana
kubectl apply -n open-match -f https://open-match.dev/install/v1.8.0/yaml/04-grafana-chart.yaml
# Install Jaeger
kubectl apply -n open-match -f https://open-match.dev/install/v1.8.0/yaml/05-jaeger-chart.yaml
# Install the above telemetry backends with Open Match core services
kubectl apply -n open-match -f https://open-match.dev/install/v1.8.0/yaml/install.yaml
# Install Stackdriver
# No need to install, provided by GKE by default.

Install via helm

$ helm install my-release --namespace open-match open-match/open-match \
--set open-match-override.enabled=true \ # Install the default override configmap
--set open-match-telemetry.enabled=true \ # Enable telemetry logic in Open Match core
--set global.telemetry.jaeger.enabled # Install jaeger with Open Match core

Enable telemetry exporters

Note

The configurations to enable/disable the telemetry exporters are not read at runtime. Please re-deploy Open Match if you want to change the telemetry settings. Also note that helm users can skip this section as overriding helm values change the default telemetry configs at the same time.

You can configure where and what telemetry you want to emit from your Open Match deployment. To instrument Open Match with telemetry supports, modify your 06-open-match-override-configmap.yaml to override the related settings. Please refer to how to customize your Open Match deployment for more details.

Below is an example of the data section in om-configmap-override configmap to enable all of the supported telemetry exporters:

matchmaker_config_override.yaml:
  telemetry:
    zpages:
      enable: 'true'
    jaeger:
      enable: 'true'
    prometheus:
      enable: 'true'
    stackdriver:
      enable: 'true'

Prometheus

Prometheus is the server for capturing and analyzing metrics. You can use it to draft metric queries that can be used to debug a service in the moment or tweak a query that can be used in a Grafana dashboard.

To access Prometheus you’ll need to kubectl port-forward port 9090 from your Kubernetes cluster.

# Port forward Prometheus to your machine, http://localhost:9090.
kubectl port-forward --namespace open-match service/open-match-prometheus-server 9090:9090

Next visit http://localhost:9090 in your browser.

Let’s try an example query, sum(rate(backend_tickets_assigned[5m]))

This returns the number of tickets assigned by the backend server over a 5-minute sliding window period.

Let’s break down each part of the query.

sum() - This will take the sum over each individual rate produced by each server (there may be more than 1 backend server).
rate(metric_name[5m]) - Take the rate of a particular metric. Since a rate is not instantaneous we must use a sliding window where we take metric_name[now] - metric_name[5m ago] at every time point. The smaller the time window the more exact but choppy the data appears.

You can learn how to query your metrics from the Prometheus Query Basics tutorial.

Grafana Dashboard

The easiest way to view the health of your Open Match cluster is to browse the Grafana dashboards. Open Match ships with a few dashboards to monitor RPC traffic, database transaction rates, and database transaction time.

To access Grafana you’ll need to kubectl port-forward port 3000 from your Kubernetes cluster.

# Port forward Grafana to your machine, http://localhost:3000.
kubectl port-forward -n open-match service/open-match-grafana 3000:3000

Next visit http://localhost:3000 in your browser with Username: admin Password: openmatch.

Once you’re in the Home Dashboard try selecting gRPC. You’ll see a page like this:

Jaeger

Jaeger is a distributed tracing tool. To view the Jaeger frontend, please run

# Port forward Jaeger frontend UI to your machine, http://localhost:16686.
kubectl port-forward -n open-match service/open-match-jaeger-query 16686:16686

Next visit http://localhost:16686 in your browser. You’ll see a page like this:

What’s Next?

Docs: Evaluator and Synchronizer

Mon, 01 Jan 0001 00:00:00 +0000

Overlapping Match Profiles

In a basic flow to generate Matches, the Director calls FetchMatches on Open Match backend for one or more MatchProfiles. Open Match triggers execution of MatchFunctions for each of these requested MatchProfiles concurrently. Each MatchFunction queries Open Match for the Tickets that Match the filtering criteria specified for the Pools in their MatchProfile. The MatchFunction then generates Matches using the Tickets it fetched and returns them as proposals.

If the MatchProfiles perfectly partition the concurrent Ticket Pool (their Pool filters are such that each MatchProfile will only Match a non overlapping set of Tickets), then the Matches generated by the MatchFunction execution do not have any overlap.

However, this may not always be the case. There are scenarios in which, the Pools specified in concurrent MatchProfiles may have an overlap and thus concurrently executing MatchFunctions could use overlapping Ticket in its proposals.

Here is an example scenario for clarification: Lets assume a simple case where a Ticket has an attribute ‘mmr’ that can be a value between 0 and 10.

Case1 - Non-overlapping MatchProfiles

The Director could request for the following Profiles:

Profile1 has a Pool that filters all Tickets with mmr 0 to 5.
Profile2 has a Pool that filters all Tickets with mmr 6 to 10.

Given the mmr ranges, there is no overlap between the Tickets fetched by the MatchFunction execution for each of these profiles. Hence there is no overlap in the generated result matches.

Case2 - Overlapping MatchProfiles

The Director could request for the following Profiles:

Profile1 has a Pool that filters all Tickets with mmr 0 to 5.
Profile2 has a Pool that filters all Tickets with mmr 2 to 8.
Profile3 has a Pool that filters all Tickets with mmr 6 to 10.

In this case, the MatchFunction executing Profile1 and Profile2 will have some common players they consider for generating Matches (Similarly with Profile2 and Profile3). Thus, the generated Matches may have overlapping Tickets.

This overlap is desired in some real matchmaking scenarios, . Inorder to generate high quality Matches, you may want your Ticket to be considered concurrently for multiple possible MatchProfiles and then be able to compare the quality of these overlapping Matches, picking the highest quality Match and discarding the rest. For instance in the above example, a Ticket with mmr 3 could be a part of a Matches generated by Profile1 and Profile2. Based on the game design and the mmr of other Tickets in these overlapping Matches, one of these two Matches may actually be a better fit for this Ticket - and thus, searching for Matches with overlapping mmr ranges resulted in considering 2 options for this Ticket and picking the better fit.

The ability to provide this model of executing concurrent MatchFunctions on overlapping Ticket Pool is a core value proposition of Open Match.

Evaluator

Why do we need an Evaluator

To allow generation of proposals using overlapping Tickets but still provide unique Matches to the Director, we need to decollide the proposals. To achieve this, Open Match has an Evaluator component that accepts all concurrent Match proposals, evaluates them for collision and returns non-overlapping results back to Open Match. If there are multiple overlapping Matches, the Evaluator evaluates their quality, promotes the highest quality Match as a result and discards the others. The key responsibility of the Evaluator is to ensure that there are no overlapping Tickets in the result Matches returned.

How to use an Evaluator

Open Match provides a default score based Evaluator that can be used in most common scenarios but it also provides the user the ability to build your own Evaluator and to customize Open Match to use the user’s Evaluator. At a high level, there are three evaluation scenarios for you to consider when using Open Match:

Your MatchProfiles completely partition the Ticket Pool and so you will never have collisions. The default Evaluator handles this scenario and you do not have to make any changes.
Your MatchProfiles may use overlapping Ticket Pools - but each MatchFunction can simply generate a quality score for that Match based on certain game specific criteria (latency, mmr etc.). The default Evaluator can be used for this scenario. See Default Evaluator Tutorial
You have complex evaluation logic that cannot be simplified to a score for a Match but rather involves actually comparing some details of the overlapping Matches. Open Match provides you with the ability to plug in your custom Evaluator to handle this case. See the Custom Evaluator Tutorial for details.

How to configure an Evaluator

Inorder to invoke the Evaluator at runtime, Open Match needs to know the location of the Evaluator endpoint. This is provided to Open Match as an override ConfigMap. Here are the field in that config map that specifically configure an Evaluator in Open Match:

  api:
    evaluator:
      hostname: <Fully qualified host name>
      grpcport: <service port for the Evaluator>

Synchronizer

Why do we need a Synchronizer

Concurrently executing MatchFunctions fetch Tickets for their MatchProfiles from Open Match at the same time. Thus, the same active Ticket Pool is visible to multiple MatchFunctions (resulting in collision). Although Evaluation handles decolliding the overlapping results, there is a race condition. While the Evaluator is evaluating Match proposals, if MatchFunction execution is permitted, then the MatchFunctions may possible fetch Tickets that were already promoted as results. There is thus a need to stop execution of MatchFunctions (thereby halting new observations from being made) while existing results are being evaluated. This is called Synchronization.

Overview of Synchronizer States

Synchronizer’s key role is to make sure that all the proposals generated by concurrent MatchFunctions get evaluated together. Further, the Synchronizer needs to halt function execution while the results are evaluated. Given that the MatchFunction is authored by the user, Open Match has no control over its execution time. However, Open Match has to make sure its results are evaluated with the other concurrent Match Functions - or are discarded if they missed the Evaluation time. To achieve this, the singleton Synchronizer component in Open Match has the following states:

Idle State

The synchronizer is in Idle state when there is no ongoing MatchFunction execution.

Registration

Open Match backend registers each FetchMatch request with the Sychronizer before it calls into the MatchFunction. The Synchronizer enters the Registration window (configurable) upon first FetchMatches registration . All the MatchFunctions that get registered in this registration window will be evaluated together.

ProposalCollection

After the Registration Window is over, The synchronizer can be configured to give some time for the registered functions to return. This is called the ProposalCollection window. This is only so that even the last registered FetchMatches request has a chance to generate proposals before triggering Evaluation.

Evaluation

This is where all the generated proposals from all the registered MatchFunctions are sent to the Evaluator for evaluation. Synchronization is achieved by primarily two behaviors:

Any new Match Function that attempts to Register in the ProposalCollection or Evaluation window is blocked till the evaluation is complete and registration opens again.
Proposals from a MatchFunction execution that return after the ProposalCollection window are discarded (they missed their evaluation window)

How to configure a Synchronizer

Open Match enables the following two window durations to be configurable by the user. These are specified in the override ConfigMap.

  registrationIntervalMs: <millisecond value>
  proposalCollectionIntervalMs: <millisecond value>

Here is what you should consider when configuring these:

registrationIntervalMs: This is the amount of time within which we expect the Director to send all the FetchMatch requests for new matches and backfills.
proposalCollectionIntervalMs: This is the maximum amount of time it would take to execute a MatchFunction after which, the results from the MatchFunction will be ignored.

Docs: Backfill

Mon, 01 Jan 0001 00:00:00 +0000

Background

Some scenarios that could happen when playing multiplayer games include:

A multiplayer game has a player(s) leave the game. To keep the game interesting and fun, matchmaking will replace the missing player with someone new.
A large player count game (e.g.: battle royale) will create a non-full match and allow players to connect to the game server prior to it being full. Players will wait in a lobby on the game server while new players are backfilled into the game until the server is fully populated.
A social or world place in a game allows players to see others in the space, but other players are not strictly required. When players enter the game it is preferred to put players into an existing server, preventing players from joining low population servers. As existing servers reach capacity, the game server management service starts up new empty servers. The new empty servers request players to be backfilled into them, allowing more players to join world servers. Servers may be very long living, as new players replace the leaving players.
A multiplayer game has a player(s), leave the game, and makes a request for new players to be backfilled into the game. One of the players in the game invites a friend to join using a social platform. The friend joins the game, and the game server cancels its backfill request.

Main algorithm

A Match Function example using Backfill can be found here.

Player 1 creates a Ticket (T1).
Director calls Backend.FetchMatches with a MatchProfile.
MMF runs QueryBackfills and QueryTickets using the MatchProfile. It returns T1.
MMF constructs a new Backfill (B1) and returns a Match containing ticket T1 and Backfill B1. The Backfill should be created with field AllocateGameServer set to true, so the game server orchestrator (e.g.: Agones) knows it has to create a new Game Server. The Director handles making calls to create the Game Server.
Director starts allocating a Game Server with Backfill B1 information.
Player 2 creates a Ticket (T2).
Director calls Backend.FetchMatches.
MMF runs QueryBackfills and QueryTickets using the MatchProfile. They return B1 and T2 accordingly.
MMF determines B1 could be used based on data set on B1 by the last run of the MMF (e.g. a number of open slots). T2 and B1 form a new Match.
When the Game Server has started, it begins polling Open Match to acknowledge the backfill with Frontend.AcknowledgeBackfill using the backfill’s ID (B1.ID) and supplies connection information for the server, to be returned as the Assignment data of the tickets T1 and T2. The Game Server also receives the ticket data of the tickets that were assigned so that any data supplied on the ticket (e.g. player IDs) may be used by the game server.

Considerations

In order to use Backfill you need to customize your MMF and your Game Servers. Backfill IDs should be propagated to each Game Server during Game Server allocation to be used in AcknowledgeBackfill requests to define Backfill which is tied to the Game Server.

If Backfill is not being used, no changes are required since the Backfill API doesn’t change the behavior of the current Open Match API state.

Acknowledging Backfills

AcknowledgeBackfill is a request made to the Open Match Frontend that serves to notify Open Match that server still interested in the backfill, and to confirm the assignment of the tickets associated with this backfill in the same manner as Backend.AssignTickets. The Game Server receives the current Backfill status as well as a list of new tickets that were just acknowledged by the request as a response. The list is not idempotent, but can be appended to prior lists received from AcknowledgeBackfill.

GameServers must run AcknowledgeBackfill within backfillTTL seconds after the Backfill is created and within backfillTTL of the last AcknowledgeBackfill call after that, where backfillTTL is 80% of the pendingReleaseTimeout configuration value.

Note: You must keep acknowledging a backfill until it is no longer needed, and if it expires the Backfill object must be recreated and the new ID communicated to the Game Server to begin the AcknowledgeBackfill process anew.

Cleaning up Backfills

This process deletes expired Backfill objects from the state store. Backfills become expired if they are not acknowledged in the backfillTTL period.

Docs: Customize Core Open Match

Mon, 01 Jan 0001 00:00:00 +0000

Open Match needs to be customized to your Matchmaker. This custom configuration is provided to the Open Match components via a ConfigMap (om-configmap-override). Thus, starting the core service pods waits on this config map being available.

To customize an Open Match installation, we need to provide a ConfigMap with the sample configuration. Here is a sample YAML that contains the customization ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: om-configmap-override
  namespace: open-match
data:
  matchmaker_config_override.yaml: |-
    # Specifies the hostname and port of the evaluator that Open Match should talk to.
    api:
      evaluator:
        hostname: "om-evaluator"
        grpcport: "50508"
        httpport: "51508"
    # Specifies if we should turn on/off the synchronizer.
    #   - If on, specifies the registrationInterval and proposalCollectionInterval in milliseconds.
    synchronizer:
      enabled: true
      registrationIntervalMs: 3000ms
      proposalCollectionIntervalMs: 2000ms

To configure Open Match with your custom config, create a new YAML file with this content, make your edits and run this command:

kubectl apply -f <filename_here>

The Evaluator Guide gives an overview of Synchronization, Evaluation and all the related configuration.

Configurations

The following tables lists the configurable parameters of the Open Match override configmap and their default values.

Telemetry

Supported Backend	Parameter	Description	Default
zPages		Serves HTTP server runtime profiling data in the format expected by the pprof visualization tool under `/debug/pprof` path
	`telemetry.zpages.enabled`	Enable zPages support on Open Match core services	`true`
Jaeger		An open source, end-to-end distributed tracing.
	`telemetry.jaeger.enabled`	Enable Jaeger exporter on Open Match core services	`false`
	`telemetry.jaeger.samplerFraction`	Setup Jaeger exporter to randomly sample a trace with this probability. For example, with `samplerFraction: 0.5` approximately 1 out of 2 traces will be sampled	`1`
	`telemetry.jaeger.agentEndpoint`	AgentEndpoint instructs exporter to send spans to jaeger-agent at this address.	`open-match-jaeger-agent:6831`
	`telemetry.jaeger.collectorEndpoint`	CollectorEndpoint is the full url to the Jaeger HTTP Thrift collector.	`open-match-jaeger-collector:14268/api/traces`
Prometheus
	`telemetry.prometheus.enabled`	Enable Prometheus exporter on Open Match core services	`false`
	`telemetry.prometheus.endpoint`	Bind the Prometheus exporters to the specified endpoint handler, also configures the `prometheus.io/path` k8s scraping annotations	`/metrics`
	`telemetry.prometheus.serviceDiscovery`	If Prometheus is enabled and `serviceDiscover: true`, add the Prometheus scraping annotations to each Pod of the Open Match core services	`true`
Stackdriver
	`telemetry.stackdriverMetrics.enabled`	Enable Stackdriver Metric exporter on Open Match core services	`false`
	`telemetry.stackdriverMetrics.prefix`	MetricPrefix overrides the prefix of a Stackdriver metric display names to help you better identifies your metrics	`open_match`
Grafana
	`telemetry.grafana.enabled`	Enable Grafana exporter on Open Match core services	`false`

Docs: Access Open Match Endpoints

Mon, 01 Jan 0001 00:00:00 +0000

You can talk to Open Match either via an in-cluster or out-of-cluster service client.

Access Open Match via an out-of-cluster service client

To access Open Match via an out-of-cluster client, the first step is to expose a Service onto an external IP address. We recommend exposing your service using Kubernetes Load Balancers in production for a public IP if needed. Open Match provides two different ways to configure your service with Load Balancer. The following provides examples to expose the backend service with code examples in Go.

1 Exposing the Service using Cloud Load Balancer

a. Modify the install.yaml files of the latest release

# Download the latest install.yaml file
wget http://open-match.dev/install/v1.8.0/yaml/01-open-match-core.yaml

Find and modify the spec.type fields of the Service that you want to expose to LoadBalancer.

kind: Service
apiVersion: v1
metadata:
  name: om-backend
  ...
spec:
  type: LoadBalancer

Save the changes and deploy Open Match

kubectl apply -f ./install.yaml

b. Configure Load Balancer using the Helm chart

If you don’t have Helm installed locally, or Tiller installed in your Kubernetes cluster, read the Using Helm documentation to get started.

Open Match’s helm chart lives under the install/helm/open-match folder of the root directory. To install Open Match with Load Balancer enabled:

# Expose Service onto a public IP using helm cli
helm upgrade [YOUR_HELM_RELEASE_NAME] --install --wait --debug -n open-match \
    --set global.gcpProjectId=[YOUR_GCPPROJECT_ID] \
    --set backend.portType=LoadBalancer

2. Wait for the external IP address

$ kubectl get services -n open-match om-backend
# The output is similar to this
# Note: If the external IP address is shown as <pending>, wait for a minute and enter the same command again.
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)               AGE
om-backend   LoadBalancer   10.0.4.7     104.198.205.71        50505/TCP,51505/TCP   3h2m

3. Talk to Open Match using an out-of-cluster client with the external IP

// Create a gRPC backend client
conn, err := grpc.Dial("<external-ip>:50505", grpc.WithInsecure())
...
beClient := pb.NewBackendClient(conn)
...

Note

If you want to obtain the external IP programmatically, look at the out of cluster configuration example from the Kubernetes client and use the code below for your reference.

import (
    ...
    "errors"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
)

...
// Get the Open Match backend service object using the kubernetes go client
svc, err := kubeClient.CoreV1().Services("open-match").Get("om-backend", metav1.GetOptions{})
if err != nil {
    panic(err)
}

if len(svc.Status.LoadBalancer.Ingress) == 0 {
    panic(errors.New(fmt.Sprintf("service: %s does not have ingress exposed.\n", svcName)))
}

externalIP := svc.Status.LoadBalancer.Ingress[0].IP

Access Open Match via an in-cluster service client

To access Open Match with a service client deployed in the same namespace as Open Match

// Create a gRPC backend client
conn, err := grpc.Dial("om-backend:50505", grpc.WithInsecure())
...
beClient := pb.NewBackendClient(conn)
...

To access Open Match with a service client deployed in other namespaces than Open Match

// Create a gRPC backend client
// [OPEN-MATCH-NAMESPACE] is usually open-match if you install Open Match via the official yaml files.
conn, err := grpc.Dial("om-backend.[OPEN-MATCH-NAMESPACE]:50505", grpc.WithInsecure())
...
beClient := pb.NewBackendClient(conn)
...

Docs: Production Best Practices

Mon, 01 Jan 0001 00:00:00 +0000

Here are a list of settings that may help you to improve Open Match’s scalability, stability, and availability under production environments.

Setup grpc.KeepaliveEnforcementPolicy for your MMF and evaluator server

We recommend adding the grpc.WithKeepaliveParams and grpc.KeepaliveEnforcementPolicy to your customizable components. The Keepalive settings will allow the client to ping the server after a duration of time to see if it is still alive. This could help the application to cleanup the invalid connections due to pod changes from Kubernetes' HorizontalPodAutoscaler. Here is a code snippet in go about these settings:

import (
    "google.golang.org/grpc/keepalive"
    "google.golang.org/grpc"
)

...
// Client side settings
conn, err := grpc.Dial("some_address", grpc.WithKeepaliveParams(
    keepalive.ClientParameters{
        Time: 20 * time.Second,
        Timeout: 10* time.Second,
        PermitWithoutStream: true,
    })
)
...
// Server side settings
server := grpc.NewServer(
    grpc.KeepaliveEnforcementPolicy(
        keepalive.EnforcementPolicy{
            MinTime:             10 * time.Second,
            PermitWithoutStream: true,
        },
    ),
)

Enable the client-side load balancer for your gRPC client

You need to add the a load balancer config to your gRPC client to enable Open Match’s horizontal scaling feature. Here are the two steps to turn on this setting and make it compatible with Kubernetes' environment.

Changing the Kubernetes' services of your customized components into HeadlessServices, if any. e.g.:

# https://kubernetes.io/docs/concepts/services-networking/service/#headless-services
spec:
  selector:
    app: mmf
    component: mmf
    release: open-match
  clusterIP: None
  type: ClusterIP

Using gRPC’s DNS resolver when creating your clients and enable the client side load balancer:

import (
    "google.golang.org/grpc/resolver"
)

func init() {
    // Using gRPC's DNS resolver to create clients.
    // This is a workaround for load balancing gRPC applications under k8s environments.
    // See https://kubernetes.io/blog/2018/11/07/grpc-load-balancing-on-kubernetes-without-tears/ for more details.
    // https://godoc.org/google.golang.org/grpc/resolver#SetDefaultScheme
    resolver.SetDefaultScheme("dns")
}

// Adding the client side load balancing config
conn, err := grpc.Dial(
    "some_address",
    grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`),
)

Enable Redis Sentinel for high-availability and failover supports

Redis Sentinel provides high-availability for Redis. By default, the sentinel is disabled to lower the resource requirements required by Open Match. If you prefer to turn on the sentinel, please override the value of redis.sentinel.enabled to true when installing Open Match via helm.

helm install my-release -n open-match open-match/open-match --set redis.sentinel.enabled=true

Deploy Open Match based on `values-production.yaml` for production setup

Open Match project provides a recommended helm config file for production setups under install/helm/open-match/values-production.yaml. Note that this setup requires significant resources for running production workload.

helm install my-release -n open-match open-match/open-match -f values-production.yaml

Use Envoy or other load balancing solution if you plan to connect to Open Match via an out-of-cluster client

The above load balancing solution is sufficient if you have both the client and the server deployment within the same cluster. However, some game architectures may require connecting to Open Match services from an out-of-cluster client. We recommend Envoy as a solution. Alternatives like Kubernetes Ingress or platform specific L7 Load Balancer can also work.

Use alternative to included Redis (Bitnami) image

Open Match comes included with a Redis image with a high-availability option. If your needs require a Redis offering with features unavailable with the included image, Open Match provides configs to bring your own Redis. First, you must disable the use of the included image by setting the open-match-core.redis.enabled param to false. To connect to your alternative Redis setup, provide the instance’s hostname/IP address to the open-match-core.redis.hostname.

Note: If Using Memorystore

To connect to Memorystore from a GKE cluster it requires a VPC-native cluster. If you’ve created a cluster, you will have to re-create it as VPC-native cluster using Alias-IPs.

helm install open-match --create-namespace --namespace open-match open-match/open-match \
  --set open-match-customize.enabled=true \
  --set open-match-customize.evaluator.enabled=true \
  --set open-match-override.enabled=true \
  --set open-match-core.redis.enabled=false \
  --set open-match-core.redis.hostname= # Your redis server address

Docs: TLS Encryption

Mon, 01 Jan 0001 00:00:00 +0000

Open Match API is served in HTTP and gRPC. By default, these are served over unencrypted channels which are not good for security posture. Open Match provides support for TLS transport mode.

Certificates

Before we can enable TLS transport mode we’ll need to generate certificate-private key pairs. These files will contain the encryption keys used to secure communication between each server.

Certgen

Certgen uses the golang tls library and has not been reviewed by a security expert. Use it at your own risk.

Open Match provides a certgen tool for your convenience to generate certificates. This tool is used to generate the TLS certificates for development and may be used to quickly configure a secure cluster while waiting for your final certificates.

# Create Root-CA directory.
mkdir -p secrets/tls/root-ca

# Generate Root-CA public certificate-private key pair.
./certgen -ca=true -publiccertificate=secrets/tls/root-ca/public.cert -privatekey=secrets/tls/root-ca/private.key

# Create Server Keys directory.
mkdir -p secrets/tls/server/

# Create server public certificate-private key pair derived from the Root-CA keys.
./certgen -publiccertificate=secrets/tls/server/public.cert -privatekey=secrets/tls/server/private.key -rootpubliccertificate=secrets/tls/root-ca/public.cert -rootprivatekey=secrets/tls/root-ca/private.key

The instructions above only generate 1 server key for all servers. For 1 key per server follow the OpenSSL instructions below.

OpenSSL

OpenSSL is a popular library to support SSL/TLS. We’ll be using the tool OpenSSL ships with the generated public certificate-private key pairs for each Open Match server you run.

The steps below will do the following:

Create a self-signed certificate authority. Basically, this is the root key. The public key will be trusted by all servers and clients and the private key must be stored securely somewhere, it’s only used to generate derived keys. A compromise of the root private key compromises the whole Open Match cluster.
Create self-signed public certificate-private key pair for each server {om-frontend, om-backend, etc…}. A compromise of this private key will only compromise that particular server.
Enable TLS mode on Open Match servers.
Mount the TLS certificate for the Open Match server.

# Create a Root-CA (Certificate Authority) private key.
# This will prompt for a password.
# This key will only be used for generating derived public certificate-private key pairs.
# DO NOT USE on a server.
openssl genrsa -des3 -out rootCA.key 4096

# Generate the Root-CA public certificate. This is the "trusted certificate"
# that each client and server needs to know about so they can trust traffic
# from each other.
openssl req -x509 -subj "/C=US/ST=Open/L=Match/O=Open Match/OU=Match Making/CN=open-match.dev" -new -nodes -key rootCA.key -sha256 -days 1825 -out rootCA.cert

# Create a derived public certificate-private key pair
openssl req -new -sha256 -key om-frontend.key -subj "/C=US/ST=Open/L=Match/O=Open Match/OU=Frontend/CN=open-match.dev" -out om-frontend.cert

# The steps below are to generate a public certificate-private key pair for the
# om-frontend server. Repeat the following steps for each server. The address
# of the server is specified by the Common Name (CN) parameter.
#
# The list of servers you'll want to generate keys for are:
# om-backend:51505,om-demo:51507,om-demoevaluator:51508,om-demofunction:51502,om-e2eevaluator:51518,om-e2ematchfunction:51512,om-frontend:51504,om-query:51503,om-swaggerui:51500,om-synchronizer:51506

# Create a private key for the om-frontend server. (No password)
openssl genrsa -out om-frontend.key 4096

# Create a certificate signing request (CSR). This is basically a proposal to
# create a certificate based on the parameters you specify here.
openssl req -new -sha256 -key om-frontend.key -subj "/C=US/ST=Open/L=Match/O=Open Match/OU=Match Making/CN=om-frontend:51504" -out om-frontend.csr

# (optional) Verify that the CSR looks like you expected it to.
openssl req -in om-frontend.csr -noout -text

# Create the certificate for the frontend server based on the CSR just created.
# The certificate will be backed by the Root-CA you created in the first step.
openssl x509 -req -in om-frontend.csr -CA rootCA.cert -CAkey rootCA.key -CAcreateserial -out om-frontend.cert -days 1825 -sha256

# (optional) Verify that the om-frontend certificate looks like you expected
# it to.
openssl x509 -in om-frontend.cert -text -noout

If the hostname does not match the server the client will report a hostname verification error.

Configuration

To enable TLS-mode you’ll need to make the following changes:

matchmaker_config.yaml

api:
  tls:
    certificatefile: '/app/secrets/tls/server/public.cert'
    privatekey: '/app/secrets/tls/server/private.key'
    rootcertificatefile: '/app/secrets/tls/rootca/public.cert'
    trustedCertificatePath: '/app/secrets/tls/rootca/public.cert'

Run these Kubernetes commands to install the secrets in your cluster:

# Create Secret for Root-CA
kubectl create secret generic om-tls-rootca --namespace open-match --from-file=secrets/tls/root-ca/public.cert
# Create Secret for Server certificate.
kubectl create secret generic om-frontend-tls --namespace open-match --from-file=secrets/tls/server/om-frontend.key --from-file=secrets/tls/server/om-frontend.cert

Add the following to your existing deployment for Frontend. This is not a complete Deployment it only contains the TLS configuration.

kind: Deployment
metadata:
  name: om-frontend
spec:
  template:
    spec:
      volumes:
      - name: root-ca-volume
        secret:
          secretName: om-tls-rootca
      - name: tls-server-volume
        secret:
          # This may be different for each server based on how granular you want your keys.
          secretName: om-frontend-tls
      containers:
      - name: om-frontend
        volumeMounts:
        - name: om-config-volume
          mountPath: /app/config
        - name: root-ca-volume
          mountPath: /app/secrets/tls/rootca
        - name: tls-server-volume
          mountPath: /app/secrets/tls/server

Limitations

There may only be 1 trusted CA installed in an Open Match server. Each server may have it’s own.
The om-swaggerui and om-demo services do not support serving TLS but will communicate with other Open Match servers using TLS encryption if configured.
Health checks and monitoring may not work. Prometheus supports TLS but the current Kubernetes configurations do not enable it at this time.
Mixed TLS and unencrypted transport modes are not supported. If you’re enabling TLS all servers must have it enabled.

Open Match – Guides

Docs: Matchmaking using Open Match

Objective

Game Services Architecture

Open Match Core

Open Match FrontendService

Open Match BackendService

Open Match QueryService

Customized Components

Match Function

Evaluator (Optional)

External Components

Game Frontend

Director

Matchmaking Flow

Creating a Matchmaking request

Creating a Match

Setting an Assignment

Next Steps

Components Deep Dive

Docs: Invoking Open Match APIs

Open Match default hostnames and endpoints

Invoking Open Match gRPC APIs

Invoking Open Match HTTP APIs

Invoking Open Match APIs via the Swagger UI

Google Kubernetes Engine

Locally

Using the Swagger UI

Disable Swagger UI

Docs: Instrument Open Match with Telemetry Supports

Install telemetry backend

Enable telemetry exporters

Note

Prometheus

Grafana Dashboard

Jaeger

What’s Next?

Docs: Evaluator and Synchronizer

Overlapping Match Profiles

Case1 - Non-overlapping MatchProfiles

Case2 - Overlapping MatchProfiles

Evaluator

Why do we need an Evaluator

How to use an Evaluator

How to configure an Evaluator

Synchronizer

Why do we need a Synchronizer

Overview of Synchronizer States

Idle State

Registration

ProposalCollection

Evaluation

How to configure a Synchronizer

Docs: Backfill

Background

Main algorithm

Considerations

Acknowledging Backfills

Cleaning up Backfills

Docs: Customize Core Open Match

Configurations

Telemetry

Docs: Access Open Match Endpoints

Access Open Match via an out-of-cluster service client

1 Exposing the Service using Cloud Load Balancer

a. Modify the install.yaml files of the latest release

b. Configure Load Balancer using the Helm chart

2. Wait for the external IP address

3. Talk to Open Match using an out-of-cluster client with the external IP

Note

Access Open Match via an in-cluster service client

Docs: Production Best Practices

Setup grpc.KeepaliveEnforcementPolicy for your MMF and evaluator server

Enable the client-side load balancer for your gRPC client

Enable Redis Sentinel for high-availability and failover supports

Deploy Open Match based on values-production.yaml for production setup

Use Envoy or other load balancing solution if you plan to connect to Open Match via an out-of-cluster client

Use alternative to included Redis (Bitnami) image

Note: If Using Memorystore

Docs: TLS Encryption

Deploy Open Match based on `values-production.yaml` for production setup