Ri Xu Online

In 2001, C10k problem are raised, there are already many ways to solve this problem. In this post, I will implement related problem to create a WebSocket server by Go.

A WebSocket server is a TCP application listening on any port of a server that follows a specific protocol, simple as that. The task of creating a custom server tends to scare people; however, it can be easy to implement a simple WebSocket server on your platform of choice.

You will need to already know how HTTP works and have medium programming experience. Depending on language support, knowledge of TCP sockets may be required. The scope of this guide is to present the minimum knowledge you need to write a WebSocket server.

Exchanging Data Frames

Either the client or the server can choose to send a message at any time — that's the magic of WebSockets. However, extracting information from these so-called "frames" of data is a not-so-magical experience. Although all frames follow the same specific format, data going from the client to the server is masked using XOR encryption (with a 32-bit key). Section 5 of the specification describes this in detail.

• Format

Each data frame (from the client to the server or vice-versa) follows this same format:

 0               1               2               3
 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
+-+-+-+-+-------+-+-------------+-------------------------------+
|F|R|R|R| opcode|M| Payload len |    Extended payload length    |
|I|S|S|S|  (4)  |A|     (7)     |             (16/64)           |
|N|V|V|V|       |S|             |   (if payload len==126/127)   |
| |1|2|3|       |K|             |                               |
+-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
 4               5               6               7
+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
|     Extended payload length continued, if payload len == 127  |
+ - - - - - - - - - - - - - - - +-------------------------------+
 8               9               10              11
+ - - - - - - - - - - - - - - - +-------------------------------+
|                               |Masking-key, if MASK set to 1  |
+-------------------------------+-------------------------------+
 12              13              14              15
+-------------------------------+-------------------------------+
| Masking-key (continued)       |          Payload Data         |
+-------------------------------- - - - - - - - - - - - - - - - +
:                     Payload Data continued ...                :
+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
|                     Payload Data continued ...                |
+---------------------------------------------------------------+

RSV1-3 can be ignored, they are for extensions.

The MASK bit simply tells whether the message is encoded. Messages from the client must be masked, so your server should expect this to be 1. (In fact, section 5.1 of the spec says that your server must disconnect from a client if that client sends an unmasked message.) When sending a frame back to the client, do not mask it and do not set the mask bit. We'll explain masking later. Note: You have to mask messages even when using a secure socket.

The opcode field defines how to interpret the payload data: 0x0 for continuation, 0x1 for text (which is always encoded in UTF-8), 0x2 for binary, and other so-called "control codes" that will be discussed later. In this version of WebSockets, 0x3 to 0x7 and 0xB to 0xF have no meaning.

The FIN bit tells whether this is the last message in a series. If it's 0, then the server will keep listening for more parts of the message; otherwise, the server should consider the message delivered. More on this later.

• Decoding Payload Length

To read the payload data, you must know when to stop reading. That's why the payload length is important to know. Unfortunately, this is somewhat complicated. To read it, follow these steps:

1. Read bits 9-15 (inclusive) and interpret that as an unsigned integer. If it's 125 or less, then that's the length; you're done. If it's 126, go to step 2. If it's 127, go to step 3.
2. Read the next 16 bits and interpret those as an unsigned integer. You're done.
3. Read the next 64 bits and interpret those as an unsigned integer (The most significant bit MUST be 0). You're done.

The following package has been created Websocket abstract connection method, you can use their to create WebSocket server with Go.

$ go get github.com/gorilla/websocket
$ go get github.com/rgamba/evtwebsocket

WebSocket Server

hub.go

package main

// hub maintains the set of active clients and broadcasts messages to the clients.
type hub struct {
    // Registered clients.
    connections map[*connection]bool

    // Inbound messages from the clients.
    broadcast chan []byte

    // Register requests from the clients.
    register chan *connection

    // Unregister requests from clients.
    unregister chan *connection
}

var h = hub{
    broadcast:   make(chan []byte),
    register:    make(chan *connection),
    unregister:  make(chan *connection),
    connections: make(map[*connection]bool),
}

func (h *hub) run() {
    for {
        select {
        case c := <-h.register:
            h.connections[c] = true
        case c := <-h.unregister:
            if _, ok := h.connections[c]; ok {
                delete(h.connections, c)
                close(c.send)
            }
        case m := <-h.broadcast:
            for c := range h.connections {
                select {
                case c.send <- m:
                default:
                    delete(h.connections, c)
                    close(c.send)
                }
            }
        }
    }
}

connection.go

package main

import (
    "fat"

    "github.com/gorilla/websocket"
    "net/http"
)

var online int

// connection is an middleman between the websocket connection and the hub.
type connection struct {
    // The web socket connection
    ws *websocket.Conn

    // Buffered channel of outbound messages.
    send chan []byte
}

// reader pumps messages from the websocket connection to the hub.
func (c *connection) reader() {
    for {
        _, message, err := c.ws.ReadMessage()
        if err != nil {
            fmt.Println(err)
            break
        }
        h.broadcast <- message
    }
    c.ws.Close()
    online -= 1
    fmt.Println(online)
}

// write writes a message with the given message type and payload.
func (c *connection) writer() {
    for message := range c.send {
        err := c.ws.WriteMessage(websocket.TextMessage, message)
        if err != nil {
            fmt.Println(err)
            break
        }
    }
    c.ws.Close()
}

var upgrader = &websocket.Upgrader{ReadBufferSize: 1024, WriteBufferSize: 1024, CheckOrigin: func(r *http.Request) bool {
    return true
}}

// wsHandler handles websocket requests from the peer.
func wsHandler(w http.ResponseWriter, r *http.Request) {
    ws, err := upgrader.Upgrade(w, r, nil)
    if err != nil {
        fmt.Println(err)
        return
    }
    online += 1
    fmt.Println(online)
    c := &connection{send: make(chan []byte, 256), ws: ws}
    h.register <- c
    defer func() { h.unregister <- c }()
    go c.writer()
    c.reader()
}

main.go

package main

import (
    "flag"
    "go/build"
    "log"
    "net/http"
    "path/filepath"
    "text/template"
)

var (
    addr      = flag.String("addr", ":8080", "http service address")
    assets    = flag.String("assets", defaultAssetPath(), "path to assets")
    homeTempl *template.Template
)

func defaultAssetPath() string {
    p, err := build.Default.Import("github.com/gorilla/websocket", "", build.FindOnly)
    if err != nil {
        return "."
    }
    return p.Dir
}

func homeHandler(c http.ResponseWriter, req *http.Request) {
    homeTempl.Execute(c, req.Host)
}

func benchmarkHandler(c http.ResponseWriter, req *http.Request) {
    h.broadcast <- []byte("test message")
}

func main() {
    flag.Parse()
    homeTempl = template.Must(template.ParseFiles(filepath.Join(*assets, "home.html")))
    go h.run()
    http.HandleFunc("/", homeHandler)
    http.HandleFunc("/ws", wsHandler)
    http.HandleFunc("/benchmark", benchmarkHandler)
    if err := http.ListenAndServe(*addr, nil); err != nil {
        log.Fatal("ListenAndServe:", err)
    }
}

WebSocket Client

home.html

<html>
<head>
<title>Chat Example</title>
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.0/jquery.min.js"></script>
<script type="text/javascript">
    $(function() {

    var conn;
    var msg = $("#msg");
    var log = $("#log");

    function appendLog(msg) {
        var d = log[0]
        var doScroll = d.scrollTop == d.scrollHeight - d.clientHeight;
        msg.appendTo(log)
        if (doScroll) {
            d.scrollTop = d.scrollHeight - d.clientHeight;
        }
    }

    $("#form").submit(function() {
        if (!conn) {
            return false;
        }
        if (!msg.val()) {
            return false;
        }
        conn.send(msg.val());
        msg.val("");
        return false
    });

    if (window["WebSocket"]) {
        conn = new WebSocket("ws://<server_ip:server_port>/ws");
        conn.onclose = function(evt) {
            appendLog($("<div><b>Connection closed.</b></div>"))
        }
        conn.onmessage = function(evt) {
            appendLog($("<div/>").text(evt.data))
        }
    } else {
        appendLog($("<div><b>Your browser does not support WebSockets.</b></div>"))
    }
    });
</script>
<style type="text/css">
html {
    overflow: hidden;
}

body {
    overflow: hidden;
    padding: 0;
    margin: 0;
    width: 100%;
    height: 100%;
    background: gray;
}

#log {
    background: white;
    margin: 0;
    padding: 0.5em 0.5em 0.5em 0.5em;
    position: absolute;
    top: 0.5em;
    left: 0.5em;
    right: 0.5em;
    bottom: 3em;
    overflow: auto;
}

#form {
    padding: 0 0.5em 0 0.5em;
    margin: 0;
    position: absolute;
    bottom: 1em;
    left: 0px;
    width: 100%;
    overflow: hidden;
}

</style>
</head>
<body>
<div id="log"></div>
<form id="form">
    <input type="submit" value="Send" />
    <input type="text" id="msg" size="64"/>
</form>
</body>
</html>

benchmark.go

package main

import (
    "fat"
    "log"

    "github.com/rgamba/evtwebsocket"
    "golang.org/x/net/websocket"
)

func main() {
    for i := 0; i < 1000; i++ {
        go func() {
            c := evtwebsocket.Conn{

                // When connection is established
                OnConnected: func(w *websocket.Conn) {
                    fmt.Println("Connected")
                },

                // When a message arrives
                OnMessage: func(msg []byte) {
                    log.Printf("Received uncatched message: %s\n", msg)
                },

                // When the client disconnects for any reason
                OnError: func(err error) {
                    fmt.Printf("** ERROR **\n%s\n", err.Error())
                },

                // This is used to match the request and response messages
                MatchMsg: func(req, resp []byte) bool {
                    return string(req) == string(rest)
                },
            }

            // Connect
            if err := c.Dial("ws://<server_ip:server_port>/ws"); err != nil {
                log.Fatal(err)
            }

            // Create the message with a callback
            msg := evtwebsocket.Msg{
                Body: nil,
                Callback: func(resp []byte) {
                    fmt.Printf("Got back: %s\n", resp)
                },
            }

            log.Printf("%s\n", msg.Body)
        }()
    }
    select {}
}

Server Parameters Optimization

• TCP/IP parameter

Modify /etc/sysctl.conf, add following code:

net.ipv4.tcp_wmem = 4096 87380 4161536
net.ipv4.tcp_rmem = 4096 87380 4161536
net.ipv4.tcp_mem = 786432 2097152 3145728

• The file-max parameter

The file-max parameter sets the maximum number of file-handles that the Linux kernel will allocate.

Temporary settings by run:

$ sudo echo 1000000 > /proc/sys/fs/file-max

Persistent settings edit the /etc/sysctl.conf file and add the following line:

fs.file-max = 1000000

• Optimization ulimit parameter

Linux itself has a Max Processes per user limit. This feature allows us to control the number of processes an existing user on the server may be authorized to have. To improve performance, you can safely set the limit of processes for the super-user root to be unlimited.

A hard limit can only be raised by root (any process can lower it). So it is useful for security: a non-root process cannot overstep a hard limit. But it's inconvenient in that a non-root process can't have a lower limit than its children.

A soft limit can be changed by the process at any time. So it's convenient as long as processes cooperate, but no good for security.

Check current ulimit settings by command:

$ ulimit -a

Increase ulimit parameter edit the /etc/security/limits.conf file and add the following line:

*         hard    no file      1000000
*         soft    no file      1000000
root      hard    no file      1000000
root      soft    no file      1000000

Note that the hard limit can't greater than /proc/sys/fs/nr_open value, NR_OPEN is the maximum number of files that can be opened by process, so you maybe change nr_open value:

$ sudo echo 2000000 > /proc/sys/fs/nr_open

Client Parameters Optimization

By using following virtual IP address configuration, you can create (6535-1024)*4=258044 connections:

$ ifconfig eth0:0 172.17.0.2 netmask 255.255.255.0 up
$ ifconfig eth0:0 172.17.0.3 netmask 255.255.255.0 up
$ ifconfig eth0:0 172.17.0.4 netmask 255.255.255.0 up
$ ifconfig eth0:0 172.17.0.5 netmask 255.255.255.0 up

Open the /etc/sysctl.conf file and add the following line:

net.ipv4.ip_local_port_range = 1024 65535

Make configuration to take effect:

$ sudo /sbin/sysctl -p

• OutOfMemory Killer

If RAM of server not enough in some case might have a process "Killed" problems. You can close oom-killer to solve this problem with execult following command, but the better way is increase RAM.

$ sudo echo -17 > /proc/$(pidof java)/oom_adj

Websocket Server Environment

CPU     : Intel(R) Xeon(R) CPU E5-2620 @ 2.10GHz (6 Core, 12 Thread)
RAM     : 4GB DDR3 1333 MHz
HDD     : 64GB HDD
OS      : CentOS Linux release 7.2.1511 (Core)
Kernel  : Linux 3.10.0-327.22.2.el7.x86_6

Each benchmark program can initiate 5000 connection, deploy the benchmark program on the other two servers, simulation 10000 client. Use nmon as performance monitor and use Hey to send HTTP request to WebSocket server. Hey is a tiny program that sends some load to a web application. It's similar to Apache Bench (ab), but with better availability across different platforms and a less troubling installation experience. once the server received request, it will send a broadcast to all client.

$ sudo nmon16e_x86_rhel65 -f nmon -s 1 -c 180 -t
$ hey -n 1000 -c 10 http://<server_ip:server_port>/benchmark
Summary:
  Total:    31.9855 secs
  Slowest:  5.6196 secs
  Fastest:  0.0007 secs
  Average:  0.3034 secs
  Requests/sec: 31.2642

Status code distribution:
  [200] 1000 responses

Response time histogram:
  0.001 [1]     |
  0.563 [921]   |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.124 [47]    |∎∎
  1.686 [12]    |
  2.248 [4]     |
  2.810 [11]    |
  3.372 [1]     |
  3.934 [1]     |
  4.496 [1]     |
  5.058 [0]     |
  5.620 [1]     |

Latency distribution:
  10% in 0.1193 secs
  25% in 0.1606 secs
  50% in 0.2246 secs
  75% in 0.2715 secs
  90% in 0.4822 secs
  95% in 0.7415 secs
  99% in 2.5077 secs

You will be get a .nmon file when benchmark finished. Open that file with NMONVisualizer, it's a Java GUI tool for analyzing nmon system files from both AIX and Linux. It also parses IOStat files, IBM verbose GC logs, Windows Perfmon & ESXTop CSV data and JSON data. You can get some benchmark charts.

Reference Mozilla Developer Network - Writing WebSocket servers