Ask HN: Why does my Node.js multiplayer game lag at 500 players with low CPU?

So when running `top` WCHAN shows `ep_poll` most of the time and sometimes `-`. Even when the game starts lagging this pattern stays pretty consistent.

There is no cross-room communication. I could spawn a process per room but I was trying to address this issue with my current Docker setup where I have multiple `game` containers that run a single node.js process and each process can host multiple rooms.

Not having to use Docker sounds simpler but it's that's where I'm at atm haha.

I agree that the network load feels very small. Maybe it's a socket.io related issue where when many broadcasts are being fired at once, then a shared I/O step gets bottlenecked?

Here's my actual typing broadcast code, I was originally broadcasting from the socket event callback itself but I found performance improved slightly by batching broadcasts per player in a setInterval loop (also note that only 1 player in a given room can be typing at once, so batching broadcasts per room shouldn't address the bottleneck).

  /**
   * Used to handle very frequent typing events more gracefully to avoid overloading CPU
   */
  const TypingUsersMap = new Map<
    ConnectionId,
    {
      socketId: string | null; // doesn't exist for bots
      roomId: PublicRoomId;
      userId: UserId;
      currentInput: string;
    }
  >();

  type ConnectionId = `${UserId}:${PublicRoomId}`;

  // ! this should be same as client throttle interval
  const TYPING_BROADCAST_INTERVAL = 200;

  export let typingBroadcastInterval: NodeJS.Timeout | undefined = undefined;
  export const startTypingBroadcastJob = () => {
    typingBroadcastInterval = setInterval(() => {
      const freshTypingUsersMap = new Map(TypingUsersMap);
      TypingUsersMap.clear();

      if (freshTypingUsersMap.size === 0) return; // Nothing to do

      // Go through each user that has a pending update
      for (const [_connectionId, data] of freshTypingUsersMap.entries()) {
        const socket = data.socketId
          ? io.sockets.sockets.get(data.socketId)
          : undefined;

        // Use the data we stored to perform the broadcast
        if (socket) {
          // emit to other players
          socket
            .to(data.roomId)
            .volatile.emit(
              SOCKET_EVENT_NAMES.USER_TYPING_RES,
              data.userId,
              data.currentInput
            );
        } else {
          // bots emit to everyone
          io.to(data.roomId).volatile.emit(
            SOCKET_EVENT_NAMES.USER_TYPING_RES,
            data.userId,
            data.currentInput
          );
        }
      }
    }, TYPING_BROADCAST_INTERVAL);
  };

  export const stopTypingBroadcastJob = () => {
    if (typingBroadcastInterval) {
      clearInterval(typingBroadcastInterval);
      typingBroadcastInterval = undefined;
    }
  };

  // this is called from the USER_TYPING socket event callback. so effectively every throttled keystroke by the user gets queued.
  export const queueTypingEvent = ({
    socketId,
    roomId,
    userId,
    currentInput,
  }: {
    socketId: string | null;
    roomId: PublicRoomId;
    userId: UserId;
    currentInput: string;
  }) => {
    const connectionId: ConnectionId = `${userId}:${roomId}`;
    TypingUsersMap.set(connectionId, {
      socketId,
      roomId,
      userId,
      currentInput,
    });
  };

Yeah changing the sysctl options was a shot in the dark... I really hope it's my app code. But the fact that the same bottleneck occurs even when I add more containers which decreases the load per container confuses me. I mentioned this in another comment but I wonder if socket.io broadcast calls share the same I/O resource or something. Maybe a lock?

I'm somewhat buffering right now - Everytime the current turn player types I buffer their input on the backend, and I have a job setup that broadcasts typing events every ~200ms using this buffer.

I could increase this interval, but I'd like to keep it as short as I can afford to to keep that realtime feel (i.e. other players can see what the current turn player is typing).

I thought this might've been the issue too, but because the game is turn-based there should only ever be 1 person typing at once (in a given room).