Lately we’ve been working on improving the experience of having many avatars in-world at the same time. This is particularly important for hosting large events. We updated the system today with many performance improvements related to this goal.
The following is a video of a load test we performed on a copy of the Kitely Welcome Center running on an Advanced World, using 80 bots controlled from a remote server. The video was taken after all the bots completed their login process. The CPU usage for this world remained at around 60% while the video was taken.
Each bot simulated all of the network activity that a real viewer performs, including asset downloads, terrain downloads, uploading the avatar’s own baked textures and downloading other avatars’ appearance, and all network activity from the time the client begins the login process until it is inworld. Once logged into the world each bot started walking around randomly in straight lines, circles and jumps. Bots also sent instant messages to local chat at a combined rate of one chat message every two seconds (you can see the chat button flashing each time such an IM was received).
Downloading Assets from CloudFront
The biggest change we made was to our assets system. OpenSim viewers such as Firestorm download most assets (specifically, textures and meshes) using a protocol called HTTP. Previously, these assets were downloaded from Kitely’s servers. But this could be slow if many users entered a world at once, and they all tried to download the world’s assets at the same time. In order to eliminate this bottleneck we’ve moved these assets to a service called CloudFront. CloudFront is Amazon’s CDN, or Content Delivery Network. It’s a worldwide collection of very fast servers whose only purpose is to serve HTTP files. CloudFront has practically unlimited bandwidth, and it has servers in many cities around the world (see this map), so it can serve assets very quickly to an unlimited number of users.
This improvement is going to make a big difference when many users enter a world at the same time. Individual visitors might not feel much of a difference, because viewers such as Firestorm have an artificial limit on how quickly they download assets. If you want, try increasing this limit and see if it makes worlds appear faster. This setting is called “Maximum bandwidth”, and it’s set to 1500 Kbps by default in Firestorm.
CloudFront is quite expensive, so in order to limit our costs we’re only using the CloudFront edge servers in the U.S., Canada and Europe. This accounts for the vast majority of Kitely’s users. Users in other countries are still going to get a performance improvement from our switch to CloudFront, but not as much as users in the aforementioned countries.
Optimized UDP Packets
Most communications between the viewer and OpenSim are done using UDP packets. In the past we’ve already done many rounds of optimization on UDP packets, and we’ve optimized them even more in this update to reduce network lag during periods of high network activity.
Eliminated a Source of Lag
Sometimes OpenSim needs to perform maintenance work on regions. Users in worlds that are currently undergoing such maintenance can experience server lag. In order to prevent this, we’ve created a mechanism a few years ago called Shadow Worlds. A Shadow World is a copy of a real world, but it doesn’t allow visitors: it’s only used to run maintenance work. For example, we’ve been using Shadow Worlds to update the World Maps. We’ve now eliminated maintenance-related server lag by moving additional types of maintenance work to the Shadow Worlds.
Centralized Logging and Metrics
This feature is not something users can see, but it helps us a great deal in finding performance problems and fixing them. All of Kitely’s servers now send very detailed metrics (performance measurements) and logs to a centralized location. This is useful in many ways. For example, if we get a report about a world that was behaving slowly then we can look back and see what activity was taking place in the world, right down to how many UDP packets of each type were being sent and received. This will help us further optimize our system and assist us in addressing any issues that are reported to us.
Today’s update also includes a couple of features that were requested by our users.
We’ve fixed a bug related to searches in the World Map; see this forum post for discussion. The bug happened if the search used a full SLURL (vs just a world name), and there were multiple worlds with similar names.