Monitor World Performance

We added a new tool for monitoring world performance. We also wrote a new guide for detecting and fixing world performance problems, and added an option to manually stop a running world.

Monitor World Performance

OpenSim provides world owners with the freedom to create large worlds with many prims and complex scripts, but this freedom can come at the cost of performance. As a world owner it’s important to make sure that your worlds perform well, because slow worlds cause a bad user experience: the world appears slowly around the user; animations are choppy; and moving in the world is frustrating because the avatar appears to jump from place to place instead of moving smoothly.

You can now check the performance of your world using the Active World panel, which appears in the world’s World Page while the world is active. This panel contains the following information:

  • Active Time – how long the world has been active in the current session.
  • Visitors – the current number of avatars in the world.
  • CPU (Number) – the current CPU usage of the world.
  • CPU (Graph) – a history of the CPU usage.
  • Packets (Graph) – the number of networking packets sent between the world and the viewers that are connected to it.

Active World

The graphs appear shortly after the world has started, and they update automatically every 30 seconds.

  • In both graphs, the horizontal axis shows the time (in the UTC time zone).
  • In both graphs, the left vertical axis shows the number of visitors in the world. This statistic is drawn using the yellow line.
  • In the CPU graph, the right vertical axis shows the CPU usage.
  • In the Packets graph, the right vertical axis shows the number of packets.

These graphs make it easy to see how the number of visitors in the world affects the CPU and bandwidth usage. It’s natural for worlds to use more CPU resources as the number of visitors increases. However, if the world is using a lot of CPU resources even when it has few visitors then the world probably needs to be optimized.

Stop world

You can use the Stop world link to force the world to stop. This is rarely needed, since worlds automatically stop a few minutes after the last visitor has left them. However, sometimes you might believe that the world is behaving badly and you want to restart it, and in that case stopping the world explicitly saves the normal delay that occurs between the time that the last visitor has left the world and when the world is stopped. This also enables you to stop a world even if there are avatars inside it.

Stopping a world is also useful for fixing the rare error of “ghosted avatars”. Avatar ghosting occurs when the simulator fails to detect that a user has closed their viewer. This leaves their avatar in the world, and prevents them from entering it again. Restarting the world removes the ghosted avatar.

After the world has stopped, enter the world again in order to restart it.

A Guide for Optimizing World Performance

We wrote a detailed new guide that describes all the tools available for checking the world’s performance, and how to fix various performance problems. See: Optimize World Performance. The guide contains the information that was included in this blog post, and much more.

If you see that your world is using a lot of CPU then you should consult this guide to find how to make the world perform better, which will improve the experience of your visitors.

New Tools for Fixing Slow Scripts

We updated our system today with several new features. First, we vastly improved the ability to find and fix slow scripts; this has a big impact on world performance. We’ll contribute this feature to the OpenSim project soon. Next, we continued to improve support for items that were obtained in other grids. Finally, we added a few features that improve usability.

How to Fix Slow Worlds

Don’t you hate it when you visit a slow world, where every step you take causes rubber-banding, and chatting is slow and jerky? When worlds are slow, that’s almost always caused by either slow physics or slow scripts. World Managers are responsible for making sure that their worlds are fast and provide a good experience to visitors. But how can you, the world manager, know if your world is fast or slow?

The first thing you need to check is the Statistics panel. You can open this panel by pressing Ctrl+Shift+1 in the viewer. The panel shows many statistics, but the two most important statistics are the Physics Time and Script Time. If they show high values then that tells you that you have a problem that needs to be fixed.

Statistics

This screenshot was taken in the Kitely Welcome Center, which is a highly optimized world: a Script Time under 1 ms is a good result. However, if you see a high Script Time or Physics Time then the next step is to find which objects are causing the slowdown. In the Region/Estate dialog you can open one of two dialogs: Top Colliders and Top Scripts. The Top Colliders dialog shows the objects that use the most physics time, and the Top Scripts dialog shows the scripts that take the most time. With this information you can decide what to do: optimize the objects, or disable them.

Debug Region

We’ve already improved the Top Colliders dialog significantly in the past, and today we made a similar improvement for Scripts.

Find and Fix Slow Scripts

Until today, the Script Time shown in the Statistics panel was always 0! That statistic had never been implemented in OpenSim. Today’s update, at last, implements the statistic. The “Script Time” value now shows the amount of time spent executing the world’s scripts in the last frame. This value should usually be very low: ideally under 1 ms. Some worlds might legitimately need more Script Time, if they have many scripts or many animations, but usually if the Script Time is higher than a few ms then you should try to optimize some of the slowest scripts in your world.

By the way, the Script Time shown in the Statistics panel is independent from all the other times shown in this panel. That’s because scripts execute on separate threads, outside of the frame processing time. This has two important implications. First, because scripts execute outside of the frame, it’s possible that the frame time will appear to be good, even though in fact the world is operating slowly due to heavy scripts activity. Therefore, if you have a high Script Time then you should optimize your scripts, regardless of what the other statistics show. Second, it’s possible for the Script Time to be very high, in some cases even hundreds of ms, i.e. longer than the frame time.

If you see that your world is using a lot of script time then the next step is to find which scripts are causing the problem. To do this, open the Top Scripts dialog. This dialog has been vastly improved in today’s update. Previously, the values it had shown were practically useless. But now, this dialog shows an accurate list of the slowest scripts in the world, and measures their execution time precisely. The value shown for each script is the total time spent executing that script in the last 30 seconds. (Note that 30 seconds is much longer than a single frame, and therefore the times shown here have little correlation with the Script Time shown in the statistics panel.)

Top Scripts

This screenshot was also taken in the Kitely Welcome Center. The top scripts here are pretty fast: the slowest script took just 5 ms, and remember that this is the total execution time over a 30-second period.

If your world contains slow scripts then they’ll appear at the top of the list in this dialog, and you’ll be able to see that they take much more time than the rest of the scripts. Alternatively, another cause of slow performance is if your scripts are efficient, but you have thousands of them. In that case you should consider why you need so many scripts, and whether you can reduce their number. (Note that this dialog won’t show thousands of scripts, but if you see a hundred identical scripts then you may assume that there are thousands more behind them…)

Once you’ve found the slow scripts, edit them to make them faster. How to do this is out of scope for this blog post; if you need help, try asking in our Scripting Forum. Whenever you save the script, the new version starts executing immediately. Click “Refresh” in the Top Scripts dialog to see if your changes made the script faster. Keep in mind that this dialog shows the total execution time in the last 30 seconds, so it will take some time for your changes to have an effect.

Once you’ve optimized enough of the slow scripts, check the Script Time in the Statistics panel to see if it has dropped significantly.

If you don’t want to take the time to edit the scripts, or you’re in the middle of an event and you need to improve the world’s performance quickly, then you can use the “Return Selected” or “Disable Selected” buttons in the dialog. “Return Selected” removes the selected objects from the world and adds them to your inventory. “Disable Selected” disables scripts and physics on the selected objects. After using either of these buttons, you can immediately check the Statistics panel to see if the Script Time has dropped. If you Disabled the objects then remember to re-enable their scripts (and possibly their physics, if they’re physical objects) after you have optimized the scripts.

We’re very proud of this feature. One area in which OpenSim has lagged behind Second Life is in ensuring a good (i.e., fast) experience for users. Every ex-Second Life user who switches to OpenSim enjoys the freedom to create large worlds with many prims and complex scripts, but this freedom can come at the cost of performance. At Kitely we’ve done a lot of work to make sure that our side of the service is fast, by using powerful servers and optimized asset and inventory systems. But slow scripts can bring any world to its knees, no matter how fast the hardware it’s running on. Today’s update gives world managers the information they need to ensure that their worlds are snappy and enjoyable.

Contributing the New Script Features to OpenSim

After we’ve had some time to test this new feature on Kitely and ensure its quality, we’ll contribute this patch to the OpenSim project. This should improve the performance of scripts everywhere in the Metaverse, to the benefit of the entire OpenSim community.

Other Improvements

We continued improving support for taking items from other grids and having them remain Exportable. This is a continuation of the work described in our last update. If you encountered problems after the previous update then please get the affected objects from the third-party grid again now.

When you view the list of members in a Group, the list now shows for each member whether they’re online, or otherwise the last time that they were online.

The OSSL function osKey2Name now returns the full name (including grid) for HG users. For example, local users return a name such as “First Last”, while Hypergrid users return a name such as “First.Last @grid.example.com:8002”. (Previously Hypergrid users also returned a name such as “First Last”, so they couldn’t be differentiated from local users.)