Faster World Startups, Less Server Lag and Faster Rez Times

It’s been a while since we last updated our site. We spent a lot of time developing our backend and today we finally rolled out the following major upgrades to our service.

Opensim 0.7.2 Final

We were one of the first commercial grids to run OpenSim 0.7.2 RC1 and now we’re one of the first to run the release version of OpenSim 0.7.2.

New Assets System

Assets are pieces of content such as textures, sounds, animations, etc. Assets are by far the largest part of the data used in OpenSim, and therefore handling assets efficiently is important for making worlds run smoothly. We’ve created a new cloud-based assets system, which is faster and more scalable than the asset server that is part of OpenSim. We call it Cassy: the Comprehensive Assets System. Cassy enables Kitely to have faster world startup times, less server lag and provide faster viewer rezz times than grids based on standard OpenSim. Here are a few of the improvements in Cassy (there are many more).

OpenSim normally stores assets in a database. This database is managed by a single Assets Server, which needs to provide its service to all the active worlds. The database and the assets server are bottlenecks, slowing down all the worlds. To remove this bottleneck, we no longer store assets in the database: we store them in Amazon’s cloud storage system, S3. Unlike the database, which can only be accessed by the Assets Server, any instance of OpenSim can access S3. This allows us to distribute the work of the assets server among all of the sims: whenever any instance of OpenSim encounters a new asset, it uploads it directly to S3. This is faster and more scalable than sending the asset to the Assets Server, where it would have stored it in the database. Similarly, when OpenSim needs to load an asset, it downloads it directly from S3 instead of requesting the asset from the assets server.

Another improvement is how we handle duplicate assets. It’s quite common for several users to upload the same asset in a grid such as ours where many users get their content from third-party OpenSim sites. Normally this would cause two copies (or more) of the asset to be stored in the assets server. But Cassy detects such duplicates and stores only a single copy of the asset, improving storage and cache efficiency.

We also improved how viewers download textures. Textures are the largest and most common type of asset, and viewers spend a lot of time downloading them from OpenSim. Modern viewers, such as the Second Life Viewer versions 2 and 3, load textures using HTTP (older viewers used a different protocol, called UDP). OpenSim spends a lot of time sending the textures to the viewers. To lighten the load, we now serve textures from web servers running Apache. When a viewer requests a texture OpenSim redirects it to Apache, where it can download the texture. Apache is much more efficient than OpenSim at serving files so this makes the server run more smoothly, and viewers can display worlds faster.

Unfortunately not all viewers support HTTP redirection, so we only enable this feature for viewers that we’ve tested and we know can do it. This currently includes the Second Life Viewer 2.x and 3.x, and Firestorm 2.x. (Firestorm 3.x doesn’t work with Kitely at present, but they will fix this).

We also improved handling of scripts. We already save compiled scripts; this was explained in a previous blog post. Cassy now employs several additional techniques that help it load scripts significantly faster than standard OpenSim. This means that in most cases script-heavy worlds that would take minutes for all the scripts they include to start running in standard OpenSim will take seconds to start running in Kitely. Scripts still need to be compiled when a newly imported OAR file is first started, which may result in slower startup times, but after the scripts have been compiled worlds should now become fully functional much faster than they did previously.

Other Improvements

There was a problem where the terrain would appear striped. It was caused, curiously enough, by having worlds whose locations in the Kitely grid had very high coordinates. We moved all the worlds closer together and the problem disappeared.

We enabled physics for meshes so you should no longer encounter invisible boundaries around mesh objects.

People started uploading very big worlds into Kitely so we increased the upload size limit for OAR files from 250 MB to 1 GB. In addition, we made the system more robust to prevent such uploads from failing occasionally.

Copy World and Export World Respect Permissions

Backend Improvements

We rolled out a major system upgrade today, improving security and robustness amongst many other backend enhancements. We’re always making such improvements, but now they’re a particular priority as we’re getting closer to turning on billing.

Copy World and Export World now Respect Permissions

One user-visible change that we’ve made is an improvement to Copy World and Export World. To recap: Copy World lets you create multiple copies of a world in your Kitely account. Export World converts the world into an OAR file, which you can then download from Kitely and upload into other OpenSim grids.

Both of these features now respect object permissions: they will only copy or export objects (prims) that you are allowed to copy. Objects for which you have insufficient permissions will not be copied/exported. This change makes Kitely a better platform for content creators to sell their work, because it’s now much harder to create unauthorized copies of other people’s content.

These are the object permissions we check for when someone attempts to copy or export a world:

  • If they don’t own the object then the object must have the “Everyone Copy” permission in order for it to be copied into the new world.
  • If they own the object then it must have the “Owner Copy” permission in order for it to be copied. If the world is being Exported then the object must also have the “Owner Transfer” permission, because exported worlds can be given to other people.

When dealing with linked prims, we use the same logic as Second Life does: if any of the prims in the linkset have insufficient permissions then we don’t copy/export the entire linkset. We believe this behavior provides the strongest security for content creators. However, we did consider other alternatives, such as removing only the individual prims that lack permissions, and perhaps replacing them with placeholders (simple boxes). If you have thoughts about how this should work then please let us know in the comments.

This feature works on the level of objects, not individual assets. If someone is allowed to copy an object then they’re automatically allowed to copy all the assets used in that object as well. And conversely, if someone isn’t allowed to copy an object then that object’s assets will not be included in the copied/exported world.

For example, here’s a world with a few test objects. The screenshot doesn’t show the permissions of the objects, so I’ll describe them. The closer row (short cylinders) contains objects that are owned by the current user. The objects contain various combinations of Move, Copy and Transfer permissions. The farther row (tall cylinders) contains objects that aren’t owned by the current user. These objects all have the “Everyone Copy” permissions, except for the left-most cylinder.

Permissions Test - Original

Here are the results of a Copy World operation. A few of the objects have not been copied:

Permissions Test - Copy

Here are the results of an Export World operation. A few more objects have not been copied. Export World is more restrictive than Copy World, because we require objects to have the Transfer permission as well (in addition to the Copy permission).

Permissions Test - Exported

That’s it! If you have comments about how this feature works, or you find bugs, then please let us know. After we get some feedback, we will contribute this feature to OpenSim so more content creators will feel comfortable offering their wares to the OpenSim community.

Modified OpenSim to Improve World Startup Time

We made another improvement to OpenSim to make worlds start faster. When a world starts, OpenSim begins loading its assets. Some assets, e.g. textures or scripts, may be used in many prims. When that happens, OpenSim requests the asset multiple times from the assets service. In normal use this isn’t a problem because the asset is in the local cache so these requests are instantaneous. However, when the world starts the asset isn’t in the cache yet, so OpenSim downloads it multiple times. This is wasteful and slows down the world startup.

To fix this, we modified OpenSim so that additional requests for an asset are delayed until the first request has finished. This means the additional requests will be fulfilled from the local cache, which is very fast.

Faster Startup for Worlds with Many Assets

We updated our system today with various performance improvements, compatibility upgrades and requested features. The following updates may enhance your user experience:

Faster Startup for Worlds with Many Assets

We improved OpenSim to allow worlds with many assets to start faster. One of the things OpenSim does when a world starts is to update the world’s map image. Usually this is very fast (less than a second). However, it can take much longer for worlds that include many assets. We’ve seen delays of 20-100 seconds for complex worlds such as Xhyra’s Place, Chakryn Forest and Universal Campus.

We fixed this by moving the updating of the map image out of the main flow and into a separate thread. This means it no longer affects the time it takes the world to start. Thus, the startup time is reduced by up to 100 seconds!

LinkedIn Button

We added a button that allows you to share the Kitely homepage, or an individual world, with your LinkedIn contacts. This means that you can now easily spread the word about your Kitely worlds to four major social media sites: Facebook, Twitter, LinkedIn and Google+.

Improved Internet Explorer 9 Support

We now support Internet Explorer 9 natively. We did support IE9 previously, but this was done by setting it to use IE8 compatibility mode. Now we run in native IE9 mode.

Improved Support for Non-English Names

We updated Kitely today to address several recently discovered issues. While most of them were backend-related, we also fixed two user-visible problems:

Improved Support for Non-English Names

We automatically create avatars based on users’ names in Facebook. This is not always straightforward because Facebook allows names that contain non-English characters, but OpenSim does not. To bridge this gap, we convert such names into the closest possible English approximation.

This process was not as robust as it could have been, but now we’ve improved it significantly. Users who haven’t been able to login to virtual worlds previously, because their names contain non-English characters (including characters with accents), should find that they can now login successfully.

Improved “Enter World” Reliability

There was a problem that was encountered by a few users, where clicking “Enter World” didn’t succeed in logging in to the world at first. Clicking “Enter World” a second time did work. This problem occurred mainly with large and complicated worlds, and it has now been fixed. It should now be possible to log into all worlds on the first try.

Backend and OpenSim Improvements

We’ve updated Kitely today. Most of the updates are on the backend, e.g. to improve stability and create some of the infrastructure that we’ll need in order to implement billing. We’ve also added some user requested features and made some improvements to OpenSim:

Faster script loading

We’ve added script caching. When a region starts, OpenSim normally compiles all of the scripts used in the region. If the region has many scripts then this can take a long time. To make this process faster, we now save the compiled scripts and reuse them whenever the region is started. This can produce significant improvements! For example, one script-heavy world (containing about 900 scripts, 500 of which are unique) takes 10:42 minutes to compile the scripts initially. But from then on, each time the world starts it takes only 2:46 minutes to load the scripts. It should be noted that the world becomes usable even before the scripts are all ready. These timings only indicate when all of the scripts become ready, but people can enter and start using the world even before the first one is compiled.

Exported OAR files include creator information

When we export a world, we now include additional creator information. Previously, only the Creator UUID was saved. Now we also save the creator’s name. So if you export your world, and someone else imports it, you will still be listed as the creator of the objects in the world and the asset permissions you set will be retained.

Direct URL for downloading Kitely Plugin

We’ve added a permanent URL that can be used to download the latest version of Kitely Plugin: https://www.kitely.com/file?type=plugin. Most users don’t need to use this URL, because the World Page automatically downloads the plugin. However, in some cases it’s useful to have a direct URL, so we made it available.

Updated Signup Bonuses

During the beta period we give free Kitely Credits to new users. We used to give a different amount of KC to users depending on the number of Facebook friends they had: 50 KC if they had at least 20 Facebook friends, and just 10 KC if they didn’t. We had chosen these amounts in order to prevent people from abusing our system to get a lot of free Kitely Credits using throwaway Facebook accounts.

However, this policy prevented many honest people from trying out Kitely, and now that we’re topping off peoples’  accounts to 1000 KC each week it doesn’t even serve its intended purpose anymore.

Therefore we’ve decided to increase the number of free Kitely Credits that we give to new users to 200 KC regardless of the number of Facebook friends they have.

We’ve also updated the World Manager bonus to 400 KC, so that new users who create a world can get a total of 600 KC to play with without needing to contact us or wait for the weekly KC top-off.

Kitely Plugin 1.2.0 and Export World

We’ve made many updates and added two of the most commonly requested features: the ability to select which virtual world viewer to use, and exporting worlds to OAR files! Read on for the details.

Choose which virtual world viewer to use

We’ve updated Kitely Plugin to version 1.2.0. You can now choose which virtual world viewer will be launched when you click “Enter World” in a world page. Normally, the default viewer is launched; that is usually the last viewer that was installed. However, it is now possible to override this setting by entering the full path to the viewer in the file kitely-config.txt.

For details, see How can I choose which virtual world viewer to use? in our support site.

This feature is currently intended for advanced users, as they are the only ones likely to have more than one viewer installed. In the future we will make this feature accessible using a GUI instead of requiring you to edit a file manually.

Digitally signed plugin

The Windows version of Kitely Plugin is now digitally signed. This provides peace of mind when downloading the plugin. In addition, some anti-virus programs wouldn’t allow you to install the plugin because it didn’t have a digital signature (we’re looking at you, Norton AntiVirus). That should no longer be a problem.

Safari on Mac OS X

Kitely Plugin now supports the Safari browser on Mac OS X so you’ll be able to use your Mac’s default browser with Kitely.

Get notified when a newer version of the plugin is available

We’ve added notifications to the World Page so that you’ll know when a newer version of the plugin is available even if you missed our blog announcements and Tweets.

Export World

We’ve given you the ability to export worlds. Exporting a world creates an OAR file (OpenSim Archive) that contains the complete state of the world. You can download this file to your computer for backup.

To export a world, click on the “Export” link that appears next to its name in the My Worlds tab in your account. This can take a while (up to 15 minutes for large worlds) and you’ll be notified once the OAR file is ready.

Exporting and copying worlds can require a lot of server resources for a significant amount of time. We’ve therefore decided to set the price for both these actions to 10 KC.

Better World Descriptions

In Kitely, virtual worlds have a name and a description. The description used to be limited to simple text, but we’ve received numerous requests to allow richer descriptions. This is now possible! World descriptions can now include hyperlinks, images, separators and various text formatting options. You can easily take advantage of these options using the rich text editor in the New World and Edit World dialogs.

Today’s update also upgrades our system to the recently released OpenSim 0.7.1.1.