Performance Improvements for Large Events

Lately we’ve been working on improving the experience of having many avatars in-world at the same time. This is particularly important for hosting large events. We updated the system today with many performance improvements related to this goal.

The following is a video of a load test we performed on a copy of the Kitely Welcome Center running on an Advanced World, using 80 bots controlled from a remote server. The video was taken after all the bots completed their login process. The CPU usage for this world remained at around 60% while the video was taken.

Each bot simulated all of the network activity that a real viewer performs, including asset downloads, terrain downloads, uploading the avatar’s own baked textures and downloading other avatars’ appearance, and all network activity from the time the client begins the login process until it is inworld. Once logged into the world each bot started walking around randomly in straight lines, circles and jumps. Bots also sent instant messages to local chat at a combined rate of one chat message every two seconds (you can see the chat button flashing each time such an IM was received).

Downloading Assets from CloudFront

The biggest change we made was to our assets system. OpenSim viewers such as Firestorm download most assets (specifically, textures and meshes) using a protocol called HTTP. Previously, these assets were downloaded from Kitely’s servers. But this could be slow if many users entered a world at once, and they all tried to download the world’s assets at the same time. In order to eliminate this bottleneck we’ve moved these assets to a service called CloudFront. CloudFront is Amazon’s CDN, or Content Delivery Network. It’s a worldwide collection of very fast servers whose only purpose is to serve HTTP files. CloudFront has practically unlimited bandwidth, and it has servers in many cities around the world (see this map), so it can serve assets very quickly to an unlimited number of users.

This improvement is going to make a big difference when many users enter a world at the same time. Individual visitors might not feel much of a difference, because viewers such as Firestorm have an artificial limit on how quickly they download assets. If you want, try increasing this limit and see if it makes worlds appear faster. This setting is called “Maximum bandwidth”, and it’s set to 1500 Kbps by default in Firestorm.

CloudFront is quite expensive, so in order to limit our costs we’re only using the CloudFront edge servers in the U.S., Canada and Europe. This accounts for the vast majority of Kitely’s users. Users in other countries are still going to get a performance improvement from our switch to CloudFront, but not as much as users in the aforementioned countries.

Optimized UDP Packets

Most communications between the viewer and OpenSim are done using UDP packets. In the past we’ve already done many rounds of optimization on UDP packets, and we’ve optimized them even more in this update to reduce network lag during periods of high network activity.

Eliminated a Source of Lag

Sometimes OpenSim needs to perform maintenance work on regions. Users in worlds that are currently undergoing such maintenance can experience server lag. In order to prevent this, we’ve created a mechanism a few years ago called Shadow Worlds. A Shadow World is a copy of a real world, but it doesn’t allow visitors: it’s only used to run maintenance work. For example, we’ve been using Shadow Worlds to update the World Maps. We’ve now eliminated maintenance-related server lag by moving additional types of maintenance work to the Shadow Worlds.

Centralized Logging and Metrics

This feature is not something users can see, but it helps us a great deal in finding performance problems and fixing them. All of Kitely’s servers now send very detailed metrics (performance measurements) and logs to a centralized location. This is useful in many ways. For example, if we get a report about a world that was behaving slowly then we can look back and see what activity was taking place in the world, right down to how many UDP packets of each type were being sent and received. This will help us further optimize our system and assist us in addressing any issues that are reported to us.

Other Improvements

Today’s update also includes a couple of features that were requested by our users.

We’ve fixed a bug related to searches in the World Map; see this forum post for discussion. The bug happened if the search used a full SLURL (vs just a world name), and there were multiple worlds with similar names.

We’ve also added three fonts that you can use in LSL scripts. Their names are: Noto Sans, Noto Serif, and Architects Daughter.

Kitely Market Passes $250,000 in Sales

We are proud to announce that we’ve passed a new milestone: Kitely Market has sold over $250,000 worth of virtual goods to date.

Kitely Market first opened to the public on August 2013. Since then, Kitely Market has become the main marketplace serving the OpenSim metaverse (the Hypergrid).

Here are a few noteworthy stats:

  • When Kitely Market first opened it had less than 1,000 product variations listed. Now it offers close to 34,000 product variations.
  • Kitely Market’s sales revenue over the last year was over 3.6 times higher than its sales revenue in its first year.
  • When Kitely Market opened, Kitely was still a closed grid and only Kitely users could buy from it. In March 2014 Kitely Market started delivering to the Hypergrid, and has since delivered to avatars on more than 420 different OpenSim grids.
  • Buyers have viewed more than 1 million product pages to date, and 8.7% of product page views have resulted in a sale.
  • Our top merchant has sold more than $20,000 worth of virtual goods.
  • Our top 10 merchants have each sold at least $4,000 worth of virtual goods.
  • Our top 100 merchants have each sold at least $300 worth of virtual goods.

If you aren’t familiar with Kitely Market then this video will bring you up to speed:

One of our main goals for Kitely Market was to enable content creators to easily make money selling their creations to users from multiple OpenSim grids, without having to manually deliver their goods to all of those grids. Kitely Market’s growth over the last few years demonstrates that talented content creators can see a nice return from selling to an open metaverse, even when it still includes less than 100,000 active users spread over thousands of different OpenSim grids.

Thank you for using Kitely Market! Your help in spreading the word about our marketplace is what enabled it to grow, and provide you with more content buying options. With your continued support, we can attract even more professional content creators to sell their goods to the OpenSim community.

Case Study: Multi-College Diversity Course in Kitely

Since the Coronavirus pandemic has started we’ve been getting a lot of interest from organizations that are looking for virtual-world alternatives to real-world education, training, meetings and events. Our Organizations offering was designed for just such purposes, and we think that it will be instructive to show how one large project has been using it.

The project is the Introduction to Multiculturalism and Cultural Diversity course. This 14-week course is now in its second year and has already been taken by more than 1200 students from 7 different colleges in 3 different languages (English, Hebrew and Arabic).

Participants in the course are divided into six-person groups. Each group contains students from different religions, genders and cultural backgrounds. Each group gets a private virtual world (inaccessible to the other groups) that contains the content and activities for the course. Each week additional areas of the world are opened to the students, and they are instructed how to complete that week’s assignments. The system tracks attendance and completion of assignments throughout the course, and the colleges can import this information into their analysis tools.

The course’s designers have performed statistical analysis after running this course for three semesters, and they have found that participants gained a significant long-term increase in empathy to people from different religions, cultural groups, sexual identities, and races. Participants also showed a long term decrease in bias and prejudice, which persisted in follow-up testing more than eight months after completing the course.

The Introduction to Multiculturalism and Cultural Diversity course was one of the finalists at the Reimagine Education Oscars (London 2019). This was their presentation:

Seven teaching colleges worked together to produce three online courses, in English, Arabic and Hebrew. The course “Introduction to Multiculturalism and Cultural diversity” was selected for production after competing with all universities and academic colleges in Israel. It is provided as a MOOC (massive open online course) and funded by Digital Israel, representing the Ministry for Social Equality, together with the “malag”- the council for higher education in Israel and the Ministry of Education. 

All the courses selected had to appear on the Campus site for all the open online approved courses in Israel. However, in order to allow the participants to not just learn about multiculturalism, but to experience it through working in multicultural groups on a virtual world and carryout simulations and roleplays, we needed an additional learning environment. This is how we connected with Kitely which provided us with an excellent professional service.

Using virtual worlds and personalized worlds can be a daunting experience for users new to this technology. However, Kitely made this process seamless for both windows and mac users. They developed a small application for us to get the participants directly into their own world. A system was developed to make multiple worlds, duplicated one for each small group and registration was also easy for the participants. Building the worlds was also aided by the massive depository of items available for purchase on the Kitely site.

We have had more than 1200 students taking our courses on Kitely worlds and we are very satisfied with Kitely’s service and professional approach, always willing to add to the infrastructure to make the course an excellent experience for the students. 

In our research we have been able to show that through using the virtual worlds our students have more empathy for the other. Using virtual worlds and experiential learning makes the learning experience so much more powerful. The course reached the finals in London 2019 for the Oscar in Education for the best project in the world to incorporate 21st century skills.

Dr. Elaine Hoter, Talpiot College of Education, Project Leader

Technical Challenges and Requirements

This project was one of the first to use Kitely’s Organizations feature. At the time that this project started we had already had years of experience providing virtual worlds to educators on top of our consumer offering, but addressing this project’s evolving requirements helped us define what our organization-focused offering needed to include.

The first challenge was handling registration. Registration to this course is done at different times by each college, and sometimes the participants or study groups need to change even after the users have already enrolled in the course.

Our system includes an administration website that can be used to manage users and groups, but it quickly became clear that adding users one at a time was not going to be practical. We therefore developed the Batch Create Users feature to enable creating user accounts and dividing them into the desired groups while avoiding various synchronization and double registration issues that can occur in this scenario. We later developed the Kitely API, which the project is now planning to use in order to automate enrollment from it own systems.

Next came user setup issues. The course participants use a wide variety of machines running Windows and MacOS, from high end computers running the latest software to very old machines that use outdated operating systems. Users were going to need to get the correct virtual world viewer for their machine, configure it to access the course’s system, select an initial avatar from culturally appropriate options, and then enter the virtual world that belongs to the group that the user was assigned to. There are potential problems that could be encountered in each of these steps. It was therefore clear that we needed to automate this process as much as possible, in order to reduce the amount of customer support that the course administrators would need to provide to their students.

To address this we developed the Setup Kitely application. It helps users download the appropriate viewer for their operating system, and configures the viewer to login each user to the Organization that user belongs to. This app also enables users to select their initial avatar from the options that the organization had chosen in the administration website.

The next challenge we had to deal with was that the virtual worlds used in the course weren’t static: they had to change from week to week, in order to reveal that week’s assignments. The course uses three variations for its virtual worlds, one in each language, and each of these variations was being used in multiple copies by dozens of study groups. It was going to be a challenge to update all of these worlds every week. Initially this was done by having the developer upload an OAR (OpenSim Archive) file into each of these world copies every week. That process was very time consuming and error prone.

To address this need we developed the Worlds Library, where the developer could store World Archives. These archives enable quick saving of world states, which can then be used to quickly create new worlds or replace the contents of existing ones. This operation can be done either manually, using the administration website, or automatically by using the Kitely API. This system is much faster and simpler than using OAR files.

Finally, managing the permissions of many users with respect to many different virtual worlds was very complicated. We addressed that complexity by designing our system to enable administrators to manage relationships between groups of users and groups of worlds. Administrators add their users to a hierarchical tree of user groups; they add their worlds to a hierarchical tree of world groups; and then they define the relationships between these user groups and world groups. Using this method, it is possible to easily define complex relationships between many types of users and many types of worlds. It’s even possible to define permissions for cross-organization interactions (although this feature wasn’t used for this course).

Using Kitely and its suite of management tools was an excellent choice for our project. We had everything we needed to create virtual worlds and archive, duplicate and replace them at will. Kitely’s support team was always there for us when we had questions or requests and even added the functions and features we requested. The system was robust enough for us to deploy our project to more than a thousand students in multiple languages and various countries. We were also able to manage users, track visits and user engagement with our project. All in all, I highly recommend using Kitely and will definitely be using it for upcoming projects.

Jeremy Finkelstein, Digital Jelly, Virtual World Developer

There were various other issues that our offering helped the project address. Contact us to learn more about them, and about how our Organizations offering can be used to address your own organization’s needs. Or, if you want to try this immediately, then go to our website and order an Organization.

Upgraded Database, and Improved Uploading Models

We’ve upgraded our system today. The biggest change is that we’ve upgraded our database in order to support big events with thousands of users.

We’ve also implemented a feature that was requested by our users: In OpenSim 0.8, when users uploaded Models the system would also create inventory items for the Textures used in the models. This behavior was disabled in OpenSim 0.9, but now we’ve re-enabled it.

Upgrading Kitely’s Database Infrastructure

We’ve been getting a lot of new interest from organizations recently, who want to use Kitely to replace some of their large in-person gatherings. So we decided to improve our ability to handle events with tens of thousands of concurrent users.

Kitely’s system was designed to scale, as exemplified by our ability to dynamically add servers when additional worlds are used. However, our system also has some centralized components that are harder to scale. One of these component is a MySQL database that we have been running on our own server in the Amazon cloud. The problem with this system is that the performance of the database is limited by the server that it’s running on.

In order to improve our scalability, we are going to migrate to an Amazon database system called Aurora Serverless. This database is capable of scaling up automatically when necessary, and scaling back down when the additional resources are no longer required. This behavior is ideal for supporting very large events.

This major upgrade to our capabilities requires that we take down the entire system for 6 hours on April 17 at 12:01am PDT (midnight).

Kitely Market Payouts in KC

We’ve updated our system with a couple of improvements that were requested by our users: Kitely Market merchants can now convert their cleared USD earnings to Kitely Credits (KC); and you can now more easily share your worlds to Facebook. We’ve also added a feature to Organizations that allows users from one organization to visit the private worlds of another organization.

Kitely Market Payouts in KC

Kitely Market merchants can choose to sell their products for USD. These USD aren’t transferred to you immediately: we accumulate them, and once you have at least $10 (USD) in cleared sales we schedule them for the next weekly payout (sales are cleared once they pass the PayPal 45-day dispute period). This minimum payout amount exists because of PayPal’s transfer fees, which make transferring smaller amounts uneconomical.

Some merchants sell mostly in KC, so it can take them a long time to earn 10 USD. Previously this money was “stuck”, waiting until the merchant earned enough USD for a payout to PayPal. But now we’ve added a new option: you can convert any cleared funds your store has earned to KC, even if you have less than $10 cleared. The KC will be converted at a rate of 200 KC per 1 USD and will be added to your Kitely account balance immediately. You can then use those KC to pay for world hosting, buy items from Kitely Market, etc.

If you’re a Kitely Market merchant in this situation then you’ll see a new option in your Manage Store page:

This merchant has two options:

  1. They currently have only $5.20 in cleared USD. If they wait, then eventually the rest of their money will clear. Then they’ll have a total of $18.80 in cleared USD, and we’ll automatically transfer it to them.
  2. Alternatively, they can get their earnings immediately by clicking on the link in the highlighted box. The $5.20 will be converted to KC and added to their Kitely account balance. The other $13.60 will remain withheld until enough time passes for it to clear the PayPal dispute period (at which point it will start accumulating for a payout to PayPal as usual).

Better Facebook Sharing

The Kitely website has buttons for sharing stuff to Facebook. You can share worlds, Kitely Market products, or Kitely itself (from the homepage). We’ve switched to a new style of sharing, which lets you enter your own thoughts about what you’re sharing. (Previously you could only “Like” something, without entering your own text.)

Shared Groups in Organizations

Organizations can control who may visit their worlds. They can choose to make some worlds available to everyone (including users that aren’t members of their organization), and make other worlds available only to specific groups in the organization.

But what if an organization wants to make a world open only to some external users? (“External users” are users that don’t belong to the organization.) To address this need we’ve created a new feature, called Shared User Groups, that lets organizations cooperate: users from one organization can visit specific private worlds the belong to another organization.

For example, suppose there’s an organization called “Acme University”, and another organization called “The Egyptology Center”. Acme University wants its students to visit Egypt-themed worlds that belong to The Egyptology Center. These worlds are not open for public visits, so Acme University has to ask The Egyptology Center to allow its user group Students to visit these worlds:

Once The Egyptology Center agrees, the user group “Students” becomes shared. This means that it appears in the groups tree of The Egyptology Center, and the admins of that organization can give the group permissions: e.g., permission to visit some of the organization’s private worlds.

This feature is explained in more detail here.

Enabling Kitely Market Sales Campaigns for Black Friday

Kitely Market supports sales campaigns that help merchants promote their products, and help buyers find great deals in our marketplace.

To ensure that buyers get truly discounted rates, we have a policy that a product can’t be always on sale: once a product has been put on sale, the merchant must wait at least 60 days before placing the same product on sale again.

However, this limitation is currently preventing merchants who had a Halloween sale from placing the same items on sale again for Black Friday. We’ve decided that these are both legitimate occasions for sales, and therefore we’re temporarily reducing the waiting period between sales to 14 days. This limit will be raised back to 60 days after Cyber Monday (December 2, 2019).

Please Help Kitely by Participating in the Hypergrid Business Annual Grid Survey

The website Hypergrid Business is now holding its tenth annual OpenSim grid survey, and we need your help to be able to attract additional people to use our grid and marketplace. If you’re willing to spend a few minutes to help spread the word about Kitely then please answer this short multiple choice survey.

Last year many of you participated in the survey and your endorsement helped convince more people to join our community and attract additional content creators to list their items in Kitely Market. Your continued support this year is crucial for our continued growth.

Please take the time to complete this survey. Thank you!

Kitely API Beta and Custom Viewer Login Page

We updated Kitely today with a couple of important improvements for Organizations (virtual grids). You can now set up a custom viewer login page for your organization. You can also apply to join our Kitely API private beta.

Kitely API Beta

We’ve created an API that enables Organizations to automate their operations. The API allows you to programmatically create and delete users and groups; change their permissions; and replace the contents of worlds.

Kitely API is currently in private beta, and is being used by select organizations. If you have an organization and would like to use Kitely API then please contact us.

Custom Viewer Login Page

The Viewer Login Page is what appears in Firestorm when you select a grid you wish to log into. Kitely Organizations are virtual grids, and as such they each have a Viewer Login Page.

Previously, each organization’s Viewer Login Page was automatically created by Kitely, and just displayed the organization’s name and some links. This is still the default Viewer Login Page, but it’s now possible for organizations to specify their own Viewer Login Page URL instead. This enables better branding and customization.

Setting a custom Viewer Login Page is only available for Standard Organizations and above.

Improved Searching for Users and Worlds

We’ve updated Kitely today with new features for searching for users and worlds. We also made improvements to the advanced permissions system in Organizations.

Improved Search on Our Website

The Explore Worlds page now lets you select which Organizations you want to search in:

The default option is “Kitely and All Organizations”. This option finds the most possible worlds, since it searches for worlds in three places:

  • Public worlds in Kitely
  • Public worlds in any Organization
  • Private worlds in Organizations in which you are a member

Another option is “Kitely”. This option searches only for worlds that belong to Kitely, and excludes worlds that belong to Organizations.

If you belong to any Organizations then they’ll also appear in this listbox. This lets you search only for worlds that belong to a particular Organization. In the screenshot above, the option “Durion School of Wizardry” appears because I’m a member of that Organization.

In the search results, worlds that belong to an Organization have the Organization’s name appended to the world’s name. For example, see the search result “Orcs of a Feather @ Durion School of Wizardry”.

Find New Worlds

There are now over 15,000 Kitely worlds, and that number continues to grow. In order to help you discover new worlds, we’ve added the ability to choose the sort order in the Explore Worlds page. You can choose between these options:

  • Visitors – sorts worlds based on how many people are currently visiting them (the most crowded worlds appear first).
  • Newest – sort worlds based on when they were created (the most recently created worlds appear first).

Improved Search in the Viewer

As with the “Explore Worlds” page on our website, the search window in the OpenSim Viewer (Firestorm) now also lets you search by Organization, and enables you to sort the search results.

Another improvement is that the viewer search window now displays a list of worlds when you open the Search window (before you’ve even typed in a search query). This is the same behavior as in the Explore Worlds page, and it enables you to easily find new and active words.

We’ve also improved how you can find other users from inside the viewer. You can now search for users in a particular Organization, just as you can when searching for worlds. (However, unlike with worlds, you won’t see a list of users in the viewer search window unless you enter a specific query. We also don’t include Sort options for users, since they can create privacy issues.)

The following screenshot shows some search results, including a user that belongs to a different Organization. That user’s Organization name is displayed next to the user’s name.

As with World searches, you’ll only be able to see private Organization users if you belong to that Organization. But you’ll always be able to see any public users that belong to the Organization. The next section explains how Organization Admins can choose whether an Organization user can be found in searches.

Search Permissions for Managed Users

By default, Managed Users can’t search for external users or worlds. (“External” users and worlds are users/worlds that don’t belong to the Organization). Also, by default, Managed Users can’t be found in searches performed by external users. If you are an Organization Admin then you can change these behaviors by giving additional permissions to some of the Organization’s users.

For example, users that belong to the following User Group (“Teachers”) can find and be found by external users:

The permission to search for external users is explicit: you enable or disable it by clicking on a checkbox. In contrast, the permission to search for external worlds is implicit: it’s automatically enabled for users that are allowed to visit Kitely worlds. There’s no explicit checkbox for this permission, because it doesn’t make sense to allow someone to search for a world but not to visit it, or vice-versa.

As always, you can use the User Permissions tab to see which permissions a user has. For example:

Other Improvements

We’ve improved the process of adding custom grids to Kitely Market. This process is used when you want to deliver products from Kitely Market to a different grid. You’ll now get better validation that the Grid URI that you’ve entered is valid and leads to an active grid, so that if there’s a problem with the grid information that you’ve entered then you’ll be notified about it faster and more clearly.

We’ve improved the performance of the llListen LSL function.