Kitely Blog

Lately we’ve been working on improving the experience of having many avatars in-world at the same time. This is particularly important for hosting large events. We updated the system today with many performance improvements related to this goal.

The following is a video of a load test we performed on a copy of the Kitely Welcome Center running on an Advanced World, using 80 bots controlled from a remote server. The video was taken after all the bots completed their login process. The CPU usage for this world remained at around 60% while the video was taken.

Each bot simulated all of the network activity that a real viewer performs, including asset downloads, terrain downloads, uploading the avatar’s own baked textures and downloading other avatars’ appearance, and all network activity from the time the client begins the login process until it is inworld. Once logged into the world each bot started walking around randomly in straight lines, circles and jumps. Bots also sent instant messages to local chat at a combined rate of one chat message every two seconds (you can see the chat button flashing each time such an IM was received).

Downloading Assets from CloudFront

The biggest change we made was to our assets system. OpenSim viewers such as Firestorm download most assets (specifically, textures and meshes) using a protocol called HTTP. Previously, these assets were downloaded from Kitely’s servers. But this could be slow if many users entered a world at once, and they all tried to download the world’s assets at the same time. In order to eliminate this bottleneck we’ve moved these assets to a service called CloudFront. CloudFront is Amazon’s CDN, or Content Delivery Network. It’s a worldwide collection of very fast servers whose only purpose is to serve HTTP files. CloudFront has practically unlimited bandwidth, and it has servers in many cities around the world (see this map), so it can serve assets very quickly to an unlimited number of users.

This improvement is going to make a big difference when many users enter a world at the same time. Individual visitors might not feel much of a difference, because viewers such as Firestorm have an artificial limit on how quickly they download assets. If you want, try increasing this limit and see if it makes worlds appear faster. This setting is called “Maximum bandwidth”, and it’s set to 1500 Kbps by default in Firestorm.

CloudFront is quite expensive, so in order to limit our costs we’re only using the CloudFront edge servers in the U.S., Canada and Europe. This accounts for the vast majority of Kitely’s users. Users in other countries are still going to get a performance improvement from our switch to CloudFront, but not as much as users in the aforementioned countries.

Optimized UDP Packets

Most communications between the viewer and OpenSim are done using UDP packets. In the past we’ve already done many rounds of optimization on UDP packets, and we’ve optimized them even more in this update to reduce network lag during periods of high network activity.

Eliminated a Source of Lag

Sometimes OpenSim needs to perform maintenance work on regions. Users in worlds that are currently undergoing such maintenance can experience server lag. In order to prevent this, we’ve created a mechanism a few years ago called Shadow Worlds. A Shadow World is a copy of a real world, but it doesn’t allow visitors: it’s only used to run maintenance work. For example, we’ve been using Shadow Worlds to update the World Maps. We’ve now eliminated maintenance-related server lag by moving additional types of maintenance work to the Shadow Worlds.

Centralized Logging and Metrics

This feature is not something users can see, but it helps us a great deal in finding performance problems and fixing them. All of Kitely’s servers now send very detailed metrics (performance measurements) and logs to a centralized location. This is useful in many ways. For example, if we get a report about a world that was behaving slowly then we can look back and see what activity was taking place in the world, right down to how many UDP packets of each type were being sent and received. This will help us further optimize our system and assist us in addressing any issues that are reported to us.

Other Improvements

Today’s update also includes a couple of features that were requested by our users.

We’ve fixed a bug related to searches in the World Map; see this forum post for discussion. The bug happened if the search used a full SLURL (vs just a world name), and there were multiple worlds with similar names.

We’ve also added three fonts that you can use in LSL scripts. Their names are: Noto Sans, Noto Serif, and Architects Daughter.

We are proud to announce that we’ve passed a new milestone: Kitely Market has sold over $250,000 worth of virtual goods to date.

Kitely Market first opened to the public on August 2013. Since then, Kitely Market has become the main marketplace serving the OpenSim metaverse (the Hypergrid).

Here are a few noteworthy stats:

• When Kitely Market first opened it had less than 1,000 product variations listed. Now it offers close to 34,000 product variations.
• Kitely Market’s sales revenue over the last year was over 3.6 times higher than its sales revenue in its first year.
• When Kitely Market opened, Kitely was still a closed grid and only Kitely users could buy from it. In March 2014 Kitely Market started delivering to the Hypergrid, and has since delivered to avatars on more than 420 different OpenSim grids.
• Buyers have viewed more than 1 million product pages to date, and 8.7% of product page views have resulted in a sale.
• Our top merchant has sold more than $20,000 worth of virtual goods.
• Our top 10 merchants have each sold at least $4,000 worth of virtual goods.
• Our top 100 merchants have each sold at least $300 worth of virtual goods.

If you aren’t familiar with Kitely Market then this video will bring you up to speed:

One of our main goals for Kitely Market was to enable content creators to easily make money selling their creations to users from multiple OpenSim grids, without having to manually deliver their goods to all of those grids. Kitely Market’s growth over the last few years demonstrates that talented content creators can see a nice return from selling to an open metaverse, even when it still includes less than 100,000 active users spread over thousands of different OpenSim grids.

Thank you for using Kitely Market! Your help in spreading the word about our marketplace is what enabled it to grow, and provide you with more content buying options. With your continued support, we can attract even more professional content creators to sell their goods to the OpenSim community.

Since the Coronavirus pandemic has started we’ve been getting a lot of interest from organizations that are looking for virtual-world alternatives to real-world education, training, meetings and events. Our Organizations offering was designed for just such purposes, and we think that it will be instructive to show how one large project has been using it.

The project is the Introduction to Multiculturalism and Cultural Diversity course. This 14-week course is now in its second year and has already been taken by more than 1200 students from 7 different colleges in 3 different languages (English, Hebrew and Arabic).

Participants in the course are divided into six-person groups. Each group contains students from different religions, genders and cultural backgrounds. Each group gets a private virtual world (inaccessible to the other groups) that contains the content and activities for the course. Each week additional areas of the world are opened to the students, and they are instructed how to complete that week’s assignments. The system tracks attendance and completion of assignments throughout the course, and the colleges can import this information into their analysis tools.

The course’s designers have performed statistical analysis after running this course for three semesters, and they have found that participants gained a significant long-term increase in empathy to people from different religions, cultural groups, sexual identities, and races. Participants also showed a long term decrease in bias and prejudice, which persisted in follow-up testing more than eight months after completing the course.

The Introduction to Multiculturalism and Cultural Diversity course was one of the finalists at the Reimagine Education Oscars (London 2019). This was their presentation:

Seven teaching colleges worked together to produce three online courses, in English, Arabic and Hebrew. The course “Introduction to Multiculturalism and Cultural diversity” was selected for production after competing with all universities and academic colleges in Israel. It is provided as a MOOC (massive open online course) and funded by Digital Israel, representing the Ministry for Social Equality, together with the “malag”- the council for higher education in Israel and the Ministry of Education. 

All the courses selected had to appear on the Campus site for all the open online approved courses in Israel. However, in order to allow the participants to not just learn about multiculturalism, but to experience it through working in multicultural groups on a virtual world and carryout simulations and roleplays, we needed an additional learning environment. This is how we connected with Kitely which provided us with an excellent professional service.

Using virtual worlds and personalized worlds can be a daunting experience for users new to this technology. However, Kitely made this process seamless for both windows and mac users. They developed a small application for us to get the participants directly into their own world. A system was developed to make multiple worlds, duplicated one for each small group and registration was also easy for the participants. Building the worlds was also aided by the massive depository of items available for purchase on the Kitely site.

We have had more than 1200 students taking our courses on Kitely worlds and we are very satisfied with Kitely’s service and professional approach, always willing to add to the infrastructure to make the course an excellent experience for the students. 

In our research we have been able to show that through using the virtual worlds our students have more empathy for the other. Using virtual worlds and experiential learning makes the learning experience so much more powerful. The course reached the finals in London 2019 for the Oscar in Education for the best project in the world to incorporate 21st century skills.

Dr. Elaine Hoter, Talpiot College of Education, Project Leader

Technical Challenges and Requirements

This project was one of the first to use Kitely’s Organizations feature. At the time that this project started we had already had years of experience providing virtual worlds to educators on top of our consumer offering, but addressing this project’s evolving requirements helped us define what our organization-focused offering needed to include.

The first challenge was handling registration. Registration to this course is done at different times by each college, and sometimes the participants or study groups need to change even after the users have already enrolled in the course.

Our system includes an administration website that can be used to manage users and groups, but it quickly became clear that adding users one at a time was not going to be practical. We therefore developed the Batch Create Users feature to enable creating user accounts and dividing them into the desired groups while avoiding various synchronization and double registration issues that can occur in this scenario. We later developed the Kitely API, which the project is now planning to use in order to automate enrollment from it own systems.

Next came user setup issues. The course participants use a wide variety of machines running Windows and MacOS, from high end computers running the latest software to very old machines that use outdated operating systems. Users were going to need to get the correct virtual world viewer for their machine, configure it to access the course’s system, select an initial avatar from culturally appropriate options, and then enter the virtual world that belongs to the group that the user was assigned to. There are potential problems that could be encountered in each of these steps. It was therefore clear that we needed to automate this process as much as possible, in order to reduce the amount of customer support that the course administrators would need to provide to their students.

To address this we developed the Setup Kitely application. It helps users download the appropriate viewer for their operating system, and configures the viewer to login each user to the Organization that user belongs to. This app also enables users to select their initial avatar from the options that the organization had chosen in the administration website.

The next challenge we had to deal with was that the virtual worlds used in the course weren’t static: they had to change from week to week, in order to reveal that week’s assignments. The course uses three variations for its virtual worlds, one in each language, and each of these variations was being used in multiple copies by dozens of study groups. It was going to be a challenge to update all of these worlds every week. Initially this was done by having the developer upload an OAR (OpenSim Archive) file into each of these world copies every week. That process was very time consuming and error prone.

To address this need we developed the Worlds Library, where the developer could store World Archives. These archives enable quick saving of world states, which can then be used to quickly create new worlds or replace the contents of existing ones. This operation can be done either manually, using the administration website, or automatically by using the Kitely API. This system is much faster and simpler than using OAR files.

Finally, managing the permissions of many users with respect to many different virtual worlds was very complicated. We addressed that complexity by designing our system to enable administrators to manage relationships between groups of users and groups of worlds. Administrators add their users to a hierarchical tree of user groups; they add their worlds to a hierarchical tree of world groups; and then they define the relationships between these user groups and world groups. Using this method, it is possible to easily define complex relationships between many types of users and many types of worlds. It’s even possible to define permissions for cross-organization interactions (although this feature wasn’t used for this course).

Using Kitely and its suite of management tools was an excellent choice for our project. We had everything we needed to create virtual worlds and archive, duplicate and replace them at will. Kitely’s support team was always there for us when we had questions or requests and even added the functions and features we requested. The system was robust enough for us to deploy our project to more than a thousand students in multiple languages and various countries. We were also able to manage users, track visits and user engagement with our project. All in all, I highly recommend using Kitely and will definitely be using it for upcoming projects.

Jeremy Finkelstein, Digital Jelly, Virtual World Developer

There were various other issues that our offering helped the project address. Contact us to learn more about them, and about how our Organizations offering can be used to address your own organization’s needs. Or, if you want to try this immediately, then go to our website and order an Organization.