(Edit) Kitely is now back from maintenance

Provide feedback, criticism or praise about Kitely, or suggest new features
Post Reply
User avatar
Ilan Tochner
Posts: 6503
Joined: Sun Dec 23, 2012 8:44 am
Has thanked: 4942 times
Been thanked: 4454 times
Contact:

(Edit) Kitely is now back from maintenance

Post by Ilan Tochner »

Due to very heavy load on our database in the last 24 hours we decided to upgrade our database infrastructure.

This operation requires taking down our system in order to migrate the data to the new infrastructure. Our site will be in maintenance mode until we complete this operation.

We'll take down this forums post once our system is fully operational again.
User avatar
Ilan Tochner
Posts: 6503
Joined: Sun Dec 23, 2012 8:44 am
Has thanked: 4942 times
Been thanked: 4454 times
Contact:

Re: (Edit) Kitely is now back from maintenance

Post by Ilan Tochner »

We brought our system back up and you should be able to resume using Kitely now. We were down for 2:45 hours.

We found the source of the problem, it wasn't hardware related. It was due to the service we use to generate the performance graphs we display when worlds are active. The growing number of active worlds caused this system to consume so much IO that it slowed down the storage system the database runs on which then slowed down our entire system. We've now disabled performance graph generation until we have faster storage and/or optimize this system.
These users thanked the author Ilan Tochner for the post (total 2):
Zuza RittConstance Peregrine
User avatar
Mike Lorrey
Posts: 361
Joined: Sun Sep 04, 2016 5:40 pm
Has thanked: 71 times
Been thanked: 269 times

Re: (Edit) Kitely is now back from maintenance

Post by Mike Lorrey »

Thanks Ilan, are there options to pipe that data to an external chart generation service? Sounds to me like that needs to be a separate performance data historical archive.
User avatar
Mike Lorrey
Posts: 361
Joined: Sun Sep 04, 2016 5:40 pm
Has thanked: 71 times
Been thanked: 269 times

Re: (Edit) Kitely is now back from maintenance

Post by Mike Lorrey »

I am still getting teleport/login conflict bugs.
I first tried to log in about 11:10AM SLT. Firestorm crashed after login. I then relogged at around 11:14, and got a message that I could not log in because I was still teleporting to Naboo. I closed the client and tried again. This time I got into Naboo, then tried to teleport to Panthalassa for an event, and got the following notice:
Attachments
teleport-bug.jpg
Agatha Ebeler
Posts: 19
Joined: Sun Oct 02, 2016 1:20 pm
Has thanked: 8 times
Been thanked: 13 times

Re: (Edit) Kitely is now back from maintenance

Post by Agatha Ebeler »

I have the same Problems and now my firestorm Viewer doesnt work anymore :(
User avatar
Ilan Tochner
Posts: 6503
Joined: Sun Dec 23, 2012 8:44 am
Has thanked: 4942 times
Been thanked: 4454 times
Contact:

Re: (Edit) Kitely is now back from maintenance

Post by Ilan Tochner »

In order to bring our system back online quickly, we restarted it before finishing the database storage system upgrade. This means that it is still susceptible to high usage spikes which can cause transient problems such as the one you described.

We've set an automatic 10 minute timeout for teleport attempts so that our system will automatically stop the teleport process if you fail to enter a world within 10 minutes of trying to enter it. Once those ten minutes pass you should be able to retry entering worlds again.

Please try doing so now.

Please be advised that our system will remain under heavy load until we complete the upgrade. This may take a couple of days as we're trying to do the migration while keeping the system active in order to minimize the time we'll need to take it down to complete the upgrade.
Agatha Ebeler
Posts: 19
Joined: Sun Oct 02, 2016 1:20 pm
Has thanked: 8 times
Been thanked: 13 times

Same login problem again ...

Post by Agatha Ebeler »

Today again it is not possible to log in to my world, there is the same fault again, mike already made a pic ...


Now it worked ... would like to know why ....
Attachments
Screenshot_1.png
User avatar
Ilan Tochner
Posts: 6503
Joined: Sun Dec 23, 2012 8:44 am
Has thanked: 4942 times
Been thanked: 4454 times
Contact:

Re: (Edit) Kitely is now back from maintenance

Post by Ilan Tochner »

Hi Agatha,

If there are communication problems between your viewer and the world simulator you're trying to log into then your avatar may enter a "ghost" state on that world until that world is restarted. This can, in turn, cause your account to enter a state which prevents logging into other worlds until your avatar is no longer ghosted.

In order to minimize the chance of this happening, I recommend that when you log into a grid you do so into a region that has the lowest chance to result in communication issues. In Kitely that means logging into the Kitely Welcome Center that is both always on, has a lot of server resources allocated to it, and is rest periodically to reduce the effects of various issues that OpenSim (and Second Life) simulators start having after they've been active for a while. Once you complete the login process teleport from the Kitely Welcome Center to your desired destination.

Alternatively, when you encounter such issues click the "disconnect from world" link that appears under the Enter World button of the world you were in the process of logging into when the process first failed.

Lastly, if you do nothing, our system will eventually detect you are not actually logged in and will automatically disconnect your account, thus enabling you to log in again.
These users thanked the author Ilan Tochner for the post:
Selby Evans
Agatha Ebeler
Posts: 19
Joined: Sun Oct 02, 2016 1:20 pm
Has thanked: 8 times
Been thanked: 13 times

Re: (Edit) Kitely is now back from maintenance

Post by Agatha Ebeler »

Thank you, Ilan :)
Post Reply