Page 3 of 4

Re: Kitely will be down for 1 hour on October 22

Posted: Fri Oct 23, 2020 5:52 pm
by John Mela
Thanks Ilan, I was able to enter the John's Lab world with avatar John Mela using this method.

Re: Kitely will be down for 1 hour on October 22

Posted: Fri Oct 23, 2020 10:21 pm
by Ilan Tochner
Update: we rolled back the Linux version that our servers use and so far it looks like that has solved the problem.

We are really sorry that this happened. We were unaware Amazon's minor Linux version change would cause issues and it took us time to track that down as what caused teleports to fail in some cases.

Please let us know if you encounter this teleport problem again. If there is something else that can create it then we want to track it down and fix it as well.

Re: Kitely will be down for 1 hour on October 22

Posted: Fri Oct 23, 2020 11:40 pm
by Kayaker Magic
Ilan, you are using Amazon's home grown Linux? I use Ubuntu 16.04 and 18.04 on several servers for regions on OSGrid and I wonder if I should hold off on updates for a bit. Also I'm using mono 6.12.0.90, did your linux rollback include rolling back to a different version of mono? Which one are you using?

Re: Kitely will be down for 1 hour on October 22

Posted: Sat Oct 24, 2020 12:09 am
by Ilan Tochner
Hi Kayaker,

Yes, we're basing our servers on the Amazon Linux AMI. If I recall correctly we're currently using Mono 6.10, which is what we were using before the system update as well.

Re: Kitely will be down for 1 hour on October 22

Posted: Sat Oct 24, 2020 6:02 am
by Zyzzyx Qinan
I just tried repeatedly to log into my sim, I did the whole procedure you outlined above several times, then tried with my alt as well, i had no success with either. My partner Katya can't get in either. All of us have cleared caches and can get to the welcome center, we followed your instructions to use the website and to also use the "last location". I imagine you are still working on a fix, i'm going to bed :)

Re: Kitely will be down for 1 hour on October 22

Posted: Sat Oct 24, 2020 11:28 am
by Dot Matrix
Try the following:

1 Make sure the world closes down completely via the world page on the website.
2 Leave it a few minutes, then try logging into it again.

This will enable the world to be in a fresh instance of the server software running the rolled-back version of Linux (as Ilan said: "We're currently suspecting a Linux kernel update to be the cause of this new issue and are working on rolling back to using an earlier version. We'll update this thread with a progress report once we have additional information.")

This worked for me a few hours ago, but the world does need to be completely closed down to make sure the rolled-back version is used again.

Ilan -- do correct anything I've got wrong here. Not so hot on all the techie stuff!

Re: Kitely will be down for 1 hour on October 22

Posted: Sat Oct 24, 2020 1:54 pm
by senga lesley
Hi, once again i cant get into My world, New Eden. i did this morning, (Saturday) no problem at all, but now, (Saturday afternoon 3pm GMT) i get the usual error message. I start logging in and it fails when i get to regional handshake.
I tried the suggested method, SEVERAL TIMES, using "last position" and trying when the website says Start your viewer, but no luck.
Dot says:make sure the world closes down completely; how do i find out if it closed completely; i asSume it is closed completely when i see ENTER WORLD or Start your viewer

Re: Kitely will be down for 1 hour on October 22

Posted: Sat Oct 24, 2020 2:10 pm
by Ilan Tochner
Hi Senga,

Our investigations are showing that this is a result of Xengine taking up too many system resources which prevents some time-sensitive operations like teleports from completing. As we haven't changed any Xengine code in this last update, we're thinking the cause is the underlying operating system we're using which has been updated in our last system update a couple of days ago.

When we rolled out the system update it was built on the latest Linux version and it included some automatic updates to Mono and a few other system components as well. We've already rolled back the Linux version but apparently this isn't enough and we're now working on rolling back the Mono version that was updated to the latest Mono 6.12 release.

We're already testing this on our QA environment and will roll out the new (rolled back) server software in the next few minutes. After that's done currently running worlds will need to shut down before they can be switched to the updated server software.

Re: Kitely will be down for 1 hour on October 22

Posted: Sat Oct 24, 2020 2:20 pm
by Chris Namaste
:)
thx ilan for info & efforts,
thx everyone,to share experiences & stories,

atm, saturday 24 oct 7.20am in kitely, just to help (if it helps) i ve done several attempts-efforts-etc
nothing else-nowhere else is possible than kitely welcome & kitely sandbox,
hope i thelps
thx , christa

Re: Kitely will be down for 1 hour on October 22

Posted: Sat Oct 24, 2020 2:31 pm
by Zyzzyx Qinan
I was just able to successfully log into my world this morning. Thank you for all the info and ongoing work.