Has something changed? Event timeouts

Ask questions about creating worlds, using worlds, etc.
Post Reply
User avatar
Dot Matrix
Posts: 1625
Joined: Sun Jul 28, 2013 3:26 am
Has thanked: 1208 times
Been thanked: 2324 times

Has something changed? Event timeouts

Post by Dot Matrix »

Has something changed recently in the settings for Kitely?

Scripts are raising event timeouts much more readily than they used to. It seems something has changed that causes the elapsed time of events to increase, breaking the threshold. This means that when logging into some of our worlds, we get hit by loads of script errors (this was from Intersections yesterday, but friends reported a similar happening when they logged in today):

Code: Select all

[01:41] bookmark: Restarting script due to event timeout (Script: "Bookmark v1.0.1", State: default, Prim: "bookmark"/707233ab-2f1a-4757-8d24-287ab786e3c4 at <131.097, 154.7367, 25.6585>)
[01:41] Text display TL: Restarting script due to event timeout (Script: "Text display v1.3.1", State: default, Prim: "Text display TL"/576c887a-2a4e-42ec-873e-14100cedeadb at <136.791, 147.3007, 25.4595>)
[01:41] bookmark: Restarting script due to event timeout (Script: "Bookmark v1.0.1", State: default, Prim: "bookmark"/ad12ca2e-7fc3-4f31-aa87-224c5b17e3ea at <131.464, 154.7397, 25.6585>)
Could there be an increase in the available time for events? That would enable functions like osSetDynamicTextureDataBlendFace to complete within the timeframe allowed even when the server is somewhat loaded.

By the way, I've removed the objects in question temporarily while this is sorted out.
These users thanked the author Dot Matrix for the post:
John Mela
User avatar
Snoots Dwagon
Posts: 422
Joined: Mon Jul 30, 2018 9:45 pm
Has thanked: 442 times
Been thanked: 779 times

Re: Has something changed? Event timeouts

Post by Snoots Dwagon »

I've noticed this myself Dot. Event timeouts can be set for a region /grid (I figure you already know that). I had to totally re-code a significant portion of a script recently because it remained in the timer() event longer than 30 seconds. Without knowing how the system is set, it's my guess that the event timeout figure is set way too low. I don't know why Events have a timeout at all (I'm sure there's a decent reason, maybe)... but it does foul up scripts that enter an event and stay there for a bit.

The real problem is this is what you pointed out above:
Restarting script due to event timeout

It doesn't just exit the event and the script keep running. It re-starts the script, which in some cases effectively breaks its function (such as in a visitor list system for example). My preference would be this figure be extended to 30 minutes, but I don't know all the tech ins and outs of the deal. :(
These users thanked the author Snoots Dwagon for the post (total 2):
Dot MatrixJohn Mela
~~~~~~~
I'm a dwagon in real life too. (Ask my sister, who totally agrees.)

~~~~~~~
User avatar
Oren Hurvitz
Posts: 361
Joined: Sun Dec 23, 2012 8:42 am
Has thanked: 19 times
Been thanked: 499 times
Contact:

Re: Has something changed? Event timeouts

Post by Oren Hurvitz »

The event timeout in Kitely is 30 seconds, which is the OpenSim default. We haven't changed it.

There are several reason that events need a timeout.

The main reason is that the scripting engine (XEngine) has a limited number of threads. While an event is being executed one of these threads is tied up, until the event finishes. If many scripts are executing slow events then there won't be many threads left free, so all of the other scripts in the world would become slower. In extreme cases they might even stop working completely.

A secondary reason is that the script itself (the one that has a slow event) can't do anything until the event finishes, even if another event happens meanwhile. E.g., suppose the Timer event takes 30 seconds to execute, and meanwhile an avatar touches the object that contains the script: nothing will happen until the Timer event finishes. By then, the user will probably give up and walk away.

I've looked at a few of the event timeouts mentioned in this thread, and it seems to me like this problem happens mostly during world startup. Is this correct? That could be because all of the scripts are starting at once so they're fighting each other over the threads in XEngine. If so, then perhaps a solution could be to increase the Event Timeout during world startup (but reduce it back to 30 seconds afterwards).
These users thanked the author Oren Hurvitz for the post (total 3):
Snoots DwagonJohn MelaDot Matrix
User avatar
John Mela
Posts: 91
Joined: Tue Feb 04, 2014 9:50 pm
Has thanked: 139 times
Been thanked: 127 times
Contact:

Re: Has something changed? Event timeouts

Post by John Mela »

Thanks Oren. The issue is encountered during world startup as well as during normal operations. If the event timeout could be disabled during startup, that would go some way towards alleviating this issue.

Something does seem to have changed - tests that once I would have once expected to succeed without issue are now raising event timeouts. Is it possible that there has been a performance loss on the AWS servers?

As you may recall, we at RezMela have battled before with both event timeouts and event queue size issues, and made substantial changes to the way that large amounts of data (typically, loading a complex scene in a RezMela App containing dozens of script objects) are processed. In order to work round these issues, I have previously introduced queuing methods into the main code which divide the rezzing and triggering of these objects into batches. In order to work round the recent issues, I plan to extend this and allow further throttling of this process. All this, of course, adds more complexity to the code (and ironically risks worsening the performance hit). If event timeouts could be disabled during region start, this would help to reduce the additional complexity.

I've grumbled before about the unfairness of OpenSim terminating events on the basis of elapsed time (and it's worth emphasising that this is part of the vanilla code, not something that Kitely has done). In effect, this means that script A can put heavy load on the script engine and cause script B to fail, even if A and B are in different objects and created by different people. A much more just method would be to limit the number of function calls in an event (or CPU cycles if that's feasible), so that if a script works on a lightly-loaded system it won't fail when there is heavy load.

Out of interest, the investigation I've been doing into this has shown that the primary culprit is osSetDynamicTextureDataBlendFace(). Under load in my tests, this function is taking around 30 seconds to execute, even with quite a simple command list.
These users thanked the author John Mela for the post (total 2):
Snoots DwagonDot Matrix
User avatar
Snoots Dwagon
Posts: 422
Joined: Mon Jul 30, 2018 9:45 pm
Has thanked: 442 times
Been thanked: 779 times

Re: Has something changed? Event timeouts

Post by Snoots Dwagon »

John Mela wrote:I've grumbled before about the unfairness of OpenSim terminating events on the basis of elapsed time (and it's worth emphasising that this is part of the vanilla code, not something that Kitely has done). In effect, this means that script A can put heavy load on the script engine and cause script B to fail, even if A and B are in different objects and created by different people. A much more just method would be to limit the number of function calls in an event (or CPU cycles if that's feasible), so that if a script works on a lightly-loaded system it won't fail when there is heavy load.
I agree John. I know this is a function of OpenSim because I've seen the specific setting line in the .ini file (and altered it on my VARs). Interestingly though, like you I've never experienced an event timeout in any of my code before Kitely (and I've been scripting for about 15 years on various grids). I'm not pointing fingers; merely suggesting the setting / issues be reconsidered. To be honest, before this I was never aware of the downside of an event running a lengthy period of time, however I believe your solution to be a better one (though possibly difficult to implement). I heard such has been implemented to an extent, but not sufficient to this need, obviously.

One of the reasons given for a 30 second limit is that "if one event is running another can't be triggered" and the example given of being in a timer event and a touch event not working (fairly similar to an extensive loop). In my opinion that's an issue for the coder to deal with and is irrelevant to the system. Maybe the coder wants the item to not be touchable while that timer event is in operation. Beginning coders learn by experience and mistakes; experienced coders know to avoid excessive loops to begin with. That's more a scripting issue, less a system-level one. I know that "in extreme situations" one bad script can affect an entire world... but that will always be the case, no matter what. One test I've run in the past which has proved valuable is to rez 50 of an item and see what impact it has on the world. If 50 doesn't impact, the script is likely trustworthy. One can also check Top Scripts for badly-behaving scripts.

It's up to coders to make scripts work as desired, and up to world owners to implement scripts wisely. But to have a script cease to function because something might be coded badly... seems to be counter-productive.

(For example, what is it about a script writing an internal notecard that causes this to be a "security risk" function in OSSL? Would someone involved in that decision please explain to me their reasoning? Perish forbid a script would write a "deadly notecard".) :mrgreen:

There's only so many "safeguards" that can be imposed on a system... and some safeguards are more brick walls than helpful. I believe this event timeout concept is one of those brick walls. But I leave that to those that work with deep system code on a daily basis. I'm a user and don't know all the pros and cons of system functions.
These users thanked the author Snoots Dwagon for the post:
John Mela
~~~~~~~
I'm a dwagon in real life too. (Ask my sister, who totally agrees.)

~~~~~~~
User avatar
Oren Hurvitz
Posts: 361
Joined: Sun Dec 23, 2012 8:42 am
Has thanked: 19 times
Been thanked: 499 times
Contact:

Re: Has something changed? Event timeouts

Post by Oren Hurvitz »

John Mela wrote:
Tue Dec 08, 2020 3:33 pm
Thanks Oren. The issue is encountered during world startup as well as during normal operations. If the event timeout could be disabled during startup, that would go some way towards alleviating this issue.
We're going to increase the event timeout during world startup to 5 minutes. After the world finishes initializing the timeout will return to the usual value of 30 seconds. We are very hesitant to increase the overall timeout (because of the downsides), but hopefully this will be a significant help.
These users thanked the author Oren Hurvitz for the post (total 2):
Dot MatrixJohn Mela
User avatar
John Mela
Posts: 91
Joined: Tue Feb 04, 2014 9:50 pm
Has thanked: 139 times
Been thanked: 127 times
Contact:

Re: Has something changed? Event timeouts

Post by John Mela »

Many thanks, Oren. That will certainly go a long way to addressing this problem.

I'm currently in the middle of implementing more queuing within the RezMela core code to lighten the load on the script engine at other crucial times, especially the loading of a large scene containing objects with dynamic textures.
User avatar
Oren Hurvitz
Posts: 361
Joined: Sun Dec 23, 2012 8:42 am
Has thanked: 19 times
Been thanked: 499 times
Contact:

Re: Has something changed? Event timeouts

Post by Oren Hurvitz »

We've updated the system with the feature that increases the event timeout to 5 minutes while the world is starting.
These users thanked the author Oren Hurvitz for the post (total 2):
Snoots DwagonJohn Mela
User avatar
Dot Matrix
Posts: 1625
Joined: Sun Jul 28, 2013 3:26 am
Has thanked: 1208 times
Been thanked: 2324 times

Re: Has something changed? Event timeouts

Post by Dot Matrix »

Thank you very much, Oren.
User avatar
John Mela
Posts: 91
Joined: Tue Feb 04, 2014 9:50 pm
Has thanked: 139 times
Been thanked: 127 times
Contact:

Re: Has something changed? Event timeouts

Post by John Mela »

Thanks, Oren. This will make a big difference.
Post Reply