the material texture error bug is real in Naboo

Ask questions about creating worlds, using worlds, etc.
User avatar
Ilan Tochner
Posts: 6503
Joined: Sun Dec 23, 2012 8:44 am
Has thanked: 4942 times
Been thanked: 4454 times
Contact:

Re: the material texture error bug is real in Naboo

Post by Ilan Tochner »

Mike, the server delivered the correct assets to your viewer from the start and hasn't stopped doing so. The changes in how they appear are differences in the object data not in the assets data that the viewer used when rendering the object. It's the object data that tells the viewer which assets to use, specifically which textures assets are used by the mesh.

The viewer used the wrong object data initially in Naboo, you clicked it and that caused it to switch to using the correct object data immediately (according to your own report). Note that your viewer could not have been able to do so if it didn't already have the correct asset data cached already. At this point the viewer cached that correct object information and used it when you rezzed that object in SPACE FORCE. You teleported to your home, waited 5 minutes and returned to find the viewer using the old object data that it initially used in Naboo before you clicked the object. What changed is that the viewer, for whatever reason, stopped using the most up-to-date object data it cached just a few minutes earlier and switched to using the old object data.

When Koshari entered SPACE FORCE her viewer got the same object data from the server that your viewer got when it didn't have the proper object data cached so of course she should have seen the same thing that you have. My question, that you've still not answered, is whether you clicking the object after she's standing next to you and sees it inworld with the wrong texture causes both your and her viewers to then display the object with the proper texture.

---

Now let's consider how that could have happened. When you teleported back to SPACE FORCE the viewer either asked the server for object data information and used whatever it got from the server or it used the object data that it had cached a few minutes ago. If it used the cached object data information and now rendered it differently then that's a viewer bug.

If, however, it downloaded the object data from the server again and rendered it improperly then that means that either the server doesn't hold the correct object data, in which case clicking the object wouldn't have resulted in the viewer having the correct data to use to properly render the object. Or, the server holds the correct object data but the viewer, for a TBD reason doesn't use that object data and instead uses something else that looks like the object data with some changes but starts using the correct object data after the object has been clicked.

This last case is what's happening and the question is why. As stated in my previous comment, when we investigated similar issues when we upgraded to OpenSim 0.9.1.1 we found that there were bugs in both the viewer and in OpenSim that relate to the Viewer Object Cache mechanism. We couldn't fix all those issues without also fixing the viewer bugs that caused them so we used the OpenSim.ini SceneObjectGroup.IsViewerCachable setting to disable this mechanism which resulted in the elimination of those problems in Kitely with the viewer versions that existed at that time.

However, it's not impossible that later viewer versions changed how they implemented the Viewer Object Cache mechanism and that this resulted in new issues. It's also possible that the specific issue you're encountering was never fixed even when we disabled the Viewer Object Cache mechanism in OpenSim. Whatever the case, that doesn't change the fact that the viewer seems to be reacting to the same nudge when an object is clicked inworld. A nudge that it shouldn't need when the viewer is working properly. I asked you to upgrade the viewer version you were using because it was old and more recent versions fixed issues that may have been related. That, unfortunately, didn't solve this issue but upgrading the viewer is often the first thing you should do when you encounter rendering issues.
These users thanked the author Ilan Tochner for the post:
Shandon Loring
User avatar
Ilan Tochner
Posts: 6503
Joined: Sun Dec 23, 2012 8:44 am
Has thanked: 4942 times
Been thanked: 4454 times
Contact:

Re: the material texture error bug is real in Naboo

Post by Ilan Tochner »

FYI Beq, the main Firestorm dev that works on OpenSim support, contacted me and I shared the information we have about this issue with her.
These users thanked the author Ilan Tochner for the post (total 4):
Tess JuelShandon LoringAda RadiusChris Namaste
Beq Janus
Posts: 7
Joined: Fri May 15, 2020 2:06 pm
Has thanked: 2 times
Been thanked: 24 times

Re: the material texture error bug is real in Naboo

Post by Beq Janus »

Hi all,
Happy to try to get to the bottom of this, ultimately a bug is a bug, I don't care if it is in FS/OS or the Asset itself. I've not seen anything so readily repeatable as this before, so I can only assume it to be OpenSim specific (which does not narrow down the source, just the exposure) or at least if it happens in SL there is some other mitigation that prevents it from being seen.

I visited Naboo and found the problematic item(s). I've no doubt you have all seen the following but I am recording it here as a baseline.
There are 4 objects that we care about. Three are single objects (seats) and one is a linkset (two seat parts and a table as root). Perhaps notably the one that "works" is the linkset.

If I right-click on a "broken seat" it will fix itself (no need to edit, the act of selection is enough to get the refresh)
Going forward it will remain fixed across logins, unless I clear the cache.

If you can manage to stop staring at my awesome avatar for a few seconds then the following video clip shows a fresh login with a cleared cache. https://i.gyazo.com/d0cbd76b7e47d0e6a28 ... 013c18.mp4

This suggests that the problem lies in the initial update, what we need to determine is whether FS is misinterpreting an update, or whether the update is somehow incomplete. In that regard, I've not got any closer to understanding what is happening.

I would like to see the various updates, but the scene is far too busy to make any sense of that. If it is possible to reproduce the object that fails in an empty region, it will make life a lot easier; of course, it might not repro in a simple scene; that'd be sod's law.

best regards

Beq
These users thanked the author Beq Janus for the post (total 7):
Ilan TochnerTess JuelAda RadiusChris NamasteShandon LoringDot MatrixChristine Nyn
Beq Janus
Posts: 7
Joined: Fri May 15, 2020 2:06 pm
Has thanked: 2 times
Been thanked: 24 times

Re: the material texture error bug is real in Naboo

Post by Beq Janus »

Hi all,
Just following up. Is there any way to get the asset in question in an otherwise quiet region? And confirm that it is still misbehavng.
These users thanked the author Beq Janus for the post:
Christine Nyn
User avatar
Mike Lorrey
Posts: 361
Joined: Sun Sep 04, 2016 5:40 pm
Has thanked: 71 times
Been thanked: 269 times

Re: the material texture error bug is real in Naboo

Post by Mike Lorrey »

Beq Janus wrote:
Mon Jan 23, 2023 8:51 pm
Hi all,
Just following up. Is there any way to get the asset in question in an otherwise quiet region? And confirm that it is still misbehavng.
Hi Beq, I have the asset in the Kitely Merchants Sandbox which is very quiet. Its a standalone seat for a Dejarik table. Standing next to it is a dejarik table with two SIMILAR but different seats that use the exact same list of materials (created in the same blender file). Location is <247.62801, 202.13300, 21.88730>

I logged in earlier at Naboo where the objects are natively sited. The standalone seat was wrongly textured with the speaker grill texture on the frame/backside material (face 7). That texture belongs on face 5, which it does appear on. When I clicked on the object to grab it into inventory it flipped to the correct texture on face 7. I then picked up the dejarik table with its two similar but different seats into inventory as well. I teleported to Kitely Merchants Sandbox and rezzed them. Both had the right texture on face 7 of the seats.

I teleported to naboo then to kitely welcome center and ilan reset the sandbox region. I teleported back and it was the same right texture. I then relogged, and it was the same. I cleared my cache and relogged. When I relogged the second time in the region, the wrong texture, the speaker grill texture, was on face 7 of the standalone seat again.

This behavior is only different in that in Naboo and SPACE FORCE regions, the same object almost always reverts to the wrong texture regardless of teleporting out and back or relogging. Clicking on it flips it back to the right texture. One test which mike mchugh witnessed, we teleported back and it was the right texture, but when we clicked on it it flipped to the wrong texture.
These users thanked the author Mike Lorrey for the post:
Ilan Tochner
User avatar
Mike Lorrey
Posts: 361
Joined: Sun Sep 04, 2016 5:40 pm
Has thanked: 71 times
Been thanked: 269 times

Re: the material texture error bug is real in Naboo

Post by Mike Lorrey »

Mike Lorrey wrote:
Mon Jan 23, 2023 11:03 pm
Beq Janus wrote:
Mon Jan 23, 2023 8:51 pm
Hi all,
Just following up. Is there any way to get the asset in question in an otherwise quiet region? And confirm that it is still misbehavng.
Hi Beq, I have the asset in the Kitely Merchants Sandbox which is very quiet. Its a standalone seat for a Dejarik table. Standing next to it is a dejarik table with two SIMILAR but different seats that use the exact same list of materials (created in the same blender file). Location is <247.62801, 202.13300, 21.88730>

I logged in earlier at Naboo where the objects are natively sited. The standalone seat was wrongly textured with the speaker grill texture on the frame/backside material (face 7). That texture belongs on face 5, which it does appear on. When I clicked on the object to grab it into inventory it flipped to the correct texture on face 7. I then picked up the dejarik table with its two similar but different seats into inventory as well. I teleported to Kitely Merchants Sandbox and rezzed them. Both had the right texture on face 7 of the seats.

I teleported to naboo then to kitely welcome center and ilan reset the sandbox region. I teleported back and it was the same right texture. I then relogged, and it was the same. I cleared my cache and relogged. When I relogged the second time in the region, the wrong texture, the speaker grill texture, was on face 7 of the standalone seat again.

This behavior is only different in that in Naboo and SPACE FORCE regions, the same object almost always reverts to the wrong texture regardless of teleporting out and back or relogging. Clicking on it flips it back to the right texture. One test which mike mchugh witnessed, we teleported back and it was the right texture, but when we clicked on it it flipped to the wrong texture.
additional testing:
I drag copied the offending item in the sandbox. On the one I dragged, I edited face 7 and set it to the default plywood texture. Then reedited to put the correct texture on there. I left the duplicated one in the original position alone. I deleted the sit script (nothing but a set sit pos function) in one of the three originals in Naboo. I teleported back and forth between the tewo regions and nothing went haywire. I relogged, nothing. Then I cleared my caches on my viewer and relogged in the kitely merchants sandbox and both the original object I rezzed there (that I dragged and then reverted to default plywood and back to the proper texture) and the copy, were displaying the wrong texture.

Are there caches in the firestorm install other than the ones that get cleared?: If the caches are cleared, where would the viewer be getting the wrong texture other than the asset server (it couldn't be the merchant sandbox unless the region server was transferring a texture from face 5 to face 7) so where else other than the asset server find that speaker texture and wrongly apply it to the back and frame (face 7)?

Note that, ironically, I've been teleporting back to Naboo in the last couple logins and those are behaving fine for some reason. Am going there now since I logged into the sandbox, to see if that reverted.

Ok so the original items in Naboo hat been resisting changing to the wrong color through the period of this test, UNTIL I went to them in Naboo and Logged out there, then relogged into the same location. It seems the bug is specifically in the code for logging into a region and whatever data it gets from the server then. I cannot imagine how this could be coming from caches that have been cleared unless there are other caches that the viewer doesnt clear.
User avatar
Mike Lorrey
Posts: 361
Joined: Sun Sep 04, 2016 5:40 pm
Has thanked: 71 times
Been thanked: 269 times

Re: the material texture error bug is real in Naboo

Post by Mike Lorrey »

Tess Juel wrote:
Wed Jan 11, 2023 2:15 am
Mike Lorrey wrote:
Tue Jan 10, 2023 9:21 pm
a) the correct texture was already on another material of the neighboring mesh, so you are wrong. It took no time to download it cause it already had it
Ok. You didn't mention that and it makes a difference of course. Even so it must mean the viewer already had the right texture UUID so it can't be purely a server side issue.

I've seen something similar before so I can at least confirm that such a weird bug is possible. That was on a different grid running the very first distribution of opensim 0.9. What happened then was that the meshed showed up with the right textures but next time I went to the sim, face 7 had switched to a texture used by another face of the same mesh. Clicking on the object didn't solve the issue then and the bug did not show up in everybody's viewer. Unfortunately we never figured out what was happening back then.
Well, I've cleared my caches and done clean installs multiple times, so you'd have to come up with some place in hyperspace that data can be coming from. I am not the only person who has seen the bug on this object. Rosa sees it, as has Kayaker and mike mchugh and Koshari. everyone I show it to sees it, and they even see the texture flip when I touch the object, even if they've never been in my region where the object was native.

Furthermore: this flipping behavior has not happened when I've rezzed a copy of the object on other grids that I had copied to my Suitcase. so those other grids dont seem to be confused about what the proper texture is required for that material. This is why I am certain it is a server side bug, because when it gets rezzed on another grid, its no longer a kitely object..
User avatar
Shandon Loring
Posts: 1341
Joined: Sat Oct 26, 2013 3:25 am
Has thanked: 961 times
Been thanked: 1581 times
Contact:

Re: the material texture error bug is real in Naboo

Post by Shandon Loring »

anecdote time, likely unrelated?
and certainly TL;DR...

in SL..
i once had a single prim curvy wall that was a sculpty. it was linked to a large castle. usually it was a curvy wall piece. but sometimes it spontaneously became a giant sculpted light switch. sometimes it would oscillate back and forth between being a wall piece and a lightswitch while you watched it.

my best guess at the time was that somehow the sculpt textures for those two items got confused on asset server(s), or maybe, is it even possible?, somehow picked up the same uuid??

it was consistent and repeatable and years ago. helpful in this present situation? probably not. just thought i'd mention.

we solved it by deleting the prim and its sculpt texture and reuploading.

here in kitely i was building a sewer for the labyrinth sim, and one hollowed cube prim when linked with other prims and subsequently unlinked would force the whole link set apart by several 1/10ths of a meter. just that one prim.
even when deleted from world and rerezzed from inventory.

deleted it from inventory, created new one. solved.

my point? there has to be a point?? i guess that as someone in the thread mentioned earlier, its a complex system and things get weird sometimes.

i take it that since we're spending so much time on this, that folks are not interested in a resolution so much as cause though. it is indeed curious..

anyway, if you read my diatribe this far... wow!

cheers!
Last edited by Shandon Loring on Wed Jan 25, 2023 5:45 pm, edited 1 time in total.
These users thanked the author Shandon Loring for the post:
Graham Mills
Graham Mills
Posts: 1314
Joined: Sun Dec 23, 2012 2:26 pm
Has thanked: 1134 times
Been thanked: 1141 times

Re: the material texture error bug is real in Naboo

Post by Graham Mills »

I always think of OpenSim in terms of the old metaphor of a swan gliding serenely on the surface while its legs are paddling furiously below. Things that fail intermittently or are hard to reproduce are thankfully rare these days and, as Shandon says, invite pragmatic workrounds.

I was curious as to whether anyone has tried Alchemy or SceneGate which supposedly use a different rendering engine (how different is, of course, another matter).
These users thanked the author Graham Mills for the post (total 2):
Ilan TochnerShandon Loring
Beq Janus
Posts: 7
Joined: Fri May 15, 2020 2:06 pm
Has thanked: 2 times
Been thanked: 24 times

Re: the material texture error bug is real in Naboo

Post by Beq Janus »

Mike Lorrey wrote:
Tue Jan 24, 2023 12:32 am
Are there caches in the firestorm install other than the ones that get cleared?: If the caches are cleared, where would the viewer be getting the wrong texture other than the asset server (it couldn't be the merchant sandbox unless the region server was transferring a texture from face 5 to face 7) so where else other than the asset server find that speaker texture and wrongly apply it to the back and frame (face 7)?
There are a couple of different things that conceivably could be happening, the one that sits foremost in my mind is texture batching.
In the render pipeline faces that have the same set of render parameters are batched to reduce the overhead of transferring between CPU and GPU and the state changes of the drivers. There were a few bugs introduced in LL's performance updates that allowed weird behaviour in batches. Much of the recent viewer development time I have had was spent tracking these through detailed debugging sessions. As far as I recall every single one of these was applicable to rigged mesh only. Though that does not mean that they could not affect status meshes. The majority were also related to alpha-blended surfaces.

The behaviour that we see "could" be a batching issue. Somehow the viewer is seeing the textures as catchable and then when they get rendered it uses the same UUID when it should be a separate one. It doesn't sit well with me, it doesn't feel like that's what is going on but there are certainly similarities.

Typically it takes me a week or more of sitting with the asset and stepping through the logic, working out what is going on. In SL, I ask for the item to be made available to me and I have an empty space where I can take it and see it in isolation.

I was very clearly able to see the problem in Naboo, it was only ever visible after a cache clear, but it was fairly reliably repeatable. The problem I had is that it is surrounded by literally 1000s of other objects all communicating back and forth with the asset server etc.

things that seem familiar:
1) The face that corrupts is the same face on each object,
2) it corrupts in the same way in each, suggesting the underlying mesh asset not the top level "container" is the issue.
3) It is typically fixed simultaneously, also suggestive of the common feature.

things that do not fit:
1) The instance that is part of the link set with the table is not affected.
2) It does not repeat on demand but needs a clean cache
3) no edit needs to be made, simply highlighting is enough.
4) it's static mesh, for the most part nothing has changed in static mesh batching.
5) I don't think there are materials involved, this is just a simple diffuse?

why is #5 relevant? A number of the rigged mesh bugs manifest because of artful manipulation by creators to try to force alpha blended faces to render in a specific order. Frankly, those games are a pain to me as it increasingly means that just a change of compiler setting can upset someone's pet workaround. The kind of gimmicks used are the setting of an alpha mask cutoff threshold, but using alpha blend/None, of adding a blank normal map, or a blank normal+spec. These tricks are exploring the fact that the pipeline code has to deal with different things in different ways, in som cases, it forces a different render batch in others it simply changes the order in which things get drawn. In ALL cases, there is no guarantee it will continue to work tomorrow or anytime in the future, though boy do people scream when things break. HOWEVER, with the limited insight I have been able to apply, I do not see any such features (intended or not) that might explain it, these seem common or garden meshes.

@mike, given your other testing (where you set the misbehaving face to ply) are you pretty confident that the issue survives replication and transfer? If so could you pass the object to me (full perm if at all possible) my kitely account is Beq Janus.

I'll then try to find a quiet place where I can rez and hopefully reproduce the issue and then observe it both in the rendering and in the conversation with the region.

As Graham suggests, testing on another viewer would be useful. I don't know much about scene gate internals and whether they are aligned with the more recent linden lab update. Alchemy is from the same root as Firestorm but has various tweaks of its own.

It would be interesting to know which version(s) of Firestorm have exhibited this behaviour too.

I always tend to assume a viewer bug until I can demonstrate otherwise, just because there is more to go wrong.
These users thanked the author Beq Janus for the post (total 4):
Graham MillsShandon LoringIlan TochnerTess Juel
Post Reply