Downloaded RAW terrain files are corrupt?

Tips and discussions about using OpenSim
Post Reply
User avatar
Dale Innis
Posts: 20
Joined: Thu Mar 13, 2014 1:59 am
Has thanked: 26 times
Been thanked: 18 times

Downloaded RAW terrain files are corrupt?

Post by Dale Innis » Sun Jul 30, 2017 5:03 am

I tried downloading the RAW file for the terrain of my Kitely sim, and got a file that's 851,968 bytes in size, and no program I've pointed at it has been able to open. Looking at it in a hex editor, it has sequences of 00 00 00 00 FF FF FF FF inserted very often, like every five to nine bytes, variably.

I downloaded one of the standard RAW files from the site, and it was as expected 262,144 bytes (256 x 256 x 4), and in the hex editor it's clearly just a series of 4-byte integers, as expected.

I asked a friend to download the RAW file one of her Kitely regions and send it to me, and it's also 851,968 bytes, with constant 00 00 00 00 FF FF FF FF sequences in it.

Is it possible that there's something wrong with terrain RAW file download in the current Kitely? Does anyone else get a different result, or alternately understand what the result means?

Thanks!

(P.S. I want to download the RAW terrain so that I can do clever things to it using tools and programs and things, and then re-upload it as modified.)
User avatar
Ilan Tochner
Posts: 4648
Joined: Sun Dec 23, 2012 8:44 am
Has thanked: 2309 times
Been thanked: 2482 times
Contact:

Re: Downloaded RAW terrain files are corrupt?

Post by Ilan Tochner » Sun Jul 30, 2017 6:12 am

Hi Dale,

All RAW file operations in OpenSim are very sensitive to network noise as they are transferred to and from the viewer over an unreliable UDP protocol. This can often result in corrupt RAW file transfers.

Have you tried getting the RAW file by exporing your world to an OAR file then unzipping the OAR to get to the files it contains?
User avatar
Dale Innis
Posts: 20
Joined: Thu Mar 13, 2014 1:59 am
Has thanked: 26 times
Been thanked: 18 times

Re: Downloaded RAW terrain files are corrupt?

Post by Dale Innis » Sun Jul 30, 2017 1:58 pm

Ah, an interesting thought! Funny both downloads would be exactly the same size, if that's the problem, but who knows!

I will try the OAR idea.

Is RAW terrain file upload also over UDP with no reliable protocol on top of it? That could be disastrous!
User avatar
Ilan Tochner
Posts: 4648
Joined: Sun Dec 23, 2012 8:44 am
Has thanked: 2309 times
Been thanked: 2482 times
Contact:

Re: Downloaded RAW terrain files are corrupt?

Post by Ilan Tochner » Sun Jul 30, 2017 2:19 pm

Yes, RAW file uploads are done over UDP as well. That is why they can easily break if you try to upload a RAW file to a sim that is currently experiencing network load.

Have you tried this with different viewers? Viewers sometime differ in how they implement various network-related operations.
These users thanked the author Ilan Tochner for the post:
Dale Innis
Graham Mills
Posts: 985
Joined: Sun Dec 23, 2012 2:26 pm
Has thanked: 589 times
Been thanked: 736 times

Re: Downloaded RAW terrain files are corrupt?

Post by Graham Mills » Sun Jul 30, 2017 2:47 pm

Don't forget you can code basic terrain operations using OSSL so you could save the data as notecard text inworld and reconstitute it offline.

Scratch code: viewtopic.php?f=26&t=2408#p22150
These users thanked the author Graham Mills for the post (total 2):
Ilan TochnerDale Innis
User avatar
Dale Innis
Posts: 20
Joined: Thu Mar 13, 2014 1:59 am
Has thanked: 26 times
Been thanked: 18 times

Re: Downloaded RAW terrain files are corrupt?

Post by Dale Innis » Sun Jul 30, 2017 3:35 pm

Ilan Tochner wrote:Yes, RAW file uploads are done over UDP as well. That is why they can easily break if you try to upload a RAW file to a sim that is currently experiencing network load.

Have you tried this with different viewers? Viewers sometime differ in how they implement various network-related operations.
Is it at least CRC checked on upload, or something? I mean, file transfer over UDP without some kind of error-checking is madness! :)

I think we've now tried it (the download) with both FS and Singularity, on two different sims, and from two different continents. So it seems oddly consistent!
User avatar
Dale Innis
Posts: 20
Joined: Thu Mar 13, 2014 1:59 am
Has thanked: 26 times
Been thanked: 18 times

Re: Downloaded RAW terrain files are corrupt?

Post by Dale Innis » Sun Jul 30, 2017 3:43 pm

Graham Mills wrote:Don't forget you can code basic terrain operations using OSSL so you could save the data as notecard text inworld and reconstitute it offline.

Scratch code: viewtopic.php?f=26&t=2408#p22150
Woot! I was not aware of the OsXetTerrainHeight() functions; that is wonderful! I assume they do not have the wonderful "do completely random things to lots of nearby other terrain points" side-effects that the llModifyTerrain ones do? :)

I will begin playing with that at once; thanks much!
Graham Mills
Posts: 985
Joined: Sun Dec 23, 2012 2:26 pm
Has thanked: 589 times
Been thanked: 736 times

Re: Downloaded RAW terrain files are corrupt?

Post by Graham Mills » Sun Jul 30, 2017 3:57 pm

It was adequate for my purposes but Kayaker waxes eloquently on the limitations in a notecard that comes with one of his terrain products so I doubt it's 100% glitch-immune.
User avatar
Dale Innis
Posts: 20
Joined: Thu Mar 13, 2014 1:59 am
Has thanked: 26 times
Been thanked: 18 times

Re: Downloaded RAW terrain files are corrupt?

Post by Dale Innis » Sun Jul 30, 2017 5:22 pm

Graham Mills wrote:Don't forget you can code basic terrain operations using OSSL so you could save the data as notecard text inworld and reconstitute it offline.

Scratch code: viewtopic.php?f=26&t=2408#p22150
Just to follow up, that worked perfectly! My "Riverbottom Smoother" script:

Code: Select all

// This smooths the terrain of river bottoms (and anything else with a low elevation so BEWARE).
// Requires Estate Manager, or equivalent High Risk permissions

float HIGHEST = 14.0;  // The script will touch no points higher than this
float TOWARDS = 9.0;  // The script will move all points lower than HIGHEST toward this height
float FRACTION = 0.9;  // Set to 1.0 to make the bottoms completely flat, or lower numbers to keep some bumps
integer HALFSIDE = 63;  // The script will act on a square this many meters N, E, S, and W of the prim's location

default
{
    state_entry()
    {
        llSay(0, "Script running");
    }
    
    touch_start(integer n) {
        vector v = llGetPos();
        integer i = llFloor(0.5+v.x);
        integer j = llFloor(0.5+v.y);
        integer x;
        integer y;
        for (x=i-HALFSIDE; x<=i+HALFSIDE; x++) {
            for (y=j-HALFSIDE; y<=j+HALFSIDE; y++) {
                float h = osGetTerrainHeight(x,y);
                if (h<=HIGHEST) {
                    float newh = h + FRACTION * (TOWARDS - h);
                    // llOwnerSay("Would set "+(string)x+", "+(string)y+" to "+(string)newh);
                    osSetTerrainHeight(x,y,newh);
                }
            }
        }
        osTerrainFlush();
        llOwnerSay("Done");
    }
}
These users thanked the author Dale Innis for the post (total 4):
Dot MatrixKarima HoisanGraham MillsIlan Tochner
Post Reply