OpenSimWorld is the directory of 3D Virtual Worlds based on OpenSimulator and connected through the HyperGrid. Learn More
Has anyone experienced anything strange with their grids lately? A friend, (honestly not me), thinks that perhaps a worm is travelling through different grids causing the database to lose connectivity with the asset server files. (That's what he said). Apparently osgrid, digiworldz, barefoot grid, and some dreamgrids have been mysteriously affected by this supposed worm. Anyone out there have any information? Maybe it has happened to you? Maybe you know what it is and it is something else?
Liked by: 
Comments
At WT Grid we have had our fair share of asset weirdness and still do occasionally. this includes
- Users suddenly losing all the links in the outfits folder
- Assets coming up as not found when bought from other grids - I have a theory on this one but we're still looking at it after 3 years
I think there are other things that can affect grids *ESPECIALLY* if running on Windows
- IMHO Only run grids on SERVER GRADE SSD / NVME drives.
- If you are running windows, the master fat table corrupts quite regularly and this can cause all kinds of issues. Start a CMD prompt AS ADMINISTRATOR, run chkdsk /f and reboot
- Database Latency (and this is just me theorising) really affects the performance of opensim. If you have lag on your region, sure you can clear off all the items on the region but I pretty much guarantee that you have a lot of latency between your database and the grid and your servers need tuning up.
I have rescued a few grids now that were running on hard disks, it's pretty hard work if you imagine all that writing. Every time someone moves, that's a write to the database, every time an object moves around ,.. another write, so if you have hard disks you are hammering them.
One trick I found as well was to shut down the offending region after backup and clear the fsassets folder in the bin directory (you can do this from the console as well I think its something like fcache clear). That will force it to pull all the assets from the main asset store again.
like(0)
- Users suddenly losing all the links in the outfits folder
- Assets coming up as not found when bought from other grids - I have a theory on this one but we're still looking at it after 3 years
I think there are other things that can affect grids *ESPECIALLY* if running on Windows
- IMHO Only run grids on SERVER GRADE SSD / NVME drives.
- If you are running windows, the master fat table corrupts quite regularly and this can cause all kinds of issues. Start a CMD prompt AS ADMINISTRATOR, run chkdsk /f and reboot
- Database Latency (and this is just me theorising) really affects the performance of opensim. If you have lag on your region, sure you can clear off all the items on the region but I pretty much guarantee that you have a lot of latency between your database and the grid and your servers need tuning up.
I have rescued a few grids now that were running on hard disks, it's pretty hard work if you imagine all that writing. Every time someone moves, that's a write to the database, every time an object moves around ,.. another write, so if you have hard disks you are hammering them.
One trick I found as well was to shut down the offending region after backup and clear the fsassets folder in the bin directory (you can do this from the console as well I think its something like fcache clear). That will force it to pull all the assets from the main asset store again.
I had a message from Glenn, first he said :
"please let Cyber know a very big thank you for his post!"
And about the matter here :
"
Ok I have tested and through smashing, bashing, and trashing, in every way possible, trying to find the source of the problem. I use FS .6.6. 86 And upon restoring Amoa, from an original build Oar, all of the missing items and textures that are affecting Barefoot Amoa, are not a problem. So that eliminates the FS question. As for the OS versions, Bezerkly Amoa is running on .0.9.1.1 Snail release Diva Wifi. It is slower but NONE of the issues exist that I find existing in the Yeti .0.9.2.2. version. So now it points to the problem being in .0.9.2.2. on up as a culrpit.
I tested by rezzing similar items in OSGrid, same problem exists, I tested visiting other grids, same problem existed.
"
like(2)
"please let Cyber know a very big thank you for his post!"
And about the matter here :
"
Ok I have tested and through smashing, bashing, and trashing, in every way possible, trying to find the source of the problem. I use FS .6.6. 86 And upon restoring Amoa, from an original build Oar, all of the missing items and textures that are affecting Barefoot Amoa, are not a problem. So that eliminates the FS question. As for the OS versions, Bezerkly Amoa is running on .0.9.1.1 Snail release Diva Wifi. It is slower but NONE of the issues exist that I find existing in the Yeti .0.9.2.2. version. So now it points to the problem being in .0.9.2.2. on up as a culrpit.
I tested by rezzing similar items in OSGrid, same problem exists, I tested visiting other grids, same problem existed.
"
Glenn reply :
"
No, I did not test this with any other browser. This points to the issue though of only existing in OS .0.9.2.2. on up. Yeti, possibly Sasquatch, definitely Nelly. The textures and scripts, and any damaged assets were originally OARed from 0.9.1.1. Diva D2. IAR shows they exist. So the issue appears to be traveling through multiple versions of .0.9.2.2. + and not a matter of IF but a matter of WHEN. This does not dispute the other person, who's comments are gone regarding the possibility of assets taken from grids that are now gone and don't exist anymore.
I also suspect, the fact that 3rd Rock which was very old, is now gone, and some assets that were originally created there and then copied and shoved into every tiny mall, is now possibly broken across all current grids. If they didn't constantly chat ban me, I wouldn't have to deal with this type of communication. So this will take longer to solve utilizing 3rd party communcation methods.
This all began around June 21. In the early morning, barefoot was rocked by MASSIVE lag. Then, within a few hours, it started with eyeballs missing and then avalanched into something bigger. Restoring from an OAR on barefoot, only replaced newer work. Because the oar was only 1 week old. The OAR I tested with is from 2020. However, all additions and modifications have since been wiped out. Since I have to rebuild the newer sections, I am just improving it.
"
like(0)
"
No, I did not test this with any other browser. This points to the issue though of only existing in OS .0.9.2.2. on up. Yeti, possibly Sasquatch, definitely Nelly. The textures and scripts, and any damaged assets were originally OARed from 0.9.1.1. Diva D2. IAR shows they exist. So the issue appears to be traveling through multiple versions of .0.9.2.2. + and not a matter of IF but a matter of WHEN. This does not dispute the other person, who's comments are gone regarding the possibility of assets taken from grids that are now gone and don't exist anymore.
I also suspect, the fact that 3rd Rock which was very old, is now gone, and some assets that were originally created there and then copied and shoved into every tiny mall, is now possibly broken across all current grids. If they didn't constantly chat ban me, I wouldn't have to deal with this type of communication. So this will take longer to solve utilizing 3rd party communcation methods.
This all began around June 21. In the early morning, barefoot was rocked by MASSIVE lag. Then, within a few hours, it started with eyeballs missing and then avalanched into something bigger. Restoring from an OAR on barefoot, only replaced newer work. Because the oar was only 1 week old. The OAR I tested with is from 2020. However, all additions and modifications have since been wiped out. Since I have to rebuild the newer sections, I am just improving it.
"
Pagane works for microsoft in their bug finding department. We should listen to her because she runs a version of opensimulator without a cache. This is very special b/c it lets her see the errors the instant they start happening. So Pagane says that later versions of firestorm on 9.2.2 are causing this problem, and I believe her. I think you Glen are just seeing the same thing from the server side of it. But Pagane is able to see it instantly on both viewer and server side b/c of her unique build. We should listen to Pagane.
like(1)
Two things that I've noticed:
One, do you know the Swiss Cafe? That two-floor building with the Swiss flag, "CAFE" written on the outside, bar counter inside but hardly any other furniture, baked-on spotlight highlights on the walls etc. etc.? There are several derivatives of this building already.
All of them, original and derivatives, have the same bug now: Ground floor. Second window from the left. The frame and the upper part are plain white, they've lost their texture. Everywhere, all over the Hypergrid, no matter where I go.
Two, I keep seeing Adonis 3 with a white right hand and a Bento-less starfish left hand. This could depend on the skin, though.
I see both with multiple avatars on multiple grids each, running very different OpenSim versions. And I see this in all Firestorm versions from 6.3.9 to 7.1.9 as well as in the Cool VL Viewer.
like(1)
One, do you know the Swiss Cafe? That two-floor building with the Swiss flag, "CAFE" written on the outside, bar counter inside but hardly any other furniture, baked-on spotlight highlights on the walls etc. etc.? There are several derivatives of this building already.
All of them, original and derivatives, have the same bug now: Ground floor. Second window from the left. The frame and the upper part are plain white, they've lost their texture. Everywhere, all over the Hypergrid, no matter where I go.
Two, I keep seeing Adonis 3 with a white right hand and a Bento-less starfish left hand. This could depend on the skin, though.
I see both with multiple avatars on multiple grids each, running very different OpenSim versions. And I see this in all Firestorm versions from 6.3.9 to 7.1.9 as well as in the Cool VL Viewer.
When I start seeing white textures, I clear my viewer caches and that usually resolves the issue. It seems to happen a lot if and when I use the Opensimulator version of FS to log in to SL. Logging back to my Opensim grid, results in some content turning white.
So I try to always use the Opensim version of FS for Opensim and the SL version for SL.
like(0)
So I try to always use the Opensim version of FS for Opensim and the SL version for SL.
Clearing the viewer caches did nothing to solve my problem. In fact, I've tried deleting the entire user directory for Firestorm after saving the settings and having it regenerated. The issue persisted.
And if
* Firestorm on my tower with dedicated AMD graphics
* the Cool VL Viewer on my tower with dedicated AMD graphics
* Firestorm on my laptop with on-board Intel graphics
* the Cool VL Viewer on my laptop with on-board Intel graphics
all show the exact same symptoms and do so for four avatars on two grids, one running OpenSim 0.9.2.0, one running OpenSim 0.9.3.0, then it's highly unlikely that the viewer cache is the issue. Firestorm and the Cool VL Viewer don't have a shared cache, nor do they share a cache across multiple independent machines.
like(2)
And if
* Firestorm on my tower with dedicated AMD graphics
* the Cool VL Viewer on my tower with dedicated AMD graphics
* Firestorm on my laptop with on-board Intel graphics
* the Cool VL Viewer on my laptop with on-board Intel graphics
all show the exact same symptoms and do so for four avatars on two grids, one running OpenSim 0.9.2.0, one running OpenSim 0.9.3.0, then it's highly unlikely that the viewer cache is the issue. Firestorm and the Cool VL Viewer don't have a shared cache, nor do they share a cache across multiple independent machines.
Out of curiosity has anyone tried clearing the region's cache? Assets basically flow like this: Robust -> Sim (cached) -> Viewer (cached). If there's a network hiccup between the sim and robust during the initial fetch, then what's in the sim and viewer cache will be broken. If there's a hiccup on the disk the sim is caching to, then what's in the sim and viewer will be broken.
In the case of an HG transfer, I believe the assets going to the foreign grid are going through the region's cache so... Those could likely be affected to.
like(2)
In the case of an HG transfer, I believe the assets going to the foreign grid are going through the region's cache so... Those could likely be affected to.
yes we have tried clearing the regions cache, the grid cache, any & all cache that we could clear, we have done this clearing of cache's & then dropping an old backup of each region and it did not work, I have tons of space on our servers so it is not that we lack the space for all of our data
like(0)
We had some issues with things vanishing, including our avatar bodies and heads, it was explained to me that it was simply that one of the hard drives did not have enough space on it to carry the load. We cleared a lot of stuff from the grid that proved useless and unnecessary and it appears to be doing better now. I'm thankful it was something minor. Happy Independence Day!!!
like(3)
Are there any other big brains who could look into all this so that we have certainties and especially solutions? thank you!!!!!
(Happy 4th of July USA!!!)
like(2)
(Happy 4th of July USA!!!)
Open your Inventory, Outfits folder
If you have Saved appearances you must see folders with links to assets in every one appearance.
But if you see empty folders or broken links...... OOOOPS is late!
If you see how 🢒 sign before folders disapear one by one...... This is NOT worm!!! This is your FS and now he kill your SQL database.
Similar must see and in other folders.
like(2)
If you have Saved appearances you must see folders with links to assets in every one appearance.
But if you see empty folders or broken links...... OOOOPS is late!
If you see how 🢒 sign before folders disapear one by one...... This is NOT worm!!! This is your FS and now he kill your SQL database.
Similar must see and in other folders.
I believe the new version of firestorm has issues, i rolled back and havent seen any since. However I have one region being built that has things mysteriously disappear including terrain. Replaced a few things here and there.
like(1)
High latency or loss of data packets between the client and server can result in items not being displayed correctly or disappearing.
like(0)
I cant imagine considering its the same items, but they just poof from the region. i think its something in the region causing it or just opensim being opensim or whatever or dg lol. so far all is good in the region.
like(0)
I think it is more likely the owner having way to many assets. Keep in mind it all has to rez not just fetch up. Also badly created meshes may never rez.
like(0)
so far everything seems to rez okay, but yea its built by others on a test region owned by one of my members and she has a lot of assests in her invent so could be.
like(1)
I have a ton of assets, and on the land I try to keep it within reason. I know that when I visit other grids if they have lots of mesh buildings they rez super slow but it all depends too on how the mesh is created. Scripts are another thing to look at on land to make sure you don't have bad scripts about.
like(1)
After reading that the latest versions of Firestorm could be related to our asset issues I remember having a lot of things disappearing while I was in OSGrid with version 6.6.17 and Leonitas had told me that the last reliable version was 6.6.14. And it's true that miraculously I haven't had (for now) these problems in OSGrid since going back to 6.6.14. They persist in Barefoot though :-(
like(2)
And if it the cause of all our lost it is really a calamity.
How to tell everybody to go back to old versions if it is that?....
like(1)
How to tell everybody to go back to old versions if it is that?....
There are proper channels to address and troubleshoot issues with your system. Firestorm Help Ticket. When you have issues with Dreamgrid the correct protocol is to file a Mantis report, ( https://www.outworldz.com/bugreport/my_view_page.php?refre... ) this helps to track down bugs and to see how many are experiencing the issue. You also must keep in mind that each grid is maintained by each grid owner and their systems, so you have to understand those could be what is corrupt. Use the proper channels of submitting bug reports. Check your logs, it has all the errors it will help when you submit your reports.
like(0)
The worst symptoms I saw in Barefoot Dreamer grid and a little in Virtualife and I dont remember where else but there were at least two other where I saw problems I thought was because HG jump...
The most often it is on body colors, some skins (bom I think) making top of the body white or only lips, I lost "asset not present in database" for ears or AO too. Parts of Athena body changed colors or turn transparent (and even seem as not existing at all when pointing on them with the mouse).
Scripts are vanishing too, sometimes not entirely, in some hud (emoticon for example) you can still have parts functionning while the other half is dead...
like(1)
The most often it is on body colors, some skins (bom I think) making top of the body white or only lips, I lost "asset not present in database" for ears or AO too. Parts of Athena body changed colors or turn transparent (and even seem as not existing at all when pointing on them with the mouse).
Scripts are vanishing too, sometimes not entirely, in some hud (emoticon for example) you can still have parts functionning while the other half is dead...
Appearance issues sometimes happen to me when grid hopping. I can normally just edit my appearance and exit without making any changes and it will fix me. I make sure to keep all my outfits/body parts in my suitcase so I have access to them while grid hopping and can reattach things if things get wonky.
If all else fails then as soon as I return home I look normal again. My appearance never seems to mess up if I'm teleporting around within my own grid.
I have not found any assets missing from my home grid either. Though I'm running Diva Distribution that has not had an update for a long time, I think its still on 9.2.1 and I am using FS 6-6-17 Linux Mint.
The only asset related thing I notice is the occasional yellow message in the console about request for missing material on some objects. Though this has been a thing for as long as I can remember and is just a bad mesh I picked up on my travels. I never get that message for any meshes I have made myself, its always on something I picked up.
like(1)
If all else fails then as soon as I return home I look normal again. My appearance never seems to mess up if I'm teleporting around within my own grid.
I have not found any assets missing from my home grid either. Though I'm running Diva Distribution that has not had an update for a long time, I think its still on 9.2.1 and I am using FS 6-6-17 Linux Mint.
The only asset related thing I notice is the occasional yellow message in the console about request for missing material on some objects. Though this has been a thing for as long as I can remember and is just a bad mesh I picked up on my travels. I never get that message for any meshes I have made myself, its always on something I picked up.
My DG is fine, no worm and all assets fine on my end. No reports at the Mantis from other users https://www.outworldz.com/bugreport/my_view_page.php?refre... We have invited the reporters to post a Mantis for further investigation. No reports from said reporters.
like(0)
Yehhh, Pagane said it's a problem with the last 3 firestorm viewers writing edited assets incorrectly back to the asset server. She said she has personally observed the errors happening in real time.
like(1)
The viewer has no ability to "rewrite" assets to the asset server. Could cause a new asset to be saved of course but there is no rampant deconstruction of assets occurring due to the viewer code.
like(0)
Yes, and may repeat how many times want!
Full server backup/restore is 15-20min. Next is enough to save new appearance and ... assets links in SQL server is dead... (My server NOT have FSCache and bug is imediatelly and total!)
In full server with cache etc errors will be visible later...
like(1)
Full server backup/restore is 15-20min. Next is enough to save new appearance and ... assets links in SQL server is dead... (My server NOT have FSCache and bug is imediatelly and total!)
In full server with cache etc errors will be visible later...
Jupiter Rowland said "Can it be that assets can be damaged by faulty asset servers or something, and that damage spreads all across the Hypergrid and wrecks all copies and all derivatives of the affected content?"
like(3)
I see your post1: last 2 versions of FS have lots of errors with saving assets after edit. Depending of client and server OS and OpenSIM server version errors is from unpleasant like your up to fatal.
Try to make this same with Debian client connected to SQLITE based Linux OpenSIM server and will get total crash and destroy all server....
I simple downgrade to FS 6.6.14 and wait bug fix:(
and post 2:Before 4 months i try to warn about this FS bug.
https://opensimworld.com/post/106551#cmt59567
Peoples continute to not understand whats happen.
Last 3 versions of FS contains bug and when owner edit assets FS broke assets on server.
Keep wide closed eyes and wait:)
like(1)
Try to make this same with Debian client connected to SQLITE based Linux OpenSIM server and will get total crash and destroy all server....
I simple downgrade to FS 6.6.14 and wait bug fix:(
and post 2:Before 4 months i try to warn about this FS bug.
https://opensimworld.com/post/106551#cmt59567
Peoples continute to not understand whats happen.
Last 3 versions of FS contains bug and when owner edit assets FS broke assets on server.
Keep wide closed eyes and wait:)
Before 4 months i try to warn about this FS bug.
https://opensimworld.com/post/106551#cmt59567
Peoples continute to not understand whats happen.
Before 11 days AGAIN try to warn :
https://opensimworld.com/post/110398#cmt65470
Last 3 versions of FS contains bug and when owner edit assets FS broke assets on server.
And whats happen???
Super dupper mega experts know better...
Keep wide closed eyes and wait:)
like(2)
https://opensimworld.com/post/106551#cmt59567
Peoples continute to not understand whats happen.
Before 11 days AGAIN try to warn :
https://opensimworld.com/post/110398#cmt65470
Last 3 versions of FS contains bug and when owner edit assets FS broke assets on server.
And whats happen???
Super dupper mega experts know better...
Keep wide closed eyes and wait:)
Pagane can you provide more details about the bug? Do you know where in the code the bug is? Do you know how to fix it?
like(1)
Not know WHERE is bug bot know WHY and HOW:
Last versions of FS and server have major misunderstanding about SQL, server cache, file system and file system cache. ALL THIS in last 3-4 versions is one big omelette!!! And every next version is so bad.
Problem start in 9.2 version of server (I try to warn but was kicked like street cat) where someone invite mixture from filesystem caches and SQL and assets in file system .... may add soma jam, juice and what you want!
Next FS start to use same omlette from SQL, ancient Flosam cache, bad understanded BTRFS, SQLITE and essence of NTFS bugs!
Short:
Server read from SQL DB but write in OS file system CACHES!!! And wait OS to return this to SQL DB....
Viewer read whats come..... But when you edit write direct to SQL. And may be not write data, may be write indexes, file names or idk whats write but..... Sometimes server decode and repair bugs.
(If your server not use any caches like my > destroy SQL DB imediatelly)
How to find more than few major bugs multiplied in all code of server and viewer?
How to find where is assets and to return from files to SQL???
May be must ask some priest or Native American shaman!!!
like(3)
Last versions of FS and server have major misunderstanding about SQL, server cache, file system and file system cache. ALL THIS in last 3-4 versions is one big omelette!!! And every next version is so bad.
Problem start in 9.2 version of server (I try to warn but was kicked like street cat) where someone invite mixture from filesystem caches and SQL and assets in file system .... may add soma jam, juice and what you want!
Next FS start to use same omlette from SQL, ancient Flosam cache, bad understanded BTRFS, SQLITE and essence of NTFS bugs!
Short:
Server read from SQL DB but write in OS file system CACHES!!! And wait OS to return this to SQL DB....
Viewer read whats come..... But when you edit write direct to SQL. And may be not write data, may be write indexes, file names or idk whats write but..... Sometimes server decode and repair bugs.
(If your server not use any caches like my > destroy SQL DB imediatelly)
How to find more than few major bugs multiplied in all code of server and viewer?
How to find where is assets and to return from files to SQL???
May be must ask some priest or Native American shaman!!!