Neural Texture Upscaling?

[Terrapin2190]

Hey everybody! I'm new here

So, I've been installing all the various source ports scattered around the web lately and stumbled on some info about something called AI Neural Texture Upscaling. Apparently, it's something people can do themselves according to a post I found that describes it in detail.

A few texture packs have already been made for several games including Doom, Hexen, Heretic, Shadow Warrior, and Morrowind. And they look AMAZING when applied to the games.

There are a few games I would like to see this done for and I would love to try my hand at it if I can manage to figure out how it works.

Does anyone know if something like this could be done for Shadow Warrior? The current (and now somewhat dated) HRP pack truly is great, but it doesn't really reflect some of the original textures like the Neural Texture packs do in other games.

I am a quick study and have quite a bit of technical know-how, but I assume (from what little I have read so far) that it would require a decent processor and graphics unit. Two things that I lack with my current system.

I would imagine that this may be possible since the HRP pack is set up as a zip archive for use with Proasm and/or Hendricks's SWP. Would anyone be up for giving this a shot?



Here's the link that describes how Neural Upscaling works and provides some comparison pics in the comments:
www.resetera.com/threads/ai-neural-networks-being-used-to-generate-hq-textures-for-older-games-you-can-do-it-yourself.88272/

A link to the upscaling guide:
kingdomakrillic.tumblr.com/post/178254875891/i-figured-out-how-to-get-esrgan-and-sftgan

And a link to the Hexen Neural Textures Pack to see how impressive the results can be:
talk.dengine.net/discussion/2680/hexen-neural-texture-pack

[Ninjakitty]

I'd definitely be up to it
There seems to be a lot of setup, but I'm P sure my laptop would be more than capable, and I already have literally all the tiles saved as .PNG files. I'll tell ya when I get around to trying it ;3

[Ninjakitty]

So I'ma be honest, everything it's outputting that isn't just cobblestone looks kinda trash

[Ninjakitty]

It makes everything else look scaly and like it's tearing apart, or too blurry and it doesn't know what to do with square pixels

[Ninjakitty]

I think the real problem is it just doesn't like SW's palette or its way of color blending. I've heard the AI can be trained, but I have no idea how to do that.

[Robman]

Neat ... Going to have to keep my eye on this, I hope a new true to form texture pack comes of it!
Welcome to the forum!

[Ninjakitty]

Hecc yea

[Robman]

Holy sh*t... So, this is the AI doing its work? With some elbow grease this could be applied to every texture in the game?
Do some of the images have anomalies created by the AI?

[Ninjakitty]

Oct 4, 2019 19:29:03 GMT -5 Robman said:

Holy sh*t... So, this is the AI doing its work? With some elbow grease this could be applied to every texture in the game?
Do some of the images have anomalies created by the AI?

There are some things I need to find solutions to before this can be finalized, but yeah most textures actually look pretty good after I found a few new AI models and interpolated them

Right now I'm just tryna figure out how to export transparency instead of black and how to perform accurate pallete changes before upscaling (but I won't need to worry about the pallete changes if VoidSW comes out with the same true color pallete thingy ICD's port had) and also how to keep seamless tiles seamless

I've got some pretty nice character sprites now tho

(It's probably worth noting that I'm really only showing you the best cases, but honestly these just look so damn cool )

[Ninjakitty]

I'm currently in the process of finalizing and trying to find the best combination of AI model and anti-alias/no anti alias for every sprite

[Robman]

Some look sharp, some just appear to look softer? Anyway, so if I understand you correctly you're undertaking a full sprite upgrade???

[Ninjakitty]

I've only found one model that does a good job on these (except in the case of semi-muscular characters there's 2), so basically when it does too good of a job and makes everything too sharp, I gotta anti alias it (removes hard edges, but unfortunately can lose some detail in the process), but when it doesn't do a good enough job I can split the color channels up and process each one seperately (though this leads to some minor chromatic aberation and usually only works for gray or single color textures)

For each texture I do whatever combination of these I have to in order to get the Sprite to look as good as possible, but some of these might still look a little blurry (or hairy), so I apologise for that 

I believe the guardian, and stone lantern I had to use some anti-aliasing on, I just couldn't get it to look really good while sharp, but I believe they're still much better looking than the original sprites. I can assure you, tho, this thing loves muscle and the sumo and ripper look amazing. 

[Ninjakitty]

And yea, I'm upscaling this whole game 

[Ninjakitty]

So I spent some time trying to get some of the blurry stuff to look sharper, found out I didn't know what the interpolation value meant so I thought I was interpolating like just a little bit of the blurry smooth model, not like almost 100%

I've got a lot of nice looking sharp stuff now

[Robman]

There we go, very nice

Once you've discovered the recipe for good conversions, next will be the full sprite sheet undertaking? Will it be an arduous task of 1 by one editing or can the beast do it in bulk?

[Ninjakitty]

Well, I gotta look through every sprite and get the correct combinations to make it look good, but also I've got a small problem: some of the sprites are partially transparent where they shouldn't be, and not fully transparent where they should be, not really sure what's causing this (although it's not very noticeable with enhanced alpha channel turned off in video settings). If I make transparency all or nothing, it sort of fixes this, but the edges of all the sprites look kinda hard. So I'll probably release two versions when I'm done: one with threshold alpha, and the other with softer edges that you need to turn off enhanced alpha.

Overall, though, I'm really liking how this is turning out and I hope to have this completed as soon as possible

I probably would've had this thing done by now if I had more free time

[Ninjakitty]

I'm not sure I understood your question, but to answer as best I can, basically it can do as many as I want it to (though it does take time) at once.

But my current work flow is:

I take a batch of tiles usually less than 100 (one batch consists of only sprites or only textures, because seamless tiling for textures can't be used on the sprites or it'll mess them up) fatality model, put all the results in the Final folder

Next, I try different combinations. For each of the combinations I check if the new combination sprite is better than the one currently in Final folder, if it is I replace it

Fatality model is generally pretty good, but if it doesn't do a good job, I interpolate it 0.1 or 0.2 with scalenx (this blurs it a very tiny bit, but gets rid of any details that are too harsh. I almost always have to do this with characters)

If Fatality isn't working (had a case where it was turning stone textures into hair for some reason), then I try the manga109 model, which is pretty good just doesn't fill in as many details

And of course then I also try with and without anti-aliasing and splitting RGB

And then occasionally I'll notice a weird artifact on a sprite and go into GIMP and manually edit it out

It's definitely quite the process, but I'm not settling for low quality anything when it comes to Shadow Warrior

[Terrapin2190]

Holy WOW those output textures look amazing!
So, the 2D sprites are different than the textures (or won't run through the neural algorithm correctly)? Or are these the voxel textures you're working on?

That would be super cool, having an option of pixelated or smooth-edged sprites. Yep. I'm hyped.

[Ninjakitty]

Ok, I've found the solution to the transparent sprite problem. IEU's alpha tiles it creates to merge with color tiles to get the final tile have color for some reason

They supposed to be black an white

Yanno, not bRoWn

How the hecc u get brown from black and white?

This make me agrey

But anyway, I just gotta intercept the alpha tiles before the program merges them, and fix them (grayscale, then posterize to allow only 2 colors), and then allow it to merge

Unfortunately, that takes even longer than the process I currently use to output tiles, but I really think these sprites look a lot better, I can guarantee it'll be worth the wait

[Robman]

Oct 10, 2019 20:20:46 GMT -5 Ninjakitty said:

I can guarantee it'll be worth the wait

Exciting

[Ninjakitty]

Ok so there's basically nothing I can do to fix this, but I still kinda want to get opinions.

Does the black outline bother or distract you at all?

[Robman]

Well, now that you mention it.....sure there's no work around?

[Ninjakitty]

There's like a 5 step process that I'd have to perform on each and every sprite (including making many copies of that sprite) and it would get rid of small details in the outline of the transparency, so I'm afraid this is the best I can do. At least until IEU gets a smarter and more accurate way to define alpha

It's not too noticeable in my opinion, like you definitely wouldn't notice it unless you stopped and looked really hard, but it still kinda bothers me and I wish there was a feasible way to fix it

On the bright side tho, I can't say there are any sprites or textures I've upscaled that I genuinely don't think look somewhat better than the originals

[Ninjakitty]

Happy Halloween!

[Terrapin2190]

Sorry for the late reply! I haven't checked in for a while.
I don't really mind the black outline. It's hard to tell without seeing it in-game, but I doubt it would really be noticeable, like you said. Might make things looks 'slightly' cartoon-ish maybe, but that could just add to the charm of the pack.

Things are looking good!

[Terrapin2190]

You might find some info here useful. Talks about Non-Duke3D ESRGAN Upscaling.

forums.duke4.net/topic/10406-the-esrgan-ai-upscale-non-duke-thread/page__st__90

[Terrapin2190]

Hey, been a while (again lol, almost a year!). Ninjakitty, were you still working on the upscale pack or end up dropping it with the work needing to be done on the models? Was wondering if it would be possible to compile the textures separately as a release without models and/or voxels as a sort of alpha pack. I would love to see what these textures look like in-game! But, mostly just checking in to say hey. I go on a modding spree every now and then, looking up HD textures and remastered soundtracks for various things. Anyways, hope things are well! Or as well as well can be with everything that's happened this year.

[Ninjakitty]

Not going to get any upscales worth looking at without a customized AI model for sprites with SW's palette. So I've been trying to make one for a while now. It's not going so well.

Pheedreeke released his own upscale pack for SW on ModDB already. It's pretty good but it does look a little cartoony imo, and it doesn't get rid of the paletting on the sprites.

[Robman]

Sept 25, 2020 15:19:21 GMT -5 Ninjakitty said:

Pheedreeke released his own upscale pack for SW on ModDB already. It's pretty good but it does look a little cartoony imo, and it doesn't get rid of the paletting on the sprites.

As a casual observer to this topic, if you have the time/ambition/knowhow to release a superior pack to Pheeedreeke's .. I'd say do it up.

[Ninjakitty]

Yeah, I'm trying. Making an AI model specific to SW sprites is my first step in trying to make a full HD (4x res with no palette) upscale pack.