Remastering old anime

[deama]

So I've been doing some remastering using AI algorithms on an old anime, hajime no ippo, a boxing anime and have just finished it.
The type of remastering I did was to enhance the resolution, clean up artifacts/noise, and frame interpolation.
I've made my own subreddit, in there I've got more info, as well as some FAQs.
Check it out!

r/InterpolateAndEnhance

For now I'll be doing anime, the next series will be the old hunter x hunter. At some point I want to try out live action content, specifically action stuff like martial arts movies, or straight up action, but for now the interpolation algorithms aren't quite good enough, or well actually they kinda are, at least the high end ones, but good luck trying to interpolate anything with those on anything other than a RTX 3090.

 

[Dan]

Hajime No Ippo has been released on blu ray in the US hasn’t it. Wouldn’t that be really good quality anyway?

 

[Lordhippos]

Pretty much all tv/film is 24 fps, interpolation is adding additional frames that weren't in the original source by using algorithms.

In the example above you can see that in the one punch man it's 48fps instead of 24.

The result can be quite jarring for some people, a bit like when you enable motion enhancements on your TV, you can get soap opera effect, everything appears to move faster/more smoothly, but coming from 24fps content it can certainly look unnatural.

As I have an OLED TV, any slow panning shots at 24fps on any content can appear to be juddery, interpolation does fix that but it also changes the cadence of the original frames at other times.

 

[IdiomaticLynx]

Interpolation never really looks good to me and I'm not just referring to the "soap opera effect", but mainly the artefacts it introduces, especially in animation. There are just some inherit problems that simply can't be overcome.

Especially when the animation is on 3s (or even on 4s) the algorithms tend to struggle. They only tend to consider frames where the key frame changes to actually contain movement/changes. As a result, all the additional temporal resolution you get is focussed on these frame transitions, causing a weird pacing where it's not consistently smooth.

Another big problem are the more stylistic animation techniques. For example grass, hair or cloth moving rapidly in the wind. Each frame places the strands of hair or grass in a semi-random spot, creating a chaotic/unruly effect. But when interpolating, sometimes a smooth transition is created between the frames, which actually creates the exact opposite effect. Smear frames also don't really fare that well, but they are often already in a high-motion moment, that most artefacts don't really become noticeable. In general, I think the ideal output would be more frames as if the animators drew more frames, which is sadly not what these algorithms focus on.

And ultimately you're also left with the non-animation specific problems. Personally I tend to notice it more with small/thin objects, especially if they are close together (e.g. iron bars, corn fields, fences, etc) where it sometimes confuses two instances and creates a weird merged in-between frame. But to be fair, the existing solutions (even the ones built into TVs) have come a long way in this regard.

The only time where I actually prefer it, is panning shots. I also have an OLED TV and as @Lordhippos mentioned, slow panning shots at 24fps are not the nicest to look at. It's also these panning shots that can be interpolated without any real artefacting. Even older TVs with pretty bad interpolation algorithms do tend to manage panning shots without too much trouble.

But at the end of the day, it's just not worth it to me. I'd much rather watch something close to the original and see any shortcoming, be it resolution or framerate as part of its charm.

 

[ayase]

I can't actually see your videos @deama since they either don't load or when they do it asks me to sign up for an account, which I won't be doing, but I'm not really a fan of added interpolation/motion smoothing when it comes to anime. I feel it makes traditional animation look like flash tweening which, as @Lordhippos has already pointed out, is essentially what it is: using software to auto-generate inbetween frames. Imo this has the effect of making it look like a more cheaply produced show since that's its usual use of software tweening, to save time and money by not having to animate things by hand.

I do however quite like 60fps when it comes to live action, since it solves a serious issue I have with a lot of fast action scenes in modern live action film and TV that I often can't see what the hell is going on and it just looks like a blurry mess.

 

[Lordhippos]

Higher framerates on live action stuff is something I'd like to see in the future, as well as more IMAX type shots on films to remove letterboxing. For anime it would be a bigger problem though as you'd need more frames to be drawn/animated, which is increasing the workload a lot!

 

[Dan]

Oh I’ve totally not read/understood this correctly. God I hate higher frame rates. That stupid true motion they put on TVs does something similar and it makes me want to vomit I remember when I first got my TV and it made Harry Potter look like Coronation Street, I really don’t get why people leave it on. I’m sure it’s a company ploy to get us to move away from 24fps... no thanks

 

[Lordhippos]

I don't know why the film industry settled on 24 fps for live action a long, long time ago, it can feel quite "cinematic" at times because it's actually a lot slower than real life would be if viewing the same thing in person.

I think 60 fps for cinema could work but it would need to be recorded and shown 1:1 at the same 60 fps, so not using interpolation or tricks. I believe that a couple of films have 60 fps versions but I've not checked them out (Hobbit was one I think). Truemotion and other similar TV based interpolation techniques are definitely adding frames that weren't present in the source which is the cause of things like SOE and artifacts. For the most part nothing wrong with 24 fps content to me, very rarely see noticeable issues with it.

If you play any games consoles or pc games, then they will always feel a bit slow at say 30 fps, but can render in a variable fashion, 60 fps or above is normally preferred on games.

 

[Dan]

If you play any games consoles or pc games, then they will always feel a bit slow at say 30 fps, but can render in a variable fashion, 60 fps or above is normally preferred on games.

Yeah games are definitely better at 60fps or even higher but I don’t like movies higher than 24fps. Just doesn’t look right. Also sports having a higher frame rate is really good too.

 

[deama]

Interpolation never really looks good to me and I'm not just referring to the "soap opera effect", but mainly the artefacts it introduces, especially in animation. There are just some inherit problems that simply can't be overcome.

Especially when the animation is on 3s (or even on 4s) the algorithms tend to struggle. They only tend to consider frames where the key frame changes to actually contain movement/changes. As a result, all the additional temporal resolution you get is focussed on these frame transitions, causing a weird pacing where it's not consistently smooth.

Another big problem are the more stylistic animation techniques. For example grass, hair or cloth moving rapidly in the wind. Each frame places the strands of hair or grass in a semi-random spot, creating a chaotic/unruly effect. But when interpolating, sometimes a smooth transition is created between the frames, which actually creates the exact opposite effect. Smear frames also don't really fare that well, but they are often already in a high-motion moment, that most artefacts don't really become noticeable. In general, I think the ideal output would be more frames as if the animators drew more frames, which is sadly not what these algorithms focus on.

And ultimately you're also left with the non-animation specific problems. Personally I tend to notice it more with small/thin objects, especially if they are close together (e.g. iron bars, corn fields, fences, etc) where it sometimes confuses two instances and creates a weird merged in-between frame. But to be fair, the existing solutions (even the ones built into TVs) have come a long way in this regard.

The only time where I actually prefer it, is panning shots. I also have an OLED TV and as @Lordhippos mentioned, slow panning shots at 24fps are not the nicest to look at. It's also these panning shots that can be interpolated without any real artefacting. Even older TVs with pretty bad interpolation algorithms do tend to manage panning shots without too much trouble.

But at the end of the day, it's just not worth it to me. I'd much rather watch something close to the original and see any shortcoming, be it resolution or framerate as part of its charm.

Did you look at some of the examples I had? I have an OLED myself and hate the motion smoothing it has, but the AI algorithm I use works pretty well.

I can't actually see your videos @deama since they either don't load or when they do it asks me to sign up for an account, which I won't be doing, but I'm not really a fan of added interpolation/motion smoothing when it comes to anime. I feel it makes traditional animation look like flash tweening which, as @Lordhippos has already pointed out, is essentially what it is: using software to auto-generate inbetween frames. Imo this has the effect of making it look like a more cheaply produced show since that's its usual use of software tweening, to save time and money by not having to animate things by hand.

I do however quite like 60fps when it comes to live action, since it solves a serious issue I have with a lot of fast action scenes in modern live action film and TV that I often can't see what the hell is going on and it just looks like a blurry mess.

I made the first 3 eps free as the later ones require the LBC currency which you can get by signing up and using the website daily.
If they don't load then might be internet issues, either on your end or the website's as unfortunately I made a mistake when rendering these.
I basically originally rendered them in x265, but the website doesn't support those, so I had to re-render them in x264 which increased the size too much, I'll fix it in the next series I'll be doing.
The reason I can't just render the sources to x264 is cause I don't have the raws anymore as each episode's raws are like 100GB and I only got 2TB storage, 500GB of that is used for general storage.

Try the one punch man janos fight, that one is just an example and it's small too in comparison to the others, should load.
You can also just download them and watch them later, the button is next to the share button.

I don't know why the film industry settled on 24 fps for live action a long, long time ago, it can feel quite "cinematic" at times because it's actually a lot slower than real life would be if viewing the same thing in person.

I think 60 fps for cinema could work but it would need to be recorded and shown 1:1 at the same 60 fps, so not using interpolation or tricks. I believe that a couple of films have 60 fps versions but I've not checked them out (Hobbit was one I think). Truemotion and other similar TV based interpolation techniques are definitely adding frames that weren't present in the source which is the cause of things like SOE and artifacts. For the most part nothing wrong with 24 fps content to me, very rarely see noticeable issues with it.

If you play any games consoles or pc games, then they will always feel a bit slow at say 30 fps, but can render in a variable fashion, 60 fps or above is normally preferred on games.

Originally the reason they stuck with lower fps for movies and content was due to cost, I believe there was one guy arguing that it would have been much better if they went with 48fps as the standard, but they didn't listen to him cause they wanted to save cost and the audience didn't seem to (sadly) mind it much.

There's about two main reasons why higher fps content looks bad on live action if you film it with a high fps camera. One of them is due to the directors/editors/cameramen just not used to actually filming in high fps, e.g. the lighting will look sometimes completely different at higher fps because of how the shutter speed works on cameras, among many other things. The other reason being motion and CGI. You see, lower fps helps hide the "smoothness" of CGI because in reality, nothing except maybe robots move that smooth, and so it makes it very unrealistic, but at lower fps it judders more and thereby tricks the brain into making it more real, at the cost of more processing power from the brain. Humans and all things don't move smoothly, if you look closely sometimes there are these imperfections in our movements, or shaking, or just randomness, this isn't emulated in CGI, so when they put CGI next to live action it just looks very, very weird.

As an example, if you can get a hold of the gemini man's 60fps release, I find the general scenes to look pretty good, sure the lighting seems off, but ignore that for now. When it comes to the action, if you look real close, you'll actually be able to see that they used some sort of CGI in the fights, like, they CGI'd the actors, due to this the fights end up looking way too unnaturally smooth because they're using CGI for the fights lol. You could probably not tell that they were using CGI if it was 24fps because of not just it not being as smooth, but also because your brain gets less information, meaning it'll just be harder to recognize it unless you pay extra, extra attention.

 

[deama]

Hajime No Ippo has been released on blu ray in the US hasn’t it. Wouldn’t that be really good quality anyway?

Unfortunately they used a simple upscale, the originals will always be 480p and for the bluray they used simple upscaling, nothing fancy like mine. I think they might have done some colour correction as well? Anyway, my one also does interpolation.

 

[ActionFaust]

For anime I definitely don't like the idea of adding frames as it often ruins the creator's vision. Scenes will often feature a limited amount of frames to convey the tone of a scene more effectively.

For example this cut starting at 15 seconds just wouldn't be as effective if it used more frames

 

[Lordhippos]

@deama You should consider posting before and after samples to youtube, whatever site you are linking them from is very slow. Youtube may hit you with a copyright thing though, not sure. If you post it as a private link you might get away with it, if the duration is short.

 

[IdiomaticLynx]

Did you look at some of the examples I had? I have an OLED myself and hate the motion smoothing it has, but the AI algorithm I use works pretty well.

I did take a look and honestly it doesn't really change my opinion. I only looked at the OPM fight and the start of episode one of Hajime no Ippo. The OPM fight is mostly animated on 2s and thus fares pretty well, and as expected the panning shots are buttery smooth. But there are still too many artifacts for me to even consider watching anime like this. The following I observed in my initial viewing without slowing down or pausing (you'll have to believe me on that ):

• At 2:21 you see the rocks merge/morph in a weird way
• At 3:58 the lines seem to "move" instead of "flash"
• At 4:10 when Saitama walks off screen it's clear something strange happens with his head

There were more points where things felt off, but I was only able to discern why/what after slowing it down when watching it a second time. While those might've "fooled" me, it's also part of this uncertainty whether or not the interpolation is "correct" that makes it highly unlikely I would ever watch animation in this manner.

As for Hajime no Ippo it clearly shows my main gripe of using interpolation algorithms with animation. All the movement seems to constantly speed up and slow down. While I haven't seen Hajime no Ippo in it's original form, it's undoubtedly caused by interpolation of animation that is on 3s (or 4s).
(Also, it seems there is an animation mistake where the cardboard box flap just straight up disappears. )

The example @ActionFaust linked is a good example where any interpolation algorithm will most likely fail to retain what makes that cut work. Personally I'm quite curious when we'll see algorithms that focus on animation instead of being general purpose. Instead of "interpolating" they might even attempt to "redraw" cuts from being on 3s to 2s. Undoubtedly they'll be rubbish, but I'm curious nonetheless

Anyway, it's really nice to see that you take it seriously and put in the effort. Going by the amount of anime clips on YT that are "60fps", there seems to be quite a lot of interest in it, and sadly also a lot of lousy/atrocious interpolations. From what I've seen, your approach is comparatively a lot better. But at the end of the day, it's simply not for me.

 

[deama]

@deama You should consider posting before and after samples to youtube, whatever site you are linking them from is very slow. Youtube may hit you with a copyright thing though, not sure. If you post it as a private link you might get away with it, if the duration is short.

I've already tried it a bunch, even tried cutting the music out, it takes it down everytime, no idea how other people get away with it lol.
You can try downloading it, the button is next to the share button, it may take time but won't stutter when you finally watch it.

I did take a look and honestly it doesn't really change my opinion. I only looked at the OPM fight and the start of episode one of Hajime no Ippo. The OPM fight is mostly animated on 2s and thus fares pretty well, and as expected the panning shots are buttery smooth. But there are still too many artifacts for me to even consider watching anime like this. The following I observed in my initial viewing without slowing down or pausing (you'll have to believe me on that ):

• At 2:21 you see the rocks merge/morph in a weird way
• At 3:58 the lines seem to "move" instead of "flash"
• At 4:10 when Saitama walks off screen it's clear something strange happens with his head

There were more points where things felt off, but I was only able to discern why/what after slowing it down when watching it a second time. While those might've "fooled" me, it's also part of this uncertainty whether or not the interpolation is "correct" that makes it highly unlikely I would ever watch animation in this manner.

As for Hajime no Ippo it clearly shows my main gripe of using interpolation algorithms with animation. All the movement seems to constantly speed up and slow down. While I haven't seen Hajime no Ippo in it's original form, it's undoubtedly caused by interpolation of animation that is on 3s (or 4s).
(Also, it seems there is an animation mistake where the cardboard box flap just straight up disappears. )

The example @ActionFaust linked is a good example where any interpolation algorithm will most likely fail to retain what makes that cut work. Personally I'm quite curious when we'll see algorithms that focus on animation instead of being general purpose. Instead of "interpolating" they might even attempt to "redraw" cuts from being on 3s to 2s. Undoubtedly they'll be rubbish, but I'm curious nonetheless

Anyway, it's really nice to see that you take it seriously and put in the effort. Going by the amount of anime clips on YT that are "60fps", there seems to be quite a lot of interest in it, and sadly also a lot of lousy/atrocious interpolations. From what I've seen, your approach is comparatively a lot better. But at the end of the day, it's simply not for me.

To each his own I suppose. There are better algorithms out there that I wish I could use, but sadly I need at least a 3090 to try anything other than a clip on those lol.

 

[Neil.T]

I don't know why the film industry settled on 24 fps for live action a long, long time ago

I can answer this one, actually. Even better, I found the actual Wikipedia article where I first read the answer.

From the Wikipedia article on Frame rate:

Sound films

When sound film was introduced in 1926, variations in film speed were no longer tolerated, as the human ear is more sensitive than the eye to changes in frequency. Many theaters had shown silent films at 22 to 26 FPS, which is why the industry chose 24 FPS for sound films as a compromise. From 1927 to 1930, as various studios updated equipment, the rate of 24 FPS became standard for 35 mm sound film.​

Frame rate - Wikipedia

en.m.wikipedia.org

[Under "Film and video"]