TR Forums

VideoReDo has released V6 with x265 HEVC (BETA Test Trial).
I tried that and then Handbrake.

Both only used about 1\3 of my 3930 6core/12thread cpu. (4 threads or so)
So a quad core is all anyone is going to need? Is this normal?

The simplest x265.TS output was almost real time at 27fps.
Up the quality with a SLOW to VERY SLOW preset and it is ten times slower. 2.5fps

Handbrake has trouble scaling beyond 6 cores, but it should scale pretty well from 4 to 6.

https://handbrake.fr/docs/en/latest/tec ... mance.html

Sometimes Handbreak doesn't max-out my Core i7 4790k, depending on the film being encoded, so it could just be varying the amount of threading based on complexity of source content? I would try multiple movies to see how performance is?

jmc2 wrote:

VideoReDo has released V6 with x265 HEVC (BETA Test Trial).
I tried that and then Handbrake.

Both only used about 1\3 of my 3930 6core/12thread cpu. (4 threads or so)
So a quad core is all anyone is going to need? Is this normal?

The simplest x265.TS output was almost real time at 27fps.
Up the quality with a SLOW to VERY SLOW preset and it is ten times slower. 2.5fps

For Handbrake, if you want faster encodes you have to reduce the quality. I use 2-pass encoding with turbo enabled for the 1st pass and an encoder preset of "fast" when I have a large batch and the source material is good quality. When the source is poor quality I turn the video preset down to medium and turn on denoise.

defaultluser wrote:

Handbrake has trouble scaling beyond 6 cores, but it should scale pretty well from 4 to 6.

https://handbrake.fr/docs/en/latest/tec ... mance.html

Sometimes Handbreak doesn't max-out my Core i7 4790k, depending on the film being encoded, so it could just be varying the amount of threading based on complexity of source content? I would try multiple movies to see how performance is?

Handbrake can scale well beyond 6 cores. The people that say this aren't using the right settings. I have a 16 core 7960x at 4GHz in my media server/encoding workstation and I regularly see usage hover near 80% on the 1st pass (swings between 75-85), dropping down to around 60% on the 2nd pass. That 1st pass runs at about 650-750 FPS with the 2nd being 300-350.

Here's proof: note that there is a 3rd pass in this particular job so "pass 2" is actually the 1st encoding pass

[quote=[/quote]
Handbrake can scale well beyond 6 cores. The people that say this aren't using the right settings. I have a 16 core 7960x at 4GHz in my media server/encoding workstation and I regularly see usage hover near 80% on the 1st pass (swings between 75-85), dropping down to around 60% on the 2nd pass. That 1st pass runs at about 650-750 FPS with the 2nd being 300-350.[/quote]

That is interesting, on my first gen 16 core threadripper, x264 would only use 30 something %. So 10-11 cores(threads?).
Have not tried HEVC on the threadripper yet.

On my 6 core, x264 and a HD file can sit at 95-100% for one pass. But HEVC was 30+% both passes.
I tend to convert dvd level .mpgs to x264.mp4s so not a demanding job.

If you know what the "magic" switch is to get 80% use out of 16 core please let me know.

Thanks,
jmc

The number of threads involved is dependent on the codec and the source material. Encoding an old DVD won't scale very well no matter what you try to do to it, but encoding a 4k video will.

In general, shouldn't your encoding process be tuned towards the best visual results rather than using up all the threads from your leet CPU?

In the reencoding of drone footage I've been doing recently I've been using x265 via an ffmpeg interface. No problem railing my cores but I'm doing it on a quad core so that's not as hard to do.

My question is about the first pass/second pass operation. I've seen some online sources saying that's an older methodolgy that's been replaced by just doing everything in one pass. Is that true or is a two pass process still the best way to encode the video either for quality or performance?

Since streaming is the most common use case these days I think two-pass encoding is just falling out of favor a bit. It takes a long time which is often a deal breaker for live streaming, and unless you put a ceiling on the bitrate it can go past your target bandwidth and cause performance issues. Setting the CRF somewhere in the 17-28 range and just doing a single pass is usually "good enough", but as far as I know the only way to really know what's going on with your source material is to do a first pass, and then do the encode. I haven't looked into how things work with VP9 or AV1, so maybe there's some trick there, but for playing local content using x264 and HVEC I'd probably stick with two-pass and just wait a little bit longer.

Found this link detailing the various switches related to threading with x265.
https://x265.readthedocs.io/en/default/threading.html

x265 via Handbrake loads up all 8 cores on my SB-E Xeon and 9900k just fine, but certain options might be relevant to Threadripper as AFAIK it does make use of multiple NUMA nodes.

Thread Pools

x265 creates one or more thread pools per encoder, one pool per NUMA node (typically a CPU socket). --pools specifies the number of pools and the number of threads per pool the encoder will allocate. By default x265 allocates one thread per (hyperthreaded) CPU core on each NUMA node.

Redocbew wrote:

The number of threads involved is dependent on the codec and the source material. Encoding an old DVD won't scale very well no matter what you try to do to it, but encoding a 4k video will.

In general, shouldn't your encoding process be tuned towards the best visual results rather than using up all the threads from your leet CPU?

I do go for quality first. The "slow" to "very slow" (x264) handbrake preset drops the fps from 7x real time to 4-5x real time and that is ok.

HEVC goes from almost real time(27fps) to 1/10 real time (2.5fps)(very slow preset) and that is not ok.
And seeing only 1/3 cpu use, gotta hope there is something that can be done to speed things up to an acceptable level by using more of the cpu.
With HEVC finally coming to VideoReDo this is the first time I've explored x265 and have lots to learn.

chuckula wrote:

In the reencoding of drone footage I've been doing recently I've been using x265 via an ffmpeg interface. No problem railing my cores but I'm doing it on a quad core so that's not as hard to do.

My question is about the first pass/second pass operation. I've seen some online sources saying that's an older methodolgy that's been replaced by just doing everything in one pass. Is that true or is a two pass process still the best way to encode the video either for quality or performance?

VideoReDo seems to recommend the one pass CRF now. I've played with it and have just stuck with my old two pass method.
It is frustrating to change settings and not see any difference. Wish there was a program that could analyze the video so I could KNOW that this tweak or that tweak made a positive difference. Add all the right unseeable positive tweaks together and maybe I could see an improvement.
https://www.videoredo.com/en/index.htm
https://videoredo.net/msgBoard/index.ph ... oad.36740/ (very,very beta)


http://www.oodlestechnologies.com/blogs ... -CRF-Guide
https://forum.videohelp.com/threads/378 ... Frame-(CRF)
http://slhck.info/video/2017/02/24/crf-guide.html

techguy wrote:

[quote=

Handbrake can scale well beyond 6 cores. The people that say this aren't using the right settings. I have a 16 core 7960x at 4GHz in my media server/encoding workstation and I regularly see usage hover near 80% on the 1st pass (swings between 75-85), dropping down to around 60% on the 2nd pass. That 1st pass runs at about 650-750 FPS with the 2nd being 300-350.[/quote]

So you are using other video processing options to get the high cpu use?

I've used (with x264) some other options that did jump the cpu use on soft fuzzy faded digitized tapes...
Trim the black pillars off
Enhance contrast and color.
But the most important option that I've found is "Contour" in TMPGEnc. It is different then just sharpening and with soft fuzzy tapes it makes a big improvement. A lot of the soft fuzzy is just gone. Now I am losing the nice 5x real time encoding but it is still encoding better then real time and worth it for old fuzzy tapes!
No idea what the "contour" setting is but I love it.

jihadjoe wrote:

Found this link detailing the various switches related to threading with x265.
https://x265.readthedocs.io/en/default/threading.html

x265 via Handbrake loads up all 8 cores on my SB-E Xeon and 9900k just fine, but certain options might be relevant to Threadripper as AFAIK it does make use of multiple NUMA nodes.

Thanks for the link. Lots to learn.
It's painful to see less then real time encoding AND see the cpu barely being used.I probably won't be switching over to HEVC till I can just stick a flashstick into a random tv and have it play the HEVC file.

jmc2 wrote:

That is interesting, on my first gen 16 core threadripper, x264 would only use 30 something %. So 10-11 cores(threads?).
Have not tried HEVC on the threadripper yet.

On my 6 core, x264 and a HD file can sit at 95-100% for one pass. But HEVC was 30+% both passes.
I tend to convert dvd level .mpgs to x264.mp4s so not a demanding job.

If you know what the "magic" switch is to get 80% use out of 16 core please let me know.

Thanks,
jmc

Encoding speed is affected by numerous factors so I can't guarantee that my settings will work for you, but you can give them a shot and see if you like the results.

Here's my Video settings tab:


And the filters I use:


I have a different profile I use to extract more picture quality from lower quality source material but it sounds like you're looking for more speed now so those settings won't help.

I'm curious to see your results. I considered a Threadripper system when I upgraded to X299 but early Handbrake results from reviews weren't promising so I went with Skylake X instead.

Redocbew wrote:

The number of threads involved is dependent on the codec and the source material. Encoding an old DVD won't scale very well no matter what you try to do to it, but encoding a 4k video will.

In general, shouldn't your encoding process be tuned towards the best visual results rather than using up all the threads from your leet CPU?

This is just plain wrong.

The screenshot I showed earlier with 80% CPU utilization and over 700 FPS was a DVD.

It's all in the settings folks.

jmc2 wrote:

Redocbew wrote:

The number of threads involved is dependent on the codec and the source material. Encoding an old DVD won't scale very well no matter what you try to do to it, but encoding a 4k video will.

In general, shouldn't your encoding process be tuned towards the best visual results rather than using up all the threads from your leet CPU?

I do go for quality first. The "slow" to "very slow" (x264) handbrake preset drops the fps from 7x real time to 4-5x real time and that is ok.

HEVC goes from almost real time(27fps) to 1/10 real time (2.5fps)(very slow preset) and that is not ok.
And seeing only 1/3 cpu use, gotta hope there is something that can be done to speed things up to an acceptable level by using more of the cpu.
With HEVC finally coming to VideoReDo this is the first time I've explored x265 and have lots to learn.

The video encoder preset is the single most important slider in terms of speed. I've tried the various settings here and compared the results - anything to the right of "fast" results in a huge drop off in encoding rate. In my findings, a 2-pass job with a high enough bitrate achieves quality results that are indistinguishable from a single pass, slow encoder preset job.

Give it a shot and see how you like the results. Your source material may differ from mine by enough that the slower encoder preset can produce better quality results. Keep in mind that I don't configure Handbrake to produce smaller files than the source material so if your storage is limited you may not want to use the same high bitrates that I do, though that will definitely effect quality with a faster preset.

jmc2 wrote:

HEVC goes from almost real time(27fps) to 1/10 real time (2.5fps)(very slow preset) and that is not ok

The bit about you seeing relatively low CPU usage does make it sound like something isn't right, but I hope you're not expecting HEVC to behave exactly the same way that h.264 does.

techguy wrote:

Redocbew wrote:

The number of threads involved is dependent on the codec and the source material. Encoding an old DVD won't scale very well no matter what you try to do to it, but encoding a 4k video will.

In general, shouldn't your encoding process be tuned towards the best visual results rather than using up all the threads from your leet CPU?

This is just plain wrong.

The screenshot I showed earlier with 80% CPU utilization and over 700 FPS was a DVD.

It's all in the settings folks.

I'll rephrase the point just to make it a bit clearer. On a 12 core CPU, the fact that an encoder may be using 12 cores instead of 8 or 10 does not necessarily mean the encoder is doing a "better job". Handbrake doesn't even let you set the number of threads explicitly, and clearly that's not a problem. It just lets the codec handle that. I suppose it's still possible to limit effective scaling just by providing the encoder with really oddball source material, but in general that seems unlikely.

Again, that doesn't mean all of your various "settings" aren't doing anything, but they're not affecting how your encode scales.

Redocbew wrote:

techguy wrote:

Redocbew wrote:

The number of threads involved is dependent on the codec and the source material. Encoding an old DVD won't scale very well no matter what you try to do to it, but encoding a 4k video will.

In general, shouldn't your encoding process be tuned towards the best visual results rather than using up all the threads from your leet CPU?

This is just plain wrong.

The screenshot I showed earlier with 80% CPU utilization and over 700 FPS was a DVD.

It's all in the settings folks.

I'll rephrase the point just to make it a bit clearer. On a 12 core CPU, the fact that an encoder may be using 12 cores instead of 8 or 10 does not necessarily mean the encoder is doing a "better job". Handbrake doesn't even let you set the number of threads explicitly, and clearly that's not a problem. It just lets the codec handle that. I suppose it's still possible to limit effective scaling just by providing the encoder with really oddball source material, but in general that seems unlikely.

Again, that doesn't mean all of your various "settings" aren't doing anything, but they're not affecting how your encode scales.

If 2 PCs running Handbrake produce the same output (from the same source material) and PC 2 gets the job done faster because it has 16 cores while PC 1 has 12 cores, what do you call that?

I would say that indicates the encode engine is capable of "scaling" beyond 12 cores. I.E. "better scaling" with a greater core count.

Let me add some additional perspective. When I first built this machine I purchased a 7900x and left it stock (no overclock, no de-lid). I had an opportunity to upgrade to a 7960x for a reasonable price so I re-checked my research and found that while most review outlets indicate little to no "scaling" from 7900x to 7960x in most Handbrake workloads, some outlets did in fact show scaling. When I compared the source material and settings in use I found that some reviewers had the wrong methodology for comparing Handbrake encoding speed. If you want speed, you have to use a faster video encoder preset. Long story short, after the upgrade I've seen my encode times speed up by somewhere between 30-40% on average.

That's not the same example. The point(again) is that it's beyond your control using Handbrake, so there's no point in telling the OP to look for some magic knob or dial that'll help improve scaling because there isn't one. The problem, whatever it is, has to be somewhere else.

Redocbew wrote:

That's not the same example. The point(again) is that it's beyond your control using Handbrake, so there's no point in telling the OP to look for some magic knob or dial that'll help improve scaling because there isn't one. The problem, whatever it is, has to be somewhere else.

This isn't a subjective matter where we argue about theories, these claims are testable, the outcomes are measurable. I proved your earlier statements to be patently false with proof of both the scaling capabilities of Handbrake as well as the resultant performance, in the form of the screenshots posted earlier.

But I have better things to do than argue on forums. Hopefully the OP gets what he's looking for.

Redocbew wrote:

That's not the same example. The point(again) is that it's beyond your control using Handbrake, so there's no point in telling the OP to look for some magic knob or dial that'll help improve scaling because there isn't one. The problem, whatever it is, has to be somewhere else.

The whole "Extra Options" box in Handbrake's Video encode settings is specifically there so advanced users can tweak their encode settings. Not everything has to be manipulated with a GUI dropdown or slider.

jihadjoe wrote:

Redocbew wrote:

That's not the same example. The point(again) is that it's beyond your control using Handbrake, so there's no point in telling the OP to look for some magic knob or dial that'll help improve scaling because there isn't one. The problem, whatever it is, has to be somewhere else.

The whole "Extra Options" box in Handbrake's Video encode settings is specifically there so advanced users can tweak their encode settings. Not everything has to be manipulated with a GUI dropdown or slider.

Those option change what the encoding is doing. It's not checking a "do the same encode but use more cores".

Usacomp2k3 wrote:

those option change what the encoding is doing. It's not checking a "do the same encode but use more cores".

I think this gets at the heart of the miscommunication here.

The same encode settings have no slider for "more cores please" nor do they scale the same in relation to core count. This was show by defaultluser's link to handbrake itself: Broadwell with 3.5x cores was only *twice* as fast as Westmere (with only a 12% clock advantage, but no AVX!) in "Very Fast 1080p30" x264.

As the results show, the presets scaled better (in relation to cores[but also architecture--there are some confounding factors like the aforementioned AVX support]) as you went higher in quality. By the time they compared "Super HQ", it was almost a 4x speed improvement.

So, in relation specifically to HEVC with the -same- core count, Yes, "Very Fast" is nearly 3x faster than "Super HQ", and "Fast" is more than 2x faster. "Quality" makes a big difference.

I don't think anyone disputes that.

Nor do I think anyone particularly disputes that same people get no benefit or honestly just can't tell between "Fast" and "Super HQ" in HEVC quality. I use "Fast" HEVC myself.

---

But just because someone can't tell the different between the encoding options doesn't really mean they are same. And just because some of the preset scale better (or even encoders---x265/HEVC is generally better in this regard than x264) doesn't mean that core counts will always influence your results positively or meaningfully.

As I demonstrated, from handbrake's discussion, there can be incredible differences in scaling depending on the presets chosen.

And it gets even more granular, because DVD's are actually a classic example: some of the deinterlace options were notorious concurrency bottlenecks. It's been awhile, but arguing that DVD's weren't a pain point because "just use these settings" isn't very helpful because deinterlacing/decombing/etc used to have huge impacts on speed -AND- scaling, and they aren't automatically equivalent to each other in every possible scenario. In fact, I'm pretty insensitive to video quality, but that's a huge one I would totally notice when it was done wrong or inappropriately.

Like it's probably a lot better now scaling-wise (as I said I haven't done in this in awhile), but "decomb" isn't always the right option, and it can be -very- noticeable.

And there are lots and lots of little options like that, so the experience can be pretty variable based on what you pick.


---

Anyway, until -very- recently (as in, less than six months ago--same major release as now) that link had this statement in it:

Handbrake wrote:

The hardware you run on can have a large effect on performance. HandBrake can scale well up to 6 to 8 CPU cores with diminishing returns thereafter. So a 4 core CPU can be nearly twice as fast as a dual Core equivalent, however a 16 core may not be twice as fast as an 8 core but may still offer significant increases in performance. The CPU scaling curve does vary greatly by source and settings used.

Once you beyond the 16 cores, in most scenarios the performance increase is minimal. And the scenarios for which it has a big impact are almost universally the slower ones.

I was ignoring diminishing returns to try to keep things simple, but yeah there's that also. I think it's also easy to forget that in terms of technology DVDs are old. Anyone creating something similar today would be completely bonkers to include the kind of roadblocks to concurrency that we found in DVDs, but back then given the state of current hardware it wasn't as big of a deal breaker as it would be now.

Redocbew wrote:

I was ignoring diminishing returns to try to keep things simple, but yeah there's that also.

Well, that's the entire concept of "performance scaling" in relation to cores, isn't it? I mean, if the benefit of "more cores" diminishes, then no, performance doesn't scale very well, right?

Redocbew wrote:

I think it's also easy to forget that in terms of technology DVDs are old.

And that's definitely the part that I think is valuable: I haven't done that in a *long* time. So it is patently possible, and indeed very likely, that my impressions are out-of-date and just wrong now. I appreciate someone giving a modern update, because without it my "knowlwedge" just grows increasingly long-in-the-tooth.

My issue is that I don't think it's very fair or accurate to simply say "just use these settings" and everything is fine. With DVDs, of course, the whole decomb/detelecine/deinterlace thing is very relevant, and while it's totally possible that it's all the same now performance-wise (particularly scaling-wise), they definitely aren't the same output-wise when given arbitrary selection of input.

As I said, I'm pretty indifferent to a lot of "quality" stuff, but I'm very sensitive to someone screwing that up.

It's just not the same, so if the argument is "ALWAYS just use THIS preset" as a "solution", I have to reject that off the bat.

Glorious wrote:

As I said, I'm pretty indifferent to a lot of "quality" stuff, but I'm very sensitive to someone screwing that up.

It's just not the same, so if the argument is "ALWAYS just use THIS preset" as a "solution", I have to reject that off the bat.

I'm not very sensitive to picture of audio quality either. It rarely makes a bad movie more enjoyable and rarely makes a good movie not enjoyable. I'm ok with a "good enough" preset that "just works" and wold gladly take longer render time (ie 2-pass) and larger file size without having to muck with settings.

Glorious wrote:

Well, that's the entire concept of "performance scaling" in relation to cores, isn't it? I mean, if the benefit of "more cores" diminishes, then no, performance doesn't scale very well, right?

True, but I was having enough trouble just trying to explain that scaling wasn't a thing available for user control without explaining the concept in general.

Glorious wrote:

And that's definitely the part that I think is valuable: I haven't done that in a *long* time. So it is patently possible, and indeed very likely, that my impressions are out-of-date and just wrong now. I appreciate someone giving a modern update, because without it my "knowlwedge" just grows increasingly long-in-the-tooth.

I haven't ripped a DVD from source in quite some time also, and it wouldn't surprise me either if people had chipped away at this problem over time until it wasn't much of a concern anymore. That's why I didn't mind backing off on the bit about source material affecting the scalability of the encode too much. That doesn't seem to happen as much as it used to these days.

Usacomp2k3 wrote:

I'm not very sensitive to picture of audio quality either. It rarely makes a bad movie more enjoyable and rarely makes a good movie not enjoyable. I'm ok with a "good enough" preset that "just works" and wold gladly take longer render time (ie 2-pass) and larger file size without having to muck with settings.

Yeah, I think that gets at the heart of the matter:

I think we all agree that, for the most part, the "less quality" settings are "good enough".

In other words, "Fast" or whatever is generally just fine, in almost all cases. Thus, yes, if you simply want to speed up the clock-time of your encodes tremendously, don't pick "Ultra-slow/Placebo" or "Super" or whatever they are calling the presets this year.

I don't think there is any objection to that. That's good advice and a good point.

----

The pushback is over a related, but different, context, which is essentially "core utilization". Because not only does handbrake self-admittedly not scale tremendously well (for a variety of reasons and differently over a variety of scenarios), but it appears to scale better as you approach "Super/Ultra-slow".

Which is really what OP was asking about--why doesn't Handbrake appear to fully utilize his 6C12T machine?

And the answer to that is, well, that's complicated. Different settings can definitely help, for sure, but *which* settings? As I just said, it generally scales better with slower settings, not faster. So simply suggesting use "Fast" doesn't seem to address the question, and if anything, points the arrow in the wrong way.

Usacomp2k3 wrote:

jihadjoe wrote:

Redocbew wrote:

That's not the same example. The point(again) is that it's beyond your control using Handbrake, so there's no point in telling the OP to look for some magic knob or dial that'll help improve scaling because there isn't one. The problem, whatever it is, has to be somewhere else.

The whole "Extra Options" box in Handbrake's Video encode settings is specifically there so advanced users can tweak their encode settings. Not everything has to be manipulated with a GUI dropdown or slider.

Those option change what the encoding is doing. It's not checking a "do the same encode but use more cores".

Some options actually do.
--pools=n[,n] changes the number of threads per NUMA node
--frame-threads=n will allow encoding multiple frames simultaneously, thus allowing the encoder to use more cores/threads with a corresponding penalty in memory use.

The defaults try to provide good one-size-fits-all settings, but they're not necessarily the best settings for every PC configuration or type of encode which is why the advanced switches are still available. Some of these settings are actually what the fast/slow presets change. A lower CTU size as used by the fast preset for example is somewhat akin to using a smaller dictionary in a zip archive, trading file size for parallelism and thus speed.

Redocbew wrote:

True, but I was having enough trouble just trying to explain that scaling wasn't a thing available for user control without explaining the concept in general

Right, and we also have to discuss that while metrics such "time to complete" or "FPS" of the encodes are related to the utilization of the cores available, the lack of full utilization of all available cores is simply a separate concept.

And, as you note, it's not something that the user has any sort of direct control over, only very indirect( the only thing that comes close, IIRC, is that certain encoders at least USED to have options to LIMIT thread use--but that's not handbrake as a total product, not really a slider, and it can only choke core utilization, not improve it).

---

Speaking of core scaling, I took a peak at a recent TR review because they've used Handbrake for awhile now:

https://techreport.com/review/34253/int ... reviewed/7

The i9-9900k took 385 seconds to complete a 4k iphone video to 1080p30 HEVC clip with "default" settings.

The i9-9980XE took 269 seconds.

The 9900k is a 8C16T at 3.6Ghz base/all-core 4.7

The 9980XE is a 18C36T at a 3.0 Ghz base/all-core 3.8

So while the 9900k is roughly 20%-25% faster in clocks, the 9980XE has 125% more cores.

And yet the 9980XE is only is only ~43% faster than the i9.

What's interesting, though, is that the i9-9980XE is basically just a 7980XE clocked higher by like 300-400mhz, and that jumped the performance 15% between them. If you compare the 7980XE to the 9900K, it's only 25% faster.