Personal computing discussed

Moderators: renee, Dposcorp

 
ClickClick5
Gerbil Elite
Topic Author
Posts: 581
Joined: Fri Aug 15, 2008 8:28 pm
Location: Alaska
Contact:

Come Press Me! (Compression Review)

Sun Feb 09, 2014 3:11 pm

*post removed*
Last edited by ClickClick5 on Mon Feb 10, 2014 10:27 am, edited 1 time in total.
i7-5930k (4.1GHz), 32GB DDR4-2400, 2080Ti, GA-X99-UD3, Samsung 860 Pro 256GB (os/programs), Toshiba 5TB (games), WD VelociRaptor 500GB (games/scratch disk), Win 10 Pro x64
Check out my site for trance sets and other goodies! clickclick5.com!
 
toki
Gerbil XP
Posts: 353
Joined: Fri Nov 08, 2013 3:55 pm

Re: Come Press Me! (Compression Review)

Sun Feb 09, 2014 3:40 pm

wow, thank you for the time and the graphs!
 
LostCat
Minister of Gerbil Affairs
Posts: 2107
Joined: Thu Aug 26, 2004 6:18 am
Location: Earth

Re: Come Press Me! (Compression Review)

Sun Feb 09, 2014 3:57 pm

Did you try the WinZip OpenCL acceleration? I don't think its CPU optimization really means much anymore.
Meow.
 
Deanjo
Graphmaster Gerbil
Posts: 1212
Joined: Tue Mar 03, 2009 11:31 am

Re: Come Press Me! (Compression Review)

Sun Feb 09, 2014 3:59 pm

1) The first three episodes of the last season of Breaking Bad in MP4 format.


I quit reading right there.
 
ClickClick5
Gerbil Elite
Topic Author
Posts: 581
Joined: Fri Aug 15, 2008 8:28 pm
Location: Alaska
Contact:

Re: Come Press Me! (Compression Review)

Sun Feb 09, 2014 4:36 pm

*post removed*
Last edited by ClickClick5 on Mon Feb 10, 2014 10:27 am, edited 1 time in total.
i7-5930k (4.1GHz), 32GB DDR4-2400, 2080Ti, GA-X99-UD3, Samsung 860 Pro 256GB (os/programs), Toshiba 5TB (games), WD VelociRaptor 500GB (games/scratch disk), Win 10 Pro x64
Check out my site for trance sets and other goodies! clickclick5.com!
 
LostCat
Minister of Gerbil Affairs
Posts: 2107
Joined: Thu Aug 26, 2004 6:18 am
Location: Earth

Re: Come Press Me! (Compression Review)

Sun Feb 09, 2014 4:44 pm

Toms workload has some interesting results on that front. 7zip does notably worse wrt time needed.

http://www.tomshardware.com/reviews/a10 ... 25-13.html

I don't know if their settings chosen are any better/worse but it's still interesting.
Meow.
 
Deanjo
Graphmaster Gerbil
Posts: 1212
Joined: Tue Mar 03, 2009 11:31 am

Re: Come Press Me! (Compression Review)

Sun Feb 09, 2014 5:14 pm

ClickClick5 wrote:
I tried testing files that I see people compress (minus the Chrome cache). Not everyone knows what type of files are compressible or not.


If someone is accidentally pouring diesel in their gasoline engine car do you tell them to stop or do you tell them what the price of diesel is?
 
ClickClick5
Gerbil Elite
Topic Author
Posts: 581
Joined: Fri Aug 15, 2008 8:28 pm
Location: Alaska
Contact:

Re: Come Press Me! (Compression Review)

Sun Feb 09, 2014 5:32 pm

Deanjo wrote:
ClickClick5 wrote:
I tried testing files that I see people compress (minus the Chrome cache). Not everyone knows what type of files are compressible or not.


If someone is accidentally pouring diesel in their gasoline engine car do you tell them to stop or do you tell them what the price of diesel is?


Haha! And if you bothered to read instead of judge, you would see i mention compressing already encoded/compressed files offer no benifit. Try reading before jumping to conclusions next time.
i7-5930k (4.1GHz), 32GB DDR4-2400, 2080Ti, GA-X99-UD3, Samsung 860 Pro 256GB (os/programs), Toshiba 5TB (games), WD VelociRaptor 500GB (games/scratch disk), Win 10 Pro x64
Check out my site for trance sets and other goodies! clickclick5.com!
 
The Egg
Minister of Gerbil Affairs
Posts: 2938
Joined: Sun Apr 06, 2008 4:46 pm

Re: Come Press Me! (Compression Review)

Sun Feb 09, 2014 5:44 pm

Wow, thanks for all your work ClickClick. I've been a WinRAR user for many years, but lately I've found myself using 7-Zip more and more frequently. The reason was originally due to something which reviews never take into account: Amount of time to open a very large compressed archive.

At my place of work, I have an archive which has grown to almost 250GB and several hundred thousand files. These are blueprints which need to be saved, and up until recently, I had an older machine with limited space, so I needed to keep them compressed. I also semi-frequently need to access files in the archive, and sometimes the boss wants them quickly.

When using WinRAR, simply opening an archive of this size could take upwards of 20 minutes before I'm able to see the files, regardless of the archive settings (at least on my old machine). When using 7-zip and the "Fast" preset, I was generally able to open the same archive in less than a minute (even though extraction takes slightly longer). In this case, the greater configurability of 7-zip made it much more valuable.
 
Usacomp2k3
Gerbil God
Posts: 23043
Joined: Thu Apr 01, 2004 4:53 pm
Location: Orlando, FL
Contact:

Re: Come Press Me! (Compression Review)

Sun Feb 09, 2014 5:58 pm

Was the .7z files created in 7-zip or regular .zip files?
 
ClickClick5
Gerbil Elite
Topic Author
Posts: 581
Joined: Fri Aug 15, 2008 8:28 pm
Location: Alaska
Contact:

Re: Come Press Me! (Compression Review)

Sun Feb 09, 2014 6:03 pm

Usacomp2k3 wrote:
Was the .7z files created in 7-zip or regular .zip files?


The .7z files were created in 7Zip.
i7-5930k (4.1GHz), 32GB DDR4-2400, 2080Ti, GA-X99-UD3, Samsung 860 Pro 256GB (os/programs), Toshiba 5TB (games), WD VelociRaptor 500GB (games/scratch disk), Win 10 Pro x64
Check out my site for trance sets and other goodies! clickclick5.com!
 
Deanjo
Graphmaster Gerbil
Posts: 1212
Joined: Tue Mar 03, 2009 11:31 am

Re: Come Press Me! (Compression Review)

Sun Feb 09, 2014 7:48 pm

ClickClick5 wrote:
Deanjo wrote:
ClickClick5 wrote:
I tried testing files that I see people compress (minus the Chrome cache). Not everyone knows what type of files are compressible or not.


If someone is accidentally pouring diesel in their gasoline engine car do you tell them to stop or do you tell them what the price of diesel is?


Haha! And if you bothered to read instead of judge, you would see i mention compressing already encoded/compressed files offer no benifit. Try reading before jumping to conclusions next time.


I don't need to read results of flawed methodology to know the results. You still waited until the tank was full before notifying.
 
Flatland_Spider
Graphmaster Gerbil
Posts: 1324
Joined: Mon Sep 13, 2004 8:33 pm

Re: Come Press Me! (Compression Review)

Sun Feb 09, 2014 9:00 pm

Deanjo wrote:
ClickClick5 wrote:
I tried testing files that I see people compress (minus the Chrome cache). Not everyone knows what type of files are compressible or not.


If someone is accidentally pouring diesel in their gasoline engine car do you tell them to stop or do you tell them what the price of diesel is?


I give them the business card of my mechanic. Then I tell my mechanic.
 
Glorious
Gerbilus Supremus
Posts: 12343
Joined: Tue Aug 27, 2002 6:35 pm

Re: Come Press Me! (Compression Review)

Mon Feb 10, 2014 9:46 am

I hate to be critical of someone who takes the time to investigate things, but...

Can I make a few suggestions?

First off, let's address what deanjo already brought up:

ClickClick wrote:
Haha! And if you bothered to read instead of judge, you would see i mention compressing already encoded/compressed files offer no benifit. Try reading before jumping to conclusions next time.


Yeah, but almost all of your tests are already compressed:

ClickClick wrote:
1) The first three episodes of the last season of Breaking Bad in MP4 format. I took these from my own HTPC (HandBrake encodes of legally bought BluRay discs) for this test. Totaling 6.05GB in size.


That's your 1st test, and it's a horrible one. The resultant file obviously isn't going to be any smaller, if anything it'll only be larger.

ClickClick wrote:
2) 2,278 JPG pictures I have collected through the years of internet Fails, Imgur, etc. This folder totals to 636MB in size.


JPGs are generally very compressed already. Again, not a good test, albeit slightly better than the first one.

ClickClick wrote:
3) Something truly random here, the cache folder for Google Chrome. For the test, there were 2,988 files and they totaled to 362MB


This test is actually probably somewhat compressible, for once, but it's also completely transient data (web cache)! What use case is this supposed to represent?

ClickClick wrote:
4) Before modern fast internet, when downloading in Steam, I would keep a backup of my games in compressed form on a disc in case I had to reinstall or my HDD died, so found my old Half-Life backup folder. This folder contained 4,510 files and totaled to 409MB for me.


Whether or not this will be compressible probably depends on the game(but I'd assume most have a compressed package format), but it's still misplaced. For one thing, Steam already has a built-in backup system you can use. For another, who really bothers these days (as you even say)? "Modern fast internet" makes it somewhat silly for most users.

ClickClick wrote:
5) Next, 180 MP3/AAC files totaling 1.03GB in size


Again, already very compressed.

Out of your 5 tests, only one is even moderately compressible, and it's of data that no one would ever want to save. A better selection of test cases would be much more interesting.

Second off, you constantly change the color scheme of your graphs. That makes reading them very difficult and you probably shouldn't do it.

Third off, you include graphs that you self-admittedly don't seem to understand the relevancy of, such as thread count. I don't really understand it either, so I'm not really sure what the point of it was.

Fourth off, you split the file size graphs into two different graphs, based on scale. I guess I could understand why you did that, but it's a little confusing and I'm not really sure why the absolute size even matters. The real relevant metric would be the compression percentage, which you have, so why bother with file size?

---

Again, this might seem overly critical, but if you want people to appreciate the work you've obviously done, they are all things you ought to pay some attention to.
 
Scrotos
Graphmaster Gerbil
Posts: 1109
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Come Press Me! (Compression Review)

Mon Feb 10, 2014 10:57 am

Well, since he took his ball and went home, here's some real-world comparison in compression formats I've run into lately.

6+ GB SQL database. With 7zip as the compressor.

Going to ZIP gets it down to around 600+ MB. Takes quite some time.
Going to 7z gets it down to around 200 MB. Much faster.

With 7z it is able to use all 8 threads on the CPUs whereas with the ZIP it only uses 2. I thought that only mattered if there were multiple files being included rather that just one big .BAK file, but I guess not. I don't believe the compression is enabled on the database so it's ripe fer compressin'.

With this in mind, I'm going to redo all our schedule task compress and backup scripts to use 7z instead of ZIP. I had done ZIP in the past because it could be opened natively in Windows but hell, 7zip is easy enough to get if need be for recovery purposes.
 
ClickClick5
Gerbil Elite
Topic Author
Posts: 581
Joined: Fri Aug 15, 2008 8:28 pm
Location: Alaska
Contact:

Re: Come Press Me! (Compression Review)

Mon Feb 10, 2014 11:01 am

Scrotos wrote:
Well, since he took his ball and went home, here's some real-world comparison in compression formats I've run into lately.

6+ GB SQL database. With 7zip as the compressor.

Going to ZIP gets it down to around 600+ MB. Takes quite some time.
Going to 7z gets it down to around 200 MB. Much faster.

With 7z it is able to use all 8 threads on the CPUs whereas with the ZIP it only uses 2. I thought that only mattered if there were multiple files being included rather that just one big .BAK file, but I guess not. I don't believe the compression is enabled on the database so it's ripe fer compressin'.

With this in mind, I'm going to redo all our schedule task compress and backup scripts to use 7z instead of ZIP. I had done ZIP in the past because it could be opened natively in Windows but hell, 7zip is easy enough to get if need be for recovery purposes.


Again, missed the point of the original post. Seems everyone did. If was for the normal computer user, not Admins, power users, etc. I'll post it elsewhere.
i7-5930k (4.1GHz), 32GB DDR4-2400, 2080Ti, GA-X99-UD3, Samsung 860 Pro 256GB (os/programs), Toshiba 5TB (games), WD VelociRaptor 500GB (games/scratch disk), Win 10 Pro x64
Check out my site for trance sets and other goodies! clickclick5.com!
 
Glorious
Gerbilus Supremus
Posts: 12343
Joined: Tue Aug 27, 2002 6:35 pm

Re: Come Press Me! (Compression Review)

Mon Feb 10, 2014 11:15 am

Scrotos wrote:
Going to ZIP gets it down to around 600+ MB. Takes quite some time.
Going to 7z gets it down to around 200 MB. Much faster.

With 7z it is able to use all 8 threads on the CPUs whereas with the ZIP it only uses 2. I thought that only mattered if there were multiple files being included rather that just one big .BAK file, but I guess not. I don't believe the compression is enabled on the database so it's ripe fer compressin'.


Eh,

For the compression ratio, generic ZIP is just ages-old DEFLATE applied per file (not archive), which obviously isn't all that great (especially as DEFLATE was written to work around a patent that is no longer extant) ...

7z is basically xz, which usually beats bzip2 ratio-wise, so that's cool. I'm not surprised it devastates generic ZIP.

As to thread-usage, yeah... That appears to be a widespread issue. I usually use pbzip2, for instance, because bzip2 still isn't multi-threaded. :o I'm therefore not surprised, at all, that generic ZIP has issues as well.

I'd probably mess around with xz, but pbzip2 is generally good enough for my purposes.

ClickClick wrote:
Again, missed the point of the original post. Seems everyone did. If was for the normal computer user, not Admins, power users, etc. I'll post it elsewhere.


Err, no. Information theory doesn't have exceptions for the "normal computer user." If such users are futilely trying to compress the incompressible, the correct action is to tell them not to do that, not inform them how long it will take to increase the size of their files.
 
ClickClick5
Gerbil Elite
Topic Author
Posts: 581
Joined: Fri Aug 15, 2008 8:28 pm
Location: Alaska
Contact:

Re: Come Press Me! (Compression Review)

Mon Feb 10, 2014 11:37 am

ClickClick wrote:
Again, missed the point of the original post. Seems everyone did. If was for the normal computer user, not Admins, power users, etc. I'll post it elsewhere.


Err, no. Information theory doesn't have exceptions for the "normal computer user." If such users are futilely trying to compress the incompressible, the correct action is to tell them not to do that, not inform them how long it will take to increase the size of their files.

It only increased on one file. Of which I noted.
This thread got killed in the way of Fred Silver's comics...
i7-5930k (4.1GHz), 32GB DDR4-2400, 2080Ti, GA-X99-UD3, Samsung 860 Pro 256GB (os/programs), Toshiba 5TB (games), WD VelociRaptor 500GB (games/scratch disk), Win 10 Pro x64
Check out my site for trance sets and other goodies! clickclick5.com!
 
anotherengineer
Gerbil Jedi
Posts: 1688
Joined: Fri Sep 25, 2009 1:53 pm
Location: Northern, ON Canada, Yes I know, Up in the sticks

Re: Come Press Me! (Compression Review)

Mon Feb 10, 2014 2:35 pm

ClickClick5 wrote:
This thread got killed in the way of Fred Silver's comics...


Which way is that?
Life doesn't change after marriage, it changes after children!
 
Scrotos
Graphmaster Gerbil
Posts: 1109
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Come Press Me! (Compression Review)

Mon Feb 10, 2014 4:20 pm

ClickClick5 wrote:
Scrotos wrote:
Well, since he took his ball and went home, here's some real-world comparison in compression formats I've run into lately.

6+ GB SQL database. With 7zip as the compressor.

Going to ZIP gets it down to around 600+ MB. Takes quite some time.
Going to 7z gets it down to around 200 MB. Much faster.

With 7z it is able to use all 8 threads on the CPUs whereas with the ZIP it only uses 2. I thought that only mattered if there were multiple files being included rather that just one big .BAK file, but I guess not. I don't believe the compression is enabled on the database so it's ripe fer compressin'.

With this in mind, I'm going to redo all our schedule task compress and backup scripts to use 7z instead of ZIP. I had done ZIP in the past because it could be opened natively in Windows but hell, 7zip is easy enough to get if need be for recovery purposes.


Again, missed the point of the original post. Seems everyone did. If was for the normal computer user, not Admins, power users, etc. I'll post it elsewhere.


I only saw the thread after you had deleted everything so I only got the gist of it. All I'm providing is a different data point showing compressible data, that's all. I could be compressing PDFs or TIFFs or whatnot, either business or regular user use. It's just different data.
 
The Egg
Minister of Gerbil Affairs
Posts: 2938
Joined: Sun Apr 06, 2008 4:46 pm

Re: Come Press Me! (Compression Review)

Mon Feb 10, 2014 7:14 pm

I think some of you were overly harsh. I did notice that most of his tests used file types which had already been compressed, but there was still some value to it, and I at least appreciate the time he spent on it.
Glorious wrote:
ClickClick wrote:
4) Before modern fast internet, when downloading in Steam, I would keep a backup of my games in compressed form on a disc in case I had to reinstall or my HDD died, so found my old Half-Life backup folder. This folder contained 4,510 files and totaled to 409MB for me.


Whether or not this will be compressible probably depends on the game(but I'd assume most have a compressed package format), but it's still misplaced. For one thing, Steam already has a built-in backup system you can use. For another, who really bothers these days (as you even say)? "Modern fast internet" makes it somewhat silly for most users.

It's not misplaced at all. The Steam Backup feature is a buggy piece of garbage that is incredibly sloooooow when it does work (rarely), and crashes nearly 100% of time once your game library surpasses a certain threshold. I have repeatedly done everything I can to get it to work, trying about once every 6 months for a period of several years, and on several different machines. I don't think Valve has updated this utility once in Steam's 10 year lifespan.

As far as compressibility of the games, my most recent 7-zip archive of the Steamapps folder took a 358GB library down to just under 244GB, using fairly conservative settings.
 
vargis14
Gerbil Jedi
Posts: 1900
Joined: Fri Aug 20, 2010 6:03 pm
Location: philly suburbs

Re: Come Press Me! (Compression Review)

Mon Feb 10, 2014 8:02 pm

Hopefully in the future kaveri gets that GDDR5 memory on a Ram Dimm Stick. If it had that you would get slightly better performance then a HD7750 since I believe the 512 shader cores are the same as in the 384 core based Oland GPU that performs pretty dang close to the HD 7750 with only 384 cores compared to the 512 in the 7750. They must have did some optimization on those cores sorta like the optimizations of the 7790 chip where its tesselation performance is close to the much more power hungry and core heavy HD 7850.

Would be pretty kewl if the APUs could use a stick of GDDR5 along with a couple sticks of DDR3 at the same time.

HD7750 crossfire scaled pretty good back in februrary 2012...now with frame pacing a GDDR5 powered a10-7850k and a HD7750 would do very well @1080p.
2600k@4848mhz @1.4v CM Nepton40XL 16gb Ram 2x EVGA GTX770 4gb Classified cards in SLI@1280mhz Stock boost on a GAP67-UD4-B3, SBlaster Z powered by TX-850 PSU pushing a 34" LG 21/9 3440-1440 IPS panel. Pieced together 2.1 sound system
 
vargis14
Gerbil Jedi
Posts: 1900
Joined: Fri Aug 20, 2010 6:03 pm
Location: philly suburbs

Re: Come Press Me! (Compression Review)

Mon Feb 10, 2014 8:11 pm

[/quote]
It's not misplaced at all. The Steam Backup feature is a buggy piece of garbage that is incredibly sloooooow when it does work (rarely), and crashes nearly 100% of time once your game library surpasses a certain threshold. I have repeatedly done everything I can to get it to work, trying about once every 6 months for a period of several years, and on several different machines. I don't think Valve has updated this utility once in Steam's 10 year lifespan.

As far as the steam backup, I have never tried it. But with them keeping track of all the games you own and you can download them to pretty much any machine you get or use I do not think it is a huge priority to them. I just with the Steam would store your game's progress and achievements for each and every game. So when you install a game on your rig it remembers your place and level in the game.

At least for games that do not do this automatically already.
2600k@4848mhz @1.4v CM Nepton40XL 16gb Ram 2x EVGA GTX770 4gb Classified cards in SLI@1280mhz Stock boost on a GAP67-UD4-B3, SBlaster Z powered by TX-850 PSU pushing a 34" LG 21/9 3440-1440 IPS panel. Pieced together 2.1 sound system
 
The Egg
Minister of Gerbil Affairs
Posts: 2938
Joined: Sun Apr 06, 2008 4:46 pm

Re: Come Press Me! (Compression Review)

Mon Feb 10, 2014 8:56 pm

vargis14 wrote:
As far as the steam backup, I have never tried it. But with them keeping track of all the games you own and you can download them to pretty much any machine you get or use I do not think it is a huge priority to them. I just with the Steam would store your game's progress and achievements for each and every game. So when you install a game on your rig it remembers your place and level in the game.
At least for games that do not do this automatically already.

True, but some people have data caps to deal with, and a large Steam library like mine can be a problem. Others might not have blazing fast internet connections. 358GB at a modest 6mbit connection would be over 129 hours.
 
Glorious
Gerbilus Supremus
Posts: 12343
Joined: Tue Aug 27, 2002 6:35 pm

Re: Come Press Me! (Compression Review)

Tue Feb 11, 2014 6:30 am

ClickClick5 wrote:
It only increased on one file. Of which I noted.


I was being a bit flippant and focusing on the worst case, but it bares noticing that the worst case was your first case. The point is that when compressing previously compressed content you'll either do very, very little for vast amounts of effort and, if it is highly compressed content (like any video file using a modern codec) you'll very likely do worse.

You can take the criticisms in stride and adjust your approach, or you can childishly reject them. <shrug>

The Egg wrote:
but there was still some value to it, and I at least appreciate the time he spent on it.


I never said there wasn't. In fact, I explicitly said:

Glorious wrote:
I hate to be critical of someone who takes the time to investigate things, but...

Can I make a few suggestions?


Glorious wrote:
Again, this might seem overly critical, but if you want people to appreciate the work you've obviously done, they are all things you ought to pay some attention to.


The Egg wrote:
It's not misplaced at all. The Steam Backup feature is a buggy piece of garbage that is incredibly sloooooow when it does work (rarely), and crashes nearly 100% of time once your game library surpasses a certain threshold. I have repeatedly done everything I can to get it to work, trying about once every 6 months for a period of several years, and on several different machines. I don't think Valve has updated this utility once in Steam's 10 year lifespan.


Well, that's good to know. I haven't used it in years, I just assumed it worked.

The Egg wrote:
As far as compressibility of the games, my most recent 7-zip archive of the Steamapps folder took a 358GB library down to just under 244GB, using fairly conservative settings.


Like I said, it really depends on the game. At least for Half-Life, the only one he did, it appears to already be a highly compressed installation according to his results.

The Egg wrote:
True, but some people have data caps to deal with, and a large Steam library like mine can be a problem. Others might not have blazing fast internet connections. 358GB at a modest 6mbit connection would be over 129 hours.


The relevant point is that you aren't very likely to need to play every game in that 358GB in those 129 hours.

As I said, for most users, there isn't much of a need anymore. Obviously it's still relevant to some, but your data is more valuable to them than his.

All I was trying to do was improve the relevancy and presentation of what he was trying to do, not dismiss the entire concept.

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On