Anti-Virus Scan Bench Machine Build

Don't see a specific place for your hardware question? This is the forum for you!

Moderators: mac_h8r1, Nemesis

Anti-Virus Scan Bench Machine Build

Postposted on Wed Feb 05, 2014 4:15 pm

I've been running a scan machine here in my office using old parts, little Core 2 Duo's and all. I'm noticing those that it takes quite awhile to get through a scan and at present I'm only doing a single drive at a time. So when taking into account the cost/time benefits of a faster machine and interesting thought came to mind. Would the benefits of an AMD CPU finally be worth using for Avast virus scans vs say a Core i3 with Hyper threading? Bare in mind, this machine would be used for simple virus scans of drives connected via USB 3.0 docking stations and making Acronis backups.

It would of course rely heavily on whether those programs (Acronis does) support a certain number of threads... The idea is to build the machine for as cheap as possible, squeezing out of it faster scans and doing 2-3 drives at the same time potentially.

Anyone here really big on this or keep a bench machine up-to-date? I'll most likely install Deep Freeze on it to prevent the drive itself from being infected between machines, or within that install create virtual machines with VMWare assuming no major performance hits (Of course a CPU with virtualization would help in this case)
"I think there is a world market for maybe five computers."
Thomas Watson, chairman of IBM, 1943

i5-2500K|Asus P67 Sabertooth|16GB Corsair 1600|MSI 7850 2GB|250gb Evo 840|Corsair 400R|ET750w PSU|Logitech G5|Dell 2420L|Corsair Vengeance 1300
Welch
Minister of Gerbil Affairs
Gold subscriber
 
 
Posts: 2639
Joined: Thu Nov 04, 2004 5:45 pm
Location: Fairbanks, Alaska

Re: Anti-Virus Scan Bench Machine Build

Postposted on Wed Feb 05, 2014 4:28 pm

My first thought is that it's IO limited. What do the stats look like while it's scanning?
Flatland_Spider
Gerbil Elite
 
Posts: 833
Joined: Mon Sep 13, 2004 8:33 pm
Location: The 918/539

Re: Anti-Virus Scan Bench Machine Build

Postposted on Wed Feb 05, 2014 4:30 pm

As someone who works for one of the big three AV firms... the bottleneck will be usb versus sata and decomposing files.

Most scanners run 3 levels deep on objects. Things like mountable virtual drives, iso, zips, mime-containers such as pdf, xls, word, emails, jar files, class files. The larger majority of your time will be spent scanning inside those file types. Most of the time the temp work of extracting and scanning these files will be the windows temp path or the Av's temp path.

optimize your scan, make sure the scanners temp file is running on a faster drive. (ssd) this will help with the large amount of disk IOp/s your going to inccur.
rather than using usb, get a esata cable and it will greatly enhance the scan times
most AV (ours included) claim to be multithreaded but wont exceed 20-30% cpu time on each core .. so number of threads is not as important as the speed of the cpu + cpu cache size.

i'd suggest a small ssd (128 gigs) for the os drive and where the virus scanner runs from
a esata doc to mount the drives your clearing
dual core cpu running 3+ghz with a decent cache size
4 gigs of ram to keep things out of the scanners swap file

if the target drive has alot of compressed data or mime-containers it is still going to take a while to run.. not much can be done there. extracting a spreadsheet and pdf are very time consuming..

cheers
Cybert said: Capitlization and periods are hard for you, aren't they? I've given over $100 to techforums. I should have you banned for my money.
maxxcool
Gerbil Elite
Silver subscriber
 
 
Posts: 645
Joined: Thu Sep 12, 2002 8:40 am
Location: %^&*%$$

Re: Anti-Virus Scan Bench Machine Build

Postposted on Wed Feb 05, 2014 4:32 pm

Flatland_Spider wrote:My first thought is that it's IO limited. What do the stats look like while it's scanning?


^ this. SSD is the best bet here. doesn't even have to be a amazing drive... just get to +15000 iops.. each file that has to be extracted is going to incur tons of IO..
Cybert said: Capitlization and periods are hard for you, aren't they? I've given over $100 to techforums. I should have you banned for my money.
maxxcool
Gerbil Elite
Silver subscriber
 
 
Posts: 645
Joined: Thu Sep 12, 2002 8:40 am
Location: %^&*%$$

Re: Anti-Virus Scan Bench Machine Build

Postposted on Wed Feb 05, 2014 11:09 pm

Thanks for the very detailed explanation Maxx, this is exactly the kind of info I was hoping for :).

I can tell you that at this point I'm using Avast IS. I'm surprised to hear that the OS of the host scan machine would play a major role in scan times. I would have thought all relevant information from the AV would be loaded into memory and be ran from there. There is no doubt in my mind about the speed benefits of SSDs in general, I'm just curious what most popular AV software actually do step-by-step in order to process a file, especially when it comes across something it has to decompress. Care to elaborate a bit more and possible explain the steps the software would generally take and why its more advantageous to have faster cores not more? As for Cache and cache speed, that makes sense to me as instructions can be called on very faster not having to go through the rest of the systems slower resources (even system memory).

So I'd assume a Core i3-4130 with 2x4GB of ram and a high IOPS(4k) SSD would drastically decrease the time of scans compared an "8-core" AMD equivalent with the same RAM and SSD?

This seems like something that would be very interesting for TR to do a write up on, considering most of these parts are something already in house. It would provide a lot of real world data for techs like myself who want to build a machine with a very specific purpose.
"I think there is a world market for maybe five computers."
Thomas Watson, chairman of IBM, 1943

i5-2500K|Asus P67 Sabertooth|16GB Corsair 1600|MSI 7850 2GB|250gb Evo 840|Corsair 400R|ET750w PSU|Logitech G5|Dell 2420L|Corsair Vengeance 1300
Welch
Minister of Gerbil Affairs
Gold subscriber
 
 
Posts: 2639
Joined: Thu Nov 04, 2004 5:45 pm
Location: Fairbanks, Alaska

Re: Anti-Virus Scan Bench Machine Build

Postposted on Thu Feb 06, 2014 2:16 pm

Welch wrote:Thanks for the very detailed explanation Maxx, this is exactly the kind of info I was hoping for :).

I can tell you that at this point I'm using Avast IS. I'm surprised to hear that the OS of the host scan machine would play a major role in scan times. I would have thought all relevant information from the AV would be loaded into memory and be ran from there. There is no doubt in my mind about the speed benefits of SSDs in general, I'm just curious what most popular AV software actually do step-by-step in order to process a file, especially when it comes across something it has to decompress. Care to elaborate a bit more and possible explain the steps the software would generally take and why its more advantageous to have faster cores not more? As for Cache and cache speed, that makes sense to me as instructions can be called on very faster not having to go through the rest of the systems slower resources (even system memory).

So I'd assume a Core i3-4130 with 2x4GB of ram and a high IOPS(4k) SSD would drastically decrease the time of scans compared an "8-core" AMD equivalent with the same RAM and SSD?

This seems like something that would be very interesting for TR to do a write up on, considering most of these parts are something already in house. It would provide a lot of real world data for techs like myself who want to build a machine with a very specific purpose.



Sure..

filescanning comes a few forms, Hash, reputation and behavior analytics.

Hashing is the #1 method of av scanning for any and all vendors. this involves the "definitions" "pattern files" "check files" what ever the vendor labels them as which are large databases of md5/sha1 hash lists of known bad things.

so to run this hash scan which is mostly what you are doing.. the scanner needs to determine a path of action.

--is the file a container i recognize ? (zip, rar, jar, class, eml, msg, msi, cab, spreadsheet, word file, power point, flash file etc...
**if it is not, it goes straight off and gets the md5/sha1 of that file and looks it up in the local database of the scanner (held in the pattern files\definitions files).. if it gets a reasonable hit. its flagged as bad

**if it is a file that can be extracted as a container object it "decomposes" or extracts the object out *after* determining how many layers and how many files it will have to scan. This extraction will always be to the install directory of the application (scanner) or a temp folder on the operating system drive specified by the scanner on install. this extraction is VERY IO intensive a it is literally like extracting a zip.. then running the hash scan on those objects.

But the catch is this.. If it is a nested zip, or a nested mime container (zips in zips.. or word docs with word docs embedded in the doc) those objects must be extracted themselves and then hash scanned as well. The extractions impact on the cpu but not as much as you'd think. But a faster cpu will help with the extraction times. But the biggest impact will be the SSD for the temp work. Any file that must be extracted that is not scanned "in memory" will have to be extracted "too" the scanners system drive then scanned.


Reputation: If the scanner does reputation lookups (sonar, download insight etc) these are not as disk intensive, there is a read to get the hash but that has is then transmitted to the reputation servers for a lookup there.

Behavior: Behavior scanning is also very cpu and disk intensive.. *but* only if the file is executed. Behavior is a runtime scan for api calls, file and folder touches that might clue the scanner into a bad file doing suspicious things (like messing with the host file).

Back to what your doing, scanning mounted drives.

#1 issue is the disk being mounted in a usb cradle, while it may have a fast transfer rate... the issue is the transfer rate is for the largest file transferred and not for multiple small file read and writes. if you use HDtune you will find a raptor drive in a usb3 cradle will have a much lower trasnfer rate and much lower IOp/s rate as compared to a Esata mounted drive. Esata is hardware accelerated .. usb is not and is cpu dependent and will not be able to keep up even with a great cpu.

So eliminate the 1st bottle neck in the chain of data handling. get better IOps and lower cpu usage by getting off the usb channel.

#2, Second bottle neck will be the host drive of the scanner. with a platter drive your averaging 70-120 ops a second. compared to 15000k for some of the cheapest 128 meg ssds out there. anytime you need to have the scanner transfer data from the guest drive to the host OS drive you will be waiting on the drives IOp rated writes .. faster is better. sooner it is written then we can scan it which is also disk IOp heavy so again limited by the 70-120 ops of a platter setup.

#3, cpu speed. if you pop open task manager during a scan, you will find (at least on my 4 core systems and VMs that i do AV testing on..) that the scanner will not peg the cpu .. not even close. this is because really most scanners (our included) may be multithreaded.. .but beyond 2 cpus little coding has been done. also.. most vendors keep the scanner "in safe mode" for scanning.. they limit the number of cycles that can be used so as to not impact the host. since your host *IS* a dedicated scanner, see if there is a performance slider to turn up. Ours has a slider for "application performance and the other side is scan performance" .. it is much faster when set to scan performance.. but even then it wont peg my vmware guest systems to protect the user. however.. even though it is limited to x# of cycles... the speed of those cycles is controlled by the speed of the cpu.. so faster ghz... faster execution of what it is doing. (hashing)

my recommendation .. quad core amd, quad core intel at 3.5 ghz. go full fat, no celerons or cheapo Athlons.. you want the bigger l3 and bigger l2 caches to maximize cache hits .. (fyi 8230 or 8350 is what i mean by a amd quad core)

#4 memory .. 2 gigs is sufficient.. but 4 is better and 8 is a waste. for us at least we load not into user space ram but -nonpage-pool space. so it is limited in what it can load there. so gobs of ram wont help. * but * during extractions and the scan itself the scanner may need more resources to run. 4 gigs is exactly what you need.
Cybert said: Capitlization and periods are hard for you, aren't they? I've given over $100 to techforums. I should have you banned for my money.
maxxcool
Gerbil Elite
Silver subscriber
 
 
Posts: 645
Joined: Thu Sep 12, 2002 8:40 am
Location: %^&*%$$

Re: Anti-Virus Scan Bench Machine Build

Postposted on Fri Feb 07, 2014 3:53 pm

Great info Maxx, again exactly what I was looking for :)! I never touch the lower bin CPUs due to their anemic nature when it comes to even the simplest tasks. Yep I was thinking the AMD FX-8xxx, realizing of course that they are "8-core" (4 modules) and being close to equivalent to the i5 series. Knowing that the AMDs are higher clocked but less IPC, I'd imagine it depends on how the AV vendor limits the cycles used by the scanner, either by CPU % or something else? I'd imagine its % based, so if that is the case the AMD cpu would need to be clocked high enough that its % being used makes up for the lower IPC.

I'll be looking at ordering some of these parts here in the near future and I'll do a test with the parts to show the difference. My aversion to eSATA came from when it was initially released and very buggy. By any chance, has the eSATA standard been improved? Before it would not recognize drives, I'd have to restart the machine in order to add a drive, ect. The one nice thing about USB that I should mention (as a backup interface) is that you can plugin drives that are failing to a certain extent and they wont lockup the entire OS. Those same drives plugged into SATA will lockup the entire SATA controller.
"I think there is a world market for maybe five computers."
Thomas Watson, chairman of IBM, 1943

i5-2500K|Asus P67 Sabertooth|16GB Corsair 1600|MSI 7850 2GB|250gb Evo 840|Corsair 400R|ET750w PSU|Logitech G5|Dell 2420L|Corsair Vengeance 1300
Welch
Minister of Gerbil Affairs
Gold subscriber
 
 
Posts: 2639
Joined: Thu Nov 04, 2004 5:45 pm
Location: Fairbanks, Alaska

Re: Anti-Virus Scan Bench Machine Build

Postposted on Mon Feb 10, 2014 12:31 pm

Well, it is a catch 22 on the esata.

For me, my intel box that is work issued, the e-sata if flawless.
For the 2 AMD thuban x6 test boxes i bought to take to work beacuse 4 cores is not enough, e-sata is .. well .. different.

For the two Thuban systems I use to build vmware test machines I had a number of issues with the e-sata driver from AMD's own website. But the chipset driver from the OEM (gigabyte and asrock, both 990fx ) site works ok as long as i did not have any IDE devices.

After i was annoyed enough i ditched the esata port that is built onto the board and opted to get a e-sata bracket that converts internal sata to e-sata.. after that .. no issue :)

So, i'd say if you have a older amd board, get the 8$ sata to e-sata bracket. hook into that. you wont have to fiddle with "hot mount" e-sata settings and the weird e-sata sleep resume issues. hook the dock into that. no issue for the two boards i'm using in that config.

just have to massage the amd systems a tad with e-sata sometimes. just make sure your ''ahci'' is set ti native, or ahci.. not "combination" or legacy.. you will loose hot swap if your in either combo or legacy.
Cybert said: Capitlization and periods are hard for you, aren't they? I've given over $100 to techforums. I should have you banned for my money.
maxxcool
Gerbil Elite
Silver subscriber
 
 
Posts: 645
Joined: Thu Sep 12, 2002 8:40 am
Location: %^&*%$$

Re: Anti-Virus Scan Bench Machine Build

Postposted on Mon Feb 10, 2014 12:53 pm

maxxcool wrote:As someone who works for one of the big three AV firms... the bottleneck will be usb versus sata and decomposing files.


Forgive my ignorance, but who would be considered the 'big three" nowadays?

I guess Norton? McAfee was bought out by Intel?
Life doesn't change after marriage, it changes after children!
anotherengineer
Gerbil Elite
 
Posts: 551
Joined: Fri Sep 25, 2009 1:53 pm
Location: Timmins, ON Canada, Yes I know, Up in the sticks

Re: Anti-Virus Scan Bench Machine Build

Postposted on Mon Feb 10, 2014 12:58 pm

anotherengineer wrote:
maxxcool wrote:As someone who works for one of the big three AV firms... the bottleneck will be usb versus sata and decomposing files.


Forgive my ignorance, but who would be considered the 'big three" nowadays?

I guess Norton? McAfee was bought out by Intel?


For the US (where i work), McAfee, Trend and Symantec... were we outside the US it'd be Sophos kypersky and ... dunno panda. But I'm here in Oregon/Cali.. ;)
Cybert said: Capitlization and periods are hard for you, aren't they? I've given over $100 to techforums. I should have you banned for my money.
maxxcool
Gerbil Elite
Silver subscriber
 
 
Posts: 645
Joined: Thu Sep 12, 2002 8:40 am
Location: %^&*%$$

Re: Anti-Virus Scan Bench Machine Build

Postposted on Mon Feb 10, 2014 10:04 pm

maxxcool wrote:
anotherengineer wrote:
maxxcool wrote:As someone who works for one of the big three AV firms... the bottleneck will be usb versus sata and decomposing files.


Forgive my ignorance, but who would be considered the 'big three" nowadays?

I guess Norton? McAfee was bought out by Intel?


For the US (where i work), McAfee, Trend and Symantec... were we outside the US it'd be Sophos kypersky and ... dunno panda. But I'm here in Oregon/Cali.. ;)



Ahh thanks. So McAfee and Trend Micro and Symantec in the US are. So M.S.E. isn't considered one of the big three?? ;)
Life doesn't change after marriage, it changes after children!
anotherengineer
Gerbil Elite
 
Posts: 551
Joined: Fri Sep 25, 2009 1:53 pm
Location: Timmins, ON Canada, Yes I know, Up in the sticks

Re: Anti-Virus Scan Bench Machine Build

Postposted on Tue Feb 11, 2014 3:44 pm

anotherengineer wrote:

Ahh thanks. So McAfee and Trend Micro and Symantec in the US are. So M.S.E. isn't considered one of the big three?? ;)



:D MS uninstalled MSE in favor of "one of those three" two + years ago on 500k units ;)
Cybert said: Capitlization and periods are hard for you, aren't they? I've given over $100 to techforums. I should have you banned for my money.
maxxcool
Gerbil Elite
Silver subscriber
 
 
Posts: 645
Joined: Thu Sep 12, 2002 8:40 am
Location: %^&*%$$

Re: Anti-Virus Scan Bench Machine Build

Postposted on Tue Feb 11, 2014 4:49 pm

maxxcool wrote:
anotherengineer wrote:

Ahh thanks. So McAfee and Trend Micro and Symantec in the US are. So M.S.E. isn't considered one of the big three?? ;)



:D MS uninstalled MSE in favor of "one of those three" two + years ago on 500k units ;)


lol that is funny, lots of faith and support in their own products I see :)

To be honest I only use malwarebytes, since I am not an avid click on everything, click yes to everything and lets see how many programs I can install in a day type person. Can't say the same for the rest of my family though :S
Life doesn't change after marriage, it changes after children!
anotherengineer
Gerbil Elite
 
Posts: 551
Joined: Fri Sep 25, 2009 1:53 pm
Location: Timmins, ON Canada, Yes I know, Up in the sticks

Re: Anti-Virus Scan Bench Machine Build

Postposted on Tue Feb 11, 2014 5:03 pm

Do any of the scanners these days use hardware acceleration for hashing? It seems like that would be a massive boost to throughput.
Desktop: FX-8350 | 32 GB | XFX Radeon 6950 | Windows 7 x64
Laptop: i7 740QM | 12 GB | Mobility Radeon 5850 | Windows 8.1.1.1.1 x64
SuperSpy
Gerbil Jedi
Gold subscriber
 
 
Posts: 1570
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: Anti-Virus Scan Bench Machine Build

Postposted on Tue Feb 11, 2014 5:21 pm

SuperSpy wrote:Do any of the scanners these days use hardware acceleration for hashing? It seems like that would be a massive boost to throughput.

Aren't most virus scanners limited by I/O speed, at least when scanning mechanical HDDs?
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37677
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer


Return to General Hardware

Who is online

Users browsing this forum: Bing [Bot] and 3 guests