TR Forums

You'd think that with as how much people despise it on Windows for randomly chewing up system resources, Linux distros would stay away from this concept. But no... periodically, I need to fight this battle over again, even on Linux. Invariably, at some point I'll notice that some process on the system is chewing up CPU and I/O. Making matters worse, these indexers frequently don't "play nice" when you've got network shares and/or VMs which are importing host file systems; because of this, a desktop will start indexing file servers over the network, or a VM will start indexing its host.

Some recently installed update on my home desktop apparently re-enabled KDE's indexing service behind my back. I just caught my primary desktop attempting to index the contents of my NFS server.

IF YOU FEEL A NEED TO INCLUDE A GODDAMN INDEXER WITH YOUR DISTRO OR DE, MAKE THE @%$&! THING OPT-IN!!

(Just felt that I had to vent, this is a pet peeve.)

I know this is a Linux rant, but my favorite is when the Windows antimalware service decides whatever the indexer is looking at is very important and must be scanned right away. So instead of laggy I/O and one loaded core, you end up with two cores fighting over the same laggy I/O.

SuperSpy wrote:

I know this is a Linux rant,

Well, only sort of. It's more of a general rant about indexing services. Nobody seems to have been able to develop one that truly "gets it right", ever... though many have tried.

Every PC OS since the early 1980s has had a hierarchical file system which allows you to create subdirectories/folders. Organize your damn files! If you're relying on an indexing service to help you find stuff in your own computer, YOU'RE USING THE COMPUTER WRONG!

By creating buggy indexing services that chew up CPU, disk space, and network bandwidth unnecessarily, software vendors are catering to the organizationally impaired people who can't keep track of where they put their own files, and penalizing everyone else.

If you want to use an indexer, fine. But it should be off by default.

Ugh, don't remind me. I caught my servers indexing all of the mounted filesystems after an upgrade. 20+ PB of scratch and archive filesystems with billions of files.

I wouldn't care if they just did it properly...but nobody does. Don't cross filesystem boundaries! How hard is that! Certainly never go over the network...

I don't even mind the fact that indexers exist. But every one I've seen breaks the cardinal sin of getting in the way of the user. What's so hard about designing them so that they:

- Only run when the computer is locked/user is away
- Only run with low priority or throttled I/O
- Only run when fully charged and plugged in
- Don't pin the CPU/core at maximum load (even better would be to not allow the CPU to exit it's idle frequency/power bracket)
- Don't allow the OS to cache indexer reads so that the scans thrash the disk cache
- Stop as soon as user activity is detected

SuperSpy wrote:

I know this is a Linux rant, but my favorite is when the Windows antimalware service decides whatever the indexer is looking at is very important and must be scanned right away. So instead of laggy I/O and one loaded core, you end up with two cores fighting over the same laggy I/O.

Oof. All of my rage right there. Coupled with that is the idiotic decision to do this on battery... I've felt the bottom of the laptop warming up while doing something mundane, only to find indexing and malware scanning have eaten a sizeable chunk of my limited mobile time.

SuperSpy wrote:

I don't even mind the fact that indexers exist. But every one I've seen breaks the cardinal sin of getting in the way of the user. What's so hard about designing them so that they:

- Only run when the computer is locked/user is away
- Only run with low priority or throttled I/O
- Only run when fully charged and plugged in
- Don't pin the CPU/core at maximum load (even better would be to not allow the CPU to exit it's idle frequency/power bracket)
- Don't allow the OS to cache indexer reads so that the scans thrash the disk cache
- Stop as soon as user activity is detected

Exactly, thank you.

SuperSpy wrote:

I know this is a Linux rant, but my favorite is when the Windows antimalware service decides whatever the indexer is looking at is very important and must be scanned right away. So instead of laggy I/O and one loaded core, you end up with two cores fighting over the same laggy I/O.

Waco wrote:

Don't cross filesystem boundaries! How hard is that! Certainly never go over the network...

This.

In one particularly egregious incident (on a work PC several years ago), I had a copy of Ubuntu running in a VM. This VM had a mount point for a directory which had been shared into the VM from the host system; the directory on the host in turn contained a symlink to another file system which was a remote NFS mount. Stupid indexer in the VM crossed not one, but TWO file system boundaries, and when I went to investigate why my workstation was hammering the network, discovered that the VM was attempting to index the entire office file server.

I'll just echo what SuperSpy said about doing at least a little effort at detecting system load before running a scan.

Somehow it seems that all indexers fail at that, and hard. For what it's worth, Windows 10 is the first OS where I haven't straight away killed the indexer. Up until now, that was one of the first things to go.

Oh, even applications aren't immune to that problem. Whoever had the brilliant idea of making that stupid Thunderbird "search with a weight and graphs" based on an indexer that "only works when the system is idle" but then proceeds to completely bog down your machine, especially the HDD... Good thing I dropped TB like a rock multiple years ago.

just brew it! wrote:

In one particularly egregious incident (on a work PC several years ago), I had a copy of Ubuntu running in a VM. This VM had a mount point for a directory which had been shared into the VM from the host system; the directory on the host in turn contained a symlink to another file system which was a remote NFS mount. Stupid indexer in the VM crossed not one, but TWO file system boundaries, and when I went to investigate why my workstation was hammering the network, discovered that the VM was attempting to index the entire office file server.

Yep, I had something similar happen. A symlink on an NFS share (that it shouldn't have been in anyway) and it followed the link to another filesystem that was remotely mounted, which had a link to something else, which linked back to the NFS share...

It didn't even recognize it was going in circles. Sigh. I caught it after a few days since it of course started over the weekend...

This is a particular pain point because us, the techies, will notice something's up and start digging and eventually reach a culprit.

Since these indexing services come enabled by default, Regular Joe's reaction is that "the computer is broken, it's really slow" and it won't show any apparent symptoms if he wishes to try and troubleshoot it, or even when getting help from a tech person. The problem itself is invisible for the most part.

I still disabled Windows Search on Windows 10 going by past experience even though Microsoft made Windows Media Player Network Sharing Service dependent on Windows Search in favor of Agent Ransack for searching.

Agent Ransack - Free File Searching Utility

Since we're in this topic, I'm a fan of BinaryFortress' FileSeek

I dunno, updatedb runs at 4:40 every morning. It's one way I know it's time for bed.