Personal computing discussed

Moderators: SpotTheCat, Nemesis

 
frumper15
Gerbil XP
Topic Author
Posts: 351
Joined: Mon Jan 18, 2010 3:25 pm

UPS Failure Mode

Mon Nov 20, 2017 1:49 pm

I came into the office this morning and was greeted by an incessant beeping from the uninterruptible power supply in my server room. It is an APC Back-UPS ES 750 and it has been happily serving at its post for 5 years or so. This isn't the first UPS I've had the batteries go bad in, I have another APC - that one a Back-UPS XS 900 - that went bad a few months back that had been serving our phone system. At the time that one went bad I simply moved the phones over to the 750 and things have been happy. We don't need more than a few seconds of backup until the generator gets spun up in the case of a power outage so the 750 was more than capable of handling our few switches, small phone system and couple of quad-core Haswell servers.

The point of this all is how the dumb thing failed. In my experience with UPSs, when they are beeping you can push the button to silence them and that is your warning to take some action. Apparently the failure mode of this unit is to simply turn off immediately and refuse to power back on. So, some scrambling for another power strip to move things over to and now we're back up and running, albeit without backup power. I took the opportunity to upgrade our quality a little to a unit with Sinewave output and a display that might communicate more when it's getting ready to completely die on me: CyberPower CP1500PFCLCD PFC Sinewave UPS System https://www.amazon.com/gp/product/B0042 ... UTF8&psc=1

I'm wondering if anyone else has experienced this failure mode or could explain to me a good reason this unit can't just act as a surge protector when it's battery goes bad? I went with the Cyberpower because I have one at home that is at least 10 years old and has never given me an issue. Maybe it will fail in the same way some day, but for now I didn't want to pay for batteries in a unit that is going to leave me totally stranded that way in the future.

EDIT: I forgot to ask - what do you guys do to protect against this kind of failure mode? Our servers aren't high-end enough to have dual power supplies that I could plug into separate UPSs, but other than that is there any other solution you guys have come up with?
i7-6700K | Z170X-Ultra Gaming | 32GB DDR4-2400 | EVGA GTX 1080 | 256GB SAMSUNG SM951 M.2 | 27" ACHIEVA 2560x1440 LED | Nanoxia Deep Silence 3 | EVGA SuperNOVA 550G2 | Win10 Pro x64.
 
Glorious
Gold subscriber
Grand Admiral Gerbil
Posts: 10302
Joined: Tue Aug 27, 2002 6:35 pm

Re: UPS Failure Mode

Mon Nov 20, 2017 2:23 pm

frumper15 wrote:
The point of this all is how the dumb thing failed. In my experience with UPSs, when they are beeping you can push the button to silence them and that is your warning to take some action. Apparently the failure mode of this unit is to simply turn off immediately and refuse to power back on.


Are you saying that it just up and failed completely without mains ever going down? Like the outlet was good the entire time, but everything connected to the UPS lost power anyway?

Or, did the power go out, the UPS failed, and upon the power coming back you still can't get the UPS to turn on?
 
just brew it!
Gold subscriber
Administrator
Posts: 49707
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: UPS Failure Mode

Mon Nov 20, 2017 2:24 pm

I believe I have/had a unit that behaved that way.

In general, consumer UPSes don't seem to be very good about letting you know when the battery is getting dodgy. The only solution I've come up with is to replace the battery every ~3 years, whether you think it needs it or not.
Nostalgia isn't what it used to be.
 
Vhalidictes
Gold subscriber
Gerbil Jedi
Posts: 1668
Joined: Fri Jan 07, 2005 2:32 pm
Location: Paragon City, RI

Re: UPS Failure Mode

Mon Nov 20, 2017 2:25 pm

Hmm. Well, I don't know that a 750 is meant for anything other than under-the-desk use. I've used a ton of those and that is the normal failure mode, they won't pass power if there's a major problem, it's part of the design. That model isn't really meant for server rack use.
 
The Egg
Gold subscriber
Minister of Gerbil Affairs
Posts: 2236
Joined: Sun Apr 06, 2008 4:46 pm

Re: UPS Failure Mode

Mon Nov 20, 2017 3:00 pm

UPS batteries need to be replaced every 3-5 years. The tone of your post gives the impression that you're expecting them to last forever. 5 years is really pushing it for a server UPS, and your 10 year old home unit is just a surge protector right now.
 
MOSFET
Silver subscriber
Gerbil Team Leader
Posts: 208
Joined: Fri Aug 08, 2014 12:42 am

Re: UPS Failure Mode

Mon Nov 20, 2017 4:23 pm

I have a few BE750's and yes, they are for under the desk use, and NO they do NOT warn you if the batteries are bad. They do not have a self-test mode. You have to find out for yourself. Ask me how I know...!

I have several consumer CyberPowers as well. Don't expect different behavior. With these consumer-level UPS, you either pull the plug yourself to find out if the battery is still good, or deal with terrible software in return for very little information.

I also don't have expensive servers, and they don't have dual PSUs. I use Cyberpowers similar in size and scope to the one you mentioned, one per server. I think all the utility stuff (cable modem, switches, wireless, etc) are plugged into a BE750G. That one lays flat and has lots of room for AC/DC adapters. I use the Cyberpower virtual appliance (Agent version) to shut down ESXi servers if necessary.

Add: I also had an APC lose the ability to sense A/C last winter. It was plugged in with mains power, but no longer believed it, and by Monday morning, pretty much all the internal circuitry was toast. It was about 8 years old with regular battery replacements.
 
ludi
Darth Gerbil
Posts: 7446
Joined: Fri Jun 21, 2002 10:47 pm
Location: Sunny Colorado front range

Re: UPS Failure Mode

Mon Nov 20, 2017 7:11 pm

All the APCs we've dealt with at our office will beep to warn you of battery failure but continue to provide mains AC power to the connected equipment; however, there is no "silence/acknowledge" mode. If you press the power button then, yes, it shuts down.

This is preferable to a few Tripp-Lite units we had for a while that would simply go dark when the battery died, no warning whatsoever(!)
Abacus Model 2.5 | Quad-Row FX with 256 Cherry Red Slider Beads | Applewood Frame | Water Cooling by Brita Filtration
 
just brew it!
Gold subscriber
Administrator
Posts: 49707
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: UPS Failure Mode

Mon Nov 20, 2017 7:18 pm

Tripp Lite has been on my "avoid" list ever since I had one die by shooting sparks and smoke out the ventilation holes.
Nostalgia isn't what it used to be.
 
frumper15
Gerbil XP
Topic Author
Posts: 351
Joined: Mon Jan 18, 2010 3:25 pm

Re: UPS Failure Mode

Mon Nov 20, 2017 7:25 pm

The Egg wrote:
UPS batteries need to be replaced every 3-5 years. The tone of your post gives the impression that you're expecting them to last forever. 5 years is really pushing it for a server UPS, and your 10 year old home unit is just a surge protector right now.


You're not wrong - I don't feel like I didn't get my money's worth out of it, it was just such a funny thing that one moment everything seemed fine and the next it wouldn't power back on for anything I tried. We never lost power - it was plugged into a good outlet the whole time.

I just checked my unit at home - still chugging along - I unplugged it from the wall and it kept everythign running happily - I don't think it has enough juice to keep things going for long, but it definitely still works as expected.

I had a similar experience at my last job with a rackmount server-grade unit - it just stopped working completely. No warning, no lights indicating a problem or even an alarm - everything connected to it just - wasn't on anymore...
i7-6700K | Z170X-Ultra Gaming | 32GB DDR4-2400 | EVGA GTX 1080 | 256GB SAMSUNG SM951 M.2 | 27" ACHIEVA 2560x1440 LED | Nanoxia Deep Silence 3 | EVGA SuperNOVA 550G2 | Win10 Pro x64.
 
The Egg
Gold subscriber
Minister of Gerbil Affairs
Posts: 2236
Joined: Sun Apr 06, 2008 4:46 pm

Re: UPS Failure Mode

Mon Nov 20, 2017 10:07 pm

frumper15 wrote:
You're not wrong - I don't feel like I didn't get my money's worth out of it, it was just such a funny thing that one moment everything seemed fine and the next it wouldn't power back on for anything I tried. We never lost power - it was plugged into a good outlet the whole time.

I just checked my unit at home - still chugging along - I unplugged it from the wall and it kept everythign running happily - I don't think it has enough juice to keep things going for long, but it definitely still works as expected.

I had a similar experience at my last job with a rackmount server-grade unit - it just stopped working completely. No warning, no lights indicating a problem or even an alarm - everything connected to it just - wasn't on anymore...

I dunno what to tell you......s#!t fails. It could be as ludi mentioned...that you powered it down while trying to "silence" it, and then it refused to turn back on due to the condition of the battery. In any case, if you have an important server, you may want to invest in a higher-quality UPS, and change the battery more often than 5+ years.

As for your home unit, if the battery still holds enough charge to power your setup after 10 years, it must have managed to suck up a little bit of unicorn dust. :wink:
 
Ummagumma
Gerbil
Posts: 26
Joined: Fri May 27, 2016 9:18 pm

Re: UPS Failure Mode

Mon Nov 20, 2017 10:40 pm

frumper15 wrote:
[snip] I took the opportunity to upgrade our quality a little to a unit with Sinewave output and a display that might communicate more when it's getting ready to completely die on me: CyberPower CP1500PFCLCD PFC Sinewave UPS System https://www.amazon.com/gp/product/B0042 ... UTF8&psc=1

I'm wondering if anyone else has experienced this failure mode or could explain to me a good reason this unit can't just act as a surge protector when it's battery goes bad? I went with the Cyberpower because I have one at home that is at least 10 years old and has never given me an issue. Maybe it will fail in the same way some day, but for now I didn't want to pay for batteries in a unit that is going to leave me totally stranded that way in the future.

EDIT: I forgot to ask - what do you guys do to protect against this kind of failure mode? Our servers aren't high-end enough to have dual power supplies that I could plug into separate UPSs, but other than that is there any other solution you guys have come up with?


IMHO that UPS "upgrade" is not much from a quality perspective as both are considered "backup / standby" UPS designs. Sure, the Cyberpower units are lower cost and seem to be reliable, but sometimes it's better to spend a few more $$ for a higher quality unit. When I want a high quality UPS I choose Eaton over APC. Sure, APC builds a reliable product, but I like the monitoring features of the Eaton units when using "NUT" ("Network UPS Tools") on Linux. Even the Eaton "Eclipse" series has very good reporting while the higher end Eaton "5S" workstation/home-use series monitors everything that NUT can monitor.

There is a price difference between Cyberpower and Eaton "5S"; the Eaton "5S" is almost double the price, but you get a "line interactive" UPS where batteries can be "hot swapped" whereas the Cyberpower unit has to be powered off and then almost torn apart to replace the batteries. I have owned both brands and different models in each brand for years now.

As for the UPS that powers off after it has beeped at you incessantly, yes I have seen that. I have seen 2 causes for it: (1) a bad UPS (something shorted inside that defied efforts to repair it); and (2) dead batteries in a "backup / standby" type of UPS. A "line interactive" UPS should run even with dead batteries based on my own experience, provided the AC power is on (obviously). My older model Eaton "5S" units will run with dead batteries with the AC power on. My Cyberpower units will "act strange" with dead batteries when on AC power. One of them would power off while the other would run. Having a "line interactive" UPS unit is nice because you can change batteries at any time without having to power off it's "load" or even disconnect any exterior cables.

How do I protect against UPS failures?

I buy good quality UPS units like Eaton for my servers. For my network gear at home I use Cyberpower since the network gear is POE powered and the main POE supply is on a high quality UPS unit. I monitor all of my UPS units using "NUT". I configure all of my attached servers (after tracing their power cables) for "graceful shutdown" when "NUT" reports a UPS is running on batteries. I make a plan to change UPS batteries at the 3 year mark. If a UPS unit has been "behaving well" and the "NUT" data doesn't show any battery charging or battery voltage issues, then I revisit my plan around every 6 months up to the 5 year point. After 5 years it is generally a good idea to replace the batteries.

Hint: Another nice thing about a "line interactive" UPS is the ability to remove the batteries while the AC power is on so you can check each battery block's voltage after it has been disconnected from the UPS and surrounding battery blocks. I have done that to find "weak" battery blocks, which is a battery block that is not providing it's rated DC output even when charged possibly due to dead/dying "internal battery sections". Yes, this test procedure is risky since you are "rolling the dice" that the AC power does not go out on you, but sometimes it's a necessary risk.

What do I look at in the "NUT" data?

I start with "actual" and "nominal" values for "battery voltage"; both Eaton and Cyberpower can provide this in many of their units and it can indicate if the UPS has a bad or weak battery. I look at the "Battery Charge / Load" chart which is commonly available in most UPS units since a bad/weak battery will not charge to it's full potential. I look at "Time Left"; "runtime" when on battery and generally available in many UPS units to help me decide if UPS "loads" have to be "rebalanced" or a new UPS is needed. The graphs of AC line quality can be useful for detecting AC power issues when you are not there; also commonly available in UPS units.

How do I monitor using "NUT" software?

A Raspberry Pi connected to the UPS using the USB monitoring cable that came with the UPS works well for me; about 5W power consumption, totally quiet, and no distracting lights. I have also used old "PogoPlug" units (about 5W power consumption, totally quiet, and no distracting lights) and old Intel D510 Atom-based mini PCs (about 10W power consumption, totally quiet, and no distracting lights) as UPS monitors. As long as the monitoring device can run a commonly available Linux distribution, then the NUT package is likely in the archives for that distribution; I have not used "NUT" on a Windows OS. Even an "always on" server can be used for monitoring since "NUT" requires very few system resources to run and the software is designed with "security" in mind, like "dropping root privileges" after startup. To monitor the "NUT" data I can use the tools in the "NUT" package for a quick look, but for historical data gathering I use "Monitorix" on a dedicated monitoring platform that monitors many things along with "NUT". I haven't spent enough time getting UPS monitoring into my "Cacti" installation since my "Monitorix" is simpler to configure & use.

Yes, everything that I have said so far sounds like lots of work, but the "automation" does all of the "tedious routine work" while I make the decisions on when to change batteries and add UPS units. I place a label on all of my UPS units showing when I last replaced the batteries. I add a calendar reminder for myself to periodically check the batteries on/around the anniversary date of that battery change.

I have not configured any "alerting" mechanisms for my "NUT" installation as that aspect is not important to me in my "home use" case.

A piece of advice on UPS batteries: Buy UPS batteries that are designed for use in UPS units whenever possible. I can get the UPS-rated battery blocks I need at the local "Batteries Plus" store for about $10 more than a "standard" battery block; others may prefer other battery sellers. A UPS-rated battery should be more expensive than the same size battery that is not UPS rated, when such comparisons are possible. So what's the difference? A UPS-rated battery can be "deep cycled" (completely discharged) more times than a "standard" battery block, and that requires (at least) different construction in the plates of the battery. The "standard" batteries might handle a couple (about 2 to 3 in my experience) of "deep discharges" before they fail to recharge (dead battery), and then it's time to replace them. Finally, UPS-rated batteries should be clearly identified as "rated for UPS use" by the manufacturer; when in doubt, check it out thoroughly.
I used to do networking & network security for a living. Now I just do it for fun, but I still take it seriously.
 
Convert
Grand Gerbil Poohbah
Posts: 3382
Joined: Fri Nov 14, 2003 6:47 am

Re: UPS Failure Mode

Mon Nov 20, 2017 11:34 pm

That is the standard failure mode for that model of UPS.

Never use a UPS like that for server protection. A UPS like that is fine for routers/switches where needed though. Even with a generator you should of had something better running your servers.

I've not had good luck with CyberPower, but to each their own.

I do try to split multi-psu servers between UPS units where possible. Single PSU servers I just make sure I get a good quality UPS, nothing more you can do about it.
Tachyonic Karma: Future decisions traveling backwards in time to smite you now.
 
just brew it!
Gold subscriber
Administrator
Posts: 49707
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: UPS Failure Mode

Tue Nov 21, 2017 12:07 am

One lesson I learned the hard way a number of years back: Never buy "new old stock" UPSes on clearance, unless it's still a good price after you factor in the cost of a new battery pack. They go bad just sitting, so the factory-installed pack is probably close to worthless.
Nostalgia isn't what it used to be.
 
ludi
Darth Gerbil
Posts: 7446
Joined: Fri Jun 21, 2002 10:47 pm
Location: Sunny Colorado front range

Re: UPS Failure Mode

Tue Nov 21, 2017 12:49 am

just brew it! wrote:
They go bad just sitting, so the factory-installed pack is probably close to worthless.

Yeah, lead-acid batteries minimally need a maintenance charge every 2-3 months, else they drop below their critical voltage and the plates start to sulfate.
Abacus Model 2.5 | Quad-Row FX with 256 Cherry Red Slider Beads | Applewood Frame | Water Cooling by Brita Filtration
 
Kougar
Silver subscriber
Minister of Gerbil Affairs
Posts: 2031
Joined: Tue Dec 02, 2008 2:12 am
Location: Texas

Re: UPS Failure Mode

Tue Nov 21, 2017 8:56 pm

The Egg wrote:
I dunno what to tell you......s#!t fails. It could be as ludi mentioned...that you powered it down while trying to "silence" it, and then it refused to turn back on due to the condition of the battery. In any case, if you have an important server, you may want to invest in a higher-quality UPS, and change the battery more often than 5+ years.

As for your home unit, if the battery still holds enough charge to power your setup after 10 years, it must have managed to suck up a little bit of unicorn dust. :wink:


This is what I would put my money on.

I had an APC unit like that, trying to mute it actually just turns the whole thing off. :evil: If you want to prove the theory, disconnect the battery inside the unit then it will probably turn on and act like a power strip as you desired. That is how mine worked.

But it will beep at you until the end of time because "mute" just turns it off. This was another reason I stopped buying APC units. Cyberpower actually has a dedicated mute button on its units. :lol:

Also, for batteries I suggest you source them locally from a dedicated battery supplier. Atlex Electronics sells the same batts locally that APC relabels and sells online, but I can attest from trying both sources that the Altex batteries are not old and lasted a full ~5 years. Any online batts are probably old, especially any you find at cheaper prices.
 
Flying Fox
Gerbil God
Posts: 25418
Joined: Mon May 24, 2004 2:19 am
Contact:

Re: UPS Failure Mode

Tue Nov 21, 2017 10:07 pm

At a previous job the IT guy pretty much told us to treat those Back-UPS models as "slightly better than a power bar", and he left it at that. :lol:
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
 
just brew it!
Gold subscriber
Administrator
Posts: 49707
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: UPS Failure Mode

Tue Nov 21, 2017 11:01 pm

Flying Fox wrote:
At a previous job the IT guy pretty much told us to treat those Back-UPS models as "slightly better than a power bar", and he left it at that. :lol:

Sounds about right.
Nostalgia isn't what it used to be.
 
Kougar
Silver subscriber
Minister of Gerbil Affairs
Posts: 2031
Joined: Tue Dec 02, 2008 2:12 am
Location: Texas

Re: UPS Failure Mode

Wed Nov 22, 2017 1:58 am

Flying Fox wrote:
At a previous job the IT guy pretty much told us to treat those Back-UPS models as "slightly better than a power bar", and he left it at that. :lol:


May be true, but mine kicks in almost daily due to a brief brownout when the local substation switches, or I dunno what they're doing. At one point it would happen every 9am sharp, daily. For personal electronics I will forever use them.

I've commented before about an apartment complex I lived at that experienced underground cabling going bad 5-10 years prematurely, they would blow out the van-sized giant box transformers almost monthly somewhere in the complex before the utility or somebody finally got their rear in gear and began tearing up the parking lots to lay new cable. Which still took almost six months to complete. Every time one blew there'd be a massive power surge synced to a concussive cannon shot, was fun. They toasted more than a dozen of them. :P
 
ludi
Darth Gerbil
Posts: 7446
Joined: Fri Jun 21, 2002 10:47 pm
Location: Sunny Colorado front range

Re: UPS Failure Mode

Wed Nov 22, 2017 1:41 pm

Kougar wrote:
May be true, but mine kicks in almost daily due to a brief brownout when the local substation switches, or I dunno what they're doing.

Sounds like a large industrial load switching on the same circuit. If it's happening daily you're entitled to pester the utility, and knowing the time will enable them to correlate it. They might try to push you off at first, but be persistent.
Abacus Model 2.5 | Quad-Row FX with 256 Cherry Red Slider Beads | Applewood Frame | Water Cooling by Brita Filtration
 
SuperSpy
Gold subscriber
Minister of Gerbil Affairs
Posts: 2226
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: UPS Failure Mode

Wed Nov 22, 2017 1:57 pm

I just came across a unit like this which in infuriating. I had one of the 'flat' style Tripp-Lite 900 VA units powering my home pfSense router, wifi access point, and switch (my 'core network'). I live out in the country, so when it's extremely windy the power will flicker a few times a day, and that's usually enough for the UPS to roll over to battery for a few seconds, then back to AC.

Suddenly the unit will instead just shut off and refuse to turn on when the AC comes back, until I reach up and hit the stupid button. I don't understand how anyone would design a UPS to behave that way. I mean sure, if the battery is dead and the power goes out, the unit doesn't have much choice but to shut down. BUT when the power comes back like why in the world wouldn't the unit turn itself back on. It's literally the opposite of the desired behavior.

I replaced the battery in this unit because they only cost like $12, but I'm going to be keeping my eye out for a unit that doesn't have this insane 'feature'.
Desktop: i7-4790K @4.8 GHz | 32 GB | EVGA Gefore 1060 | Windows 10 x64
Laptop: i7 740QM | 12 GB | Mobility Radeon 5850 | Windows 10 x64

Who is online

Users browsing this forum: No registered users and 1 guest