TR Forums

Hackaday wrote:

The Japanese X-ray telescope Hitomi has been declared lost after it disintegrated in orbit, torn apart when spinning out of control. The cause is still under investigation but early analysis points to bad data in a software package pushed shortly after an instrument probe was extended from the rear of the satellite. JAXA, the Japanese space agency, lost $286 million, three years of planned observations, and a possible additional 10 years of science research.

Hitomi, also known as ASTRO-H, successfully launched on February 17, 2016 but on March 26th catastrophe struck, leaving only pieces floating in space. JAXA, desperately worked to recover the satellite not knowing the extent of the failure. On April 28th they discontinued their efforts and are now working to determine the reasons for the failure, although a few weeks ago they did provide an analysis of the failure sequence at a press conference.

More to be read at the source link.
Source: http://hackaday.com/2016/05/02/software ... satellite/

Edit: Clipped the quoted article down to a couple of paragraphs.

I read this, it just makes me cringe. Such a cool project and it gets taken down by a bad patch.

Chuckaluphagus wrote:

I read this, it just makes me cringe. Such a cool project and it gets taken down by a bad patch.

Mars Climate Orbiter, lost because one sensor output data in pound-seconds and the guidance unit was expecting newton-seconds.

"Captain, she canna take it much more!"

"Oops, something happened."

There's a long list of satellites/probes done in by sloppy coding going all the way back to Mariner 1 in 1962.

Whoops.

I'm reminded of V'Ger for some reason. A different kind of destruction, I guess.

Captain Ned wrote:

There's a long list of satellites/probes done in by sloppy coding going all the way back to Mariner 1 in 1962.

"Measure twice, cut once" philosophy doesn't seem to "get through" in the software-world. It's like: "well, I can always change the code..".

Plus, if it's at all attached to Gov't contracts - it just doesn't "pay" to be really thorough if there isn't someone's life at stake. ie. I need more funding so I can f***-up again.

CScottG wrote:

Plus, if it's at all attached to Gov't contracts - it just doesn't "pay" to be really thorough if there isn't someone's life at stake. ie. I need more funding so I can f***-up again.

The real problem is that every new mission gets spread out through the entire NASA infrastructure, as every Senator has to bring home a slice of the pie. This diffuse org structure leaves no single entity in charge with any ability to tell JPL that they don't sync with Goddard, the booster team still doesn't have the final volume & mass distribution specs for the payload and neither JPL or Goddard will listen, and no one has actually looked at the entire mess from a point of command to say "quit yer funding wars".

I've spent 20 years working in state-level gov't. We have none of the cannibalistic funding fights currently going on among the last bits of our old space infrastructure, yet we're still slow and inefficient. Add the desire to preserve old empires from the Saturn & Shuttle days and it's a wonder that anything NASA launches actually works.

In the previous press briefings, JAXA has reported 3 events, “attitude anomaly”, “objects separation” and “communication anomaly”, occurred around the in-flight anomaly of ASTRO-H at 16:40(JST), March 26, 2016.

What the heck happened over at the Japan Space Agency that resulted in three separate reported failures?

Link to the pdf of the press conference.
http://global.jaxa.jp/projects/sat/astr ... 160415.pdf

From what I understand the failed update sent Hitomi spinning out of control (attitude anomaly), which led to parts of it coming off (objects separation). Of course, having spun out of control and lost parts it eventually lost communication (communication anomaly). So yeah, three reported failures, all of which point to one probable cause.

Windows 10 forced updates strikes again.

CScottG wrote:

Plus, if it's at all attached to Gov't contracts - it just doesn't "pay" to be really thorough if there isn't someone's life at stake. ie. I need more funding so I can f***-up again.

It's really not like that.

I made the mistake of doing a Linux kernel upgrade this morning and spent the rest of the day cleaning up the smoking wreckage. I have to admit, I feel a little bit better after reading this thread.

NovusBogus wrote:

I made the mistake of doing a Linux kernel upgrade this morning and spent the rest of the day cleaning up the smoking wreckage. I have to admit, I feel a little bit better after reading this thread.

Several weeks ago, I clicked the Upgrade button on my wireless controller software and by just letting it do its job, my Ubiquiti UAC-AP-PRO access point bricked itself somewhere in the process. Since part of my network (namely the SO's office) is bridged via that wireless link, I spent several hours re-building another option when I couldn't recover the AP via TFTP. And yes, I feel a little better about somebody's several hundred million satellite having the same thing happen...and I doubt they have a serial port to recover that puppy with.

Ubiquiti came through and shipped me a new access point and a free CloudKey PC-on-a-stick controller for my troubles so I don't have to run a controller VM any more. So all's well that ends well.

Darkmage wrote:

CScottG wrote:

Plus, if it's at all attached to Gov't contracts - it just doesn't "pay" to be really thorough if there isn't someone's life at stake. ie. I need more funding so I can f***-up again.

It's really not like that.

It was definitely an exaggeration. Mostly.

Still, when it comes to really large contractors, particularly US contractors and particularly US Fed. agencies "pitching" there ideas to Congress and then receiving bids and "following through" on those contracts, it's not that far off.

Lockheed Martin & Northrup Grumman are pretty awful in this regard - and that's even if someone's life is at stake. Try reading up on the F35.

As for NASA:

Search for critical commentary on Constellation. Then search for critical commentary on SLS. (..and these are just two recent examples of dominate projects within NASA.)

-you might even want to look at a major contractor involved with these programs.

biffzinker wrote:

In the previous press briefings, JAXA has reported 3 events, “attitude anomaly”, “objects separation” and “communication anomaly”, occurred around the in-flight anomaly of ASTRO-H at 16:40(JST), March 26, 2016.

What the heck happened over at the Japan Space Agency that resulted in three separate reported failures?

They are almost certainly all just consequences/symptoms of the single, larger failure. The three events you listed could be translated as: "Uh-oh, we seem to be losing control...", "WTF, it looks like it is breaking up?!??", and "Well, crap... it doesn't seem to be communicating any more... guess it's time to find another job".

Sounds like a pretty logical sequence to me! (And my version may be closer to what was actually said in the control room, for all I know...)

CScottG wrote:

It was definitely an exaggeration. Mostly.

Definitely.

It's not like there is some grand conspiracy that goes "Hey, Fred... don't actually make that code run. Because if you do, we'll be out of a job." You're trying to fulfill unique requirements in a highly restrictive environment full of byzantine rules put there for political purposes. You do the best with what you have and what you have isn't particularly good.

Hell, even if the F-35 flew like a dream and all the systems worked correctly the first time, those large companies are guaranteed a solid income for the next 30 years in parts, maintenance and training contracts. The logic of someone sandbagging their performance in order to keep the money flowing doesn't make logical sense.

I wonder if the cause was cultural "hate to disappoint the superiors" and the change in question got pushed before the final testing had completed, because the superiors were expecting that the testing had already completed.

bronek wrote:

I wonder if the cause was cultural "hate to disappoint the superiors" and the change in question got pushed before the final testing had completed, because the superiors were expecting that the testing had already completed.

Could be. But that sort of thing isn't limited to Japanese culture; a lot of organizations here in the US have a "shoot the messenger" management mentality, which fosters similar behavior up and down the chain of command.

It's (usually) not active sabotage that makes government work slower and more expensive. It's a combination of the regulations, as one person noted, and the usual total lack of any incentive to do better.

If anyone needs proof, SpaceX; they brought private sector vigor to the fight, partly because they're young and not poisoned by the culture that comes over government contractors, and partly because they've probably always known it'd be a knife fight to win business from incumbents. Compare the costs of NASA's over-engineered sky crane delivering 900kg to the Martian surface and SpaceX's plan to leave 6 tons after a propulsive landing. NASA, 2.5 billion, SpaceX hasn't said but one estimate is around $350 million. Even if one strips out a whole billion dollars from NASA's cost to make allowances for a kitchen sink of excuses, SpaceX still has a huge advantage.

That's because SpaceX knows it has to continuously do better to survive, where Boeing and Lockheed and others have known for decades they'll get to ghost write the specifications of bids, and might not even need to clear the low bar they set for themselves to get the fat contract. Boeing and Lockheed will get the enterprising spirit revived, though; I think they know their survival is at risk in the launch market.

w76 wrote:

Boeing and Lockheed will get the enterprising spirit revived, though; I think they know their survival is at risk in the launch market.

Knowing they need to do it and having the ability to pull it off are not the same thing.

w76 wrote:

Windows 10 forced updates strikes again.

First consumer-level home computers, now million-dollar satellites...what next?

How about a bricked Windows 7 system?

Microsoft inadvertently bricked a vast number of PCs running Windows 7, by changing the priority of an erroneous software update. Earlier this month, Microsoft changed the priority of an obscure-sounding security update for Windows 7 from "Optional" to "Recommended," (which by default gets automatically downloaded an installed). This update, KB3133977, bricks machines running ASUS motherboards, in the UEFI mode.

Sounds like someone here at the forums was hit by it.

anotherengineer wrote:

Doh, might miss it

http://www.techpowerup.com/222238/micro ... therboards

I'm wondering when we will be reading a story of a satellite that gets owned and malware encrypted.

johnnywheels wrote:

w76 wrote:

Windows 10 forced updates strikes again.

First consumer-level home computers, now million-dollar satellites...what next?

The Azure platform. Have fun putting all your important stuff on the 2-3 megacorps' servers, kids. Muahahahahahahahahahaha.

That's doubly unfortunate. Hate the loss of any astronomical science, but it's also bad the thing flung yet more debris into completely random orbits in the process. It won't take many more of those before losing satellites to space debris becomes a regular thing.

Over 10% of the golf ball size and larger objects in orbit around our planet were put there on January 11, 2007 when China created over 2,500 pieces of large debris and over 150,000 pieces of smaller debris... on purpose. At least one Russian satellite has already been lost due to a 2013 collision with the debris from this Chinese act. Much of that debris will still be a hazard 30+ years later.

"It's cool guys, we can cut down on testing time so we can meet the deadline/budget/whatnot." <-- says every project manager ever.

Oh, look what happened.

Not that I'm a bitter test analyst or anything.