Personal computing discussed

Moderators: renee, SecretSquirrel, just brew it!

 
SecretSquirrel
Minister of Gerbil Affairs
Posts: 2726
Joined: Tue Jan 01, 2002 7:00 pm
Location: North DFW suburb...
Contact:

Re: Getting into some pretty low-level embedded stuff again

Wed Dec 31, 2014 12:54 am

And sometimes it's your ODF...

I'm working on a project and switched from a switch based keypad to a capacitive touch based one. The new keypad is connected via i2c. Everything was working great in the test code, but when I switched the main codebase over, everything went to crap. Things worked fine until you hit a key and then the code would freeze. Now, this is an embedded system running on an 8 bit micro with 1k of RAM so in some sense it actually takes a bit to make things truly freeze. Freeze like a heartbeat LED driven by a timer overflow interrupt stops toggling. Freeze like gdb connected via JTAG stops being able to communicate with the running program.

As I said, ODF. The code has a cooperative multitasking kernel that handles several tasks as well as interfacing to the peripherals. The core of the kernel disables interrupts, checks the status of a few IO pins, checks the keypad, determines the next task to run, enables interrupts, and begins task execution. When I switched to the i2c keypad, I left the keypad check in the same location. Unfortunately, reading the touch keypad requires multiple i2c transactions. Each transaction checks to see if the prior is complete and loops if not. This loop is is implemented by looping while the i2c interrupt is enabled. This is converted to a single assembly instruction that looks something like this:

WAIT: jbs twie, WAIT


Because it is a single instruction, the debugger cannot insert a break in the loop. So, as soon as the second i2c transaction was started, program would go into an unbreakable loop. The prior i2c transaction would never complete as it was driven by a interrupt service routine and interrupts were disabled globally.

I hate these kinds of self inflicted bugs...

--SS
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Getting into some pretty low-level embedded stuff again

Wed Dec 31, 2014 10:34 am

The bug which was the original subject of this thread was also very much self-inflicted, if perhaps a bit more convoluted. Definitely a facepalm moment!

Edit: While dealing with these kinds of obscure bugs comes with the territory in "bare metal" embedded work, I really like the sense of control you have over the hardware. Everything is laid bare, oftentimes all the way down to the level of individual signals on physical wires. You're dealing with fewer "black boxes"... it's just you, the hardware, and the (admittedly sometimes buggy) development tools. I'm also starting to think that my move "down-stack" is a good thing at this point in my career; a number of factors seem to be contributing to a shortage of engineers with solid experience in this sort of low-level software development.
Nostalgia isn't what it used to be.
 
SecretSquirrel
Minister of Gerbil Affairs
Posts: 2726
Joined: Tue Jan 01, 2002 7:00 pm
Location: North DFW suburb...
Contact:

Re: Getting into some pretty low-level embedded stuff again

Wed Dec 31, 2014 1:00 pm

just brew it! wrote:
While dealing with these kinds of obscure bugs comes with the territory in "bare metal" embedded work, I really like the sense of control you have over the hardware. Everything is laid bare, oftentimes all the way down to the level of individual signals on physical wires. You're dealing with fewer "black boxes"... it's just you, the hardware, and the (admittedly sometimes buggy) development tools. I'm also starting to think that my move "down-stack" is a good thing at this point in my career; a number of factors seem to be contributing to a shortage of engineers with solid experience in this sort of low-level software development.


I agree whole heartedly. I greatly prefer low level embeded work. As far as the lack of skilled engineers go, I think that is actually driven by the prevelance of high power hardware. Low level embedded development, these days, seems to be defined as using a Raspberry PI (BeagleBone, etc) running Python. There do not seem to be a whole lot of degree programs that teach low level development on regular PC hardware, much less on bare microcontrollers. The engineers that had no choice buy to develop in PIC or 8051 assembler are aging out of the workforce and are not, seemingly, being replaced.

I also thing this is contributing to the general flakiness of our automated devices. When you use an bare microcontroller to run your coffe machine, you can pretty well -- with a bit of though -- account for every possible input/error condition and handle them in an appropriate way. When your coffee maker has an init file, internal message queues, and boots an RTOS, things are going to fail in very strange ways.

That said, I am considering moving over to a Rasberry Pi for my project. I'm weighing the trade offs of having to maintain a full Linux environment vs the ease of development, especially for the UI, and the ability to have real a properly multithreaded environment.

--SS
--SS
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Getting into some pretty low-level embedded stuff again

Wed Dec 31, 2014 1:36 pm

SecretSquirrel wrote:
just brew it! wrote:
While dealing with these kinds of obscure bugs comes with the territory in "bare metal" embedded work, I really like the sense of control you have over the hardware. Everything is laid bare, oftentimes all the way down to the level of individual signals on physical wires. You're dealing with fewer "black boxes"... it's just you, the hardware, and the (admittedly sometimes buggy) development tools. I'm also starting to think that my move "down-stack" is a good thing at this point in my career; a number of factors seem to be contributing to a shortage of engineers with solid experience in this sort of low-level software development.

I agree whole heartedly. I greatly prefer low level embeded work. As far as the lack of skilled engineers go, I think that is actually driven by the prevelance of high power hardware. Low level embedded development, these days, seems to be defined as using a Raspberry PI (BeagleBone, etc) running Python. There do not seem to be a whole lot of degree programs that teach low level development on regular PC hardware, much less on bare microcontrollers. The engineers that had no choice buy to develop in PIC or 8051 assembler are aging out of the workforce and are not, seemingly, being replaced.

Indeed. It is tempting for a product designer to "take the easy way out" and just throw hardware and a full Linux/Android (or even Windows!) stack at the problem! But this can have its own costs in complexity, security, reliability, maintainability, and a general reliance on too many third-party "black box" subsystems. And unless you're using an off-the-shelf RPi, PC, or similar device, someone still has to deal with the BSP and initial board bring-up.

FPGA and soft-core CPU also seems to be on the rise, now that FPGAs are fast enough to permit soft implementations of reasonably capable 32-bit CPU cores. A lot of the same design principles formerly used for hard-wired embedded systems carry over to the new FPGA soft-core world. Having also done a little VHDL development a few years ago (even further "down-stack" than what I'm doing now!), I think I may have a leg up on this niche since I am comfortable with both sides of FPGA development.

SecretSquirrel wrote:
I also thing this is contributing to the general flakiness of our automated devices. When you use an bare microcontroller to run your coffe machine, you can pretty well -- with a bit of though -- account for every possible input/error condition and handle them in an appropriate way. When your coffee maker has an init file, internal message queues, and boots an RTOS, things are going to fail in very strange ways.

I firmly believe that once the complexity of a software system gets to the point where it is impossible for a small team (preferably one person!) to understand *everything* that is going on from the user interface and other external I/O all the way down to the hardware, there will inevitably be some things that fall through the cracks, especially when development is constrained by time and/or budget. (And if you think there are projects that don't have time and/or budget constraints you are either smoking something, or an ivory tower academic.)

This reminds me of an incident about 10 years ago, where our (somewhat new at the time) Chrysler mini-van decided that it wanted to blink the headlights every couple of seconds. It didn't matter whether the ignition was on or off, or what position the headlight switch was in. It was just going to keep making the headlights blink, damnit! Everything else in the vehicle appeared to still be operating normally. After much head-scratching and Googling, I finally just disconnected the battery cable for a few minutes, effectively forcing a cold boot of all onboard systems. I never did figure out what was wrong, or what the on-board computer systems might have been trying to tell us (I suspect it was just the equivalent of a BSOD from the microcontroller tasked with operating the headlights...), and to this day the problem has never occurred again.

SecretSquirrel wrote:
That said, I am considering moving over to a Rasberry Pi for my project. I'm weighing the trade offs of having to maintain a full Linux environment vs the ease of development, especially for the UI, and the ability to have real a properly multithreaded environment.

Yup, there's something to be said for leveraging all of that existing code. The correct answer is really a case-by-case thing!
Nostalgia isn't what it used to be.
 
notfred
Maximum Gerbil
Posts: 4610
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: Getting into some pretty low-level embedded stuff again

Wed Dec 31, 2014 3:04 pm

Got to suggest my day job, QNX, as a solution :)
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Getting into some pretty low-level embedded stuff again

Wed Dec 31, 2014 3:11 pm

notfred wrote:
Got to suggest my day job, QNX, as a solution :)

My current day job project is using MicroC/OS-II. Not sure how it compares to QNX... QNX looks like it may be a bit higher-level (more UNIX-like)?
Nostalgia isn't what it used to be.
 
notfred
Maximum Gerbil
Posts: 4610
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: Getting into some pretty low-level embedded stuff again

Thu Jan 01, 2015 5:10 am

Yes, QNX is fully POSIX compliant. It's a micro-kernel though so a lot of the usual stuff that is in the kernel is running in user space so if you don't need it you don't run it.

It's a bit bigger than MicroC/OS-II, running on ARMv7 and x86 rather than the smaller micro controllers.
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Getting into some pretty low-level embedded stuff again

Thu Jan 01, 2015 10:49 am

notfred wrote:
Yes, QNX is fully POSIX compliant. It's a micro-kernel though so a lot of the usual stuff that is in the kernel is running in user space so if you don't need it you don't run it.

It's a bit bigger than MicroC/OS-II, running on ARMv7 and x86 rather than the smaller micro controllers.

That was more or less my impression based on a cursory evaluation.

It also amuses me that these days those "smaller microcontrollers" have an order of magnitude more RAM than an early 1990s PC! :lol:
Nostalgia isn't what it used to be.

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On