Personal computing discussed

Moderators: renee, SecretSquirrel, notfred

 
Madman
Minister of Gerbil Affairs
Topic Author
Posts: 2317
Joined: Tue Apr 01, 2003 4:55 am
Location: Latvia

How does the software compiling/install works in Linux

Sun May 27, 2012 8:48 am

Luckily, Linux has moved to automatic packages and everything, and is usable for novices nowadays. But since I'm interested in internals a little bit, I would like to understand how does the software install from sources work.

Sometimes you get the sources, makefile and config files to build the source. But I have never fully understood such distribution internals, even though I'm a programmer myself, I've been programming mostly with MS stack.

Is the approach standartized? Makefiles are not just the compile script, they can copy files to system locations, create symlinks, envvars and so on.

So basically, sometimes you get a source to ~\Download, run make config, make, and some magic happens. Unless there are a lot of intermediate steps required.

But I don't fully understand the magic here. Does the source get's copied to /bin? /sbin? /opt?, is it symlinked, how is the binary linked for other users, is it safe to delete the source afterwards? Or do you have to review every single makefile in makefile chain to figure out what's happening?

Does anyone know a short, but good description on how this all works, or should work?
Core 2 Duo E6300, MSI P45 NEO-F, Club 3D GTX 260, 4Gb DDR2-800Mhz, Audigy X-Fi Fatal1ty Champ1on ed., 0.5Tb+1Tb Seagate Barracuda 7200.12, 630W AXP, Samsung SyncMaster BX2450, ViewSonic VP171b
 
DancinJack
Maximum Gerbil
Posts: 4494
Joined: Sat Nov 25, 2006 3:21 pm
Location: Kansas

Re: How does the software compiling/install works in Linux

Sun May 27, 2012 9:11 am

Here is some info on makes. Hope this help understand them a little better.

http://frank.mtsu.edu/~csdept/Facilitie ... troduction

That gives some decent information. Obviously it doesn't go into the depth you'd like, but it's a decent place to start. I highly suggest you try and program in a *nix environment to try everything out. In college I started out programming(C++) at the command line in Gentoo and it really helped me learn a lot about how these things work.

There are quite a few people here that can explain this better. I haven't programmed in years now.
i7 6700K - Z170 - 16GiB DDR4 - GTX 1080 - 512GB SSD - 256GB SSD - 500GB SSD - 3TB HDD- 27" IPS G-sync - Win10 Pro x64 - Ubuntu/Mint x64 :: 2015 13" rMBP Sierra :: Canon EOS 80D/Sony RX100
 
bthylafh
Maximum Gerbil
Posts: 4320
Joined: Mon Dec 29, 2003 11:55 pm
Location: Southwest Missouri, USA

Re: How does the software compiling/install works in Linux

Sun May 27, 2012 9:37 am

I'd recommend installing Stow to manage your locally-compiled packages. To make use of it, you'll need to type "./configure --prefix=/usr/local/stow/PACKAGENAME-version [1]" (in addition to other arguments to the "./configure" command), and then once the package is installed you cd to /usr/local/stow and type "sudo stow PACKAGENAME-version" to install symlinks into the /usr/local/ hierarchy. When you want to remove them (e.g. to upgrade to a newer version) you cd to /usr/local/stow and type "sudo stow -D PACKAGENAME-version".

[1] for instance "/usr/local/stow/nano-2.0.4".
Hakkaa päälle!
i7-8700K|Asus Z-370 Pro|32GB DDR4|Asus Radeon RX-580|Samsung 960 EVO 1TB|1988 Model M||Logitech MX 518 & F310|Samsung C24FG70|Dell 2209WA|ATH-M50x
 
Althernai
Gerbil
Posts: 88
Joined: Thu Nov 19, 2009 8:02 pm

Re: How does the software compiling/install works in Linux

Sun May 27, 2012 11:08 am

Madman wrote:
Luckily, Linux has moved to automatic packages and everything, and is usable for novices nowadays. But since I'm interested in internals a little bit, I would like to understand how does the software install from sources work.

The easiest way is to look at some of the automatic package managers and figure out what they do. Here is the user guide for RPM. The first chapter more or less explains what must be done to install a package and why the automatic package managers are a good idea.

But I don't fully understand the magic here. Does the source get's copied to /bin? /sbin? /opt?, is it symlinked, how is the binary linked for other users, is it safe to delete the source afterwards? Or do you have to review every single makefile in makefile chain to figure out what's happening?

Once you have built the binary, this is what will be use and it won't care even if you delete the source altogether. All you need is a soft link to the binary somewhere in your PATH (e.g. /usr/bin). Try it yourself with Hello World.

I don't think there is a standard way to install stuff. Some of the software I've dealt with has a "make install" option which seems to do everything via the Makefile while other software uses shell or Python scripts for the installation and the Makefile only for compiling.
 
Madman
Minister of Gerbil Affairs
Topic Author
Posts: 2317
Joined: Tue Apr 01, 2003 4:55 am
Location: Latvia

Re: How does the software compiling/install works in Linux

Sun May 27, 2012 11:51 am

I have done some programming in Linux, the difference is that hello world is a binary, usually even without .so or .o whichever it was for dynamic link libraries. But the packages like, for example, Python are executables + libs + libs in hierarchy, like libcwhateverversion.so.

And the second thing is that, usually, linked executable is copied to source relative folder for example make at ~/Projs/SomeApp/src/* would seem to make ~/Projs/SomeApp/bin/runme, at least that's how it works in Windows. Which would break if the SomeApp folder is deleted but some sort of symmlink is left in the system.

Ok, I'll try to read the materials to understand.
Core 2 Duo E6300, MSI P45 NEO-F, Club 3D GTX 260, 4Gb DDR2-800Mhz, Audigy X-Fi Fatal1ty Champ1on ed., 0.5Tb+1Tb Seagate Barracuda 7200.12, 630W AXP, Samsung SyncMaster BX2450, ViewSonic VP171b
 
Flatland_Spider
Graphmaster Gerbil
Posts: 1324
Joined: Mon Sep 13, 2004 8:33 pm

Re: How does the software compiling/install works in Linux

Sun May 27, 2012 12:35 pm

The info on the FreeBSD Ports collection is probably the best overview of how compilation on a Unix-like operating system works.

The Ports Collection (Kind of dated, but it's clearer then the newer version. http://www.freebsd.org/doc/en_US.ISO885 ... ports.html)
http://docs.freebsd.org/doc/2.1.5-RELEA ... ook24.html

FreeBSD Developers' Handbook
http://www.freebsd.org/doc/en_US.ISO885 ... -handbook/

Porters' Handbook
http://www.freebsd.org/doc/en_US.ISO885 ... index.html

PMake - A Tutorial
http://www.freebsd.org/doc/en_US.ISO885 ... index.html

RHEL 6 Developer Guide
http://docs.redhat.com/docs/en-US/Red_H ... index.html
 
hhhoudini
Gerbil
Posts: 21
Joined: Wed Feb 15, 2012 2:09 pm

Re: How does the software compiling/install works in Linux

Sun May 27, 2012 3:34 pm

Madman wrote:
Does anyone know a short, but good description on how this all works, or should work?


Very short description of the system used to build a great deal of the software on most linux distros.

http://en.wikipedia.org/wiki/GNU_build_system

There are some alternate systems that have come into existence, due to some real or precieved shortcomings in auto tools.

http://en.wikipedia.org/wiki/CMake
http://en.wikipedia.org/wiki/SCons

The Filesystem Hierarchy Standard.

http://www.pathname.com/fhs/

A fairly concise description of the "Standard" that most distros at least pay some attention to. It should be noted that prototype versions of some system software (written by folks at RedHat/Fedora) may change some of the file system layouts in coming versions of most distros.

HTH
 
notfred
Maximum Gerbil
Posts: 4610
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: How does the software compiling/install works in Linux

Sun May 27, 2012 6:48 pm

In a short description, typically the "configure" script should sort out how it works on your distribution and create the appropriate files including Makefiles and header files. You run "make" and it will build a binary (or library) runable only by you in your own directory. After you've checked that it works you then become root and run "make install" and it will copy the binary plus any other needed bits (man files, default config files) in the places that the configure script worked out. Your program (or library) is now usable by every user.

After the "make install" you can blow away the source directory and the binaries that you used for building because it did a copy. Distribution packages like .rpm or .deb files just contain what the "make install" did, no source or other intermediate steps. That is unless you are using a distribution that builds everything from scratch like Gentoo. The Gentoo users will reply next week once they have finished building their latest updates ;-)
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: How does the software compiling/install works in Linux

Mon May 28, 2012 11:42 am

notfred summed it up really well.

Something else to keep in mind is that you don't need to be root to build a package, but you need to be root to install it to a shared location where other users (or the system, in the case of a system service) can run it. Depending on the package, it may be possible to run it locally (from your home directory) without installing it into the system directories.

The exact locations of files can vary slightly from distro to distro, and there isn't always a hard-and-fast rule for where something belongs. E.g. you will sometimes see "optional" components installed under the /opt directory hierarchy, and other times they get put under /usr.

Regarding the distinction between the different types of object code files:

A binary (typically without any extension) is a ready-to-run file containing executable machine code; in order for the system to recognize it as a runnable binary, the execute (x) bit must be set in its directory entry.

.o files are compiled object code corresponding to a single source code file. A .o file is not runnable as-is; it must be processed by the linker to resolve references to any other modules that make up the application it is a part of, and to any statically and dynamically linked libraries it uses (the linker is what produces the final binary).

.a files are searched by the linker to resolve references to library functions. They can contain actual object code of the library (for static linking), or information which the linker can embed in the binary to allow the library function to be found at runtime (dynamically linked shared libraries).

.so files contain the executable code for shared dynamically linked libraries.
Nostalgia isn't what it used to be.
 
PenGun
Gerbil Elite
Posts: 893
Joined: Fri Jun 18, 2004 1:48 pm
Location: BC Canada
Contact:

Re: How does the software compiling/install works in Linux

Mon May 28, 2012 12:27 pm

Almost right notfred. A good explanation except a program is not a library. In fact one of the main reasons a ./configure will fail is missing libraries.

A library, a dll in windows, a .so in *nix is what it says a collection of books. ;) Well a collection of data and information anyway. This is shared information any program can call on, for the information in that library.
Fuji X-E1 Leica Elmar 135 4 XF60mm 2.4 Macro | Zeiss FE 35mm 2.8
http://carnagepro.com
"Everything ... they eat everything, and fear is their bacon bits."
 
notfred
Maximum Gerbil
Posts: 4610
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: How does the software compiling/install works in Linux

Mon May 28, 2012 7:26 pm

PenGun wrote:
Almost right notfred. A good explanation except a program is not a library. In fact one of the main reasons a ./configure will fail is missing libraries.
Yup and I didn't think I said that a program was a library. Some packages build a library rather than an executable, that's how you fix the missing libraries if you cannot get it from your distribution, that's all I was trying to get across.

One of the best guides I have found on how Unix style shared libraries work was the Sun "Linkers and Libraries Guide", now owned by Oracle but still available from them on the web at http://docs.oracle.com/cd/E19253-01/817-1984/ I've also skimmed bits out of Ulrich Drepper's paper "How To Write Shared Libraries" http://www.akkadia.org/drepper/dsohowto.pdf I really need to sit down and go through it in detail
 
PenGun
Gerbil Elite
Posts: 893
Joined: Fri Jun 18, 2004 1:48 pm
Location: BC Canada
Contact:

Re: How does the software compiling/install works in Linux

Mon May 28, 2012 11:05 pm

notfred wrote:
PenGun wrote:
Almost right notfred. A good explanation except a program is not a library. In fact one of the main reasons a ./configure will fail is missing libraries.
Yup and I didn't think I said that a program was a library. Some packages build a library rather than an executable, that's how you fix the missing libraries if you cannot get it from your distribution, that's all I was trying to get across.

One of the best guides I have found on how Unix style shared libraries work was the Sun "Linkers and Libraries Guide", now owned by Oracle but still available from them on the web at http://docs.oracle.com/cd/E19253-01/817-1984/ I've also skimmed bits out of Ulrich Drepper's paper "How To Write Shared Libraries" http://www.akkadia.org/drepper/dsohowto.pdf I really need to sit down and go through it in detail

"Your program (or library)" is what you said and it's true you probably meant compiling a library. It was just a bit confusing is all. Well to me anyway. :oops:
Fuji X-E1 Leica Elmar 135 4 XF60mm 2.4 Macro | Zeiss FE 35mm 2.8
http://carnagepro.com
"Everything ... they eat everything, and fear is their bacon bits."

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On