415 lines
20 KiB
Markdown
415 lines
20 KiB
Markdown
---
|
||
created_at: '2015-05-24T20:13:35.000Z'
|
||
title: Linkers, part 1 (2007)
|
||
url: http://www.airs.com/blog/archives/38
|
||
author: weinzierl
|
||
points: 51
|
||
story_text: ''
|
||
comment_text:
|
||
num_comments: 7
|
||
story_id:
|
||
story_title:
|
||
story_url:
|
||
parent_id:
|
||
created_at_i: 1432498415
|
||
_tags:
|
||
- story
|
||
- author_weinzierl
|
||
- story_9597406
|
||
objectID: '9597406'
|
||
year: 2007
|
||
|
||
---
|
||
[Source](https://www.airs.com/blog/archives/38 "Permalink to Airs – Ian Lance Taylor » Linkers part 1")
|
||
|
||
# Airs – Ian Lance Taylor » Linkers part 1
|
||
|
||
# [Airs – Ian Lance Taylor][1]
|
||
|
||
## [Linkers part 1][2]
|
||
|
||
August 22, 2007 at 11:30 pm · Filed under [Programming][3]
|
||
|
||
I’ve been working on and off on a new linker. To my surprise, I’ve discovered in talking about this that some people, even some computer programmers, are unfamiliar with the details of the linking process. I’ve decided to write some notes about linkers, with the goal of producing an essay similar to my existing one about the GNU configure and build system.
|
||
|
||
As I only have the time to write one thing a day, I’m going to do this on my blog over time, and gather the final essay together later. I believe that I may be up to five readers, and I hope y’all will accept this digression into stuff that matters. I will return to random philosophizing and minding other people’s business soon enough.
|
||
|
||
A Personal Introduction
|
||
|
||
Who am I to write about linkers?
|
||
|
||
I wrote my first linker back in 1988, for the AMOS operating system which ran on Alpha Micro systems. (If you don’t understand the following description, don’t worry; all will be explained below). I used a single global database to register all symbols. Object files were checked into the database after they had been compiled. The link process mainly required identifying the object file holding the main function. Other objects files were pulled in by reference. I reverse engineered the object file format, which was undocumented but quite simple. The goal of all this was speed, and indeed this linker was much faster than the system one, mainly because of the speed of the database.
|
||
|
||
I wrote my second linker in 1993 and 1994. This linker was designed and prototyped by Steve Chamberlain while we both worked at Cygnus Support (later Cygnus Solutions, later part of Red Hat). This was a complete reimplementation of the BFD based linker which Steve had written a couple of years before. The primary target was a.out and COFF. Again the goal was speed, especially compared to the original BFD based linker. On SunOS 4 this linker was almost as fast as running the cat program on the input .o files.
|
||
|
||
The linker I am now working, called gold, on will be my third. It is exclusively an ELF linker. Once again, the goal is speed, in this case being faster than my second linker. That linker has been significantly slowed down over the years by adding support for ELF and for shared libraries. This support was patched in rather than being designed in. Future plans for the new linker include support for incremental linking–which is another way of increasing speed.
|
||
|
||
There is an obvious pattern here: everybody wants linkers to be faster. This is because the job which a linker does is uninteresting. The linker is a speed bump for a developer, a process which takes a relatively long time but adds no real value. So why do we have linkers at all? That brings us to our next topic.
|
||
|
||
A Technical Introduction
|
||
|
||
What does a linker do?
|
||
|
||
It’s simple: a linker converts object files into executables and shared libraries. Let’s look at what that means. For cases where a linker is used, the software development process consists of writing program code in some language: e.g., C or C++ or Fortran (but typically not Java, as Java normally works differently, using a loader rather than a linker). A compiler translates this program code, which is human readable text, into into another form of human readable text known as assembly code. Assembly code is a readable form of the machine language which the computer can execute directly. An assembler is used to turn this assembly code into an _object file_. For completeness, I’ll note that some compilers include an assembler internally, and produce an object file directly. Either way, this is where things get interesting.
|
||
|
||
In the old days, when dinosaurs roamed the data centers, many programs were complete in themselves. In those days there was generally no compiler–people wrote directly in assembly code–and the assembler actually generated an _executable file_ which the machine could execute directly. As languages liked Fortran and Cobol started to appear, people began to think in terms of libraries of subroutines, which meant that there had to be some way to run the assembler at two different times, and combine the output into a single executable file. This required the assembler to generate a different type of output, which became known as an object file (I have no idea where this name came from). And a new program was required to combine different object files together into a single executable. This new program became known as the linker (the source of this name should be obvious).
|
||
|
||
Linkers still do the same job today. In the decades that followed, one new feature has been added: shared libraries.
|
||
|
||
More tomorrow.
|
||
|
||
[Permalink][2]
|
||
|
||
## 17 Comments »
|
||
|
||
1. ### [rmathew][4] said,
|
||
|
||
August 23, 2007 @ [1:53 am][5]
|
||
|
||
I am looking forward to the rest of this series. I hope you will also touch upon sorting out template instantiations, doing link-time optimisations, etc. that put additional burdens on a linker making it slower, though also more useful.
|
||
|
||
PS: I’m glad that you have started blogging regularly. I like it that you seem to have thought through most of the issues that you blog about, even if at times I don’t find myself agreeing with your conclusions.
|
||
|
||
2. ### [ncm][6] said,
|
||
|
||
August 23, 2007 @ [7:03 pm][7]
|
||
|
||
I too am looking forward to the rest of the series. It seems to me that it’s getting hard to make something recognizable, any more, as a classical linker, now that code generation and optimization are essential parts of the job.
|
||
|
||
3. ### [movement][8] said,
|
||
|
||
August 23, 2007 @ [7:41 pm][9]
|
||
|
||
Linkers and related topics such as runtime loaders and shared libraries
|
||
are a mystery at some level to most programmers, I find. It initially took
|
||
me a lot of time at staring at assembly output to work out link-time
|
||
relocations actually worked: both the Solaris linker and the GNU one are
|
||
pretty impenetrable if you’re just browsing. It’ll be very interesting to hear
|
||
some details from you on these topics.
|
||
|
||
Some related reading:
|
||
|
||
Sun’s Linkers and Libraries Guide
|
||
<http://blogs.sun.com/rie/>
|
||
<http://blogs.sun.com/msw/>
|
||
<http://blogs.sun.com/ali/>
|
||
|
||
John Levine’s old book Linkers and Loaders too. I didn’t get much out of
|
||
this book; unfortunately it was pretty outdated, and not very clearly put
|
||
together. I suspect the exercises would prove interesting to do though.
|
||
|
||
4. ### [Ian Lance Taylor][10] said,
|
||
|
||
August 23, 2007 @ [8:27 pm][11]
|
||
|
||
Thanks for the notes.
|
||
|
||
I tend to view template instantiation and link-time optimizations as separate from the linker proper. In implementations I know about, they are done before invoking the linker itself, or they are done via plugins which the linker invokes. That is, under the hood, there is still a classical linker.
|
||
|
||
But since there is interest, perhaps I will move on to those topics after covering the linker proper.
|
||
|
||
5. ### [Ivan][12] said,
|
||
|
||
August 23, 2007 @ [8:41 pm][13]
|
||
|
||
Great Idea,
|
||
|
||
I look forward to reading these entries.
|
||
|
||
Thanks,
|
||
Ivan Novick
|
||
<http://www.0x4849.net>
|
||
|
||
6. ### [christian schorn » Blog Archive » links for 2007-08-30][14] said,
|
||
|
||
August 30, 2007 @ [1:22 pm][15]
|
||
|
||
[…] Airs – Ian Lance Taylor » Linkers part 1 (tags: programming basics) […]
|
||
|
||
7. ### [Mark J. Wielaard » Ian Lance Taylor’s Linker Notes][16] said,
|
||
|
||
August 31, 2007 @ [8:21 am][17]
|
||
|
||
[…] Linkers part 1 – A Personal Introduction and A Technical Introduction. […]
|
||
|
||
8. ### [jrlevine][18] said,
|
||
|
||
September 13, 2007 @ [3:32 pm][19]
|
||
|
||
Nice series. Believe it or not, relocating loaders predate assemblers, with the first one in the late 1940s, and linking loaders aren’t much later. This technology goes way back.
|
||
|
||
Also, I was kind of surprised at the comment that my books was outdated. One of the reasons I wrote it was that linker technology changes so slowly. There hasn’t been an interesting new idea since incremental linkers about 20 years ago, knowledge of linkers has been mostly programmer folklore, so I figured I’d write it down so it’d be at last available somewhere. The descriptions of ELF, ECOFF, and they way they support dynamic linking are as far as I know still current, nothing’s changed since I wrote the book in 2000.
|
||
|
||
9. ### [Ian Lance Taylor][10] said,
|
||
|
||
September 13, 2007 @ [8:30 pm][20]
|
||
|
||
Thanks for the note. There is a lot I don’t know about the history.
|
||
|
||
I didn’t make the comment about your book myself; it’s certainly the best description of linkers I know of. Still, unless I misremember, there are some recent important ideas which aren’t covered, such as ELF symbol versions, ELF (and Mach-O) symbol visibility, interposition with LD_PRELOAD and the like, TLS details. I don’t actually have a copy to hand, so I hope I am not misrepresenting it. These are not major ideas like incremental linking, but they are things which the relatively few people who work with linkers need to understand.
|
||
|
||
10. ### avjo said,
|
||
|
||
November 4, 2007 @ [11:30 pm][21]
|
||
|
||
Hi Ian,
|
||
|
||
This series is so educating and interesting ! Thank
|
||
you for that !
|
||
|
||
Just wondering here… will your new linker be GPLv2 or GPLv3 ?
|
||
|
||
~avjo
|
||
|
||
11. ### [Ian Lance Taylor][10] said,
|
||
|
||
November 5, 2007 @ [6:07 am][22]
|
||
|
||
The goal is for the new linker to be part of the GNU binutils, which means that it will be GPLv3.
|
||
|
||
(I’ll add that I think that in practice there is very little difference between GPLv2 and GPLv3.)
|
||
|
||
12. ### [E. Huntley - Programming & Development Blog » Blog Archive » Back to the “Basics”][23] said,
|
||
|
||
February 23, 2009 @ [8:48 am][24]
|
||
|
||
[…] The entries start here. And continue through his entry archives to mid September 2007. I highly reccommend giving at least the first few entries a quick read-through if you are like me, and want a better understading of the development tools we use every day. […]
|
||
|
||
13. ### [zur::Linux » Gold Linker][25] said,
|
||
|
||
September 1, 2011 @ [10:02 am][26]
|
||
|
||
[…] If you want to know more about linkers and Gold in particular Ian Lance Taylor has a twenty-part series about linker internals on his blog. […]
|
||
|
||
14. ### [Yearzero.flaminghorns.com − September is Linker Month!!!][27] said,
|
||
|
||
August 27, 2013 @ [6:50 am][28]
|
||
|
||
[…] There are around 20 odd articles to be read. The link series for the article is as follows: <http://www.airs.com/blog/archives/38> – <http://www.airs.com/blog/archives/57> Tagged and categorized as: General | No comments […]
|
||
|
||
15. ### [Relocatable objects - BYTEC/16][29] said,
|
||
|
||
December 16, 2013 @ [1:03 pm][30]
|
||
|
||
[…] code generation, and that's exactly what symbol tables and relocation tables are used for. I used this material by Ian Lance Taylor to understand the basics of […]
|
||
|
||
16. ### [What is PLT/GOT? | CL-UAT][31] said,
|
||
|
||
December 23, 2014 @ [6:52 pm][32]
|
||
|
||
[…] Also, Ian Lance Taylor, the author of GOLD has put up an article series on his blog which is totally worth reading (twenty parts!): entry point here “Linkers part 1″. […]
|
||
|
||
17. ### [A journey into Radare 2 – Part 2: Exploitation – Megabeets][33] said,
|
||
|
||
September 3, 2017 @ [9:32 am][34]
|
||
|
||
[…] the function address from the GOT. To read more about the linking process, I highly recommend this series of articles about linkers by Ian Lance […]
|
||
|
||
[RSS feed for comments on this post][35] · [TrackBack URI][36]
|
||
|
||
## Leave a Comment
|
||
|
||
You must be [logged in][37] to post a comment.
|
||
|
||
* * *
|
||
|
||
## Search
|
||
|
||
* * *
|
||
* * *
|
||
|
||
[landmark][38]
|
||
|
||
* ## Archives
|
||
|
||
* [July 2017][39]
|
||
* [June 2017][40]
|
||
* [February 2012][41]
|
||
* [December 2011][42]
|
||
* [November 2011][43]
|
||
* [October 2011][44]
|
||
* [September 2011][45]
|
||
* [August 2011][46]
|
||
* [July 2011][47]
|
||
* [June 2011][48]
|
||
* [May 2011][49]
|
||
* [April 2011][50]
|
||
* [March 2011][51]
|
||
* [February 2011][52]
|
||
* [January 2011][53]
|
||
* [December 2010][54]
|
||
* [November 2010][55]
|
||
* [October 2010][56]
|
||
* [September 2010][57]
|
||
* [August 2010][58]
|
||
* [July 2010][59]
|
||
* [June 2010][60]
|
||
* [May 2010][61]
|
||
* [April 2010][62]
|
||
* [March 2010][63]
|
||
* [February 2010][64]
|
||
* [January 2010][65]
|
||
* [December 2009][66]
|
||
* [November 2009][67]
|
||
* [November 2008][68]
|
||
* [October 2008][69]
|
||
* [September 2008][70]
|
||
* [August 2008][71]
|
||
* [July 2008][72]
|
||
* [June 2008][73]
|
||
* [May 2008][74]
|
||
* [April 2008][75]
|
||
* [March 2008][76]
|
||
* [February 2008][77]
|
||
* [January 2008][78]
|
||
* [December 2007][79]
|
||
* [November 2007][80]
|
||
* [October 2007][81]
|
||
* [September 2007][82]
|
||
* [August 2007][83]
|
||
* [February 2007][84]
|
||
* [January 2007][85]
|
||
* [December 2006][86]
|
||
* [July 2006][87]
|
||
* [June 2006][88]
|
||
* [April 2006][89]
|
||
* [February 2006][90]
|
||
* [January 2006][91]
|
||
* [December 2005][92]
|
||
* [November 2005][93]
|
||
* ## Categories
|
||
|
||
* [Books][94]
|
||
* [Money][95]
|
||
* [Movies][96]
|
||
* [Philosophy][97]
|
||
* [Politics][98]
|
||
* [Programming][99]
|
||
* [Random][100]
|
||
* ## Meta
|
||
|
||
* [Register][101]
|
||
* [Log in][102]
|
||
* [Entries RSS][103]
|
||
* [Comments RSS][104]
|
||
* [Wordpress][105]
|
||
|
||
Design by [Beccary][106] and [Weblogs.us][107] · [XHTML][108] · [CSS][109]
|
||
|
||
[1]: http://www.airs.com/blog
|
||
[2]: https://www.airs.com/blog/archives/38 "Permanent link to Linkers part 1"
|
||
[3]: https://www.airs.com/blog/archives/category/programming
|
||
[4]: http://rmathew.com/
|
||
[5]: https://www.airs.com#comment-3769 "Permanent link to this comment"
|
||
[6]: http://cantrip.org
|
||
[7]: https://www.airs.com#comment-3803 "Permanent link to this comment"
|
||
[8]: http://movementarian.org/
|
||
[9]: https://www.airs.com#comment-3804 "Permanent link to this comment"
|
||
[10]: http://airs.com/ian/
|
||
[11]: https://www.airs.com#comment-3806 "Permanent link to this comment"
|
||
[12]: http://www.0x4849.net
|
||
[13]: https://www.airs.com#comment-3807 "Permanent link to this comment"
|
||
[14]: http://christian-schorn.de/2007/08/links-for-2007-08-30
|
||
[15]: https://www.airs.com#comment-4008 "Permanent link to this comment"
|
||
[16]: http://gnu.wildebeest.org/diary/2007/08/31/ian-lance-taylors-linker-notes/
|
||
[17]: https://www.airs.com#comment-4025 "Permanent link to this comment"
|
||
[18]: http://weblog.johnlevine.com
|
||
[19]: https://www.airs.com#comment-4270 "Permanent link to this comment"
|
||
[20]: https://www.airs.com#comment-4285 "Permanent link to this comment"
|
||
[21]: https://www.airs.com#comment-6326 "Permanent link to this comment"
|
||
[22]: https://www.airs.com#comment-6337 "Permanent link to this comment"
|
||
[23]: http://www.ernesthuntley.com/blog/?p=10
|
||
[24]: https://www.airs.com#comment-16003 "Permanent link to this comment"
|
||
[25]: http://www.egbw.com/?p=130
|
||
[26]: https://www.airs.com#comment-18637 "Permanent link to this comment"
|
||
[27]: http://yearzero.flaminghorns.com/?p=28
|
||
[28]: https://www.airs.com#comment-18696 "Permanent link to this comment"
|
||
[29]: http://www.pilawa.org/computer/2013/12/16/relocatable-objects/
|
||
[30]: https://www.airs.com#comment-18709 "Permanent link to this comment"
|
||
[31]: http://cluat.com/question/what-is-pltgot/
|
||
[32]: https://www.airs.com#comment-19168 "Permanent link to this comment"
|
||
[33]: https://www.megabeets.net/a-journey-into-radare-2-part-2/
|
||
[34]: https://www.airs.com#comment-20194 "Permanent link to this comment"
|
||
[35]: https://www.airs.com/blog/archives/38/feed
|
||
[36]: https://www.airs.com/blog/archives/38/trackback
|
||
[37]: http://www.airs.com/blog/wp-login.php?redirect_to=https://www.airs.com/blog/archives/38
|
||
[38]: http://www.airs.com/breach.php
|
||
[39]: https://www.airs.com/blog/archives/date/2017/07
|
||
[40]: https://www.airs.com/blog/archives/date/2017/06
|
||
[41]: https://www.airs.com/blog/archives/date/2012/02
|
||
[42]: https://www.airs.com/blog/archives/date/2011/12
|
||
[43]: https://www.airs.com/blog/archives/date/2011/11
|
||
[44]: https://www.airs.com/blog/archives/date/2011/10
|
||
[45]: https://www.airs.com/blog/archives/date/2011/09
|
||
[46]: https://www.airs.com/blog/archives/date/2011/08
|
||
[47]: https://www.airs.com/blog/archives/date/2011/07
|
||
[48]: https://www.airs.com/blog/archives/date/2011/06
|
||
[49]: https://www.airs.com/blog/archives/date/2011/05
|
||
[50]: https://www.airs.com/blog/archives/date/2011/04
|
||
[51]: https://www.airs.com/blog/archives/date/2011/03
|
||
[52]: https://www.airs.com/blog/archives/date/2011/02
|
||
[53]: https://www.airs.com/blog/archives/date/2011/01
|
||
[54]: https://www.airs.com/blog/archives/date/2010/12
|
||
[55]: https://www.airs.com/blog/archives/date/2010/11
|
||
[56]: https://www.airs.com/blog/archives/date/2010/10
|
||
[57]: https://www.airs.com/blog/archives/date/2010/09
|
||
[58]: https://www.airs.com/blog/archives/date/2010/08
|
||
[59]: https://www.airs.com/blog/archives/date/2010/07
|
||
[60]: https://www.airs.com/blog/archives/date/2010/06
|
||
[61]: https://www.airs.com/blog/archives/date/2010/05
|
||
[62]: https://www.airs.com/blog/archives/date/2010/04
|
||
[63]: https://www.airs.com/blog/archives/date/2010/03
|
||
[64]: https://www.airs.com/blog/archives/date/2010/02
|
||
[65]: https://www.airs.com/blog/archives/date/2010/01
|
||
[66]: https://www.airs.com/blog/archives/date/2009/12
|
||
[67]: https://www.airs.com/blog/archives/date/2009/11
|
||
[68]: https://www.airs.com/blog/archives/date/2008/11
|
||
[69]: https://www.airs.com/blog/archives/date/2008/10
|
||
[70]: https://www.airs.com/blog/archives/date/2008/09
|
||
[71]: https://www.airs.com/blog/archives/date/2008/08
|
||
[72]: https://www.airs.com/blog/archives/date/2008/07
|
||
[73]: https://www.airs.com/blog/archives/date/2008/06
|
||
[74]: https://www.airs.com/blog/archives/date/2008/05
|
||
[75]: https://www.airs.com/blog/archives/date/2008/04
|
||
[76]: https://www.airs.com/blog/archives/date/2008/03
|
||
[77]: https://www.airs.com/blog/archives/date/2008/02
|
||
[78]: https://www.airs.com/blog/archives/date/2008/01
|
||
[79]: https://www.airs.com/blog/archives/date/2007/12
|
||
[80]: https://www.airs.com/blog/archives/date/2007/11
|
||
[81]: https://www.airs.com/blog/archives/date/2007/10
|
||
[82]: https://www.airs.com/blog/archives/date/2007/09
|
||
[83]: https://www.airs.com/blog/archives/date/2007/08
|
||
[84]: https://www.airs.com/blog/archives/date/2007/02
|
||
[85]: https://www.airs.com/blog/archives/date/2007/01
|
||
[86]: https://www.airs.com/blog/archives/date/2006/12
|
||
[87]: https://www.airs.com/blog/archives/date/2006/07
|
||
[88]: https://www.airs.com/blog/archives/date/2006/06
|
||
[89]: https://www.airs.com/blog/archives/date/2006/04
|
||
[90]: https://www.airs.com/blog/archives/date/2006/02
|
||
[91]: https://www.airs.com/blog/archives/date/2006/01
|
||
[92]: https://www.airs.com/blog/archives/date/2005/12
|
||
[93]: https://www.airs.com/blog/archives/date/2005/11
|
||
[94]: https://www.airs.com/blog/archives/category/books "Books and authors."
|
||
[95]: https://www.airs.com/blog/archives/category/money "Thoughts on money, finance, economics, business."
|
||
[96]: https://www.airs.com/blog/archives/category/movies "Movies."
|
||
[97]: https://www.airs.com/blog/archives/category/philosophy "Philosophy, religion, other things normally associated with the sophomore year in college."
|
||
[98]: https://www.airs.com/blog/archives/category/politics "Thoughts about politics."
|
||
[99]: https://www.airs.com/blog/archives/category/programming "Thoughts about computer programming."
|
||
[100]: https://www.airs.com/blog/archives/category/random "Random posts."
|
||
[101]: https://www.airs.com/blog/wp-login.php?action=register
|
||
[102]: https://www.airs.com/blog/wp-login.php
|
||
[103]: https://www.airs.com/blog/feed "Syndicate this site using RSS 2.0"
|
||
[104]: https://www.airs.com/blog/comments/feed "The latest comments to all posts in RSS"
|
||
[105]: http://wordpress.org "Powered by Wordpress, state-of-the-art semantic personal publishing platform."
|
||
[106]: http://beccary.com "Theme designed by Beccary"
|
||
[107]: http://weblogs.us "Theme sponsored by Weblogs.us"
|
||
[108]: http://validator.w3.org/check/referer "This page validates as XHTML 1.0 Transitional"
|
||
[109]: http://jigsaw.w3.org/css-validator/check/referer "This page validates as CSS"
|
||
|
||
[*[CSS]: Cascading Style Sheets
|
||
[*[RSS]: Really Simple Syndication
|
||
[*[XHTML]: eXtensible HyperText Markup Language
|
||
[*URI]: Uniform Resource Identifier
|
||
[*RSS]: Really Simple Syndication
|
||
|