nemo
/
hn-classics
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
hn-classics/_stories/2007/13805096.md

630 lines
30 KiB
Markdown
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
created_at: '2017-03-06T19:02:11.000Z'
title: Newton Storage History (2007)
url: http://www.dadhacker.com/blog/?p=948
author: jsnell
points: 78
story_text:
comment_text:
num_comments: 10
story_id:
story_title:
story_url:
parent_id:
created_at_i: 1488826931
_tags:
- story
- author_jsnell
- story_13805096
objectID: '13805096'
year: 2007
---
[Source](http://www.dadhacker.com/blog/?p=948 "Permalink to Newton Storage History | Dadhacker")
# Newton Storage History | Dadhacker
[Dadhacker][1]
Instant Wisdom Worth What You Paid
![][2]
[Skip to content][3]
* [Home][4]
* [About][5]
* [Fiction and Other Reading][6]
* [Papers read][7]
←[ I knew FORTH was evil][8]
[Warnock on continuations and my favorite beverage →][9]
# Newton Storage History
Posted on [October 23, 2007][10] by [landon][11]
I joined the Newton group in early June of 1992. I’d been at Apple since 1987 working on tools and then a database engine in the development tools group, but after a bunch of bad politics that I won’t dwell upon I wound up needing a new position pretty quickly. I managed to land a gig working on a device driver framework for the Newton.
I arrived in Newton just as the group had decided to do “Junior,” the product that was to become the first Newton MessagePad. Up to that point there were a bunch of people happily noodling away in Ralph (the object-oriented dialect of Scheme that the Newton tablet product was going to be based on), but the decision was made (over CES, I gather) to do Junior in C++ and what later would become NewtonScript. When I arrived there were a bunch of ex-LISP hackers wandering the hallways clutching C++ reference books and looking unhappy. Really unhappy. The expensive tablet product (“Senior”) wasn’t _entirely_ shelved — they kept doing revs of its plastic — but it was soft-pedalled and eventually died.
I did a simple driver model for devices (in retrospect, probably way too simple), and inherited a heap manager that I eventually made work pretty well. I worked with some early storage stuff with Tim Harrington, another engineer in the kernel group.
At first the store inside the Newton itself just kept objects in a heap in the Newton’s battery backed RAM. We also worked out a scheme for keeping heaps on removable PCMCIA RAM cards, but the problem was that the heaps had to be located at particular virtual address to work, and the virtual address of the card could vary depending on the slot you plugged the card into. We could have simply relocated the heap structures, but the problem remained that the power could fail (or the user could remove the card) in the middle of store update, whereupon you’d lose _all_ the data (or worse, it would become flakily corrupt). We needed a storage system that was address-independent and that would also survive restarts.
The group was starting to realize that the Newton platform was way over budget on RAM. There was some neat work done by the kernel team to reduce the stack and other memory needed for threads, and I did some work in the heap managers to minimize virtual memory usage and fragmentation, but it was clear that we wouldn’t have enough RAM to store as much user data as we would need to; even reducing the application heap, we’d need another 128K or more to make people (ahem: Marketing) happy.
So the hardware team added a single byte lane of RAM. Basically this was a region of memory where every fourth byte was valid (and the agreement between the hardware folks and the OS team was that they could populate more lanes of bytes, until they filled all four lanes and it became just like the rest of RAM). This was easy for the hardware folks to do (just solder another chip on the board), but the heap-based store couldn’t deal with byte lanes.
We also had to support Flash devices. The type of flash we planned to support could be read directly like RAM, but not written to without going through funny little state-machine gyrations. Given the wacky byte-lanes of RAM and the non-RAM-nature of Flash, it was clear we had to do a bunch of work on the storage layer. So I did that; the object store talked to the actual backing store through an abstraction and the callers had no idea what kind of wacky stuff the store was going through in order to make things persistent. In fact, for the internal store of the OMP we used 128K of byte-lane memory plus another 64K of ordinary memory “stolen” from the rest of the system.  The abstraction glued all this wacky memory together and the NewtonScript world never knew the difference.
But there were still windows where resets or power failures would cause the store to become corrupt; it wasn’t crash-proof or reset-proof. We all knew this, and I’d done some thinking about putting a transaction system in, but none of us thought we had time to fix things before ship.
One afternoon Michael Culbert, the Newton Technologist, came into my office to discuss battery contact bounce. I’d been nervous about the robustness of the storage system and had talked with him about it before, and now he confirmed my worst fears. Battery contact bounce is the jiggling of the power connectors when you toss a device onto a hard surface; the device can easily momentarily lose power and reset, and if the Newton was in the middle of a storage update all the data would be lost. As I said, we’d been talking about doing transactions in the stores for a while, and Michael finally had a good argument for why they were going to be critical.
“We have to do this,” I said, “Or we’re fucked.”
“We have to do this or we’re fucked,” he echoed. “I’ll go tell Fi [the software manager of Newton] and Gaston [the head of the division].”
“Okay. It’ll take a few weeks.”
“I know.”
This was maybe four months before the product was scheduled to ship. The first thing I did was to leave work for a couple of hours to visit some bookstores I liked, buy a couple of CDs I’d been thinking of getting, and take a long walk. Then I came back that evening, closed the door to my office and worked 18-20 hour days for the next five or six weeks.
A couple of evenings into this effort Gaston came into my office and tried some small talk. He wasn’t very good at it. I was concentrating on the design of the transaction log at that point, and _really_ didn’t want to be disturbed.
Gaston said, “Fi tells me you’re the guy who’s going to save the product.”
“I guess so.”
“All by yourself? You don’t need any help?”
“Not right now.” Some things aren’t partitionable.  I knew what I had to do, practically down to each subroutine to write.
After Gaston left, I tracked down Fi and told her, “No offense, but please tell Gaston not to come around my office anymore. He makes me nervous.” She did get some other folks on the team to help out on some pieces that I was able to partition and farm out, but for the most part the things that had to be done simply couldn’t be parceled out to other people, there just wasn’t _time_ to get them up to speed.  “Adding people to a late project makes it later.”
The only thing I remember clearly about the next couple of weeks was that I dreamed a whole lot about transaction systems. I stuck the changes into the system (we had mandatory code reviews; I spent the two days prior to checking-in talking myself hoarse, walking through the code with Greg Seitz, another kernel guy). I did my only all-nighter at Apple getting that build out, and I recall only a couple of bug fixes to it.
The upshot of the transaction system was that you could do a bunch of updates to the objects on a store, crash in the middle, and recover everything prior to the update that crashed. It was pretty cool.
Here’s roughly how it worked:
Imagine that you have a notepad entry, with a bunch of text. The NewtonScript storage layer breaks up the notes into a bunch of small objects, including the text itself, plus some extra objects for searching. There are objects containing structures that allow the text to be quickly searched, and so on.
Now say your original note was, “Buy milk,” and that you add “Bread, too!” to it. What the Newton does is create a whole set of shadow objects, just like the old ones but with your changes applied. Then, with a _single_ write to a single byte somewhere in the store the storage system commits your changes, then goes back and deletes the outdated objects.
If the power fails before the commit point, the Newton just deletes the new objects and you haven’t lost much work (you just have to remember to buy bread). If the power fails after the commit point, the Newton just ensures that all of the old objects have been gotten rid of (so it can re-use the space they take up).
This sounds simple in principle, but in fact it’s quite difficult to get correct. Additionally, the flash storage we were using had the amusing characteristic that bits could go bad, which is a little like having to design a car with an engine that might lose a gear or two and still have to maintain freeway speeds without the driver even noticing.
Flash-based file systems in 1992 were kind of rocket science; the existing stuff from Microsoft had to be manually compacted, which as you can well imagine was a really awful pain the rear. The Newton’s flash store did all of this transparently and in real-time, too; during a technical presentation we got some questions about this (I always assumed they were from Microsofties):
“When do users have to compact the flash cards?”
“You don’t, it’s all done for you.”
There was some refusal to believe this, and we got the question several times. That was fun. Nowadays users just expect this stuff to work, but back then all the existing Flash file systems were pretty icky.  It felt great to have something cool out there.
After 1.0 shipped we added large binary support. The idea was to use the virtual memory system of the Newton OS to dynamically page in modifiable binary objects that could be significantly larger than the amount of RAM in the system (e.g., faxes). The first couple of cuts at this were pretty miserable until I had the idea of hooking the objects into the NewtonScript garbage collector.
The idea was that a large binary not associated with a persistent slot would live in a kind of limbo; if it was garbage-collected it would be deleted, and if the system crashed before it was deleted it would be cleaned-up as part of the normal transaction “undo” phase. But if a object was persisted and had a large binary as a slot, the large binary would be kept alive. It took a while to do, but the result was a system with only 512K of RAM that could receive a multi-megabyte FAX over a modem, or work with bitmap images larger than main memory, or play sounds that were likewise really large.
There was a lot of great technology hidden away in the guts of Newton.  I worked on some other pieces that were neat, but this is what I’m most happy about having shipped. It was a lot of fun to work on. Newton was one of those projects I always felt it was a privilege to be part of.
This entry was posted in [Rantage][12]. Bookmark the [permalink][13].
←[ I knew FORTH was evil][8]
[Warnock on continuations and my favorite beverage →][9]
### 8 Responses to _Newton Storage History_
1. ![][14] Brant Sears says:
[ December 3, 2007 at 4:59 pm][15]
I was a Newton programmer for a few years writing medical records software. The persistent storage solution on the newton was really wonderful. I remember working with it very fondly. It was my first real job as a professional programmer. Our company made versions of our software for Newton and Windows CE. We only had one Newton engineer (me) and many Windows engineers. I was always able to keep the newton product ahead of CE despite working alone because of the tremendously powerful APIs of the the Newton. Even a year after cancelation our Newton software was far outselling the Windows version. I am a professional software developer today because I truly fell in love with the Newton and I like to think that a little bit of the design philosophy of the Newton rubbed off on me.
[Reply][16]
2. ![][17] [desp][18] says:
[ December 3, 2007 at 5:11 pm][19]
Thanks for writing this. I remember being highly impressed with the way storage worked on my MP2100.
Please write more about your experiences at Apple.
[Reply][20]
3. ![][21] NAL says:
[ December 4, 2007 at 5:54 pm][22]
You did an amazing job with the technology that you had. Only just a year ago, I wrote the majority of my PhD dissertation from an eMate 300 that I bought at a recycled computer store for $10 because I couldn’t afford a laptop and couldn’t take most of my research materials outside of the library. The formatting was done with MS Word after the fact, of course.
Obviously I wish I owned an MP2100 because the eMate’s processor is so slow, but still, the machine is remarkably resilient. I did have some flash memory corruption once that caused one of those crazy backlight-flashing errors, but a brain wipe did the trick.
[Reply][23]
4. ![][24] [Matthias][25] says:
[ March 11, 2008 at 4:03 pm][26]
Thanks for this great article. You probably know about the Newton emulator “Einstein”. The emulator works quite wel (great work by Paul Guyot!)l, but is too slow to run on most PDAs. Which is why I started to rewrite parts of Newton OS, using the known Netwon API’s. I know, this is probably an endless project, but it’s fun nevertheless.
Feel free to peek into the source code at code.google.com or download the disk image. Comments and suggestions are very welcome.
Matthias
[Reply][27]
5. Pingback: [DadHacker: working on Newton storage « Newton Poetry][28]
6. ![][29] [Morgan Aldridge][30] says:
[ March 24, 2008 at 12:17 pm][31]
This is such a great story! Please contact Paul Potts and get this, and any others you may have, added to the new Newton project on Folklore.org:
<http://folklore.org/ProjectView.py?name=Newton>
See Paul’s recent announcement to the NewtonTalk mailing list:
[http://marc.info/?l=newtontalk&m=120633234403859&w=2][32]
[Reply][33]
7. ![][34] [Thomas][35] says:
[ April 19, 2008 at 12:47 am][36]
This is a cool story – I remember that I read a feature story about the Newton storage system in a Byte magazine around that time…
I had already been programming for about a decade at that point, but the complexity of the Newton OS and the number of brand-new concepts in the code was staggering to me. You are my hero! 🙂
[Reply][37]
8. ![][38] Jeff Sheldon says:
[ August 5, 2014 at 12:10 pm][39]
This is a tremendous memoir which provides wonderful insight into problem solving, product design, and concern for user data integrity. I recall moments where the transaction system really saved my butt. Special thanks to myapplenewton.blogspot.com for reviving attention to Landon’s post.
[Reply][40]
### Leave a Reply [Cancel reply][41]
Your email address will not be published. Required fields are marked *
Comment
Name *
Email *
Website
* Search for:
* ### Recent Posts
* [RIP Jerry Pournelle][42]
* [When the little hand is on the two, and the big hand…][43]
* [RIP George Michael][44]
* [oh my god][45]
* [RIP Henry SF Cooper][46]
* ### Recent Comments
* Lee on [Donkey Kong and Me][47]
* [Götz][48] on [How the Atari ST almost had Real Unix][49]
* [Petari][50] on [The Atari ST, Part 2][51]
* [landon][52] on [Donkey Kong and Me][53]
* [AtariXL][54] on [Donkey Kong and Me][55]
* ### Archives
* [September 2017][56]
* [January 2017][57]
* [December 2016][58]
* [November 2016][59]
* [July 2016][60]
* [June 2016][61]
* [April 2016][62]
* [March 2016][63]
* [February 2016][64]
* [December 2015][65]
* [November 2015][66]
* [August 2015][67]
* [June 2015][68]
* [May 2015][69]
* [March 2015][70]
* [January 2015][71]
* [December 2014][72]
* [October 2014][73]
* [September 2014][74]
* [August 2014][75]
* [July 2014][76]
* [June 2014][77]
* [May 2014][78]
* [March 2014][79]
* [February 2014][80]
* [January 2014][81]
* [December 2013][82]
* [November 2013][83]
* [October 2013][84]
* [August 2013][85]
* [July 2013][86]
* [June 2013][87]
* [April 2013][88]
* [March 2013][89]
* [February 2013][90]
* [January 2013][91]
* [December 2012][92]
* [October 2012][93]
* [September 2012][94]
* [August 2012][95]
* [July 2012][96]
* [June 2012][97]
* [May 2012][98]
* [April 2012][99]
* [March 2012][100]
* [February 2012][101]
* [January 2012][102]
* [November 2011][103]
* [October 2011][104]
* [September 2011][105]
* [August 2011][106]
* [July 2011][107]
* [June 2011][108]
* [May 2011][109]
* [April 2011][110]
* [March 2011][111]
* [February 2011][112]
* [January 2011][113]
* [December 2010][114]
* [November 2010][115]
* [October 2010][116]
* [August 2010][117]
* [July 2010][118]
* [June 2010][119]
* [May 2010][120]
* [April 2010][121]
* [March 2010][122]
* [February 2010][123]
* [January 2010][124]
* [December 2009][125]
* [November 2009][126]
* [October 2009][127]
* [September 2009][128]
* [August 2009][129]
* [July 2009][130]
* [June 2009][131]
* [May 2009][132]
* [April 2009][133]
* [March 2009][134]
* [February 2009][135]
* [January 2009][136]
* [December 2008][137]
* [November 2008][138]
* [October 2008][139]
* [September 2008][140]
* [August 2008][141]
* [July 2008][142]
* [June 2008][143]
* [May 2008][144]
* [April 2008][145]
* [March 2008][146]
* [February 2008][147]
* [January 2008][148]
* [December 2007][149]
* [November 2007][150]
* [October 2007][151]
* [September 2007][152]
* [August 2007][153]
* [July 2007][154]
* [June 2007][155]
* [May 2007][156]
* [April 2007][157]
* [March 2007][158]
* [February 2007][159]
* [January 2007][160]
* [December 2006][161]
* [November 2006][162]
* [October 2006][163]
* [September 2006][164]
* [August 2006][165]
* [July 2006][166]
* [June 2006][167]
* [May 2006][168]
* [April 2006][169]
* [March 2006][170]
* [February 2006][171]
* [January 2006][172]
* [December 2005][173]
* [November 2005][174]
* [October 2005][175]
* [September 2005][176]
* [August 2005][177]
* [July 2005][178]
* [June 2005][179]
* [May 2005][180]
* [April 2005][181]
* [March 2005][182]
* [February 2005][183]
* [January 2005][184]
* [December 2004][185]
* [November 2004][186]
* [October 2004][187]
* [September 2004][188]
* [August 2004][189]
* [July 2004][190]
* [June 2004][191]
* [May 2004][192]
* [April 2004][193]
* [March 2004][194]
* [February 2004][195]
* [January 2004][196]
* [December 2003][197]
* [November 2003][198]
* [October 2003][199]
* [September 2003][200]
* [August 2003][201]
* [July 2003][202]
* [June 2003][203]
* [May 2003][204]
* [April 2003][205]
* [March 2003][206]
* [February 2003][207]
* [January 2003][208]
* [December 2002][209]
* [November 2002][210]
* [October 2002][211]
* [September 2002][212]
* [August 2002][213]
* [July 2002][214]
* ### Categories
* [Attitude][215]
* [Memo to self][216]
* [Rage can be fun][217]
* [Rantage][12]
* [Uncategorized][218]
* ### Meta
* [Log in][219]
* [Entries RSS][220]
* [Comments RSS][221]
* [WordPress.org][222]
[ Dadhacker ][1]
[Proudly powered by WordPress.][223]
[1]: http://www.dadhacker.com/blog/ "Dadhacker"
[2]: http://www.dadhacker.com/blog/wp-content/uploads/2011/03/cropped-dadhacker-background1.jpg
[3]: http://www.dadhacker.com#content "Skip to content"
[4]: http://www.dadhacker.com/blog/
[5]: http://www.dadhacker.com/blog/?page_id=1348
[6]: http://www.dadhacker.com/blog/?page_id=827
[7]: http://www.dadhacker.com/blog/?page_id=826
[8]: http://www.dadhacker.com/blog/?p=947
[9]: http://www.dadhacker.com/blog/?p=949
[10]: http://www.dadhacker.com/blog/?p=948 "11:18 pm"
[11]: http://www.dadhacker.com/blog/?author=2 "View all posts by landon"
[12]: http://www.dadhacker.com/blog/?cat=7
[13]: http://www.dadhacker.com/blog/?p=948 "Permalink to Newton Storage History"
[14]: http://2.gravatar.com/avatar/289fb188848f85fe71dff80fb582a0b8?s=40&d=mm&r=g
[15]: http://www.dadhacker.com/blog/?p=948#comment-393
[16]: http://www.dadhacker.com/blog/?p=948&replytocom=393#respond
[17]: http://1.gravatar.com/avatar/77cb644e3dec8320d05469bcbddf7815?s=40&d=mm&r=g
[18]: http://varsztat.com
[19]: http://www.dadhacker.com/blog/?p=948#comment-394
[20]: http://www.dadhacker.com/blog/?p=948&replytocom=394#respond
[21]: http://1.gravatar.com/avatar/d68d31f64638929f981fdf2b1f010844?s=40&d=mm&r=g
[22]: http://www.dadhacker.com/blog/?p=948#comment-395
[23]: http://www.dadhacker.com/blog/?p=948&replytocom=395#respond
[24]: http://0.gravatar.com/avatar/f318be7b992f1b586cce781fe156f91b?s=40&d=mm&r=g
[25]: http://www.robowerk.com
[26]: http://www.dadhacker.com/blog/?p=948#comment-396
[27]: http://www.dadhacker.com/blog/?p=948&replytocom=396#respond
[28]: http://newtonpoetry.wordpress.com/2008/03/14/dadhacker-working-on-newton-storage/
[29]: http://2.gravatar.com/avatar/b2217a363e0648dd23614752b0de75bb?s=40&d=mm&r=g
[30]: http://www.makkintosshu.com/
[31]: http://www.dadhacker.com/blog/?p=948#comment-398
[32]: http://marc.info/?l=newtontalk&m=120633234403859&w=2
[33]: http://www.dadhacker.com/blog/?p=948&replytocom=398#respond
[34]: http://2.gravatar.com/avatar/81db3fd91a98ec34d42b53f9a0e928ab?s=40&d=mm&r=g
[35]: http://www.sturm.to
[36]: http://www.dadhacker.com/blog/?p=948#comment-399
[37]: http://www.dadhacker.com/blog/?p=948&replytocom=399#respond
[38]: http://1.gravatar.com/avatar/473d8e9b1f7c06199df2681a91f23485?s=40&d=mm&r=g
[39]: http://www.dadhacker.com/blog/?p=948#comment-47063
[40]: http://www.dadhacker.com/blog/?p=948&replytocom=47063#respond
[41]: /blog/?p=948#respond
[42]: http://www.dadhacker.com/blog/?p=2528
[43]: http://www.dadhacker.com/blog/?p=2519
[44]: http://www.dadhacker.com/blog/?p=2514
[45]: http://www.dadhacker.com/blog/?p=2511
[46]: http://www.dadhacker.com/blog/?p=2501
[47]: http://www.dadhacker.com/blog/?p=987#comment-475615
[48]: http://www.3rz.org
[49]: http://www.dadhacker.com/blog/?p=1383#comment-472542
[50]: http://atari.8bitchip.info
[51]: http://www.dadhacker.com/blog/?p=1000#comment-472270
[52]: http://www.dadhacker.com
[53]: http://www.dadhacker.com/blog/?p=987#comment-471010
[54]: https://atarixl.wordpress.com/2017/11/28/127/
[55]: http://www.dadhacker.com/blog/?p=987#comment-470248
[56]: http://www.dadhacker.com/blog/?m=201709
[57]: http://www.dadhacker.com/blog/?m=201701
[58]: http://www.dadhacker.com/blog/?m=201612
[59]: http://www.dadhacker.com/blog/?m=201611
[60]: http://www.dadhacker.com/blog/?m=201607
[61]: http://www.dadhacker.com/blog/?m=201606
[62]: http://www.dadhacker.com/blog/?m=201604
[63]: http://www.dadhacker.com/blog/?m=201603
[64]: http://www.dadhacker.com/blog/?m=201602
[65]: http://www.dadhacker.com/blog/?m=201512
[66]: http://www.dadhacker.com/blog/?m=201511
[67]: http://www.dadhacker.com/blog/?m=201508
[68]: http://www.dadhacker.com/blog/?m=201506
[69]: http://www.dadhacker.com/blog/?m=201505
[70]: http://www.dadhacker.com/blog/?m=201503
[71]: http://www.dadhacker.com/blog/?m=201501
[72]: http://www.dadhacker.com/blog/?m=201412
[73]: http://www.dadhacker.com/blog/?m=201410
[74]: http://www.dadhacker.com/blog/?m=201409
[75]: http://www.dadhacker.com/blog/?m=201408
[76]: http://www.dadhacker.com/blog/?m=201407
[77]: http://www.dadhacker.com/blog/?m=201406
[78]: http://www.dadhacker.com/blog/?m=201405
[79]: http://www.dadhacker.com/blog/?m=201403
[80]: http://www.dadhacker.com/blog/?m=201402
[81]: http://www.dadhacker.com/blog/?m=201401
[82]: http://www.dadhacker.com/blog/?m=201312
[83]: http://www.dadhacker.com/blog/?m=201311
[84]: http://www.dadhacker.com/blog/?m=201310
[85]: http://www.dadhacker.com/blog/?m=201308
[86]: http://www.dadhacker.com/blog/?m=201307
[87]: http://www.dadhacker.com/blog/?m=201306
[88]: http://www.dadhacker.com/blog/?m=201304
[89]: http://www.dadhacker.com/blog/?m=201303
[90]: http://www.dadhacker.com/blog/?m=201302
[91]: http://www.dadhacker.com/blog/?m=201301
[92]: http://www.dadhacker.com/blog/?m=201212
[93]: http://www.dadhacker.com/blog/?m=201210
[94]: http://www.dadhacker.com/blog/?m=201209
[95]: http://www.dadhacker.com/blog/?m=201208
[96]: http://www.dadhacker.com/blog/?m=201207
[97]: http://www.dadhacker.com/blog/?m=201206
[98]: http://www.dadhacker.com/blog/?m=201205
[99]: http://www.dadhacker.com/blog/?m=201204
[100]: http://www.dadhacker.com/blog/?m=201203
[101]: http://www.dadhacker.com/blog/?m=201202
[102]: http://www.dadhacker.com/blog/?m=201201
[103]: http://www.dadhacker.com/blog/?m=201111
[104]: http://www.dadhacker.com/blog/?m=201110
[105]: http://www.dadhacker.com/blog/?m=201109
[106]: http://www.dadhacker.com/blog/?m=201108
[107]: http://www.dadhacker.com/blog/?m=201107
[108]: http://www.dadhacker.com/blog/?m=201106
[109]: http://www.dadhacker.com/blog/?m=201105
[110]: http://www.dadhacker.com/blog/?m=201104
[111]: http://www.dadhacker.com/blog/?m=201103
[112]: http://www.dadhacker.com/blog/?m=201102
[113]: http://www.dadhacker.com/blog/?m=201101
[114]: http://www.dadhacker.com/blog/?m=201012
[115]: http://www.dadhacker.com/blog/?m=201011
[116]: http://www.dadhacker.com/blog/?m=201010
[117]: http://www.dadhacker.com/blog/?m=201008
[118]: http://www.dadhacker.com/blog/?m=201007
[119]: http://www.dadhacker.com/blog/?m=201006
[120]: http://www.dadhacker.com/blog/?m=201005
[121]: http://www.dadhacker.com/blog/?m=201004
[122]: http://www.dadhacker.com/blog/?m=201003
[123]: http://www.dadhacker.com/blog/?m=201002
[124]: http://www.dadhacker.com/blog/?m=201001
[125]: http://www.dadhacker.com/blog/?m=200912
[126]: http://www.dadhacker.com/blog/?m=200911
[127]: http://www.dadhacker.com/blog/?m=200910
[128]: http://www.dadhacker.com/blog/?m=200909
[129]: http://www.dadhacker.com/blog/?m=200908
[130]: http://www.dadhacker.com/blog/?m=200907
[131]: http://www.dadhacker.com/blog/?m=200906
[132]: http://www.dadhacker.com/blog/?m=200905
[133]: http://www.dadhacker.com/blog/?m=200904
[134]: http://www.dadhacker.com/blog/?m=200903
[135]: http://www.dadhacker.com/blog/?m=200902
[136]: http://www.dadhacker.com/blog/?m=200901
[137]: http://www.dadhacker.com/blog/?m=200812
[138]: http://www.dadhacker.com/blog/?m=200811
[139]: http://www.dadhacker.com/blog/?m=200810
[140]: http://www.dadhacker.com/blog/?m=200809
[141]: http://www.dadhacker.com/blog/?m=200808
[142]: http://www.dadhacker.com/blog/?m=200807
[143]: http://www.dadhacker.com/blog/?m=200806
[144]: http://www.dadhacker.com/blog/?m=200805
[145]: http://www.dadhacker.com/blog/?m=200804
[146]: http://www.dadhacker.com/blog/?m=200803
[147]: http://www.dadhacker.com/blog/?m=200802
[148]: http://www.dadhacker.com/blog/?m=200801
[149]: http://www.dadhacker.com/blog/?m=200712
[150]: http://www.dadhacker.com/blog/?m=200711
[151]: http://www.dadhacker.com/blog/?m=200710
[152]: http://www.dadhacker.com/blog/?m=200709
[153]: http://www.dadhacker.com/blog/?m=200708
[154]: http://www.dadhacker.com/blog/?m=200707
[155]: http://www.dadhacker.com/blog/?m=200706
[156]: http://www.dadhacker.com/blog/?m=200705
[157]: http://www.dadhacker.com/blog/?m=200704
[158]: http://www.dadhacker.com/blog/?m=200703
[159]: http://www.dadhacker.com/blog/?m=200702
[160]: http://www.dadhacker.com/blog/?m=200701
[161]: http://www.dadhacker.com/blog/?m=200612
[162]: http://www.dadhacker.com/blog/?m=200611
[163]: http://www.dadhacker.com/blog/?m=200610
[164]: http://www.dadhacker.com/blog/?m=200609
[165]: http://www.dadhacker.com/blog/?m=200608
[166]: http://www.dadhacker.com/blog/?m=200607
[167]: http://www.dadhacker.com/blog/?m=200606
[168]: http://www.dadhacker.com/blog/?m=200605
[169]: http://www.dadhacker.com/blog/?m=200604
[170]: http://www.dadhacker.com/blog/?m=200603
[171]: http://www.dadhacker.com/blog/?m=200602
[172]: http://www.dadhacker.com/blog/?m=200601
[173]: http://www.dadhacker.com/blog/?m=200512
[174]: http://www.dadhacker.com/blog/?m=200511
[175]: http://www.dadhacker.com/blog/?m=200510
[176]: http://www.dadhacker.com/blog/?m=200509
[177]: http://www.dadhacker.com/blog/?m=200508
[178]: http://www.dadhacker.com/blog/?m=200507
[179]: http://www.dadhacker.com/blog/?m=200506
[180]: http://www.dadhacker.com/blog/?m=200505
[181]: http://www.dadhacker.com/blog/?m=200504
[182]: http://www.dadhacker.com/blog/?m=200503
[183]: http://www.dadhacker.com/blog/?m=200502
[184]: http://www.dadhacker.com/blog/?m=200501
[185]: http://www.dadhacker.com/blog/?m=200412
[186]: http://www.dadhacker.com/blog/?m=200411
[187]: http://www.dadhacker.com/blog/?m=200410
[188]: http://www.dadhacker.com/blog/?m=200409
[189]: http://www.dadhacker.com/blog/?m=200408
[190]: http://www.dadhacker.com/blog/?m=200407
[191]: http://www.dadhacker.com/blog/?m=200406
[192]: http://www.dadhacker.com/blog/?m=200405
[193]: http://www.dadhacker.com/blog/?m=200404
[194]: http://www.dadhacker.com/blog/?m=200403
[195]: http://www.dadhacker.com/blog/?m=200402
[196]: http://www.dadhacker.com/blog/?m=200401
[197]: http://www.dadhacker.com/blog/?m=200312
[198]: http://www.dadhacker.com/blog/?m=200311
[199]: http://www.dadhacker.com/blog/?m=200310
[200]: http://www.dadhacker.com/blog/?m=200309
[201]: http://www.dadhacker.com/blog/?m=200308
[202]: http://www.dadhacker.com/blog/?m=200307
[203]: http://www.dadhacker.com/blog/?m=200306
[204]: http://www.dadhacker.com/blog/?m=200305
[205]: http://www.dadhacker.com/blog/?m=200304
[206]: http://www.dadhacker.com/blog/?m=200303
[207]: http://www.dadhacker.com/blog/?m=200302
[208]: http://www.dadhacker.com/blog/?m=200301
[209]: http://www.dadhacker.com/blog/?m=200212
[210]: http://www.dadhacker.com/blog/?m=200211
[211]: http://www.dadhacker.com/blog/?m=200210
[212]: http://www.dadhacker.com/blog/?m=200209
[213]: http://www.dadhacker.com/blog/?m=200208
[214]: http://www.dadhacker.com/blog/?m=200207
[215]: http://www.dadhacker.com/blog/?cat=3
[216]: http://www.dadhacker.com/blog/?cat=4
[217]: http://www.dadhacker.com/blog/?cat=6
[218]: http://www.dadhacker.com/blog/?cat=1
[219]: http://www.dadhacker.com/blog/wp-login.php
[220]: http://www.dadhacker.com/blog/?feed=rss2
[221]: http://www.dadhacker.com/blog/?feed=comments-rss2
[222]: https://wordpress.org/ "Powered by WordPress, state-of-the-art semantic personal publishing platform."
[223]: https://wordpress.org/ "Semantic Personal Publishing Platform"
[*RSS]: Really Simple Syndication
Reference in New Issue View Git Blame Copy Permalink