What Adrian Did Subsequent — Half 2 — Solar Microsystems | by adrian cockcroft
Jun 21, 2022
I spent six years at Cambridge Consultants, constructing some fascinating techniques, managing our Solar workstations and studying quite a bit, however by then Solar had opened a gross sales workplace throughout the road, and I wished to search out out what they have been going to launch subsequent, earlier than everybody else. So in 1988 I joined the Solar gross sales power and have become a Techniques Engineer (these days we’d say Options Architect). This was an enormous leap to make, I needed to put on a go well with to work day by day, and spend all my time paired up with a gross sales rep visiting prospects. I actually loved the number of working with a number of completely different prospects day by day, on completely different issues, and being a part of an especially revolutionary and quick rising firm. I additionally discovered quite a bit about work straight with prospects, when to close up and let the gross sales man drive the dialog, and customarily how know-how gross sales works. The early days at Solar Cambridge have been particular, I absorbed quite a bit about networking and the technical aspect of the function from my fellow techniques engineer Martin Baines, and we have been driving all around the area in cool firm vehicles (I had a Citroen BX 16V) promoting a extremely scorching product. The corporate and the crew acquired larger over time, however with the preliminary gross sales crew of Paul Martin (Ford Sierra Cosworth — the quickest gross sales man in East Anglia), Martin Harris (Land Rover Discovery — recruited an inflatable skeleton as an additional gross sales man), David Morley (Vauxhall Carlton — all the time betting on the horses at Newmarket), and Gary Littlefair (BMW 320 — did over 200% of his goal a number of years operating as we took British Telecom to Solar UK’s largest account) we have been “kicking butt and having enjoyable” as Scott McNealy preferred to say. I additionally acquired to go to Solar head workplace in California often for coaching periods, and met some fascinating individuals there, specifically Reynold Jabbour from New York, who’s deeply insightful on each individuals and know-how, ran the Techniques Engineers crew for the east coast, later went to work for numerous Wall Road banks, and remains to be certainly one of my closest mates.
I grew to become the Solar UK native specialist in efficiency and {hardware}, and as Solar transitioned from a desktop workstation firm to promote excessive finish multiprocessor servers I used to be serving to prospects discover and repair scalability issues. In 1991 I wrote a white paper on efficiency that was broadly learn, and in 1993 (with assist from Brian Wong) that acquired me a job within the USA, working alongside Brian for Mike Briggs in technical product advertising. One other massive leap, however now it was my job to run benchmarks within the lab, and write white papers that defined the brand new merchandise to the world, as they have been launched. I additionally discovered how advertising labored, and started to construct my presentation and coaching expertise as I used to be despatched around the globe by Solar to show workshops and communicate at occasions. I used to be principally coding in C, tuning FORTRAN, and after I wanted to do quite a lot of knowledge evaluation of benchmark outcomes used the S-PLUS statistics language, that’s the predecessor to R. I still code in R…
One function I had for a few years was to steer technical coaching occasions for the worldwide specialist techniques engineers, which we referred to as the Ambassador Program. I had attended these occasions after I was primarily based within the UK, however grew to become one of many organizers. I needed to setup every week of talks by all of the related product groups, with 100 or so of probably the most skilled techniques engineers from all around the world as an viewers. We had specializations in {hardware}, working techniques, databases, graphics, and many others. At one level we had two conflicting excessive availability merchandise from completely different groups, who weren’t cooperating. I intentionally scheduled them again to again at one occasion in order that they have been all within the room on the identical time, and the viewers roasted them — because of this there have been product and messaging adjustments. We’d typically handle to get Solar founder Andy Bechtolsheim to present us a chat, and he’d inform us way over the product managers wished us to know, about all of the cool issues he was engaged on that haven’t occurred but. He hasn’t changed. So many world friendships have been made that proceed to immediately. I noticed Erik Fisher final time I used to be in Budapest, and Constantin Gonzalez later grew to become one of many first AWS options architects in Germany, too many names to say…
I bear in mind organizing my first week lengthy occasion, having it go effectively, and heading again to the workplace to inform my supervisor Mike Briggs. He was happy, but additionally instructed me he was leaving that day, to be one of many first workers of NetApp, on the time I used to be shocked! We ended up hiring Phil Parkman to take over managing our group, which by then additionally included my buddy Dave Fisk, who I nonetheless hang around with often. Afterward after some re-orgs we ended up reporting to Greg Papadopoulos, who was CTO of the server group on the time.
I’d been utilizing the Web since about 1984, for e mail and newsgroups, sharing recordsdata like my white papers by way of ftp websites, however within the early 1990’s the world broad net got here alongside, and Solar was a really early adopter. Company IT had no concept what was occurring, however somebody acquired a T1 line attached and we put a spare desktop machine on it and arrange www.sun.com (hyperlink to archive from 1996). I ended up with root entry serving to Shane Sigler run the location round 1995–1996 when it was one of many very first company internet sites. I discovered about tuning techniques for this new community connection intensive workload, helped Solaris engineering tune the community stack and helped prospects run a number of the early net properties. Mike Briggs instructed that Brian Wong and I write books, as a result of then we’d have extra affect on the engineering groups, which turned out to be wonderful recommendation. Brian wrote Configuration and Capacity Planning for Sun Servers, and I turned my assortment of white papers right into a e book, Sun Performance and Tuning, which grew to become broadly learn, and printed a month-to-month Q&A column at SunWorld On-line. This was adopted by a tremendously expanded second version with some further chapters by Wealthy Pettit.
Brian Wong began a “rotation program” that he and I ran for a few years, with admin assist and funding from Barb Hill, the place we’d borrow a techniques engineer from someplace on the earth for a month, and have them work with us benchmarking one thing or writing a paper. We made nice mates and located some wonderful expertise this fashion, and so many individuals ended up getting employed into everlasting roles that it was referred to as the job donation program by a number of the SE managers, who misplaced good individuals to the central groups, however have been principally completely happy to see them succeed.
Wealthy Pettit is without doubt one of the greatest programmers I ever met. He turned up on a month lengthy rotation one Monday in 1993, took a take a look at a sophisticated Perl script Brian had written to attempt to seize some tuning concepts, and determined to make use of lex and yacc to put in writing a devoted language to make it simpler. He had it up and operating on Wednesday. I adopted the language, referred to as the SE Toolkit, which was a C-based interpreter that understood straight learn all of the efficiency knowledge out of the kernel, and wrote quite a lot of scripts over time, together with one referred to as virtual_adrian.se that lots of people used. The SE toolkit was quick, had no reminiscence leaks (monitoring scripts may run for years) and let me implement plenty of cool efficiency monitoring concepts. Wealthy grew to become co-author of the second version of the Solar Efficiency Tuning e book, to explain the way it labored.
Phil Harman is a buddy of mine from Solar UK, we have been the 2 most senior techniques engineers within the UK on the time I left for the USA, and he came visiting on rotation just a few instances. He grew to become a number one skilled in Solaris multi-threading. Jim Mauro and Allan Packer visited on rotation from New Jersey, and Adelaide Australia, and labored out produce a database sizing information for Oracle Financials, which I wrote up and printed. It was the primary database sizing information Solar had ever produced. Allan moved to the USA for a number of years to work on database efficiency and wrote the e book Configuring and Tuning Databases on the Solaris Platform. He additionally labored intently with IBM as they ported DB2 to run on Solaris. Jim additionally joined the database efficiency crew.
Allan additionally launched me to Richard McDougall, who had labored with Allan for Solar Australia, and Richard visited for a rotation to construct some monitoring instruments. No-one may imagine how a lot he acquired achieved in a month, constructing a number of instruments together with one that might lay out all the reminiscence utilization of a machine in a approach nobody had ever seen earlier than. A bit later our supervisor requested Brian and I who we must always rent subsequent, and we each mentioned Richard. He wished somebody simpler to rent than a world relocation, however ultimately we made it occur. Afterward Richard teamed up with Jim Mauro to put in writing the Solaris Internals sequence of books, and whereas making an attempt to determine how one thing labored for the primary e book, ended up discovering and fixing a serious bug within the kernel code that was the largest efficiency win in Solaris 8.
Paul Reithmuller was yet one more imported Australian engineer who did wonderful work. Someday we have been chatting in a bunch assembly and he mentioned he was engaged on a TPC-C benchmark that was spending an excessive amount of time within the SCSI disk driver, so he wrote a 10-line awk script operating towards kernel reminiscence to take a look at the SPARC machine code execution path and discover which branches have been being predicted flawed, then wrote one other awk script to flip the department prediction bits in reminiscence and it confirmed a big speedup. It was a memorable jaw drop second for a room full of individuals, you probably did what!?! After that we had a operating joke that each impossibly arduous drawback can be solved with a ten line awk script.
I met Paul Robust when he was engaged on a rotation in 1998, and we grew to become shut mates, bonding over a shared curiosity in music by Hawkwind, Frank Zappa and King Crimson. He later moved to the US to work on Solaris, and for a few years we labored and performed collectively, specifically after I helped him type the progressive rock band Fractal. He’s a wonderful drummer, and I wasn’t a adequate bass participant, however I recorded lots of their rehearsals and gigs and named a few of their early instrumental songs (principally with dangerous puns).
After the success of the books that Brian and I had written, which we acquired a lower of royalties for, our administration determined that we must always write books as our day job, and we assembled a crew of specialists to put in writing up greatest follow guides as books within the Solar Blueprints sequence. I made main contributions to 2 Solar Blueprints books on Resource Management (1999) with Richard McDougall, Evert Hoogendoorn, Enrique Vargas, and Jim Bialaski, and Capacity Planning for Internet Services (2001) with Invoice Walker. Our crew additionally produced The Guide to High Availability by storage skilled Jeannie Johnstone Kobert, Sun Cluster Environment by Enrique Vargas, Solaris PC Netlink by Don Devitt (we had an incredible crew outing close to Boston on his sailboat), and several other different books. Ultimately, the books we wrote in our personal time have been extra opinionated, and I believe extra helpful, and offered higher, than the official books that have been extra “right” and reviewed by plenty of individuals. I additionally got here up with Cockcroft’s Law of Book Writing, which is that books get larger as you write them, quicker than you possibly can write them, so you’ll want to prune them to complete them.
Ultimately, after turning into the go-to individuals for capability planning and efficiency for Solar, Brian Wong and I grew to become Distinguished Engineers, and I joined the central efficiency engineering crew in 1999. I led a small group, and had plenty of enjoyable working with Elizabeth Purcell on the Menlo Park campus that now homes Fb, together with Roch Bourbonnais in France, and Bob Sneed in Virginia. We ran what our datacenter designer Rob Snevely (who wrote a Blueprint book that accommodates a chapter by Elizabeth Purcell) named the Can of Worms Venture, the place we’d set up and run efficiency exams on massive server configurations that have been extra customer-like and complicated than the same old check or benchmarking environments that engineering groups ran. We discovered many issues that didn’t work correctly collectively, filed plenty of bugs, and headed off a bunch of buyer points. We referred to as this “meltdown prevention” and needed to work arduous to get individuals to see that close to misses are value monitoring in addition to precise meltdowns.
Within the early 2000s, Solar was designing it’s subsequent technology excessive finish server, and to keep away from reliability points that had brought about issues in earlier techniques, adopted Six Sigma to create high quality greatest practices. I used to be searching for a brand new problem, so joined this system and ended up getting licensed as a “black belt”. It was like an intense and really technical MBA, re-learning quite a lot of the statistics I’d forgotten from my diploma, together with programs in change administration, necessities evaluation, failure modeling, and different issues I nonetheless discover helpful immediately. I labored with the engineering crew that was designing the backplane connector, however the venture was finally cancelled when Solar ran out of cash. I additionally utilized Six Sigma to capacity planning and offered this at a convention in 2003.
One enjoyable aspect venture I acquired concerned in was that Solar offered the again finish server techniques for the 2002 Winter Olympics in Salt Lake Metropolis, and I joined the trouble because the capability planning marketing consultant, to ensure nothing would go flawed when the occasions have been reside. I used some customized monitoring instruments I’d constructed over time as a part of Wealthy Pettit’s SE Toolkit, that included a instrument referred to as virtual-adrian, which carried out all of the concepts I had provide you with. Nothing went flawed, I acquired to see some occasions, and acquired very chilly at instances. I additionally helped out within the run-up to the 2004 Athens Olympics.
The final function I had at Solar was as Chief Architect for the Excessive Efficiency Technical Computing crew led by Shahin Khan, we had enjoyable and he was among the finest managers I’ve had, creating me in plenty of completely different instructions. Sooner or later our product boss left, and I took over the function, so I spent a 12 months or two studying be the product supervisor for our excessive efficiency computing and excessive finish graphics product traces. I used to be additionally working alongside a really skilled Enterprise Growth Supervisor, Susanna Kass, to setup relationships with many inside and exterior companions, and acquired a very good perception into how BD works. The crew partnered with Solar’s director of requirements Carl Cargill, as Solar and Oracle created a brand new requirements physique, the Enterprise Grid Alliance. I labored as the primary consultant for Solar, and on our competing relationship with the IBM dominated International Grid Discussion board. I acquired Paul Robust concerned within the EGA earlier than I left Solar, the 2 competing our bodies finally merged, and Grid and Net Providers advanced into Cloud Computing. We have been serious about the developments that finally grew to become cloud computing when AWS launched it, however Solar didn’t have the correct enterprise mannequin to go direct to builders and promote capability on a bank card, which is how AWS bypassed the resistance from company CIOs that we noticed at Solar. This job stretched me in many various methods and we have been having a very good time, however by then Solar was shrinking and one of many many layoffs included our total crew.
We had simply launched our most important HPC product at an occasion in Shanghai, racks of Intel Linux primarily based compute nodes utilizing a SPARC Solaris server to configure it and supply storage, with Infiniband or Ethernet connectivity, Solar Grid Engine and many others. Whereas I used to be there, we heard that there have been layoffs coming, however that everybody thought our crew was nonetheless wanted. On the best way again I went to a International Grid Discussion board assembly in Honolulu, representing EGA, and met Vikas Deolaliker, who mentioned it seemed as if no-one may discover the top depend to avoid wasting the HPC crew. I bear in mind sitting in a bar with Savas Parastatidis and Jim Webber (now CTO of Neo4j) and telling them what was occurring, then later that evening beginning an e mail thread with Maynard Webb, COO of eBay, on the pretext that eBay ought to begin a brand new line of enterprise auctioning compute capability on demand, however actually fishing for a job…
So in mid-2004 I used to be laid off, searching for a change and determined to get out of the enterprise of promoting computer systems to massive firms and work out client oriented know-how at eBay. I’d helped eBay get well from capability associated outages in 1999 and had setup their capability planning processes. I had additionally turned down a job supply at the moment, however they took me on this time as a Distinguished Engineer of their Operations Structure crew.
Vikas left and co-founded Sonoa techniques, which later grew to become Apigee which is now a part of Google Cloud. Shahin Khan ran advertising for Azul techniques for some time, and remains to be energetic within the HPC neighborhood. Susanna Kass is engaged on renewable power for datacenters these days, and we reconnected once more just lately.
There’s too many tales, and other people to say, so apologies to these I unnoticed, and these are my reminiscences, which can be defective. Solar was a formative a part of my profession, a really cool place to work, and was an enormous focus of expertise for a few years.