department of hack
649 stories
·
15 followers

Isolated Integration Tests in Shell

1 Comment

As I’ve been building out During, we have a handful of microservices powering things like syncing, importing, some single-purpose serving of heavy data, and so on.

If you aren’t familiar, a microservice is just a fancy word that means “let’s spread our business objects all OVER the fucking place!”.

Anyway, my microservices are all, well, really really micro. And because I’m definitely not above being an idiot, I’ve rewritten one of my web services a number of times, starting with a folder inside of Rails, then extracted out into Ruby, then Crystal, then Go, and now back to Ruby. It’s not a huge time sink, and to be honest, much of this was just for shits and giggles to try out new approaches.

I’ve since moved all these services back to Ruby. I’m really good at Ruby these days, and it’s really easy to scale Ruby to handle all of my users. Granted, the app hasn’t even launched into beta yet, so it turns out it’s also really easy to scale when you have no users. In fact, virtually everything about software development is easier when you have no users. Aside from the making money part, that is (but that hasn’t stopped Silicon Valley much either).

Anyway, I digress. Something I’ve been really enjoying as of late is specifically using shell to test these services.

POSIX shell, more like UBIQUITOUSIX shell amirite

I’ve been writing my tests for these services in shell script, and it’s been pretty great. For one, the infrastructure is obviously easy. Shell is already on all of my AWS instances, any CI service I may use, and on my local machine. Don’t have to install anything, don’t have to run any Docker instances (though that certainly doesn’t hurt).

The most important aspect, though, is that my tests are now isolated and independent from any future language testing I may use. I can bounce around between languages and frameworks, and I don’t have to change any of the tests. This is really important because if, for example, your v1 has a subtle bug that your tests were fine with, you can rewrite the whole v2 service and your tests will fail if you accidentally corrected the bug. That means that your API you expose to the rest of your services won’t break, and you can prepare the rest of your services to fix the bug at your leisure instead of being surprised by it in production.

The tooling for shell testing is pretty decent these days, too. For years I’ve used my buddy Blake Mizerany’s roundup, a lovely tiny testing tool for shell. Recently I’ve been using Sam Stephenson’s bats, which has been steadily growing into a really active community. (lol, around a shell testing tool, who would have thought?) Most of my shell tests now look like this one, in bats:

@test "Responds with events within the given timespan" {
  url_params="?starts_at=2017-05-01T00:00:00-00:00&ends_at=2017-05-31T00:00:00-00:00"
  run curl "$URL$url_params" --silent -H "Authorization: Bearer:$bearer"

  assert_output --partial "Test Event 0"
  assert_output --partial "Test Event 2"
  refute_output --partial "Test Event 5"
  refute_output --partial "No location data"
  refute_output --partial "Not included in the date span"
}

Tests are pretty simple and easy to reason about. Basically just a curl and you check the output. Done.

Integration rules everything around me

One last quick point: these microservices are very-small to fairly-small, and I can get by with ignoring writing any other tests entirely. Writing integration-only, full-stack tests is really interesting, but people get really religious about whether this is the next best idea ever or the worst idea in the world. For what it’s worth, GitHub’s Gist was happily running in production with zero unit tests whatsoever for years. I’m somewhat up in the air about the practice overall; I go back and forth for sure. There’s plenty of other posts on the topic you can read if you’re interested.

But I will say that in these cases, wowza, what a breath of fresh air. Our tests are portable, and we don’t have to write any new tests if I ever rewrite the service. I just have to conform to my shell-based tests.

Read the whole story
brennen
1 day ago
reply
"If you aren’t familiar, a microservice is just a fancy word that means “let’s spread our business objects all OVER the fucking place!”."
Boulder, CO
Share this story
Delete

Sick of Your Shit, Apple

1 Comment and 2 Shares
You know, it's my fault for not reading the microtext on the AppleCare plan. If I'd read it, I would have taken my three hundred dollars and used it to make little bonfires in the cupboards to keep the mice warm in winter and surrounded them with little hot chocolate marshmallows on toothpicks because ...
Read the whole story
brennen
1 day ago
reply
"Instead, I metaphorically burned six Grants at the Apple altar to buy a warranty that doesn’t cover accidents. I bought three years of slightly cheaper service repairs if the equipment I already spent two grand on fails because they didn’t whip a Chinese child hard enough that day. I paid extra up front to pay slightly less in the future if I have to deal with a potential situation where they screwed up, which doesn’t exactly scream confidence in the projected runtime of their hardware. My Zippo came with a free lifetime warranty, and it’s supposed to be on fire."
Boulder, CO
Share this story
Delete

‘The Peasantry is Unimportant’

1 Share

li161.1 C44 Q.R

One of the ideas that’s been crashing around in my head for years is that vernacular furniture – what I call the “furniture of necessity” – is divorced, separate and independent from the high styles of furniture that crowd the books in my office.

This idea is not commonly held.

The conventional wisdom is this: Chippenton Sheradale invents a style of furniture that is Neo-Classical Chinese. So he publishes a pattern book to illustrate his new pieces, and the style becomes all the rage. All of the rich people want pieces in Neo-Classical Chinese to replace all the pieces in their houses that were Neo-Chinese Classical.

So the local cabinetmakers oblige and (as a result) can all afford new chrome rims for their carriages.

Rich rural farmers see the pieces in the new style and return home with the crazy idea that they should also have pieces in the latest Neo-Classical Chinese style. So they get Festus, the local cabinetmaker, to build them a Neo-Classical Chinese chair. But Festus uses Redneck Maple (Holdimus beericus) because Festus can’t get New Money Mahogany (Stickusis inbutticus).

Oh, and Festus takes some liberties with the new furniture style to please his rural customers, who want a series of cupholders in the arms that can accommodate a Bigus Gulpus.

Then the poor farmers see the Redneck Maple Neo-Classical Chairs owned by the rich farmers and ask their local carpenters to make copies, who also make changes to the design (a gun rack on the back). And then the dirt farmers see that chair. And so on.

Meanwhile, back in the city, a furniture designer draws up a pattern book for Neo-Gothic Romanian furniture. The cycle begins again.

All this sounds plausible because it has been written down in almost every book of furniture history ever published. The rich make something fashionable, and the poor imitate it until the rich become annoyed or bored. So then the rich find a new style, which the poor imitate again.

The only problem with this theory of degenerate furniture forms is that the furniture record doesn’t always go along with the theory.

I think there’s furniture that is divorced from the gentry. Furniture that is divorced from architecture. Instead of beginning with a pattern book, it begins with these questions: What do I need? What materials do I have? What can I make that will take little time to build but will endure (so I don’t have to frickin’ build it again)?

bebb_IMG_8707

For several months now I have been plowing through “Welsh Furniture 1250-1950” (Saer Books) by Richard Bebb and have been thrilled to find someone who thinks the same way. Bebb has done the research on the matter when it comes to Welsh furniture. And he has convinced me that I’m not nuts.

In the first section of Vol. I, Bebb deftly eviscerates these ideas like a fishmonger filleting a brook trout. It’s an amazing thing to read. I’ll be writing more about Bebb’s research in future entries, but if you want to get right to the source, I recommend you snag your own copy of this impressive work.

— Christopher Schwarz


Filed under: The Anarchist's Design Book, Uncategorized





Read the whole story
brennen
3 days ago
reply
Boulder, CO
Share this story
Delete

Optimizing the future

2 Shares

On Saturday, an online firestorm erupted over a ten-page memo written by James Damore, a twenty-eight-year-old software engineer at Google. Titled “Google’s Ideological Echo Chamber,” it led to its author being fired a few days later, and the furor is far from over—I have the uncomfortable feeling that it’s just getting started. (Damore has said that he intends to sue, and his case has already become a cause célèbre in exactly the circles that you’d expect.) In his memo, Damore essentially argues that the acknowledged gender gap in management and engineering roles at tech firms isn’t due to bias, but to “the distribution of preferences and abilities of men and women in part due to biological causes.” In women, these differences include “openness directed towards feelings and aesthetics rather than ideas,” “extraversion expressed as gregariousness rather than assertiveness,” “higher agreeableness,” and “neuroticism,” while men have a “higher drive for status” that leads them to take positions demanding “long, stressful hours that may not be worth it if you want a balanced and fulfilling life.” He summarizes:

I’m not saying that all men differ from women in the following ways or that these differences are “just.” I’m simply stating that the distribution of preferences and abilities of men and women differ in part due to biological causes and that these differences may explain why we don’t see equal representation of women in tech and leadership. Many of these differences are small and there’s significant overlap between men and women, so you can’t say anything about an individual given these population level distributions.

Damore quotes a decade-old research paper, which I suspect that he first encountered through the libertarian site Quillette, stating that as “society becomes more prosperous and more egalitarian, innate dispositional differences between men and women have more space to develop and the gap that exists between men and women in their personality becomes wider.” And he concludes: “We need to stop assuming that gender gaps imply sexism.”

I wasn’t even going to write about this here, but it rang a bell. Back in 1968, a science fiction fan named Ron Stoloff attended the World Science Fiction Convention in Berkeley, where he was disturbed both by the lack of diversity and by the presence of at least one fan costumed as Lt. Uhura in blackface. He wrote up his thoughts in an essay titled “The Negro and Science Fiction,” which was published the following year in the fanzine The Vorpal Sword. (I haven’t been able to track down the full issue, but you can find the first page of his article here.) On May 1, 1969, the editor John W. Campbell wrote Stoloff a long letter objecting to the argument and to the way that he had been characterized. It’s a fascinating document that I wish I could quote in full, but the most revealing section comes after Campbell asks rhetorically: “Look guy—do some thinking about this. How many Negro authors are there in science fiction?” He then answers his own question:

Now consider what effect a biased, anti-Negro editor could have on that. Manuscripts come in by mail from all over the world…I haven’t the foggiest notion what most of the authors look like—and I never yet heard of an editor who demanded a photograph of an author before he’d print his work! Nor demanded a notarized document proving he was write.

If Negro authors are extremely few—it’s solely because extremely few Negroes both wish to, and can, write in open competition. There isn’t any possible field of endeavor where race, religion, and sex make less difference. If there aren’t any individuals of a particular group in the authors’ column—it’s because either they didn’t want to, or weren’t able to. It’s got to be unbiased by the very nature of the process of submission.

Campbell’s argument is fundamentally the same as Damore’s. It states that the lack of black writers in the pages of Analog, like the underrepresentation of women in engineering roles at Google, isn’t due to bias, but because “either they didn’t want to, or weren’t able to.” (Campbell, like Damore, makes a point of insisting elsewhere that he’s speaking of the statistical description of the group as a whole, not of individuals, which strikes him as a meaningful distinction.) Earlier in the letter, however, Campbell inadvertently suggests another explanation for why “Negro authors are extremely few,” and it doesn’t have anything to do with ability:

Think about it a bit, and you’ll realize why there is so little mention of blacks in science fiction; we see no reason to go saying “Lookee lookee lookee! We’re using blacks in our stories! See the Black Man! See him in a spaceship!”

It is my strongly held opinion that any Black should be thrown out of any story, spaceship, or any other place—unless he’s a black man. That he’s got no business there just because he’s black, but every right there if he’s a man. (And the masculine embraces the feminine; Lt. Uhura is portrayed as no clinging vine, and not given to the whimper, whinny, and whine type behavior. She earned her place by competence—not by having a black skin.)

There are two implications here. The first is that all protagonists should be white males by default, a stance that Campbell might not even have seen as problematic—and it’s worth noting that even if race wasn’t made explicit in the story, the magazine’s illustrations overwhelmingly depicted its characters as white. There’s also the clear sense that black heroes have to “earn” their presence in the magazine, which, given the hundreds of cardboard “competent men” that Campbell cheerfully featured over the years, is laughable in itself. In fiction, as in life, if you’re black, you’ve evidently got to be twice as good to justify yourself.

It never seems to have occurred to Campbell that the dearth of minority writers in the genre might have been caused by a lack of characters who looked like them, as well as by similar issues in the fandom, and he never believed that he had the ability or the obligation to address the situation as an editor. (Elsewhere in the same letter, he writes: “What I am against—and what has been misinterpreted by a number of people—is the idea that any member of any group has any right to preferential treatment because he is a member.”) Left to itself, the scarcity of minority voices and characters was a self-perpetuating cycle that made it easy to argue that interest and ability were to blame. The hard part about encouraging diversity in science fiction, or anywhere else, is that it doesn’t happen by accident. It requires systematic, conscious effort, and the payoff might not be visible for years. That’s as hard and apparently unrewarding for a magazine that worries mostly about managing its inventory from one month to the next as it is for a technology company focused on yearly or quarterly returns. If Campbell had really wanted to see more black writers in Analog in the late sixties, he should have put more black characters in the magazine in the early forties. You could excuse this by saying that he had different objectives, and that it’s unfair to judge him in retrospect, but it’s equally true that it was a choice that he could have made, but didn’t. And science fiction was the poorer for it. In his memo, Damore writes:

Philosophically, I don’t think we should do arbitrary social engineering of tech just to make it appealing to equal portions of both men and women. For each of these changes, we need principled reasons for why it helps Google; that is, we should be optimizing for Google—with Google’s diversity being a component of that.

Replace “tech” with “science fiction,” “men and women” with “black and white writers,” and “Google” with “Analog,” and you have a fairly accurate representation of Campbell’s position. He clearly saw his job as the optimization of science fiction. A diverse roster of writers, which would have resulted in far more interesting “analog simulations” of reality of the kind that he loved, would have gone a long way toward improving it. He didn’t make the effort, and the entire genre suffered as a result. Google, to its credit, seems to understand that diversity also offers competitive advantages when you aren’t just writing about the future, but inventing it. And anything else would be suboptimal.








Read the whole story
brennen
4 days ago
reply
Boulder, CO
Share this story
Delete

Photo

1 Comment and 2 Shares


Read the whole story
brennen
7 days ago
reply
To the cat section of this I would probably add: Pointy, liquid.
Boulder, CO
Share this story
Delete

unifying OS installation and configuration management

1 Share

Three years ago, I realized that propellor (my configuration management system that is configured using haskell) could be used as an installer for Debian (or other versions of Linux). In propellor is d-i 2.0, I guessed it would take "a month and adding a few thousand lines of code".

I've now taken that month, and written that code, and I presented the result at DebConf yesterday. I demoed propellor building a live Debian installation image, and then handed it off to a volenteer from the audience to play with its visual user interface and perform the installation. The whole demo took around 20 minutes, and ended with a standard Debian desktop installation.

The core idea is to reuse the same configuration management system for several different purposes.

  1. Building a bootable disk image that can be used as both a live system and as an OS installer.
  2. Running on that live system, to install the target system. Which can just involve copying the live system to the target disk and then letting the configuration management system make the necessary changes to get from the live system configuration to the target system configuration.
  3. To support such things as headless arm boards, building customized images tuned for the target board and use case, that can then simply be copied to the board to install.
  4. Optionally, running on the installed system later, to futher customize it. Starting from the same configuration that produced the installed system in the first place.

There can be enourmous code reuse here, and improvements made for one of those will often benefit all the rest as well.

Once everything is handled by configuration management, all user interface requirements become just a matter of editing the configuration. Including:

  • A user interface that runs on the live system and gets whatever input is needed to install to the target system. This is really just a config editor underneath. I built a prototype gamified interface that's as minimal as such an interface could get.
  • With a regular text editor, of course. This is the equivilant of preseeding in d-i, giving advanced users full control over the system that gets built. Unlike with preseeding, users have the full power of a configuration management system, so can specify precisely the system they want installed.
  • A separate user interface for customizing disk images, for arm boards and similar use cases. This would run on a server, or on the user's own laptop.

That's the gist of it. Configuration management reused for installation and image building, and multiple editor interfaces to make it widely usable.

I was glad, sitting in to a BoF session before my talk, that several people in Debian are already thinking along similar lines. And if Debian wanted to take this work and run with it, I'd be glad to assist as propellor's maintainer. But the idea is more important than the code and I hope my elaboration of it helps point a way if not the way.

While what I've built installs Debian, little of it is Debian-specific. It would probably be easy to port it to Arch Linux, which propellor already supports. There are Linux-specific parts, so porting to FreeBSD would be harder, but propellor knows, at the type level which OSs properties support, which will ease porting.

GuixSD and NixOS already use configuration management for installation, and were part of my inspiration. I've extended what they do in some ways (in other ways they remain far ahead).


The code is here. And here are some links to more details about what I built, and ideas encountered along the way:

Read the whole story
brennen
7 days ago
reply
Boulder, CO
Share this story
Delete
Next Page of Stories