department of hack
1241 stories

The Nebraska Territory – Montana, Wyoming, and The Dakotas…

1 Share

(This post is part of a virtual road trip, which traces the routes taken by characters in The Great Offshore Grounds. For more on that, go here.)

A few hours past Missoula the land changed again, flattening and turning brown, scoured by wind. They crossed the Yellowstone River, then followed it as it ran, north by north east, fed by the Yellowstone Lake. It had poured over the falls and through the Black Canyon before they ever knew it... p 60-61.

My travels through Montana, Wyoming, the Dakotas, and Nebraska have always been framed by physicality and history. Geology, as seen above, taught me that there were many living things here before I came, and before we all come. It also taught me that what is preserved is not an accurate depiction of the life that flourished because only the hard parts survive, the shells, the bones, the statues, belief systems that, when institutionalized and perpetuated, turn a point of view into a concrete thing, seemingly undeniable as truth.

Then they came to the great Powder River, which was fed by the Little Powder, as well as oil from pipeline breaks, Red Cloud’s War, and benzene as it joined the Yellowstone. The Yellowstone, which already had inherited so much, the Bighorn and the Wind and the Tongue, as it flowed toward the Missouri River, which had no choice either but to inherit the watershed as it drained toward the Mississippi, that unstoppable force. And maybe there just is no way out of history. No matter how much you want to come from a different story, you can’t...p 61

I grew up knowing that above all else, I was Czech-American. My grandfather made sure of that. The prevalence of other histories and nationalities were dismissed. Bohunk was an early word in my vocabulary, and when I read Willa Cather, it was my own personal story, even though it wasn’t. It was not the only history I was raised on. I was also raised on the American Indian Movement. We were big fans.

Digital image of original artifact.

I was taught about COINTELPRO as well as theories on rural uprisings like the Omaha Platform with all…their…complexities, and I understood that my history was not the only history, but only partly.

All of these ideas, ghosts. My Antonia, Little House on the Prairie, Agents of Repression, and the land itself, its badlands and Pine Ridges and coal-bed-proof of (extinct) life, histories that won’t let any of us go, these are things I tried to write into the “place” of the novel.

I tried scrying and got nowhere

Passing through the Black Hills, North of Mt Rushmore and Pine Ridge, south of the Bakken Shale basin and Standing Rock, they drove the Badlands…East of the coal beds, west of the Black Hills…In an unsparing white flash of midday light, they drove through miles of the cash crop ethanol, a pentimento of buffalo skins behind it, untanned and rotting in piles on the prairie; they, too, were shadows...p 62

I cannot help but think that we all live in Territories more than States. The borders keep shifting. There are waves of people and thought. Everyone wants to feel more certain in this, their own America. Livy and Cheyenne, the main characters in The Great Offshore Grounds, are no different.

The storm came from Nebraska. When the high winds began, they had to stop to wait it out. Tornados touched down and lifted. Mad ballerinas, storm goddesses, they danced across flat lands and flood plains. When the winds died, they drove again, racing power outages and wild skies…p75

If you want to read more, or read My Antonia or Agents of Repression or anything else your heart desires, think about supporting these two wonderful independent bookstores The Country Bookshelf in Bozeman, MT and the wonderful Francie & Finch in Lincoln, NE.

Next, Michigan…

Read the whole story
1 day ago
Boulder, CO
Share this story

limiting power

1 Share

As I noted in my previous post, there is no political system, no ordered social life, in which one can wholly escape being subject to power. As Burke says in his Reflections, “Men cannot enjoy the rights of an uncivil and of a civil state together. That he may obtain justice, he gives up his right of determining what it is in points the most essential to him. That he may secure some liberty, he makes a surrender in trust of the whole of it.” Rousseau tried to evade this stark choice by inventing the idea of the “general will,” to which we all supposedly give free assent even when we are being brought to the guillotine. But of course the “general will” is a pious fiction for autocrats, and Burke’s contrast, in all its starkness, is certainly true.

Anarchism, as I understand its best exponents, does not deny Burke’s contrast but rather openly acknowledges the truth of it and yet attempts to blunt its force. This can never be done completely. A spontaneously self-organized environment, an emergent order, such as anarchism favors, is still an order and in any order some people at some times have power over others. And when they have such power they will sometimes use it wrongly.

The best fictional depiction of anarchism in practice is Ursula K. Le Guin’s The Dispossesed, and I find it fascinating that the pivotal conflict of this novel — which largely celebrates the austere beauty of an anarchist world — centers on a power struggle among anarchists. Insofar as Shevek, the book’s protagonist, has an antagonist it is his fellow physicist Sabul, whose role in the Physics syndic on Anarres allows him control over the publication and distribution of scientific papers — especially papers that come from the nearby capitalist world of Urras. (Anarres is the moon of Urras, largely uninhabited until a group of anarchists were allowed to settle there two centuries before the time in which the novel is set.)

It occurred to [Shevek] once that Sabul wanted to keep the new Urrasti physics private — to own it, as a property, a source of power over his colleagues on Anarres. But this idea was so counter to Shevek’s habits of thinking that it had great difficulty getting itself clear in his mind, and when it did he suppressed it at once, with contempt, as a genuinely disgusting thought.

But disgusting or not, the thought is true, and Shevek has to find a way to work with Sabul, to deal with Sabul, so he can get his own highly speculative work into the hands of Urrasti physicists — because there is no one on Anarres who understands what he is trying to do. “He wanted to publish what he wrote and to send it to the men who could understand it, the Urrasti physicists; he needed their ideas, their criticism, their collaboration.” And in order to make that happen, “He needed Sabul.”

So they had bargained, he and Sabul, bargained like profiteers. It had not been a battle, but a sale. You give me this and I’ll give you that. Refuse me and I’ll refuse you. Sold? Sold! Shevek’s career, like the existence of his society, depended on the continuance of a fundamental, unadmitted profit contract. Not a relationship of mutual aid and solidarity, but an exploitative relationship; not organic, but mechanical. Can true function arise from basic dysfunction?

But all I want to do is get the job done, Shevek pleaded in his mind, as he walked across the mall towards the domicile quadrangle in the grey, windy afternoon. It’s my duty, it’s my joy, it’s the purpose of my whole life. The man I have to work with is competitive, a dominance-seeker, a profiteer, but I can’t change that; if I want to work, I have to work with him.

Ursula Le Guin was profoundly attracted to anarchism. As she wrote in a description of her novel’s genesis,

The Dispossessed started as a very bad short story, which I didn’t try to finish but couldn’t quite let go. There was a book in it, and I knew it, but the book had to wait for me to learn what I was writing about and how to write about it. I needed to understand my own passionate opposition to the war that we were, endlessly it seemed, waging in Vietnam, and endlessly protesting at home. If I had known then that my country would continue making aggressive wars for the rest of my life, I might have had less energy for protesting that one. But, knowing only that I didn’t want to study war no more, I studied peace. I started by reading a whole mess of utopias and learning something about pacifism and Gandhi and nonviolent resistance. This led me to the nonviolent anarchist writers such as Peter Kropotkin and Paul Goodman. With them I felt a great, immediate affinity. They made sense to me in the way Lao Tzu did. They enabled me to think about war, peace, politics, how we govern one another and ourselves, the value of failure, and the strength of what is weak.

So, when I realised that nobody had yet written an anarchist utopia, I finally began to see what my book might be.

Le Guin says she set out to write “an anarchist utopia,” but the subtitle which has been attached to the book since a copywriter for an early edition came up with the phrase calls it “an ambiguous utopia.” Le Guin as political thinker wanted construct an ideal, but Le Guin as novelist — as a close and shrewd observer of “the crooked timber of humanity” — couldn’t do that. She had to acknowledge the flaws that creep into every social order because every social order is comprised of people.

This chastened view of anarchism strikes me as the proper one. The founders of the anarchist colony on Anarres certainly wanted to make it impossible for anyone to become Sabul — thus especially their constructed language, Pravic, which tries to remove from lanugage and therefore from thought the idea of possession or ownership — but Sabuls there will always be. The feasible goal of anarchist order is to reduce the number of Sabuls and, when they arise, limit their power over others.

Read the whole story
4 days ago
Boulder, CO
Share this story

webshit weekly

1 Comment

An annotated digest of the top "Hacker" "News" posts for the third week of September, 2020.

When you browse Instagram and find Tony Abbott's passport number
September 15, 2020 (comments)
An Internet discovers some awful webshit; the reward is a conversation with a politician who attempted to secure personal information by destroying the national internet infrastructure. Hackernews has also discovered, been affected by, or written similarly awful webshit and we are treated to a comprehensive list thereof.

This electrical transmission tower has a problem
September 16, 2020 (comments)
A Twitter provides the backstory for the murder weapon of a serial killer. Hackernews is not comfortable with the Twitter's ascription of responsibility, as holding any California-based institution responsible for lax engineering and shitty maintenance practice would set a dangerous precedent and place most Hackernews at severe risk of exposure to adulthood. The thing about forest fires, says Hackernews, is that they're the fault of the forests.

Cloudflare and the Wayback Machine, joining forces for a more reliable Web
September 17, 2020 (comments)
A library makes Faustian deal with some webshits. Only time will tell if Cloudflare will manage to stay online for a long enough contiguous block of time that the agreed-upon functionality will take effect. Hackernews is relieved that the library has a chance to live through this deal, which causes the rest of Hackernews to angrily declare a free CDN the most important thing any massive corporation could ever have done for society.

Ruth Bader Ginsburg has died
September 18, 2020 (comments)
A judge has passed away. No technology is discussed.

DuckDB – An embeddable SQL database like SQLite, but supports Postgres features
September 19, 2020 (comments)
Some Internets release some software. The download page detects the user's operating system and, based on this information, presents download links to a different operating system. Hackernews is enthused by this software's similarity to whatever database they have a crush on, unless the database they have a crush on is PostgreSQL, in which case they are angry that this database is insufficiently similar to PostgreSQL.

On the Use of a Life
September 20, 2020 (comments)
An Internet explains that research is for losers and cool people write computer programs. Hackernews thinks so too. The only evidence provided to back up this claim is vague gesturing at lists of computer programs written by people who like writing computer programs, which explains why the people holding this opinion declined to pursue research careers.

I no longer build software
September 21, 2020 (comments)
A programmer escaped. Hackernews is fascinated by this eventuality, and the programmer arrives to narrate the process of digging a tunnel out of computer hell and emerging in the peaceful woodworking career by the time the credits roll. I did not search the comments to see if anyone else made the joke before I decided the title of this story should have been changed to "The Saw Shank Redemption."

Read the whole story
5 days ago
"A programmer escaped."
Boulder, CO
Share this story



Read the whole story
5 days ago
Boulder, CO
Share this story

Migrating git data with rsync

1 Share

This is a cautionary tale about keeping git data in sync between two machines with rsync. There aren’t really a lot of pitfalls here, but we stumbled into one of them, and I’ve been meaning to write this up since.

tl;dr: to keep git repos in sync using rsync use the command:

rsync --archive --verbose --delete <dir1> <dir2>


Almost a year ago we upgraded the hardware for our primary git host at work. We run our primary git server on bare metal in one of the Equinix data centers in Virginia and it was starting to show its age. Our git host was coming up on the end of its warranty, but – more importantly – we’d simply outgrown the hardware. We run Gerrit as our code review system and its hunger for heap led to more than one late night caused by java.lang.OutOfMemoryError. After spending more time than I probably should have tuning various GC parameters, I put in a request for new hardware.

The plan for the upgrade was pretty simple: Setup a new machine seeded with all of our git data and run it as a replica of the current machine until the switchover window. Prevent the new machine from writing to Gerrit’s database entirely. When the switchover window rolls around: take both machines offline, one final rsync of data, swap DNS records, allow database writes from the new machine, and bring the new machine online.

We finished up the migration at the end of my day and all seemed to go fine, we sent out the all clear and claimed victory. Over my night the European cohort began to see the first inklings of a problem: there were revisions and Gerrit comments missing on the new server! Patches that had been merged were showing up as unmerged! Day was night! Dogs and Cats were best friends! Chaos reigned.

Data integrity problems are alarming, but they are especially acute when the data that’s integrity is in doubt is the canonical source code to a gigantic open source project backing one of the most important free knowledge projects in existence. No pressure.

NoteDB and things to know

The first thing to know is that code reviews in Gerrit aren’t stored in a real database, but are stored instead in NoteDB – which is just a bunch of namespace conventions on top of git. In fact, as of today, the latest version of Gerrit stores nothing in the database and stores everything in git.

Everything being stored in git has some uhhh…I’ll say “interesting”…. side-effects. For example, users are stored in a git repo called All-Users.git and in our version of that repository there are >22,000 refs pointing to the blob ce7b81997cf51342dedaeccb071ce4ba3ed0cf52. Why tag a blob? What could be in that blob?

$ git show ce7b81997cf51342dedaeccb071ce4ba3ed0cf52

That’s right, there are 22,000 refs pointing to a single blob with the contents, star. Each ref is of the format refs/starred-changes/XX/YYYYXX/ZZZZ. This is how Gerrit stores starred changes :|

I don’t know if that’s normal or sane: there are no rules out here in git-is-your-database-now land.

All of the above background about NoteDB is to say that any knowledge you might have about how reviews might disappear from a database don’t hold in Gerrit. All the lovely persistence guarantees about RDBMS mean fuck all. This is a pop quiz about git knowledge.

How reviews are stored

OK, so Gerrit doesn’t use an RDBMS, so we’ll need to know how reviews are stored in order to understand how they might disappear.

Gerrit stores patchsets for review in refs. Gerrit uses the “changes” ref namespace for all changes. For example, the first revision for the first change for the repo “foo” would be stored in /srv/gerrit/git/foo.git under the ref refs/changes/01/0001/1. The next revision for the first change would be stored on refs/changes/01/0001/2. Any commentary about the first change is also stored in a special ref in the changes namespace in git in refs/changes/01/0001/meta.

How refs are stored

Git refs are stored in the refs directory inside a repository’s git directory. A Gerrit change stored in loose refs on disk might look like:

└── 01
    └── 0001
        ├── 1
        └── meta

Each file there points to a commit (or a tree or a blob, but in practice it’s usually a commit).

Periodically (i.e., whenever git runs a garbage collection cycle) that directory is emptied out and the info is shoved into a packed-refs file.

But what happens when there are both? When there is a refs/heads/foo and a packed-refs that references a refs/heads/foo? When you do git rev-parse which one “wins”? This is a common scenario and happens whenever you update a ref:

$ git init
Initialized empty Git repository in /home/thcipriani/tmp/git-pack/.git/
$ echo "foo" > README
$ git add . && git commit -m 'Initial commit'
[main (root-commit) 8c1ba31] Initial commit
 1 file changed, 1 insertion(+)
  create mode 100644 README
$ git update-ref refs/changes/1 HEAD
$ cat .git/refs/changes/1
$ echo 'bar' > README
$ git commit -a -m 'update'
  [main 93791e4] update
   1 file changed, 1 insertion(+), 1 deletion(-)
$ git gc
   Enumerating objects: 6, done.
   Counting objects: 100% (6/6), done.
   Delta compression using up to 4 threads
   Compressing objects: 100% (2/2), done.
   Writing objects: 100% (6/6), done.
   Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
$ ls -lh .git/refs/changes/
total 0
$ git update-ref refs/changes/1 HEAD
$ cat .git/refs/changes/1
$ cat .git/packed-refs
# pack-refs with: peeled fully-peeled sorted
8c1ba312abe6b25948011d05e0ded8bc581b6bb0 refs/changes/1
93791e4e3fbf39cd2d90d678eb2530ce03e5eaf4 refs/heads/main

The punchline

OK, so what happened to our changes? Trying to be cautious we used the rsync command:

rsync --archive --verbose <dir1> <dir2>

We purposely omitted --delete because objects in git are deterministic: who cares if they were packed? Why risk deleting things? We knew we didn’t lose any objects in the transfer. The problem was we didn’t lose any of the unpacked refs either. This meant that when we seeded the git directories on the new server a month before the maintenance window, some of these repositories had loose refs that were subsequently packed into packed-refs. Since the newer refs ended up in packed-refs while the older refs were on disk it made the Gerrit interface appear to be in an older state.

The moral of the story here is to never omit --delete from rsync if you’re trying to keep repos in sync.

Read the whole story
6 days ago
Boulder, CO
Share this story

The unrealized potential of federation

1 Comment

There are some major problems on the internet which may seem intractable. How do we prevent centralization of our communication tools under the authority of a few, whose motivations may not align with our interests? How do we build internet-scale infrastructure without a megacorp-scale budget? Can we make our systems reliable and fault-tolerant — in the face of technical and social problems?

Federation is an idea which takes a swing at all of these problems.

Note: apparently some cryptocurrency enthusiasts are parading this article around to peddle their garbage. Cryptocurrency is the digitally woke techbro’s ponzi scheme, and is a massive waste of electricity and developer effort. Anyone who tells you anything positive about anything which is even remotely connected to cryptocurrency almost certainly has ulterior motives and you should steer clear. So hopefully that settles that. And cryptocurrency is a P2P system, anyway, NOT a federation!

The key trait of a software system which is federated is that the servers are controlled by independent, sovereign entities, and that they exist together under a common web of communication protocols and social agreements. This occupies a sort of middle ground between the centralized architecture and the peer-to-peer (or “decentralized”) architecture. Federation enjoys the advantages of both, and few of the drawbacks.

In a federated software system, groups of users are built around small, neighborly instances of servers. These are usually small servers, sporting only modest resource requirements to support their correspondingly modest userbase. Crucially, these small servers speak to one another using standard protocols, allowing users of one instance to communicate seamlessly with users of other instances. You can build a culture and shared sense of identity on your instance, but also reach out and easily connect with other instances.

The governance of a federated system then becomes distributed among many operators. Every instance has the following privileges:

  1. To set the rules which govern users of their instance
  2. To set the rules which govern who they federate with

And, because there are hundreds or even thousands of instances, the users get the privilege of choosing an instance whose rules they like, and which federates with other instances they wish to talk to. This system also makes it hard for marketing and spam to get a foothold — it optimizes for a self-governing system of human beings talking to human beings, and not for corporations to push their products.

The costs of scaling up a federation is distributed manageably among these operators. Small instances, with their modest server requirements, are often cheap enough that a sysadmin can comfortably pay for the expenses out of pocket. If not, it’s usually quite easy to solicit donations from the users to keep things running. New operators appear all the time, and the federation scales up a little bit more.

Unlike P2P systems, the federated model allows volunteer sysadmins to use their skills to expand access to the service to non-technical users, without placing the burden on those non-technical users to set up, understand, maintain, or secure servers or esoteric software. The servers are also always online and provide strong identities and authenticity guarantees — eliminating an entire class of P2P problems.

A popular up-and-coming protocol for federation is ActivityPub, but it’s not the only way to build a federated system. You’re certainly familiar with another federation which is not based on ActivityPub: email. IRC and Matrix also provide federated protocols in the instant messaging domain. Personally, I don’t like ActivityPub, but AP is not necessary to reap the benefits of federation. Many different kinds of communication systems can be designed with federation in mind, and adjust their approach to accommodate their specific needs, evident in each of these examples.

In short, federation distributes governance and cost, and can allow us to tackle challenges that we couldn’t overcome without it. The free software community needs to rally behind federation, because no one else will. For all of the reasons which make it worth doing, it is not rewarding for corporations. They would much rather build walled gardens and centralize, centralize, centralize — it’s more profitable! Democratic software which puts control into the hands of the users is something we’re going to have to take for ourselves. Viva la federación!

Read the whole story
6 days ago
Shared with the usual "Drew DeVault can overall be a bit much" warning, but I pretty much share this thinking. (Also, though it's not the main part of the post, I concur with this view of cryptocurrency.)
Boulder, CO
Share this story
Next Page of Stories