department of hack
1071 stories
·
15 followers

Maintaining Wiki Userscripts via Git

1 Share

This week I’m at the Wikimedia Technical Conference in Atlanta, Georgia. I’ve attended a bunch of great sessions and I’m getting a lot out of my attendance. Today I attended a session about “user scripts” that piqued my interest.

MediaWiki user scripts are powerful user-customizations, written in JavaScript, that themselves exists as pages on the wiki and are used to modify a user’s view of MediaWiki page. There are many available user scripts, and you can also write your own directly in your common.js user page (or in any lowercase user page).

I’m enamoured by all of this. Going into this session, I wasn’t too terribly familiar with gadgets and user scripts – but I learned a ton! The folks in the community know best the kinds of customizations that will improve their lives, and here’s the mechanism by which they’re empowered to make that happen. There are, however, some issues with this approach.

I love wikipages, but I was, long-ago, crippled by Vim and I can’t edit any long-form text outside of Vim and a wikipage is not Vim. Further, code review for user scripts seems, from what I’m told anyway, a bit ad hoc.

One of the projects I spend a lot of time digging in is git; particularly in the contrib folder of the project. There are all sorts of strange and wonderful scripts there. One contrib script that I’ve been aware of for years is mw-to-git which is a perl module that provides a custom MediaWiki remote for git repos. This, to me, seems like a potential solution to this problem.

Setup and usage

After you’ve installed git-mediawiki and have that in your path and the perl module installed, the process is not too involved.

First, you initialize a git repo setting the “pages” variable on the remote to limit it only to the pages you’re interested in. In this instance, I’m interested in my user’s common.js file:

$ git init
Initialized empty Git repository in /home/thcipriani/Projects/User:Thcipriani/.git/
$ git remote add origin mediawiki::https://en.wikipedia.org/w
$ git config remote.origin.pages 'User:Thcipriani/common.js'
$ git pull
Searching revisions...
No previous mediawiki revision found, fetching from beginning.
Fetching & writing export data by pages...
Listing pages on remote wiki...
1 pages found.
page 1/1: User:Thcipriani/common.js
  Found 2 revision(s).
Namespace User not found in cache, querying the wiki ...
1/2: Revision #926066306 of User:Thcipriani%2Fcommon.js
2/2: Revision #926066361 of User:Thcipriani%2Fcommon.js
Searching revisions...
No previous mediawiki revision found, fetching from beginning.
Fetching & writing export data by pages...
Listing pages on remote wiki...
1 pages found.
page 1/1: User:Thcipriani/common.js
  Found 2 revision(s).
1/2: Revision #926066306 of User:Thcipriani%2Fcommon.js
2/2: Revision #926066361 of User:Thcipriani%2Fcommon.js
$ git checkout master

Now I have a file in that directory, User:Thcipriani%2Fcommon.js.mw, that contains the contents of my common.js page. From here I can make slight modifications to the page and push it backup via the same git remote.

$ chmod 600 .git/config  # Since we'll end up with a password in there
$ git config remote.origin.mwUser 'Thcipriani'
$ git config remote.origin.mwPassword 'NotMyPassword1234'
$ git push
Last local mediawiki revision found is 926066361.
Logged in mediawiki user "thcipriani".
Listing pages on remote wiki...
1 pages found.
Getting last revision id on tracked pages...
Last remote revision found is 926066361.
Computing path from local to remote ...
Namespace User:Thcipriani/common.js.mw not found in cache, querying the wiki ...
No such namespace User:Thcipriani/common.js.mw on MediaWiki, known namespaces: Book Book_talk Category Category_talk Draft Draft_talk Education_Progra$
 Education_Program_talk File File_talk Gadget Gadget_definition Gadget_definition_talk Gadget_talk Help Help_talk Media MediaWiki MediaWiki_talk Modul$
 Module_talk Portal Portal_talk Project Project_talk Special Talk Template Template_talk TimedText TimedText_talk User User_talk Wikipedia Wikipedia_t$
lk
Pushed file: 3f1cfc54f8c831f47aff3f05d33e5e12fd9d3df8 - User:Thcipriani/common.js
To mediawiki::https://en.wikipedia.org/w
 * [new branch]      master -> master

I also added a github remote for this repo. I could see, for example (if I had a popular user script) allowing pull-requests via github, code reviewing on github, merging, and pushing back up to Wikipedia post-merge.

There are undoubtedly issues here, but I had this idea kicking around my head and wanted to get it committed to pixels before it slipped out of my mind.

BIG GIANT DISCLAIMER OF DOOM

My views are my own. My views do not represent the views of any employer I've had/have: past, present, or future. I'm probably not even qualified to have these views. I am an egg.
Read the whole story
brennen
5 days ago
reply
Boulder, CO
Share this story
Delete

Puns

1 Share

I was reading over an older blog post, and I noticed an inadvertent pun. I opened up the file so that I could insert a comment at that point that said <!-- no pun intended -->.

But there was already a comment that said <!-- that was a pun -->.

Read the whole story
brennen
10 days ago
reply
Boulder, CO
Share this story
Delete

Software Updates

1 Share
Everything is a cloud application; the ping times just vary a lot.
Read the whole story
brennen
13 days ago
reply
Boulder, CO
Share this story
Delete

Wikipedia and the Big-Tech Backlash

1 Share

While the big-tech giants of Silicon Valley are drawing some much deserved scrutiny from the public and from lawmakers in the United States, Wikipedia – a tech giant in its own right – has remained above the fray.

This is not because Wikipedia is inherently a force for good – although it is most certainly that – rather there are two things that protect Wikipedia (and its users): Wikipedia does not sacrifice the privacy of its users and Wikipedia is not a social media site. The failure to recognize these distinctions represents an existential threat to Wikipedia and its mission of Free Knowledge. It’s easy to become complacent if you feel like your ends will always justify your means.

Wikipedia is unique among large websites simply by not being a social media site. Wikipedia is an encyclopedia – it’s a tertiary source. It does not contain original research or opinion (at least where most folks look). The proliferation of misinformation is often stymied by having to cite reliable sources (particularly when you also create guidelines for reliable sources). Further, the Wikipedia community works tirelessly to ensure the integrity of information on the platform. There are many problems with the information on Wikipedia – these problems are a reflection of the problems with the culture that created Wikipedia and not (for the most part) problems inherent to Wikipedia itself. Tertiary sources are cultural artifacts that reflect culture back on itself. Wikipedia is part of the culture, but it is not creating a cultural shift in the same way that social media does (for better or worse).

Wikipedia is also unique among large websites in that it doesn’t exploit user-privacy to turn a profit. The foundations of Wikipedia and Free Knowledge are the Free Software and Free Culture movements, which are themselves antithetical to the late-stage capitalism that has pushed many Silicon Valley unicorns into the exploitation of their user-base for short-term gains. That is not to say that this kind of exploitation is antithetical to Wikipedia itself – only that it hasn’t happened because the community involved hasn’t allowed it to happen: missing this distinction is an existential threat to Wikipedia as a bulwark of Free Culture.

The ends of Free Knowledge, however noble, can easily become corrupted if the means by which that knowledge is gained and disseminated are incompatible with the foundations of the Free Culture movement.

Much akin to the Euthyphro dilemma, Free Knowledge isn’t good because it’s what Wikipedia – a force for good – does; rather, the mission and values of the Free Knowledge movement are good for the world so (for now at least) Wikipedia is doing them.

Treating Wikipedia as a panacea for the myriad sins of Silicon Valley is not simply a disservice to Wikipedia, it endangers Wikipedia’s future by lulling those with an outsized influence on the movement and its direction into a false sense of certainty that – whatever the path – if Wikipedia’s doing it: it’ll be for the best.

BIG GIANT DISCLAIMER OF DOOM

My views are my own. My views do not represent the views of any employer I've had/have: past, present, or future. I'm probably not even qualified to have these views. I am an egg.
Read the whole story
brennen
13 days ago
reply
Boulder, CO
Share this story
Delete

Free software development doesn't have to awful

1 Share

I want to develop free software with people who lift up each other, and aren't arseholes.

A year ago I left Debian. The process is called retiring in Debian, and it's not final: if you do it in an orderly manner, you can come back again, and be re-instated as a Debian developer with a faster, more lightweight process than is used for entirely new developers. This was the third time I retired. The reasons were different than previously.

The first two times I retired because I was pursuing other passions, and did not feel I could give Debian even the minimal attention and effort required to keep a few minor, leaf packages maintained. This time, I retired because Debian was not fun; it was in fact becoming awful, and I didn't want to participate anymore.

Debian had stopped being fun for me in several ways. One was that the tools, file formats, and workflows Debian uses are getting a little archaic, and generally sub-optimal. Even mundane, everyday tasks involved much more friction than they should. Another is that making any large changes in Debian is too much of an effort, these days, partly because of inertia, partly because it involves so many people.

All of that could have been tolerable, if not for the people. Some of the nicest, most competent people I know work on Debian. It has been a privilege and a joy to work with them.

A few of the other people in Debian I don't want to be associated with in any way, any more.

Debian has some vocal people who treat other people in ways that I don't want accept. I don't want to go into specifics, or names, because that's not going help me move forward.

This is of course not a new thing. Debian has had problems with people behaving badly for years. I may have contributed to that, passively if not actively. However, as I get older, the friction from dealing with abrasive people is sanding off what thick skin I may have had when younger.

As I get older, I am also learning that some of the things I thought were OK when I was younger, are in fact harmful to other people. I don't want to harm other people, and I don't want to participate in a project where some of the people insist on what I think is harmful behaviour, because they feel it's their right.

Long after I left Debian, RMS managed to collapse the reality distortion field that's been surrounding and protecting him for many years. The triggering event for this was comments he made in a context involving Jeffrey Epstein. The comments caused a public uproar, and as a result RMS resigned from role of president of the Free Software Foundation, which he founded. He is currently still the leader of the GNU project. A lot of people are religously defending RMS and attacking his detractors. I find this to be problematic.

LWN has an excellent article on the topic. RMS has been behaving in problematic ways for a long time. He's not been publicaly confronted about it before, at the scale he has been now.

RMS has done some awesome things that he should be honoured for. He started the GNU project and gave it, and the world, a vision of being able to use computers whose entire software stack is free, and inventing copyleft, a legal tool to protect software freedom, as well as writing large amounts of the initial code in the GNU project. He has worked hard and long to help drive the vision of freedom into reality. For this, he shall always be remembered and revered.

That doesn't excuse bad behaviour, such as insisting on abortion jokes, making women feel unwelcome in the GNU project, or various other things. I'm not going to make a list of his shortcomings, because this isn't a critique of RMS specifically. The problem I want to discuss isn't RMS or his personal behaviour.

The problem I do want to discuss is that almost everywhere in the free and open source development communities there's a lot of harmful behavour, and tolerance of it.

Harmful behaviour comes in many forms. Some people, for example, say outright that they don't want women involved in free software development. Others attack gay, lesbian, trans, queer, black, old, young, Christian, Muslim, atheist, any other other group of people identified by whatever attribute the attacker happens to dislike. Yet others are more subtle, not attacking directly, but not giving people in the group they dislike the same chance to participate, learn, grow, and generally be the best person they can be in the context of free software development.

This doesn't just harm the groups of people being targeted. It harms others, who see it happen, and think they might be targeted too, later, maybe for some other reason. It harms reaching the vision of software freedom, because it shoves large parts of humanity outside the software freedom movement, robbing the movement from many voices and much effort. This makes it harder to achieve the vision.

Excluding people from the movement for irrelevant reasons also harms humanity in general. It propagates the hate, hurt, and harm that is emblematic of life and politics around the world. While the software freedom movement can't solve all of those problems, we can and should at least not make it worse.

What should we in the software freedom movement do about all this? I've come to a few conclusions so far, though my process to think about this is ongoing.

  • Most importantly, we need to stop being tolerant of intolerance and bad behaviour. It's time for all project, groups, and organisations in the movement to have and enforce at least a minimal level of civil behaviour. We are a movement consisting of many communities, and each community may want or need their own norms, and that's OK. Some norms may even be in conflict. That's also OK, if unfortunate.

    Some people react to this kind of suggestion with hyperbolic claims and conspiracy theories. I don't want to debate them. It's possible to discuss community norms in a civil and constructive way. I know this, because I've seen it happen many times. However, it requires all participants to at least agree that there's behaviour that's unwelcome, and not reject the notion of community norms outright.

  • I am by nature averse to conflicts. I will try to confront bad behaviour in the future, rather than slinking away and going elsewhere. I will at least speak out and say I think something is unacceptable, when I see it.

  • I think the era for dictatorial models of governance for large free software projects is over. For small projects, it's unavoidable, because there's only one person doing any development, but when a project grows, it doesn't work to have one person, or a small anointed group, making all decisions. It's time to have more democratic governance.

    There are technical decisions that probably can't be done well by letting everyone vote on them. However, every substantial project will have other decisions that the whole community around the project should have a say in. Voting on how to fix a bug may not be workable, but voting on the minimum criteria for determining if a bug fix is acceptable is. Should the project require adding a regression test for any bug found in production? Should any such test and bug fix be subjected to code review? Should such bugs and their fixes be documented, even announced, publicly or kept secret? How should security problems be handled?

    In Debian, one of the guiding principles is that those who do, decide. It seems time to involve those who use in the decision making process as well.

I can't force all communities in the software freedom movement to agree with me. Obviously not. I won't even try. I will, however, in the future be wary of joining dictatorial projects where bad behaviour is tolerated. I'm hoping this will have at least some effect.

What about you?

(My blog does not have comments, but you can respond to this fediverse thread: https://toot.liw.fi/@liw/103080486083100970.)

Read the whole story
brennen
14 days ago
reply
Boulder, CO
Share this story
Delete

An old-school shell hack on a line printer

1 Share

It’s been too long since I last did a good hack, for no practical reason other than great hack value. In my case, these often amount to a nostalgia for an age of computing I wasn’t present for. In a recent bid to capture more of this nostalgia, I recently picked up a dot matrix line printer, specifically the Epson LX-350 printer. This one is nice because it has a USB port, so I don’t have to break out my pile of serial cable hacks to get it talking to Linux 😁

This is the classic printer style, with infinite paper and a lovely noise during printing. They are also fairly simple to operate - you can just write text directly to /dev/lp (or /dev/usb/lp9 in my case) and it’ll print it out. Slightly more sophisticated instructions can be written to them with ANSI escape sequences, just like a terminal. They can also be rigged up to CUPS, then you can use something like man -t 5 scdoc to produce printouts like this:

Plugging the printer into Linux and writing out pages isn’t much for hack value, however. What I really wanted to make was something resembling an old-school TTY - teletypewriter. So I wrote some glue code in Golang, and soon enough I had a shell:

The glue code I wrote for this is fairly straightforward. In the simplest form, it spins up a pty (pseudo-terminal), runs /bin/sh in it, and writes the pty output into the line printer device. For those unaware, a pseudo-terminal is the key piece of software infrastructure for running interactive text applications. Applications which want to do things like print colored text, move the cursor around and draw a TUI, and so on, will open /dev/tty to open the current TTY device. For most applications used today, this is a “pseudo-terminal”, or pty, which is a terminal emulated in userspace - i.e. by your terminal emulator. However, your terminal emulator is emulating a terminal - the control sequences applications send to these are backwards-compatible with 50 years of computing history. Interfaces like these are the namesake of the TTY.

Visual terminals came onto the scene later on, and in the classic computing tradition, the old hands complained that it was less useful - you could no longer write notes on your backlog, tear off a page and hand it to a colleague, or white-out mistakes. Early visual terminals could also be plugged directly into a line printer, and you could configure them to echo to the printer or print out a screenfull of text at a time. A distinct advantage of visual terminals is not having to deal with so much bloody paper, a problem that I’ve become acutely familiar with in the past few days1.

Getting back to the glue code, I chose Golang because setting up a TTY is a bit of a hassle in C, but in Golang it’s pretty straightforward. There is a serial port and in theory I could have plugged it in and spawned a getty on the resulting serial device - but (1) it’d be write-only, so not especially interactive without hardware hacks, and (2) I didn’t feel like digging out my serial cables. So:

import "git.sr.ht/~sircmpwn/pty" // fork of github.com/kr/pty

// ...
winsize := pty.Winsize{
  Cols: 160,
  Rows: 24,
}
cmd := exec.Command("/bin/sh")
cmd.Env = append(os.Environ(),
  "TERM=lp",
  fmt.Sprintf("COLUMNS=%d", 180))
tty, err := pty.StartWithSize(cmd, &winsize)

P.S. We’re going to dive through the code in detail now. If you just want more cool videos of this in action, skip to the bottom.

I set the TERM environment variable to lp, for line printer, which doesn’t really exist but prevents most applications from trying anything too tricksy with their escape codes. The tty variable here is an io.ReadWriter whose output is sent to the printer and whose input is sourced from wherever, in my case from the stdin of this process2.

For a little more quality-of-life, I looked up Epson’s proprietary ANSI escape sequences and found out that you can tell the printer to feed back and forth in 216th” increments with the j and J escape sequences. The following code will feed 2.5” out, then back in:

f.Write([]byte("\x1BJ\xD8\x1BJ\xD8\x1BJ\x6C"))
f.Write([]byte("\x1Bj\xD8\x1Bj\xD8\x1Bj\x6C"))

Which happens to be the perfect amount to move the last-written line up out of the printer for the user to read, then back in to be written to some more. A little bit of timing logic in a goroutine manages the transition between “spool out so the user can read the output” and “spool in to write some more output”:

func lpmgr(in chan (interface{}), out chan ([]byte)) {
	// TODO: Runtime configurable option? Discover printers? dunno
	f, err := os.OpenFile("/dev/usb/lp9", os.O_RDWR, 0755)
	if err != nil {
		panic(err)
	}

	feed := false
	f.Write([]byte("\n\n\n\r"))

	timeout := 250 * time.Millisecond
	for {
		select {
		case <-in:
			// Increase the timeout after input
			timeout = 1 * time.Second
		case data := <-out:
			if feed {
				f.Write([]byte("\x1Bj\xD8\x1Bj\xD8\x1Bj\x6C"))
				feed = false
			}
			f.Write(lptl(data))
		case <-time.After(timeout):
			timeout = 200 * time.Millisecond
			if !feed {
				feed = true
				f.Write([]byte("\x1BJ\xD8\x1BJ\xD8\x1BJ\x6C"))
			}
		}
	}
}

lptl is a work-in-progress thing which tweaks the outgoing data for some quality-of-life changes, like changing backspace to ^H. Then, the main event loop looks something like this:

inch := make(chan (interface{}))
outch := make(chan ([]byte))
go lpmgr(inch, outch)

inbuf := make([]byte, 4096)
go func() {
  for {
    n, err := os.Stdin.Read(inbuf)
    if err != nil {
      panic(err)
    }
    tty.Write(inbuf[:n])
    inch <- nil
  }
}()

outbuf := make([]byte, 4096)
for {
  n, err := tty.Read(outbuf)
  if err != nil {
    panic(err)
  }
  b := make([]byte, n)
  copy(b, outbuf[:n])
  outch <- b
}

The tty will echo characters written to it, so we just write to it from stdin and increase the form feed timeout closer to the user’s input so that it’s not constantly feeding in and out as you write. The resulting system is pretty pleasant to use! I spent about hour working on improvements to it on a live stream. You can watch the system in action on the archive here:

If you were a fly on the wall when Unix was written, it would have looked a lot like this. And remember: ed is the standard text editor.

?

  1. Don’t worry, I recycled it all. 

  2. In the future I want to make this use libinput or something, or eventually make a kernel module which lets you pair a USB keyboard with a line printer to make a TTY directly. Or maybe a little microcontroller which translates a USB keyboard into serial TX and forwards RX to the printer. Possibilities! 

Read the whole story
brennen
19 days ago
reply
Boulder, CO
Share this story
Delete
Next Page of Stories