A view to the gallery of my mind

> recent entries
> calendar
> friends
> Website
> profile
> previous 20 entries

Saturday, September 13th, 2014
8:59 am - Trying out something new

I have a fashion/lifestyle blog these days.

It’s in Finnish, but the pictures should be language-neutral.

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

8:57 am - I have generous friends

My birthday fundraiser was a fantastic success – I wasn’t originally sure whether a $500 goal would be realistic, but we ended up collecting the whole sum in donations to charitable causes! Even though it’s not actually my birthday anymore, I’m still feeling warm and fuzzy over the fact that this many people were ready to pitch in for good causes, and for my birthday. ^^ Final donation breakdown:

MIRI: $420,34
GiveDirectly: $52,33
Mercy for Animals: $27,33

(And I pitched in a bit, both to cover the PayPal fees that would have subtracted from these totals, and to bring the Mercy for Animals total up to $50.)

Many, many thanks to Peter Hurford , Juha Törmänen, Ricardo Martins, Sara Väätäjä, Jasper Mattson, Matti Keränen, Lani Pratt, Daniel Armak, Zoe Williams, Kimmo Kenttälä, Ninnat Dangniam, Holly Morgan, and five anonymous donors!

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

Friday, September 5th, 2014
10:07 am - My birthday fundraiser

My birthday is today, on the 5th of September. I’d like to use my birthday to make the world a slightly better place, which is why I’m holding a birthday fundraiser! If you like me and would like to make both me and others a little happier on my birthday, please consider donating a few dollars, or euros, or Imperial Credits, or whatever your currency may happen to be. It would make me feel great to know that I’d enabled some worthy organizations to do their work just a bit better.

I’m using my birthday to collect for three organizations: the Machine Intelligence Research Institute works to make advanced artificial intelligence safer, GiveDirectly provides money directly to poor people in developing countries, and Mercy For Animals advocates for farm animals.

I chose MIRI because of my personal experience with them, GiveDirectly because they were recommended by the charity evaluation group GiveWell, and Mercy for Animals because they were recommended by Animal Charity Evaluators. You can choose to support one or more of them, or just make an unspecified donation, in which case I’ll use your money to support MIRI.

If you’re willing to donate, please do it via this link, so that we can see how much this drive manages to raise. :)

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

Monday, May 5th, 2014
4:54 pm - Arguments and relevance claims

The following once happened: I posted a link to some article on an IRC channel. A friend of mine read the article in question and brought up several criticisms. I felt that her criticisms were mostly correct though not very serious, so I indicated agreement with them.

Later on the same link was posted again. My friend commented something along the lines of “that was already posted before, we discussed this with Kaj and we found that the article was complete rubbish”. I was surprised – I had thought that I had only agreed to some minor criticisms that didn’t affect the main point of the article. But my friend had clearly thought that the criticisms were decisive and had made the article impossible to salvage.

Every argument actually has two parts, even if people often only state the first part. There’s the argument itself, and an implied claim of why the argument would matter if it were true. Call this implied part the relevance claim.

Suppose that I say “Martians are green”. Someone else says, “I have seen a blue Martian”, and means “I have seen a blue Martian (argument), therefore your claim of all Martians being green is false (relevance claim)”. But I might interpret this as them saying, “I have seen a blue Martian (argument), therefore your claim of most Martians being green is less likely (relevance claim)”. I then indicate agreement. Now I will be left with the impression that the other person made a true-but-not-very-powerful claim that left my argument mostly intact, whereas the other person is left with the impression that they made a very powerful claim that I agreed with, and therefore I admitted that I was wrong.

We could also say that the relevance claim is a claim of how much the probability of the original statement would be affected if the argument in question were true. So, for example “I have seen a blue martian (argument), therefore the probability of ‘Martians are green’ is less than .01 (relevance claim)”, or equivalently, “I have seen a blue martian” & “P(martians are green|I have seen a blue martian) < .01″.

If someone says something that I feel is entirely irrelevant to the whole topic, inferential silence may follow.

Therefore, if someone makes an argument that I agree with, but I suspect that we might disagree about its relevance, I now try to explicitly comment on what my view of the relevance of the argument is. Example.

Notice that people who are treating arguments as soldiers are more likely to do this automatically, without needing to explicitly remind themselves of it. In fact, for every argument that their opponent makes that they’re forced to concede, they’re likely to immediately say “but that doesn’t matter because X!”. The kinds of people who think that they aren’t treating arguments as soldiers will try to avoid automatically objecting “but that doesn’t matter because X” whenever our favored position gets weakened. This is a good thing, but it also means that we’re probably less likely than average to comment about an argument’s relevance even in cases where we should comment on it.

(Cross-posted on Less Wrong.)

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

Monday, April 21st, 2014
7:40 pm - Predictive brains

Whatever next? Predictive brains, situated agents, and the future of cognitive science (Andy Clark 2013, Behavioral and Brain Sciences) is an interesting paper on the computational architecture of the brain. It’s arguing that a large part of the brain is made up of hierarchical systems, where each system uses an internal model of the lower system in an attempt to predict the next outputs of the lower system. Whenever a higher system mispredicts a lower system’s next output, it will adjust itself in an attempt to make better predictions in the future.

So, suppose that we see something, and this visual data is processed by a low-level system (call it system L). A higher-level system (call it system H) attempts to predict what L’s output will be and sends its prediction down to L. L sends back a prediction error, indicating the extent to which H’s prediction matches L’s actual activity and processing of the visual stimulus. H will then adjust its own model based on the prediction error. By gradually building up a more accurate model of the various regularities behind L’s behavior, H is also building up a model of the world that causes L’s activity. At the same time, systems H+, H++ and so on that are situated “above” H build up still more sophisticated models.

So the higher-level systems have some kind of model of what kind of activity to expect from the lower-level systems. Of course, different situations elicit different kinds of activity: one example given in the paper is that of an animal “that frequently moves between a watery environment and dry land, or between a desert landscape and a verdant oasis”. The kinds of visual data that you would expect in those two situations differs, so the predictive systems should adapt their predictions based on the situation.

And apparently, that is what happens – when salamanders and rabbits are put to varying environments, half of their retinal ganglion cells rapidly adjust their predictions to keep up with the changing image predictions. Presumably, if the change of scene was unanticipated, the higher-level systems making predictions of the ganglion cells will then quickly get an error signal indicating that the ganglion cells are now behaving differently from what was expected based on how they acted just a moment ago; this should also cause them to adjust their predictions, and data about the scene change gets propagated up through the hierarchy.

This process involves the development of “novelty filters”, which learn to recognize and ignore the features of the input that most commonly occur together within some given environment. Thus, things that are “familiar” (based on previous experience) and behave in expected ways aren’t paid attention to.

So far we’ve discussed a low-level system sending the higher-level an error signal when the predictions of the higher-level system do not match the activity of the lower-level system. But the predictions sent by the higher-level system also serve a function, by acting as Bayesian priors for the lower-level systems.

Essentially, high up in the hierarchy we have high-level models of how the world works, and what might happen next based on those models. The highest-level system, call it H+++, makes a prediction of what the next activity of H++ is going to be like, and the prediction signal biases the activity of H++ in that direction. Now the activity of H++ involves making a prediction of H+, so this also causes H++ to bias the activity of H+ in some direction, and so on. When the predictions of the high-level models are accurate, this ends up minimizing the amount of error signals sent up, as the high-level systems adjust the expectations of the lower-level systems to become more accurate.

Let’s take a concrete example (this one’s not from the paper but rather one that I made up, so any mistakes are my own). Suppose that I am about to take a shower, and turn on the water. Somewhere in my brain there is a high-level world model which says that turning on the shower faucet will lead to water pouring out, and because I’m standing right below it, the model also predicts that the water will soon be falling on my body. This prediction is expressed in terms of the expected neural activity of some (set of) lower-level system(s). So the prediction is sent down to the lower systems, each of which has its own model of what it means for water to fall on my body, and each of which send that prediction down to yet more lower-level systems.

Eventually we reach some pretty low-level system, like one predicting the activity of the pressure- and temperature-sensing cells on my skin. Currently there isn’t yet water falling down on me, and this system is a pretty simple one, so it is currently predicting that the pressure- and temperature-sensing cells will continue to have roughly the same activity as they do now. But that’s about to change, and if the system did continue predicting “no change”, then it would end up being mistaken. Fortunately, the prediction originating from the high-level world-model has now propagated all the way down, and it ends up biasing the activity of this low-level system, so that the low-level system now predicts that the sensors on my skin are about to register a rush of warm water. Because this is exactly what happens, the low-level system generates no error signal to be sent up: everything happened as expected, and the overall system acted to minimize the overall prediction error.

If the prediction from the world-model would have been mistaken – if the water had been cut, or I accidentally turned on cold water when I was expecting warm water – then the biased prediction would have been mistaken, and an error signal would have been propagated upwards, possibly causing an adjustment to the overall world-model.

This ties into a number of interesting theories that I’ve read about, such as the one about conscious attention as an “error handler”: as long as things follow their familiar routines, no error signals come up, and we may become absent-minded, just carrying out familiar habits and routines. It is when something unexpected happens, or something of where we don’t have a strong prediction of what’s going to happen next, that we are jolted out of our thoughts and forced to pay attention to our surroundings.

This would also help explain why meditation is so notoriously hard: it involves paying attention to a single unchanging stimuli whose behavior is easy to predict, and our brains are hardwired to filter any unchanging stimuli whose behavior is easy to predict out of our consciousness. Interestingly, extended meditation seems to bring some of the lower-level predictions into conscious awareness. And what I said about predicting short-term sensory stimuli ties nicely into the things I discussed back in anticipation and meditation. Savants also seem to have access to lower-level sensory data. Another connection is the theory of autism as weakened priors for sensory data, i.e. as a worsened ability for the higher-level systems to either predict the activity of the lower-level ones, or to bias their activity as a consequence.

The paper has a particularly elegant explanation of how this model would explain binocular rivalry, a situation where a test subject is shown one image (for example, a house) to their left eye and another (for example, a face) to their right eye. Instead of seeing two images at once, people report seeing one at a time, with the two images alternating. Sometimes elements of unseen image are perceived as “breaking through” into the seen one, after which the perceived image flips.

The proposed explanation is that there are two high-level hypotheses of what the person might be seeing: either a house or a face. Suppose that the “face” hypothesis ends up dominating the high-level system, which then sends its prediction down the hierarchy, suppressing activity that would support the “house” interpretation. This decreases the error signal from the systems which support the “face” interpretation. But even as the error signal from those systems decreases, the error signal from the systems which are seeing the “house” increases, as their activity does not match the “face” prediction. That error signal is sent to the high-level system, decreasing its certainty in the “face” prediction until it flips its best guess prediction to be one of a house… propagating that prediction down, which eliminates the error signal from the systems making the “house” prediction but starts driving up the error from the systems making the “face” prediction, and soon the cycle repeats again. No single hypothesis of the world-state can account for all the existing sensory data, so the system ends up alternating between two conflicting hypotheses.

One particularly fascinating aspect of the whole “hierarchical error minimization” theory as presented so far is that it can also cover not only perception, but also action! As hypothesized in the theory, when we decide to do something, we are creating a prediction of ourselves doing something. The fact that we are actually not yet doing anything causes an error signal, which in turn ends up modifying the activity of our various motor systems so as to cause the predicted behavior.

As strange as it sounds, when your own behaviour is involved, your predictions not only precede sensation, they determine sensation. Thinking of going to the next pattern in a sequence causes a cascading prediction of what you should experience next. As the cascading prediction unfolds, it generates the motor commands necessary to fulfill the prediction. Thinking, predicting, and doing are all part of the same unfolding of sequences moving down the cortical hierarchy.

Everything that I’ve written here so far only covers approximately the first six pages of the paper: there are 18 more pages of it, as well as plenty of additional commentaries. I haven’t yet had the time to read the rest, so I recommend checking out the paper itself if this seemed interesting to you.

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

Saturday, April 19th, 2014
9:38 am - Avoiding unnecessary interpersonal anger

Mental checklist to go through whenever you feel angry at someone for not doing something that you expected them to do, or for doing something that you expected them not to do. Applies regardless of whether the person in question is a co-worker, friend, relative, significant other, or anything else:

  1. Ask yourself whether you clearly communicated your expectation to them.
  2. Ask yourself whether they, after hearing about your expectation, indicated that they understood it and would try to fulfill it.
  3. If the answer to both of the previous questions is “yes”, then absent any other mitigating factors, you have the right to be angry at the other person. Otherwise you certainly have a right to feel disappointed or sad, but not angry.
  4. “But it should have been obvious!” is not a substitute for “yes” for either of the first two questions. Okay, there are some situations where it is, like if they suddenly stabbed you with a knife or burned down your house for no reason. But outside such extremes, assume that it wasn’t at all as obvious as you’re thinking it was.

If you don’t like the above being expressed in what sounds like moral terms, you may substitute expressions like “you have a right to be angry” with something like “you may express anger with the reasonable expectation that this will probably improve rather than worsen your relationship, as you are now seeking to enforce the agreement that you and the other person previously entered into and are thus working to ensure that the relationship remains healthy and pleasant for everyone involved, as opposed to just hurting the other person by randomly lashing out at them for something they never realized they should’ve avoided and thus increasing the odds that they feel a need to be on their toes around you. Also, you yourself will be better off if you don’t poison your own thoughts by feeling anger at someone who didn’t actually intend to do you any harm”. But that wouldn’t have been anywhere near as concise to express.

(And of course, if we wanted to be really exact, there’d be the issue that there can be differing degrees of certainty. E.g. someone giving a sympathetic nod when you express your desire counts as consent in many situations. But it still leaves more room for misunderstanding than a situation where they first paraphrase your desire in their own words, and then explicitly say that they’ll try to fulfill it. So ideally you ought to also calibrate your level of anger to be proportionate to the probability of an earlier miscommunication.)

I still frequently catch myself needing to remind myself about points #1 and #2 after I’ve already gotten angry at someone, but at least the act of becoming angry at someone is starting to act as an automatic triggering event for the above checklist. Hopefully I can eventually get to the point where I always go through the list first.

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

Friday, April 4th, 2014
9:43 am - You close your eyes…

You close your eyes,
and you dream.

And then you wake up,
and you think you remember your dream.

But you are now in the Outer World,
and you think in terms of the Outer World.

What you remember is but a thin slice,
the part that can be understood
in terms of the Outer World.

It is not the Inner World,
it is a hint of its surface,
as seen from the outside.

So you forget,
all but a slice,
and you live your life
in the Outer World.

Until that day comes to an end,
until you close your eyes,
until you dream again.

And now you think in terms of the Inner World,
and you remember,
“I have been here before”.

And you wish you could do something
so that you would remember,
when you wake to the Outer World.

But you now think in terms of the Inner World,
and you can’t leave a trace in your mind
that could be understood
in terms of the Outer World.

So you content yourself to wander,
to explore your Inner World.

And then you wake up,
and you think you remember your dream…

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

Thursday, February 20th, 2014
6:19 pm - Trying to understand

“You’re not even trying to understand me”, my friend burst out, frustrated when I’d objected to something that they’d said. I don’t remember my exact response, nor even what the topic actually was. But I do remember being just as frustrated as they were, because I was putting quite a lot of effort into trying to understand what they were saying. It was just that the thing that I thought they were saying didn’t make any sense.

It’s only now, years later, that I suddenly realized just how symmetrical the situation was.

My friend meant X, and my best guess of what they might mean was Y. To them, it was obvious that they meant X, so if I went ahead to assume that they meant Y, then I was clearly just being uncharitable.

When I objected to Y, I meant to say that I was expressing my confusion about my best guess of what they meant. Their best guess of the meaning of my objection was that I could have understood their intended meaning, but chose to be uncharitable instead. And since it felt obvious to me that I was trying to understand them, I took their reply of “you’re not even trying to understand” to that as a sign that they weren’t even trying to understand me.

So in both cases, one of us said one thing, and when the other misinterpreted it, we took it as a sign of unreasonableness – rather than as a reasonable interpretation, given the information that the other person had available. (Which still allows for the possibility that one or both of us really were being unreasonable, of course.)

Communication usually fails, except by accident.

Originally published at Kaj Sotala. You can comment here or there.

(1 echo left behind | Leave an echo)

Saturday, February 8th, 2014
2:59 pm - Don’t trust people, trust their components

I used to think that people were either trustworthy, semi-trustworthy, or unreliable. Basically, there was a single scale, and if you were high enough on that scale, you could be counted on when it came to anything important. My model wasn’t quite this simplistic – for instance, I did acknowledge that someone like my mother could be trustworthy for me but less motivated in helping out anyone else – but it wasn’t much more sophisticated, either.

And over time, I would notice that my model – which I’ll call the Essentialist Model, because it presumes that people are essentially good or bad – wasn’t quite right. People who had seemingly proven themselves to be maximally trustworthy would nonetheless sometimes let me down on very important things. Other people would also trust someone, and seemingly have every reason to do so, and then that someone would systematically violate the trust – but not in general, just when it came to some very specific things. Outside those specific exceptions, the people-being-trusted would still remain as reliable as before.

Clearly, the Essentialist Model was somehow broken.

The Essentialist Model of Trust, with a person's reliability in different situations being a function of their overall trustworthiness.

The Essentialist Model of Trust, with a person’s reliability in different situations being a function of their overall trustworthiness.

Sarah Constantin has written a fantastic essay called Errors vs. Bugs and the End of Stupidity, in which she discusses the way we think about people’s competence at something. I encourage you to read the whole thing, but to briefly summarize it, Sarah notes that we tend to often assume the Error Model, according to which people’s skill at something is equal to a perfect performance minus some error. If someone has a large error, they’re bad and will deliver a consistently bad performance, whereas someone with a small error will tend to only perform well.

You may notice a certain similarity between the Error Model of Competence and the Essentialist Model of Trust. Both assume a single, continuous skill for the thing being measured, be it competence or trustworthiness. Sarah contrasts the Error Model with the Bug Model, which looks at specific “bugs” that are the causes of errors in people’s performance. Maybe someone has misunderstood a basic concept, which is holding them back. If you correct such a bug, someone can suddenly go from getting everything wrong to getting everything right. Sarah suggests that instead of thinking abstractly about wanting to “get better” – either when trying to improve your own skills, or when teaching someone else – it is more useful to think in terms of locating and fixing specific bugs1.

I find that trust, too, is better understood in terms of something like the Bug Model. I will call my current model of trust the Component Model, for reasons which should hopefully become apparent in a moment.

Before I elaborate on it, we need to ask – just what does it mean if I say that I trust someone with regard to something? I think it basically means that I assume them to do something, like coming to my assistance when I need them, or I assume them to not do something, like spreading bad rumors about me. Furthermore, my thoughts and actions are shaped by this assumption. I neither take precautions against the possibility of them behaving differently, nor do I worry about needing to. So trusting someone, then, means that I expect them to behave in a particular, relatively specific way2.

Now we humans are made up of components, and it is those components that ultimately cause all of our behavior. By “components”, I mean things like desires, drives, beliefs, habits, memories, and so on. I also mean external circumstances like a person’s job, their social relationships, the weather wherever they live, and so forth. And finally, there are also the components that are formed by the interaction of the person’s environment and their inner dispositions – their happiness, the amount of stress they experience, their level of blood sugar at some moment, et cetera.

A partial depiction of a Component Model of Trust.

A partial depiction of a Component Model of Trust.

Behaviors are caused by all of these components interacting. Suppose that I’m sick and can’t go to the store myself, but I’m out of food, so I ask my friend to bring me something. Maybe it’s an important part of the identity of my friend to help his friends, so he agrees. He also happens to know where the store is, has the time himself to visit it, isn’t too exhausted to do any extra shopping, and so on. All of these things happen to go right, so he makes good on his promise.

Of course, this isn’t the only possible combination of things that would cause him to carry out the promise. Maybe helping friends isn’t particularly important for his identity, but he happens to like helping people in general. Or maybe he is terribly insecure and afraid of displeasing me, so he agrees even though he doesn’t really want to. If he fulfills his promise, I know that it was due to some combination of enabling factors. But the fulfillment of the promise alone isn’t enough to tell me what those enabling factors actually were. Similarly, if he promises to go but then doesn’t, then I know that something went wrong, but I don’t know what, exactly.

The Essentialist Model of trust takes none of this into account. It merely treats trustworthiness as a single variable, and updates its estimate of the person’s trustworthiness based on whether or not the promise was fulfilled. It may happen that I ask the same person to do something a lot of times, and the responses are always similar, so I get more confident about my Essentialist estimate. But all of the trials happen in relatively similar circumstances: each time, I happen to be activating a similar set of enabling factors. The circumstances, and the type of requests, and the way my friend looks at things, can all change – and all of those things can change the way my friend behaves when I next ask him to do something.

If I am instead using the Component Model, I can try to think about which components might have been the enabling factors this time around, and only increase my confidence in those components. Maybe my friend mentions that he always likes helping out his friends, in which case I can increase my strength of confidence in the component of “likes to help his friends”. Or maybe he looks nervous and jumpy when he brings me the food and lights up when I thank him, in which case it may be reasonable to assume that poor self-esteem was the cause. Of course, I may misread the signs and thus misattribute the cause of the behavior, so I should ideally spread the strength of my belief update over several different possible causes. But this is still a more targeted approach than assuming that this is evidence about his essential nature.

Now the important thing to realize is that, for someone to behave as I expect them to, then every component that’s required for fulfilling that behavior has to work as it should. Either that, or some alternate set of components that causes exactly the same behavior has to come in play. A person can spend a very long time in a situation where the circumstances conspire to keep the exact same sets of components active. But if someone ever gets into a situation where even a single component gets temporarily blocked, or if that component is overridden by some previously dormant component that now becomes active, they may suddenly become unreliable – at least, with regard to the particular behavior enabled by those specific components.

Various things might cause someone to renege on their promises or obligations. The person might…

  • …be too exhausted.
  • …prioritize a conflicting obligation they have towards someone else who needs their help more.
  • …misjudge the situation and make a mistake.
  • …want to refuse to do something, but be too afraid to say so.
  • …genuinely forget having made a promise.
  • …not realize that we expect them to do something in the first place.
  • …act in such a heat of passion that the components creating that passion shut down every other component, in which case we might say that they “couldn’t help themselves” – though we might also hold that knowing their own poor self-control, they shouldn’t have gotten themselves in that situation in the first place.
  • …never have intended to fulfill the promise in the first place, lying to gain personal advantage.
  • …have no regard for other people, doing whatever benefits them personally the most

Some of these reasons we would often consider acceptable, others are more of a gray area, and some we consider as clearly worthy of moral blame no matter what.

That kind of a categorization does have value. It helps us assign moral blame and condemnation when it’s warranted. Some components also have more predictive value than others – if it turns out that someone has no regard for others, then they are probably unreliable in a lot of different situations. Somebody who had good intentions is more likely to succeed the next time, if the things that caused him to fail this time around are fixed.

But we should also notice that at heart, all of these are just situations where we might be disappointed if our model of other people’s behavior doesn’t take everything relevant into account. In each case, some set of components interacted to produce a behavior which wasn’t what we expected. That’s relevant information for predicting their behavior in other situations, to avoid getting disappointed again.

If we are thinking in terms of the Essentialist Model, we might be tempted to dismiss the morally acceptable situations as ones that “don’t count” for determining a person’s trustworthiness. We are then basically saying that if somebody had a good reason for doing or not doing something, then we should just forget about that and not hold it against that person.

But just forgetting about it means throwing away information. Maybe we should not count the breach of trust against the person, in moral terms, but we should still use it to update our model of them. Then we are less likely to become disappointed in the future.

Sarah mentions that when you start thinking in terms of the Bug Model of Competence, you stop thinking of people as “stupid”:

Tags like “stupid,” “bad at ____”, “sloppy,” and so on, are ways of saying “You’re performing badly and I don’t know why.”  Once you move it to “you’re performing badly because you have the wrong fingerings,” or “you’re performing badly because you don’t understand what a limit is,” it’s no longer a vague personal failing but a causal necessity.  Anyone who never understood limits will flunk calculus.  It’s not you, it’s the bug.

This also applies to “lazy.”  Lazy just means “you’re not meeting your obligations and I don’t know why.”  If it turns out that you’ve been missing appointments because you don’t keep a calendar, then you’re not intrinsically “lazy,” you were just executing the wrong procedure.  And suddenly you stop wanting to call the person “lazy” when it makes more sense to say they need organizational tools.

“Lazy” and “stupid” and “bad at ____” are terms about the map, not the territory.  Once you understand what causes mistakes, those terms are far less informative than actually describing what’s happening.

A very similar thing happens when you start thinking in terms of the Component Model of Trust. People are no longer trustworthy or non-trustworthy: that’s not a very useful description. Rather they just have different behavioral patterns in different situations.

You also stop thinking about people as good or bad, or of someone either being nice or a jerk or something in between. To say that someone is a jerk for not holding his promises, or actively deceiving you, just means that they have some behavioral patterns that you should watch out for. If you can correctly predict the situations where the person does and doesn’t exhibit harmful behaviors, you can spend time with them in the safe situations and avoid counting on them in other situations. Of course, you might be overconfident in your predictions, so it can be worth applying some extra caution.

For the matter, the question of “can I trust my friend” also loses some of its character as a special question. “Can I trust my friend” becomes “will my friend exhibit this behavioral pattern in this situation”. As a question, that’s not that different from any other prediction about them, such as “would my friend appreciate this book if I bought it for them as a gift”. Questions of trust become just special cases of general predictions of someone’s behavior.

Some of my more cynical friends say that I tend to have a better opinion of people than they do. I’m not sure that that’s quite it. Rather, I have a more situational opinion of people: I’m sure that countless folks would be certain to stab me in the back in the right circumstances. I just think that those right circumstances come up relatively rarely, and that even people with a track record of unreliability can be safe to associate with, or rely on, if you make sure to avoid the situations where the unreliability comes to play.

Bruce Schneier sums this up nicely in Liars and Outliers:

I trust Alice to return a $10 loan but not a $10,000 loan, Bob to return a $10,000 loan but not to babysit an infant, Carol to babysit but not with my house key, Dave with my house key but not my intimate secrets, and Ellen with my intimate secrets but not to return a $10 loan. I trust Frank if a friend vouches for him, a taxi driver as long as he’s displaying his license, and Gail as long as she hasn’t been drinking.

On the other hand, to some extent the Component Model means that I trust people less. I don’t think that it’s possible to trust anyone completely. Complete trust would imply that there was no possible combination of component interactions that would cause someone to let me down. That’s neither plausible on theoretical grounds, nor supported by my experience with dealing with actual humans.

Rather, I trust a person to behave in a specific way to the extent that I’ve observed them exhibiting that behavior, to the extent that their current circumstances match the circumstances where I made the observation, and to the extent that I feel that my model of the components underlying their behavior is reliable. If I haven’t actually witnessed them exhibiting the behavior, I have to fall back on what my model of their components predicts, which makes things a lot more uncertain. In either case, if their personality and circumstances seem to match the personalities and circumstances of other people who I’ve previously observed to be reliable, that provides another source of evidence3.

That’s what you get with the Component Model: you end up trusting the “bad guys” more and the “good guys” less. And hopefully, this lets you get more out of life, since your beliefs now match reality better.


1: See also muflax’s hilarious but true post about learning languages.

2: Some may feel that my definition for trust is too broad. After all, “trust” often implies some kind of an agreement that is violated, and my definition of trust says nothing about whether the other person has promised to behave in any particular way. But most situations that involve trust do not involve an explicit promise. If I invite someone to my home, I don’t explicitly ask them to promise that they won’t steal anything, yet I trust their implicit promise that they won’t.

Now the problem with implicit promises is that you can never be entirely sure that all parties have understood them in the same way. I might tell my friend something, trusting them to keep it private, but they might be under the impression that they are free to talk about it with others. In such a case, there isn’t any “true” answer to whether or not there existed an implicit promise to keep quiet – just my impression that one did, and their impression that one did not.

Even if someone does make an explicit promise, they might not realize to mention that there are exceptions when they would break that promise, ones which they think are so obvious that they don’t even count those as breaking the promise in the first place. For example, I promise to meet my friend somewhere, but don’t explicitly mention that I’ll have to cancel in case my girlfriend gets into a serious accident around that exact time. Or maybe I tell my friend something and they promise to keep it private, but neglect to say that they may mention it to someone trustworthy if they think that that will get me out of worse trouble, and if they also think that I would approve of that.

Again, problems show up if people have a differing opinion of what counts as an “obvious” exception. And again there may not be any objective answer in these situations – just one person’s opinion that some exception was obvious, and another person’s opinion that it wasn’t at all obvious.

This isn’t to say that a truth could never be found. Sometimes everyone does agree that a promise was made and broken. Society also often decides that some kinds of actions should be understood as promises. For instance, the default is to assume that one will be sexually faithful towards their spouse unless there is an explicit agreement to the contrary. Even though it may not be an objective truth that violating this assumption means a violation of trust, socially we have agreed to act as if it did. Similarly, if someone thinks that a written contract has been broken, we let the courts decide on a socially-objective truth of whether there really was a breach of contract.

Still, it suffices for my point to establish that there exist many situations where we cannot find an objective truth, nor even a socially-objective truth, of whether someone really did make a promise which they then violated. We can only talk about people’s opinions of whether such a promise existed. And then we might as well only talk about how we expect people to act, dropping the promise component entirely.

3: Incidentally, the Component Model may suggest a partial reason for why we tend to like people who are similar to ourselves. According to the affect-as-information model, emotions and feelings contain information which summarizes the subconscious judgements of lower-level processes. So if a particular person feels like someone whose company we naturally enjoy, then that might be because some set of processes within our brains has determined that this is a trustworthy person. It makes sense to assume that we would naturally trust people similar to ourselves more than people dissimilar to ourselves: not only are they likely to share our values, but their similarity may make it easier for us to predict their behavior in different kinds of situations.

Originally published at Kaj Sotala. You can comment here or there.

(2 echoes left behind | Leave an echo)

Saturday, February 1st, 2014
5:44 pm - Social media vignettes: January 2014 edition

Working together, the Elves and Sauron created sixteen Rings of Power, each of which held an advanced artificial intelligence capable of enhancing its wielder’s power. But Sauron tricked the elves, for he had installed a backdoor in each of the AIs: and he then created the One Ring, the world’s most sophisticated portable supercomputer, which he equipped with the cryptographic keys necessary for exploiting the backdoors, and into which he uploaded most of his consciousness.

(January 16th.)


Liar’s paradox: if you’re a fake hipster before it’s cool, then being X before it’s cool makes you a real hipster. But then you’re not a fake, so you can’t be fake before it’s cool, meaning that you’re not a real hipster anymore.

(January 16th.)


Funny Finnish For Foreigners: Jeesustella.

One of the amusing words in Finnish is “Jeesustella”. An application of the (language-independent) principle that All Nouns Can Be Verbed, Jeesustella literally means “to Jesus”. Thus, a literal translation of the Finnish sentence “älä jeesustele siinä” would be “don’t Jesus there”. A more idiomatic (if considerably less concise) translation would be “quit with that holier-than-thou attitude, you’re acting like you thought you were Jesus Christ or something”.

Of course, if you are inclined towards wordplay, you can sometimes get an opportunity to use the word in its more literal meaning. Just a moment ago, I told my mother that “tulin juuri jeesustelemasta, kävin kävelemässä vetten päällä” – “I just came back from being Jesus, I went walking on water”. Which is to say that I took a walk on the frozen ice of the nearby sea.

(January 22nd.)


Someone asked me: “Why are you utilitarian?”

And I replied: At heart, utilitarianism feels like what you get when you ask yourself, “would I rather see few people hurt than many, many people happy rather than few, and how important do I think that to be”, answer “I’d rather see few people hurt, rather see many people happy, and this is important”, and then apply that systematically. Or if you just imagine yourself as having one miserable or fantastic experience, and then ask yourself what it would be like to have that experience many times over, or whether the importance of that experience is at all diminished just because it happens to many different people. Basically, utilitarianism feels like applied empathy.

(January 27th.)

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

Sunday, January 26th, 2014
9:09 am - Applying reinforcement learning theory to reduce felt temporal distance

Computer science for mind hackers: it is a basic principle of reinforcement learning to distinguish between reward and value, where the reward is of a state is the immediate, intrinsic desirability of the state, whereas the value of the state is proportional to the rewards of the other states that you can reach from that state.

For example, suppose that I’m playing a competitive game of chess, and in addition to winning I happen to like capturing my opponent’s pieces, even when it doesn’t contribute to winning. I assign a reward of 10 points to winning, -10 to losing, 0 to a stalemate, and 1 point to each piece that I capture in the game. Now my opponent offers me a chance to capture one of his pawns, an action that would give me one point worth of reward. But when I look at the situation more closely, I see that it’s a trap: if I did capture the piece, I would be forced into a set of moves that would inevitably result in my defeat. So the value, or long-term reward, of that state is actually something close to -9.

Once I realize this, I also realize that making that move is almost exactly equivalent to agreeing to resign in exchange for my opponent letting me capture one of his pieces. My defeat won’t be instant, but by making that move, I would nonetheless be choosing to lose.

Now consider a dilemma that I might be faced with when coming home late some evening. I have no food at home, but I’m feeling exhausted and don’t want to bother with going to the store, and I’ve already eaten today anyway. But I also know that if I wake up with no food in the house, then I will quickly end up with low energy, which makes it harder to go to the store, which means my energy levels will drop further, and so on until I’ll finally get something to eat much later, after wasting a long time in an uncomfortable state.

Typically, temporal discounting means that I’m aware of this in the evening, but nonetheless skip the visit to the store. The penalty from not going feels remote, whereas the discomfort of going feels close, and that ends up dominating my decision-making. Besides, I can always hope that the next morning will be an exception, and I’ll actually get myself to go to the store right from the moment when I wake up!

And I haven’t tried this out for very long, but it feels like explicitly framing the different actions in terms of reward and value could be useful in reducing the impact of that experienced distance. I skip the visit to the store because being hungry in the morning is something that seems remote. But if I think that skipping the visit is exactly the same thing as choosing to be hungry in the morning, and that the value of skipping the visit is not the momentary relief of being home earlier but rather the inevitable consequence of the causal chain that it sets in motion – culminating in hours of hunger and low energy – then that feels a lot different.

And of course, I can propagate the consequences earlier back in time as well: if I think that I simply won’t have the energy to get food when I finally come home, then I should realize that I need to go buy the food before setting out on that trip. Otherwise I’ll again set in motion a causal chain whose end result is being hungry. So then not going shopping before I leave becomes exactly the same thing as being hungry next morning.

More examples of the same:

  • Slightly earlier I considered taking a shower, and realized that if I’d take a shower in my current state of mind I’d inevitably make it into a bath as well. So I wasn’t really just considering whether to take a shower, but whether to take a shower *and* a bath. That said, I wasn’t in a hurry anywhere and there didn’t seem to be a big harm in also taking the bath, so I decided to go ahead with it.
  • While in the shower/bath, I started thinking about this post, and decided that I wanted to get it written. But I also wanted to enjoy my hot bath for a while longer. Considering it, I realized that staying in the bath for too long might cause me to lose my motivation for writing this, so there was a chance that staying in the bath would become the same thing as choosing not to get this written. I decided that the risk wasn’t worth it, and got up.
  • If I’m going somewhere and I choose a route that causes me to walk past a fast-food place selling something that I know I shouldn’t eat, and I know that the sight of that fast-food place is very likely to tempt me to eat there anyway, then choosing that particular route is the same thing as choosing to go eat something that I know I shouldn’t.

Cross-posted to Less Wrong. Related post: Applied cognitive science – learning from a faux pas.

Originally published at Kaj Sotala. You can comment here or there.

(2 echoes left behind | Leave an echo)

Wednesday, January 8th, 2014
8:32 pm - Bayesian academy game: A core mechanic

I feel like my progress with my academy game has been frustratingly slow. Lots of natural language, little code. Over Christmas leave I finally put together a simple Bayes net and belief propagation implementation, but when I was about to move on to start actually implementing the game side of things, I realized that some of the things that I had planned didn’t quite work after all. Then I got stuck for a while, trying to decide what would work in the first place. But then I fortunately started having new ideas, and finally came up with a core gameplay mechanic that would seem promising.

For example, suppose that the player wants to persuade an NPC to dance with her. Below is a representation of what the player character knows of the situation, somewhat similar to how it might be shown in the game.

Some of you will recognize this as resembling an influence diagram (but note that it doesn’t follow all the conventions of one, so if you do know influence diagrams, don’t try to read it literally as one). The rectangles, which would ordinarily be called decision nodes, are two possible dialogue options that the player has. She could try to take a flattering approach and say “I heard you’re a good dancer” in an attempt to appeal to the other’s vanity, or she could rather try to focus on his kindness and nurturing instinct by admitting that she herself has never danced before.

The ovals – uncertainty nodes - represent things that influence the way that the NPC will react to the different dialogue options. A character who’s confident of their dancing skills is going to react to flattery differently than someone who’s insecure about his skills.  The ovals form a standard Bayesian network (or several of them), meaning that the player can have differing amounts of knowledge about the different uncertainty nodes. Maybe she knows that the NPC has high self-esteem, and that this makes it more likely that he’s confident about his dancing skills, but she doesn’t know how good he actually is at dancing, which also influences his confidence about his skills.

Finally, the diamonds – usually called value nodes, but I call these reaction nodes - represent the different ways that the NPC might react to the various dialogue options. Depending on the values of the uncertainty nodes, flattery may please him, intimidate him, or make him react in an indifferent manner. In order to get the NPC to dance with her, the player has to elicit some number of positive reactions from the NPC, and avoid causing any bad reactions. A simple counter sums positive reactions minus negative reactions that have been caused while discussing the topic of dance: if it exceeds some threshold, he agrees, and if it falls below another threshold, he’ll refuse to even consider the thought anymore. The thresholds are partially determined by how big of a favor the player is asking for, partially by how much the NPC likes/trusts the player character.

Suppose the player doesn’t know much about this NPC, but she also doesn’t want to risk things by just saying things that might or might not upset him. Now, in addition to picking one of the dialogue options, she can also experiment with the network to see which variables would influence his decision the most. Suppose that she is considering the “appeal to kindness” route, and has determined that he really needs to be kind in order for this strategy to work. Now she can select the kindness node, and bring up a different set of dialogue options which are aimed at revealing how kind he is. But here too she needs to be careful, because another counter also tracks the NPC’s overall interest in the conversation. If she gets too many indifferent or negative reactions while talking about his kindness, he may just end the conversation and wander off, before she has ever had a chance to get back to the topic of dance. So there’s also an element of risk there.

Alternatively, if the player doesn’t think that getting to dance is that important after all, she can just try her luck with one of the original dialogue options. Maybe those will reveal something about the NPC that’s useful for something more important. After the conversation is over, the NPC’s basic liking for the player character may be adjusted depending on the total positive and negative reactions that she got out of him.

It’s also easy to add additional different complications or advanced options: e.g. an option to pressure the NPC into something, which if successful contributes points for the goal counter, but reduces their liking for you afterwards.

So that’s the basic mechanic. Notice that this basic structure is a very flexible one: instead of talking to an NPC, the player character could have broken into his room in order to look for her stolen necklace, for instance. In that case the decision nodes could represent different places where the necklace might be, the chance nodes might offer chances to look for clues of where he might have hidden the necklace, and the “hit point” counters might represent time before someone shows up, or before the player character loses her nerve and decides to get out. So the same mechanic can be recycled for many different situations.

As for the educational content, the fact that you’ll need to read the Bayes network involved with each variable in order to decide what to do should teach a bunch of useful things, like a basic ability to read the network, evaluate the value of information related to each chance node, and notice and understand d-separation. Not to mention coming to understand how many different situations can all be seen through the same probabilistic framework. And it ought to all relate to the game content in a meaningful manner.

(I also have lots of ideas for other core mechanics. I also should really get the first version of this game done eventually. So I’m making a note of all those other ideas, but for now I’ll focus on milking this core mechanic for all that it’s worth.)

One interesting question is: should the results of the decision nodes be deterministic or random? In other words, suppose the player knows the values of every node that influences e.g. the outcome of the flattery dialogue option. Does that mean that they know what the outcome of choosing that option will be? Or do they only know the odds of each outcome?

I’m inclined with going for the deterministic route: if you know everything that influences the outcome, then you know the final result. I’m not sure that having a random factor in there would be all that realistic, and more importantly, if you’ve spent your time and resources on discovering the value of every parent node of an option, then you deserve to know exactly what outcome your choice will have, rather than getting screwed over by a 10% chance for an unexpected result. And if you don’t know the value of each parent, then there will effectively already be a random factor involved in terms of what you don’t know.

This should hopefully be my last post before I actually have a playable prototype done. Even if a very short one.

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

Friday, December 6th, 2013
3:14 pm - Bayesian academy game: Some game mechanics

So far I have spoken about the possibility of edugames being good, sketched out the basic idea of an edugame built around Bayesian networks, and outlined some design constraints. Now it’s finally the time to get to the actual meat of the matter – the game mechanics.

Note that everything here is subject to change. I’m aiming to outline things to a sufficient level of detail that I have a reasonable clue of how to start implementing the first prototype. Then, when I do have an early prototype together, I’ll keep experimenting with it and throw away anything that doesn’t seem fun or useful. Still, this post should hopefully give some idea of what the final product will be like.

(This post might not be enough for anyone else to start implementing an actual prototype, since there are a lot of missing pieces still. But I think I have a lot of those missing pieces in my head, it just wouldn’t be useful to try to write all of it down.)

To make things concrete enough to start implementing the game, I need to define a concrete goal for (a part of) the game, some of the concrete ways of achieving that goal, as well as the choices involved in achieving the goal. And of course I need to tie all that together with the educational goals.

Goal. Let’s say that, in the first part of the game, you are trying to get yourself voted into the Student Council as the representative of the first-year students. This requires you to first gain the favor of at least three other first-year students, so that you will be nominated to the position. After that, you need to gain the favor of the majority of the most influential students, so that you will actually be the winning candidate.

From that will follow the second part of the game, where you need to also persuade others in the council to support your agenda (whatever your agenda is), either by persuading them or having them replaced with people who are more supportive of you. But for now I will just focus on the first part.

I’m actually starting to think that I should possibly make this into more of a sandbox game, with no set goals and letting you freely choose your own goals. But I’ll go with this for the first prototype.

How to achieve the goals. The game keeps track of your relationship to the different characters. You achieve the nomination if at least three others characters like you enough. To be voted to the council, a majority of the characters have to both like and trust you more than the other candidates.

This seems like a good time to talk more about relationships, which are a rather crucial element of any social drama. Relationships are one of the main types of resource in the game, the rest of which include your public image, personal skills, and time. I’ll cover the rest shortly.

Relationships. Most games that model your relationships with other characters do so by assigning it a single numerical score, with different actions giving bonuses or penalties to the score. So if you give someone a gift you might get +10 to the relationship and if you insult them you might get -20, and your total relationship is the sum of all these factors.

This is a little boring and feels rather game-ish, so I would like to make it feel a little more like you’re actually interacting with real people. To keep things simple, there will still be an overall “relationship meter” (or actually several, if I can make them distinct and interesting enough), but affecting its value shouldn’t feel like just a mechanical process of giving your friends gifts until they love you.

Borrowing from The Ferrett’s three relationship meters, the ones I’m initially considering are like, trust, and infatuation/love.

Like measures the general extent to which someone, well, considers you likeable, even if they don’t necessarily trust you. A relationship that’s high on like but low on trust is the one that you might have with a nice acquaintance with whom you have intellectual conversations online, or that co-worker who’s friendly enough but who you haven’t really done any major projects with or interacted outside work. A high like makes people more inclined to help you out in ways that don’t involve any risk to them.

Things that affect liking include:

  • Having a public image that includes personality traits that other people like. Some traits are almost universally liked or reviled. Other traits are neutral, but other people tend to like people who they feel are similar to them – or, with some traits, dissimilar.
  • Acting in the interests of others, or contrary to them.
  • People having a crush on you.
  • Other people that you’re friends with.
  • Various random events.

Trust measures the extent to which people are willing to rely on you, and the extent to which they think you’re not going to stab them in the back. A high trust may make it easier to repair damage to the other meters, since it makes people more inclined to give you another chance. It also makes people more inclined to help you out in ways that involve a risk to themselves, to vote for you, to confide in you, and to ask you for help.

Things that build up trust include:

  • Telling people information about yourself, indicating that you trust them. If you tell someone a secret you haven’t told anyone else, this will impress them more than if you tell them something that’s common knowledge. (Just don’t jump to revealing all your deepest secrets on the first meeting, or you’ll come across as a weirdo.)
  • Making and keeping promises.
  • Acting in the interests of others, or contrary to them.
  • People having a crush on you.
  • Other people that you’re friends with.
  • Various random events.

Different characters have different kinds of ideals that they are attracted to. If your public image happens to match their ideal, they may develop a crush on you and build up infatuation. The infatuation will grow in strength over time, assuming that your public image continues to match their ideal. If you then also build up their like and trust, the infatuation may turn into love.

The fact that infatuation also increases their like and trust towards you makes it easier to convert their feelings to love, but the feelings may also come crashing down very quickly if you demonstrate untrustworthiness. An infatuated character is more likely to ignore minor flaws, but anything that produces a major negative modifier to any relationship meter may cause them to become completely disillusioned and start wondering what they ever saw in you in the first place. (Love is more stable, and makes it easier to take advantage of your lovers without them leaving you. But you’d never stoop that low, would you?)

A sufficiently high love makes people willing to do almost anything for you.

Besides the things that were already mentioned, love may be affected by:

  • The amount of trust and like that the person has towards you.
  • You committing yourself to a romantic relationship with them.
  • Whether you have any other lovers (some characters are fine with sharing you, others are not).
  • You choosing to genuinely fall in love with them as well – they’ll sense this and receive a considerable boost to their love meter, but you will also end up permanently prevented from ever taking certain negative actions towards them.
  • Various random events.

Every modifier to any of the relationship meters is associated with some source, which you may try to influence. For example, suppose that you promise your friend to do something by a certain time, and then fail to do so. This will produce a negative modifier to their trust rating. You can then try to talk to them and apologize, and possibly offer some compensation for the misdeed. If you play your cards right, you may be able to erase the penalty, or even turn it into a bonus.

Your public image. I have already mentioned your public image a few times, when I mentioned that your perceived personality traits influence the extent to which others like you, as well as the chance of someone developing a crush on you.

Basically, there’s a set of different personality traits that any character may or may not have. Some, like being kind or being cruel, are mutually exclusive. In the beginning, you don’t know anyone’s personality traits, nor does anyone know yours. If you act kindly, you will develop an image as a kind person, and if you act in a cruel way you’ll develop a reputation as a cruel person. And of course, not everything depends directly on your actions – your rivals may try to spread negative rumors about you.

I haven’t yet determined how exactly knowledge about your actions spreads. I don’t want all of the player’s actions to magically become common knowledge the moment the action is made, but neither do I want to keep track of every separate piece of information. And I do want to offer the player the option to try to keep some of their doings secret. My current compromise would be to keep track of who knows what for as long as the amount of people who knew a particular piece of information remained under a certain number. So if the limit was 6, then the game would keep track of any piece of information that was known to at most six people: once the seventh person found out, it would be assumed to have become common knowledge.

People who like you are less inclined to pass on negative rumors about you, as well as more inclined to pass on positive ones. You can also try to spread negative rumors about your rivals, yourself – but this risks developing a reputation as a lying gossip.

Personal skills. The best way of acting in any particular situation depends on what you know of the people involved. If you cultivate a certain kind of image, who will end up liking you more as a result of it, and who will end up liking you less? Is the concession you are offering to your offended friend sufficient to make them forgive you? How much should you trust your new lover or friend? Who might be spreading those nasty rumors about you?

There are several ways by which you could find this out. First, empathy skills can be learned by study: if you have a high relevant empathy skill, you can make a good guess of what someone is like, or how they might react in some situation, just based on your skill. Learning the skills takes considerable time that could be spent on other things, however.

Also, many personality traits correlate with each other, and various actions correlate with different personality traits. It’s almost as if their connections formed a… wait for it… Bayesian network! But in the beginning of the game, you only have a very rough idea of what the structure of the network is like, or what the relevant conditional probabilities are. So you have to figure this out yourself.

Ideally – and I’m not yet sure of how well I’ll get this to work – the game will give you a tool which you can use to build up your own model of what the underlying network structure might be like: a Bayesian network that you can try to play around with. In the beginning, nearly all of the nodes in the model will be unconnected with each other, though some of the most obvious nodes start out connected. For instance, you’ll know that cruel people are more likely to insult others than kind people are, though your estimate of the exact conditional probability is likely to be off.

If you suspect that two nodes in the underlying graph might be linked, you can try joining them together in your model and test how well your model now fits your observations. You can adjust both the conditional probability tables and linkages in order to create a model that most closely represents what you have seen, with the game automatically offering suggestions for the probabilities based on your experiences so far. (Just be careful to avoid overfitting.)

In addition, you can study various knowledge skills. Knowing psychology, for instance, may reveal some of the links in the network and their associated probabilities to you with certainty, which will automatically update your model.

Time. Everything you do takes time, and you can only be in a single place at once. Your lovers and friends will expect you to hang out with them regularly, studying skills takes time, and your plans may be completely interrupted by the friend of yours who has a mental breakdown and needs you there to comfort them RIGHT NOW. (But if you have a sufficiently good reason why you can’t make it, maybe they’ll understand. Perhaps.)

What choices do you need to make? The above discussion should have suggested some choices that you might run across in the game, such as:

  • Who do you make friends with?
  • Do you try to build a small group of strong friendships, or a large group of weak friendships?
  • Who do you trust with information that could hurt you?
  • How much time do you spend on learning the various skills?
  • How much time do you spend on chasing down the source of various rumors?
  • Do you want to spread any nasty rumors yourself?
  • Do you get romantically involved with someone?
  • If so, who will it be? One lover or many?
  • Do you risk doing things that would damage your reputation if people found out?
  • Do you generally side with the people who are your closest allies, or the ones who have actually been wronged?

And others. As you might notice, the choice you’ll want to make in most of these depends on what you’ve figured out about others… which should help us fulfill our goal of making the player genuinely interested in how to figure these things out, via the formalisms that the game employs and attempts to teach.

Free feel to suggest more the comments!

Originally published at Kaj Sotala. You can comment here or there.

(2 echoes left behind | Leave an echo)

Sunday, December 1st, 2013
2:36 pm - Bayesian academy game: Constraints

My work on my Master’s thesis and the Bayesian academy game was temporarily interrupted when I had to focus on finishing the work I had piled up for another course. Now I’m slowly trying to get back into the flow, so here’s a post on some of the things that I’ll be trying to keep in mind while creating the game, and which should help shape its design. This post is still somewhat abstract: more concrete material should hopefully follow in the next post.

I’ve also started on the actual programming, putting together what should hopefully be a flexible Bayes net framework. (I looked at some existing libraries, but it seemed easier to just put together my own implementation.) Mostly model level stuff so far, though, so not much to show yet. I’ll probably put a copy up on Github or somewhere eventually, once it stops looking terribly embarrassing.


“Design is the successive application of constraints until only a unique product is left.” — Donald Norman, The Design of Everyday Things

Having some constraints is nice. They help narrow down the space of possible designs and give an idea for what you need to do. So let’s establish some constraints for our game design.

Has to teach useful things about Bayesian networks / probabilistic reasoning. Kinda obvious, but still worth stating explicitly. At the same time, the game has to be fun even if you had no intrinsic interest in Bayesian networks. Balancing these two gets tough, since you can attract gamers by various story elements and interesting game mechanics, but then these elements might easily become ones that do nothing to teach the actual subject matter. My general approach for solving this is to build the mechanics so that they are all tied to various pieces of hidden information, with Bayes nets being your tool for uncovering that hidden information. More on that later.

Every choice should be interesting. Even if you manage to figure out exactly who knows what, that shouldn’t dictate the right option to pick, just as in XCOM, correctly figuring out the probability of killing your opponent given a certain tactic isn’t enough to dictate the choice of the best tactic. Rather, correctly using the skills that the game tries to teach you should be something that better informs you of the benefits and drawbacks of the different choices you make. If there’s only one obvious option to pick, that’s not interesting.

Of course, eventually somebody will figure out some optimal strategy for playing the game which dictates an ideal decision for various situations, but that’s fine. If we design the game right, figuring out the perfect strategy should take a while.

Must not be ruined by “save-scumming”. In my previous article, I gave an example of a choice involving Bayesian networks: you are given several options for how to act, with the best option depending on exactly which character knew what. Now, what does your average video game player do when they need to make a choice based on incomplete information, and they find out the truth state of affairs soon afterwards? They reload an earlier save and choose differently, that’s what.

Constantly reloading an earlier save in order to pick a better decision isn’t much fun, and really ruins the point of the whole game. But if that’s the optimal way to play, people will feel the temptation to do it, even if they know that it will ruin their fun. I would like to give people the freedom to play the game the way they like the most, but I’m worried that in this case, too much freedom would make the experience unfun.

Other games that rely on randomness try to combat save-scumming by saving the random seed, so reloading an earlier save doesn’t change the outcome of any die rolls. We could try doing the opposite: re-randomizing any hidden states once the player reloads the game. But this could turn out to be tricky. After all, if we have a large network of characters whose states we keep track of and who influence each other, setting up the game in such a way that their states can be re-randomized at will seems rather challenging. So I’m inclined to go for the roguelike approach, with only a single, constantly updating save slot per game, with no ability to go back to earlier saves. That gives us a new constraint:

Each individual game should be short. A constantly updating save means that if you lose the game, you have to start all over. This is fine with a game like Faster Than Light, where a single pass through the game only takes a couple of hours. It would be less fine in an huge epic game that took 50 hours to beat. Based on my own gut feel, I’m guessing that FTL’s couple of hours is quite close to the sweet spot – long enough that you get depth to a single game, short enough that your reaction to failure will be to start a new game rather than to quit the whole thing in disgust. So I will be aiming for a game that can be finished in, say, three hours.

This constraint is probably also a good one since my natural inclination would be to make a huge epic sprawling game. Better to go for something easier to handle at first – a limited-duration game is also easier to extensively playtest and debug. I can always expand this after finishing my thesis, making the game so far the first chapter or whatever.

One could also allow reloading, but restricting where you are allowed to save. Recently I have been playing Desktop Dungeons, where your kingdom grows as you go out on quests that are about 10 minutes long each. You can’t save your progress while on a quest, but you can save it between quests, and the temptation to try just one more quest makes for an addictive experience in the same way that FTL’s short length also makes for an addictive experience. But I’m not sure of whether my current design allows for any natural “units” that could be made into no-save regions in the same way as the quests in Desktop Dungeons can.

Another consequence of having constant saves is that

Failures should be interesting.

Always make failure entertaining. In fact, failure should be more entertaining than success. Success is its own reward. If you succeed at solving a puzzle, you feel good, you barely need any confirmation from the game. Just a simple ding will be satisfying. But if you’re struggling, if the game sort of is in on the joke with you and laughs about it, and gives you a funny animation, it’s actually sort of saying, yeah we want — it’s okay to be here. You can have fun while you’re here. You’ll get there eventually — you’ll get to the other end eventually, but while you’re here you can enjoy yourself; you can relax. And that’s really something I learned sort of from doing the game, but that’s really become an ongoing principal of ours in design is to make the failure — keep failure interesting.Scot Osterwald

Even a game being short doesn’t help if a couple of bad decisions mean that you’re stuck and would be better off restarting rather than playing to the end. FTL solves this by quickly killing you off when you’re starting to do badly: this is interesting, since you can almost always trace your failure back to some specific decision you made, and can restart with the intention not to make that mistake again. I’m not sure how well this would work in this game, though I have been thinking about splitting it into a number of substages, each with a limit on the number of actions that you are allowed to do. (First stage: become elected as the Student Council representative of the first year students by winning them over before the elections. Etc.) Failing to achieve the objective before the time limit would lead to a game over, thus killing you quickly once you started doing too badly to recover. Possibly each stage could also be made into a single no-save unit, allowing saves between them.

Another option would be to take the Princess Maker approach: in these games, you are trying to raise your daughter to become a princess, but even if you fail in that goal, there are a variety of other interesting endings based on the choices that you made in the game. A third option would be to ensure that even a sub-optimal choice opens some new paths through the game – but it could be difficult to ensure that you never ended up in a hopelessly unwinnable state.

Still not entirely sure of the exact form of this constraint and the previous one: will just have to try things out and see what seems to be the most fun.

The next update should either be about some concrete game mechanics (finally!) and the ways that relationships are handled in this game, or about the way that the information is represented to the player.

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

Saturday, November 9th, 2013
5:09 pm - How to make it easier to receive constructive criticism?

Typically finding out about the flaws in something that we did feels bad because we realize that our work was worse than we thought, so receiving the criticism feels like ending up in a worse state than we were in before. One way to avoid this feeling would be to reflect on the fact that the work was already flawed before we found out about it, so the criticism was a net improvement, allowing us to fix the flaws and create a better work.

But thinking about this once we’ve already received the criticism rarely helps that much, at least in my experience. It’s better be to consciously remind yourself that your work is always going to have room for improvement, and that it is certain to have plenty of flaws you’re ignorant of, before receiving the criticism. That way, your starting mental state will be “damn, this has all of these flaws that I’m ignorant about”, and ending up in the post-criticism state where some of the flaws have been pointed out, will feel like a net improvement.

Another approach would be to take the criticism as evidence of the fact that you’re working in a field where success is actually worth being proud about. Consider: if anyone could produce a perfect work in your field, would it be noteworthy that you had achieved the same thing that anyone else also achieve? Not really. And if you could easily produce a work that was perfect and had no particular flaws worth criticizing, that would also be evidence of your field not being particularly deep, and of your success not being very impressive. So if you get lots of constructive criticism, that’s evidence that your field *is* at least somewhat deep, and that success in it is non-trivial. Which means that you should be happy, since you have plenty of room to grow and develop your talents – and you’ve just been given some of the tools you need in order to do so.

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

Wednesday, October 23rd, 2013
11:03 am - Towards meaningfully gamifying Bayesian Networks, or, just what can you do with them

In my previous article, I argued that educational games could be good if they implemented their educational content in a meaningful way. This means making the player actually use the educational material to predict the possible consequences of different choices within the game, in such a manner that the choices will have both short- and long-term consequences. More specifically, I talked about my MSc thesis project, which would attempt to use these ideas to construct a learning game about Bayesian networks.

However, I said little about how exactly one would do this, or how I was intending to do it. This article will take a preliminary stab at answering that question – though of course, game design on paper only goes as far, and I will soon have to stop theorizing and start implementing prototypes to see if they’re any fun. So the final design might be something completely unlike what I’m outlining here. But advance planning should be valuable nonetheless, and perhaps this will allow others to chime in and offer suggestions or criticism.

So, let’s get started. In order to figure out the best way of meaningfully integrating something into a game, we need to know what it is and what we can do with it. What are Bayesian networks?

Bayesian networks in a nutshell

Formally, a Bayesian network a directed acyclic graph describing a joint probability distribution function over n random variables. But that definition isn’t very helpful in answering our question, so let’s try again with less jargon: a Bayesian network is a way for reasoning about situations where you would like to know a thing X, which you can’t observe directly, but you can instead observe a thing Y, which is somehow connected to X.

For example, suppose that you want to know whether or not Alice is a good person, and you believe that being a good person means caring about others. You can’t read her thoughts, so you can’t directly determine whether or not she cares about others. But you can observe the way she talks about other people, and the way that she acts towards them, and whether she keeps up with those behaviors even when it’s inconvenient for her. Combining that information will give you a pretty good estimate of whether or not she does genuinely care about others.

A Bayesian network expresses this intuitive idea in terms of probabilities: if Alice does care about people, then there’s some probability that she will exhibit these behaviors, and some probability that she does not. Likewise, if she doesn’t care about them, there’s still some other probability that she will exhibit these behaviors – she might be selfish, but still want to appear as caring, so that others would like her more. If you have an idea of what these different probabilities are like, then you can observe her actions and ask the reverse question: given these actions, does Alice care about other people (or, what is the probability of her caring)?

Towards gamifying Bayesian networks

Now how does one turn this into a game?

Possibly the simplest possible game that we can use as an example is that of Rock-Paper-Scissors. You’re playing somebody you don’t know, it’s the second round, and on the first round you both played Rock. From this fact, you need to guess what he intends to play next, and use that information to pick a move that will beat him. The observable behavior is the last round’s move, and the thing that you’re trying to predict is your opponent’s next move. (The pedants out there will correctly point out that the actual network you’d want to build for predicting RPS moves would look quite a bit different and more complicated than this one, but let’s disregard that for now.) Similarly, many games involve trying to guess what your opponent is likely to do next – based on the state of the game, your previous actions, and what you know of your opponent – and then choosing a move that most effectively counters theirs. This is particularly obvious with games such as Poker or Diplomacy.

In the previous article, we had a very simple probabilistic network involving Alice, Bob, and Charlie. From the fact that Charlie knew something, we concluded something about the probability of either Alice or Bob knowing something about it; and from the added fact that Bob knew something about it, we could refine our probability of Alice knowing something about it.

Suppose that the piece of knowledge in question was some dark secret of yours that you didn’t want anyone else to know. By revealing that secret, anyone could hurt you.

Now let’s give you some options for dealing with the situation. First, you could preemptively reveal your dark secret to people. That would hurt most people’s opinion about you, but you could put the best possible spin on it, so you wouldn’t be as badly hurt as you would if somebody else revealed it.

Second, you could try to obtain blackmail material on the people who you thought that knew, in order to stop them from revealing your secret. But obtaining that material could be risky, might make them resent you in case they had never intended to reveal the secret in the first place, or encourage them to dig up your secret if they didn’t actually know about it already. Or if you didn’t figure out everyone who knew, you might end up blackmailing only some of them, leaving you helpless against the ones you missed.

Third, you might elect to just ignore the whole situation, and hope that nobody who found out had a grudge towards you. This wouldn’t have the costs involved with the previous two alternatives, but you would risk becoming the target of blackmail or of your secret being revealed. Furthermore, from now on you would have to be extra careful about not annoying the people who you thought knew.

Fourth, you could try to improve your relationship with the people you thought knew, in the hopes that this would improve the odds of them remaining quiet. This could be a wasted effort if they were already friendly with you or hadn’t actually heard about the secret, but on the other hand, their friendship could still be useful in some other situation. Those possible future benefits might cause you to pick this option even if you weren’t entirely sure it was necessary.

Now to choose between those options. To predict the consequences of your choice, and thus the best choice, you would want to know 1) who exactly knew about your secret and 2) what they currently thought of you. As for 1, the example in our previous article was about just that – figuring out the probability of somebody knowing your secret, given some information about who else knew. As for 2, you can’t directly observe someone’s opinion of you, but you can observe the kinds of people they seem to hang out with, the way they act towards you, and so on… making this, too, a perfect example of something that you could use probabilistic reasoning to figure out.

You may also notice that your choices here have both short- and long-term consequences. Choosing to ignore the situation means that you’ll wish to be extra careful about pissing off the people who you think you know, for example. That buys you the option of focusing on something more urgent now, at the cost of narrowing your options later on. One could also imagine different overall strategies: you could try to always be as honest and open as possible about all your secrets, so that nobody could ever blackmail you. Or you could try to obtain blackmail material on everyone and keep everybody terrified of ever pissing you off, and so on. This is starting to sound like a game! (Also like your stereotypical high school, which is nice, since that was our chosen setting.)

Still, so far we have only described an isolated choice, and some consequences that might follow from it. That’s still a long way from having specified a game. After all, we haven’t answered questions like: what exactly are the player’s goals? What are the things that they could be doing, besides blackmailing or befriending these people? The player clearly wants other people to like them – so what goal does it serve to be liked? What can they do with that?

We still need to specify the “big picture” of the game in more detail. More about that in the next article.

Interlude: figuring out the learning objectives

Ideally, the designer of an edugame should have some specific learning objectives in mind, build a game around those, and then use their knowledge about the learning objectives to come up with tests that measure whether the students have learned those things. I, too, would ideally state some such learning goals and then use them to guide the way I designed the “big picture” of the game.

Now my plans for this game have one problem (if you can call it that): the more I think of it, the more it seems like many natural ways of structuring the overall game would teach various ways of applying the basic concepts of the math in question and seeing its implications. For example, it could teach the player the importance of considering various alternative interpretations about an event, instead of jumping to the most obvious-seeming conclusion, or it might effectively demonstrate the way that “echo chambers” of similar-minded people who mostly only talk with each other are likely to reach distorted conclusions. And while those are obviously valuable lessons, they are not necessarily very compatible with my initial plan of “take some existing exam on Bayes networks and figure out whether the players have learned anything by having them do the exam”. The lessons that I described primarily teach critical thinking skills that take advantage of math skills, rather than primarily teaching math skills. And critical thinking skills are notoriously hard to measure.

Also, a game which teaches those lessons does not necessarily need to teach very many different concepts relating to Bayes nets – rather, it might give a thorough understanding of a small number of basic concepts and a few somewhat more advanced ones. A typical math course, in contrast, attempts to cover a much larger set of concepts.

Of course, this isn’t necessarily a bad thing either – a firm grounding in the basic concepts may get people interested in learning the more advanced ones on their own. For example, David Shaffer’s Escher’s World was a workshop in which players became computer-aided designers working with various geometric shapes. While the game did not directly teach much that might come up on a math test, it allowed the students to see why various geometric concepts might be interesting and useful to know. As a result, the students’ grades improved both in their mathematics and art classes, as the game had made the content of those classes meaningful in a new way. The content of math classes no longer felt like just abstract nonsense, but rather something whose relevance to interesting concepts felt obvious.

Again, our previous article mentioned that choices in a game become meaningful if they have both short- and long-term consequences. Ideally, we would want to make the actions of the players meaningful even beyond the game – show them that the mathematics of probabilistic reasoning are interesting because it’s very much the kind of reasoning that we use all the time, in our daily lives.

So I will leave the exact specification of the learning objectives until later – but I do know that one important design objective will be to make the game enjoyable and compelling enough that people will be motivated to play it voluntarily, even on their spare time. And so that, ideally, they would find in it meaning that went beyond it being just a game.

For the next article, I’ll try to come up with a big-picture description of the game that seems fun – and if you people think that it does, I’ll finally get to work on the prototyping and see how much of that I can actually implement in an enjoyable way.

Next post in series: Bayesian academy game – constraints.

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

Wednesday, October 16th, 2013
2:19 pm - Teaching Bayesian networks by means of social scheming, or, why edugames don’t have to suck

As a part of my Master’s thesis in Computer Science, I am designing a game which seeks to teach its players a subfield of math known as Bayesian networks, hopefully in a fun and enjoyable way. This post explains some of the basic design and educational philosophy behind the game, and will hopefully also convince you that educational games don’t have to suck.

I will start by discussing a simple-but-rather-abstract math problem and look at some ways by which people have tried to make math problems more interesting. Then I will consider some of the reasons why the most-commonly used ways of making them interesting are failures, look at the things that make the problems in entertainment games interesting and the problems in most edutainment games uninteresting, and finally talk about how to actually make a good educational game. I’ll also talk a bit about how I’ll try to make the math concerning Bayesian networks relevant and interesting in my game, while a later post will elaborate more on the design of the game.

So as an example of the kinds of things that I’d like my game to teach, here’s an early graph from the Coursera course on Probabilistic Graphical Models. For somewhat mathy people, it doesn’t represent anything complicated: there’s a deterministic OR gate Y that takes as input two binary random variables, X1 and X2. For non-mathy people, that sentence was probably just some incomprehensible gibberish. (If you’re one of those people, don’t worry, just keep reading.)

I’m not going to go through the whole example here, but the idea is to explain why observing the state of X1 might sometimes give you information about X2. (If the following makes your eyes glaze over, again don’t worry – you can just skip ahead to the next paragraph.) Briefly, if you know that Y is true, then either X1 or X2 or both must be true, and in two out of three of those possible cases, X2 is true. But if you find out that X1 is true, then that eliminates the case where X1 was false and X2 was true, so the probability of X2 being true goes down to .5 from .66. In the course, the explanation of this simple case is then used to build up an understanding of more complicated probabilistic networks and how observing one variable may give you information about other variables.

For mathy types the full explanation is probably relatively easy to follow, at least if you put in a little bit of thought. But for someone who is unfamiliar with math – or worse, scared of it -, it might not be. So the question is, how to convert that explanation to a form that is somewhat easier to understand?

The traditional school math approach would be to convert the abstract explanation into a concrete “real-life” case. Let’s say that the variables are people. X1 becomes Alice, X2 becomes Bob, and Y becomes Charlie. A variable being true means that the person is question has heard about some piece of information – say, that Lord Deathfist the Terrible is on a rampage again. If one takes the lines to mean “Alice tells Charlie stuff and Bob tells Charlie stuff (but Alice and Bob don’t talk with each other)”, the “OR gate” thing becomes relatively easy to understand. It means simply that Charlie knows about the rampage if either Alice or Bob, or both, know about it and have told Charlie.

Now we could try to explain it in common-sense terms like this: “Suppose that Charlie knows about Lord Deathfist. That means that either Alice, or Bob, or both, know about it, and have told him. Now out of those three possibilities, Alice knows about it in two possible cases (the one where Alice knows, and the one where Alice and Bob both know) and there’s one possible case where she does not know (the scenario where only Bob knows), so the chance of Alice knowing this is ⅔. But if we are also told that Bob knows it, that only allows for the possibilities where 1) Bob knows and 2) both Alice and Bob know, so that’s 1 possibility out of 2 for Alice knowing it, so the chance of Alice knowing goes down to ½ where it used to be ⅔.”

This is… slightly better. Maybe. We still have several problems. For one, it’s still easy to lose track of what exactly the possible scenarios are, though we might be able to solve that particular problem by adding animated illustrations and stuff.

But still, the explanation takes some effort to follow, and you still need to be motivated to do so. And if we merely dress up this abstract math problem with some imaginary context, that still doesn’t make it particularly interesting. Who the heck are these people, and why should anyone care about what they know? If we are not already familiar with them, “Alice” and “Bob” aren’t much better than X1 or X2 – they are still equally meaningless.

We could try to fix that by picking names we were already familiar with – like Y was Luke Skywalker, and X1 and X2 were Han Solo and Princess Leia, and Luke would know about the Empire’s new secret plan if either Han or Leia had also found out about it, and we wanted to know the chance of all of them already knowing this important piece of information.

But we’d still be quite aware of the fact that the whole Star Wars gimmick was just coating for something we weren’t really interested in. Not to mention that the whole problem is more than a little artificial – if Leia tells Luke, why wouldn’t Luke just tell Han? And even if we understood the explanation, we couldn’t do anything interesting with it. Like, knowing the logic wouldn’t allow us to blow up the Death Star, or anything.

So some games try to provide that kind of significance for the task: work through an arithmetic problem, and you get to see the Death Star blown up as a reward. But while this might make it somewhat more motivating to play, we’d rather play an action game where we could spend all of our time shooting at the Death Star and not waste any time doing arithmetic problems. Additionally, the action game would also allow us to shoot at other things, like TIE Fighters, and that would be more fun.

Another way of putting this would be that we don’t actually find the math task itself meaningful. It’s artificial and disconnected from the things that we are actually interested in.

Let’s take a moment to contrast this to the way that one uses math in commercial entertainment games. If I’m playing XCOM: Enemy Unknown, for instance, I might see that my enemy has five hit points, while my grenade does three points of damage. Calculating the difference, I see that throwing the grenade would leave my enemy with two hit points left, enough to shoot back on his turn. Fortunately I have another guy nearby, and he hasn’t used his grenade either – but I also know that there are at least six more enemies left on the battlefield. Do I really want to use both of my remaining grenades, just to take out one enemy? Maybe I should just try shooting him… both of my guys have a 50% chance to hit him with their guns, and they’d do an average of three points of damage on a hit, so that’s an expected three points of damage if both take the shot, or – calculating it differently – a 25% chance of killing the alien dead… which aren’t very good odds, so maybe the other guy should throw the grenade and the other shoot, and since grenades magically never miss in this game, I’d then have a 50% chance of killing the alien.

So as I play XCOM, I keep running arithmetic calculations through my head. But unlike in the “solve five arithmetic problems, then you get to see the Death Star blowing up” example, these calculations aren’t just glued-on. In fact, while playing, I never actually think that I am solving a set of arithmetic and probability problems in order to be rewarded with the sight of the enemies dying and my soldiers surviving. I think that I’m out killing aliens and doing my best to keep my guys alive. (How many of you had realized that XCOM is an educational game that, among other things, drills you on arithmetic problems? Well, it is!)

This can be a bad thing in some senses – it means that I’m engaging in “stealth learning”, learning a skill without realizing it. Not realizing it means that I can’t consciously reflect and introspect on my learning, and I may have difficulties transferring the skill to other domains, since my unawareness of what I’m doing makes it harder to notice if I happen to run across a problem that employs the same principles but looks superficially different. But it does also mean that the calculations are very much meaningful, and that I don’t view them as an unnecessary annoyance that I’d rather skip and move on to the good parts.

The game scholars Katie Salen and Eric Zimmerman write:

Another component of meaningful play requires that the relationship between action and outcome is integrated into the larger context of the game. This means that an action a player takes not only has immediate significance in the game, but also affects the play experience at a later point in the game. Chess is a deep and meaningful game because the delicate opening moves directly result in the complex trajectories of the middle game-and the middle game grows into the spare and powerful encounters of the end game. Any action taken at one moment will affect possible actions at later moments.

The calculations in XCOM are meaningful because they let me predict the immediate consequences of my choices. Those immediate consequences will influence the outcome of the rest of the current battle, and the end result of the battle will influence my options when I return to the strategic layer of the game, where my choices will influence how well I will do in future battles…

In contrast, the arithmetic exercises in a simple edutainment game aren’t very meaningful: maybe they let you see the Death Star blowing up, but you don’t care about the end result of the calculations themselves, because they don’t inform any choices that you need to make. Of course, there can still be other ways by which the arithmetic “game” becomes meaningful – maybe you get scored based on how quickly you solve the problems, and then you end up wanting to maximize your score, either in competition with yourself or others. Meaning can also emerge from the way that the game fits into a broader social context, as the competition example shows. But of course, that still doesn’t make most edutainment games very fun.

So if we wish people to actually be motivated to solve problems relating to Bayesian networks, we need to embed them in a context that makes them meaningful. In principle, we could just make them into multistage puzzles: DragonBox is fantastic in the way that it turns algebraic equations into puzzles, where you need to make the right choices in the early steps of the problem in order to solve it in the most efficient manner. But while that is good for teaching abstract mathematics, it doesn’t teach much about how to apply the math. And at least I personally find games with a story to be more compelling than pure puzzle games – and also more fun to design.

So I’ll want to design a game in which our original question of “does Bob also know about this” becomes meaningful, because that knowledge will inform our choices, and because there will be long-term consequences that are either beneficial or detrimental, depending on whether or not we correctly predicted the probability of Bob knowing something.

My preliminary design for such a game is set in an academy that’s inspired both by Harry Potter’s Hogwarts (to be more specific, the Hogwarts in the fanfic Harry Potter and the Methods of Rationality) and Revolutionary Girl Utena’s Ohtori Academy. Studying both physical combat and magic, the students of the academy are a scheming lot, ruled over by an iron-fisted student council made up of seven members… And figuring out things like exactly which student is cheating on their partner and who else knows about it, may turn out to be crucial for a first-year student seeking to place herself and her chosen allies in control of the council. If only she can find out which students are trustworthy enough to become her allies… misreading the evidence about someone’s nature may come to cost her dearly later.

In my next post, I will elaborate more on the preliminary design of the game, and of the ways in which it will teach its players the mathematics of Bayesian networks.

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

Wednesday, September 11th, 2013
8:26 am - Just talk to him/her/it

My rule of thumb for romantic relationships and close friendships, for when the other person did something that bothers you and you’re wondering whether it’s worth bringing up. If you end up spending any non-trivial amount of time wondering whether to say anything, then that alone is already bothering you enough that it’s worth bringing up.

In my experience, such things are nearly always caused by misunderstandings or things that can be fixed trivially anyway, so this rule ends up eliminating long-term discomfort and improving communication for the cost of a small bit of short-term discomfort. If you’re worried about the other person’s reaction, it can help if you start by “this was almost too minor of an issue for me to bring up, but…”.

Most importantly, the rule is motivated by a desire for symmetry: if the other person was feeling bothered by some minor thing I did, would I want them to waste energy guessing whether or not to they have the courage to say anything to me? Of course not! And I can’t convincingly encourage them to speak up in such situations if I don’t show an example.

It often feels like staying quiet and accepting the discomfort would be making the other person a favor. But those small things that you didn’t feel were worth mentioning will start piling up and eating into the extent that you like your friend or partner. You’re not doing anyone any favors if you’re eroding your relationship because you don’t trust the other person to react reasonably to your feelings having been hurt.

(Disclaimer: This advice is calibrated for people with no serious self-esteem or mental health issues. There are situations when you actually can’t trust the other person to react reasonably. If you do want to stay close to them regardless, try to make sure that you also have other people with whom your relationship is more functional in this respect.

And of course there may be situations in which you know that you are being too sensitive and don’t want to bring up things because otherwise you’d be constantly expressing your unhappiness with something. Even in that case, though, it’s better for the other person to be aware of your sensitivity – if not every specific concern – so that they can help you overcome it. And knowing that you’re generally reluctant to express your concerns may make it easier for them to explicitly ensure that you’re alright with things that it would be reasonable to get upset about. Also, do make sure that it actually is you who is being too sensitive, rather than you being in a relationship where the other person doesn’t respect you and belittles your concerns.)

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

Sunday, September 8th, 2013
10:54 am - Doing Good in the Addiction Economy

The world is becoming ever-more addictive and distracting, showering us with short-term rewards. But we can still take control of those mechanisms in order to do good in the world, and make ourselves into better people.

Taking a walk in a beautiful forest, I come up with the opening for a brilliant essay or short story. Burning with the inspiration to write it down, I hurry back home and open up my computer. As I wait for my word processor to boot up, I instinctively check whether I have any new e-mails or Facebook notifications. It turns out that I do, so I stop for a moment to read them… and answer them… and look at the link that somebody shared…

Half an hour later, I wrench myself away from these distractions and return to my writing task. But both my inspiration and motivation are gone, and I scarcely even remember what I was intending to write. I stare at the blank screen for a moment, make a haphazard attempt of writing something, and then switch back to Facebook. I re-share the funny link that kept me entertained earlier, find some more things to post, and spend the rest of the evening checking back to see if there have been any new “likes” on those items. When not, I go back to my e-mails to see whether anybody has had anything to say to me. I go to bed feeling twitchy and deflated, feeling like I’ve wasted the whole day – not only have I done nothing useful, I did not even get to relax.

A few days pass, and a new computer game is released. I figure I’ll give it a try… and I end up playing it for the whole day. The next day, I feel compelled to continue playing, until I’ve finally beaten it. If I am lucky, the game continues to be fun throughout. If I am unlucky, as often happens, it loses its charm at some point, and playing becomes tedious; but something forces me to keep playing nonetheless, even when I am starting to find it actively unpleasant. When I’m finally free, several days have gone by, and I again feel like I’ve wasted them.


We are being surrounded by addictive influences from all directions, and increasing amounts of money and science are used for changing our behavior.

Marketers use sophisticated statistical profiling to identify new parents as early as possible, because the birth of a child changes entrenched habits and allows a supermarket chain to tempt the parents into becoming loyal customers. Of course, marketers and companies have been always been trying to influence our habits, and often been quite successful. Fast food is no doubt popular because it is quick and convenient, but it may also be addictive.

Still, as anyone who has gotten stuck checking their Facebook status or mail inbox knows, there are more serious dangers online. A person isn’t in a constant risk of visiting a supermarket or fast food place – going there takes too much effort. But you never know when there might be a new notification, interesting status update or worthwhile e-mail waiting for you, and it takes very little effort to check. Whenever you do check for new items and find something interesting, your brain registers a small amount of reward. Thus, the action that led to the reward – the act of checking your inbox, say – gets reinforced and more likely to be repeated when you’re in a similar context. The more you are online, the more situations there will be that your brain registers as a similar context… and pushes you to check the inbox once again.

At worst, you will be compulsively checking your status and mail, to the point where it prevents you from getting anything else done. Sure, you could go do something more productive, but then the reward would be a long time away… while the Internet always provides a potential opportunity for an instant reward.

Games are also addictive and easy to access. People refer to EverQuest as “EverCrack”, not always entirely jokingly. A bit of Googling brings up news articles about World of Warcraft addicts, and I know at least one person who broke up with her boyfriend after he became too obsessed with the game. Several authors have analyzed the various psychological and outright deceitful tricks that games like Candy Crush Saga pull on their players to keep them hooked. And as Charles Stross comments, even the news media does it:

When a particular incident like today’s bombing of the Boston marathon kicks off a news cycle, a common pattern asserts itself. First, there’s photographic evidence and rumour. Then there’s some initial information—immediate numbers of dead and injured, scary photographs. But the amount of new information coming out tapers off rapidly after the first hour or two, and gives way to rumour and speculation. There probably won’t be any meaningful updates for a couple of days: but the TV channels and newpapers have to fill the dead air somehow, to keep the eyeballs they’ve attracted on the advertisements, so they cobble together anything they can grab—usually talking heads speculating without benefit of actual information. Such speculation in turn increases anxiety levels and causes depression, bringing the onlookers back for more.


Why have things become so addictive now?

In part, it is because of general technological progress. Paul Graham discusses this point in his essay The Acceleration of Addictiveness: technological progress causes things to become more efficient. Solar cells, computers, medicine… everything gets both better and cheaper to produce, but the same process also makes it possible to create drugs, games, and services that are ever more addictive than before. Technological progress makes everything better at its intended task – whether or not everyone would consider that task a good one.

There are also some more specific reasons which are worth mentioning.

I already mentioned low switching costs. If you have a smartphone or tablet with a fast Internet connection, you can pretty much always engage in the addictive activity. You don’t need to expend much effort to do so – in fact, it soon starts requiring more effort not to engage in the activity, as it becomes a thoroughly automated habit.

Next, addictions become intertwined with useful things. I complain a lot about Facebook making me less productive, and ruining entire days of work. So why do I keep using it? Because despite all its problems, I find it really valuable in meeting new people, keeping in touch with acquaintances, inviting people to different events, finding valuable and insightful articles, and exposing my thoughts to feedback from many different people. I have occasionally taken complete breaks from Facebook, and while I have felt more productive and content during times, I have also felt more socially isolated.

Similarly, even though many people find e-mail addictive, few if any can really stop using it. Way too much communication happens via e-mail.

Dropping a psychological addiction would be much easier if we could just quit the source of the addiction entirely. But for many current-day addictions, it is impossible to “just say no”. We have to keep coming back – and then we end up spending much more time with those things than we intended. While the things in question are useful when used in moderation, overuse causes them to become harmful.

There are also economies of scale involved. If you make a silly game that costs .99 cents to play, once a week, and get a million people to play it, you are already making close to four million each month. Online, it’s easy to increase your ability to serve more people, especially if you provide minimal to no customer support – just add more servers. Zynga, the company behind FarmVille, made $1.28 billion in revenue in 2012. That’s a downright crazy sum for a company making simple online games.

So you can make a lot of money making things more addictive… which means that everyone will be eagerly looking at the companies who are successful, and copying everything that those companies did right. Meanwhile, those companies themselves have lots of money to spend on further research aimed at making their games ever more addictive.

Which brings us to tight feedback loops. A company that is selling a physical product has a harder time making improvements to it, since that requires manufacturing new physical copies and then trying to gauge the effect of the changes. An online games company, on the other hand, can tweak a couple of settings in a product, see how this influences the behavior of a randomly selected 5% of their user base, and then adopt the settings which produce the highest returns. The behavior of each user can be tracked, with extra rewards being offered to customers who are in a danger of quitting the service. The most addictive ways of hooking people can be found very quickly.

Feedback loops also show up in the broader economy. The companies which are the most successful in addicting their customers prosper and grow. Companies which fail to do this will either go entirely out of business, or at least fail to grow any further.

This gets us an addiction economy: an economy in which the most successful companies are those which are the best at addicting us, and where their success feeds into itself, making the world ever-more addictive.


What is wrong with this? If people are voluntarily engaging in these activities, is that not a good thing?

Well, if people genuinely enjoy those activities, then maybe so. And sometimes that is what happens. But often, my experience at least is quite different: I just end up feeling drained and miserable – either afterwards, or even during the activity itself. When I do manage to avoid the addictions, I often feel happier and get more done.

Most people want to feel like they’re doing something meaningful with their lives. And many people do not feel like these kinds of addictions cause them to do anything meaningful. Quite the opposite.


But this is not another doom and gloom essay about how technology will destroy us. (If you are into that kind of thing, I do have another essay for you.) For even though these things may be addictive and reduce our productivity, they can also increase it.

I’ve written essays that have started off as Facebook updates and then been expanded as people critiqued them and forced me to express a more nuanced position, until I could just copy-paste pieces of my original update and of my later comments into a whole essay. Where traditional essay-writing may be a lonely activity requiring a lot of focus and motivation, having a dialogue with people is easy, and good tools allows us to merge the two processes.

Or let’s look at the potential of addictive games for education, a topic I have a special interest in. Why do I have problems gathering the motivation to do many traditional math exercises, whereas games that might require just as much intellectual effort addict me without problems?

At least one of the differences is that with traditional exercises, I often need to do quite a bit of work to understand the problem and the desired method well enough that I can get even started on it. Even after I do get started, solving the exercise is often a laborious process. Research on procrastination suggests that we are much more likely to procrastinate on tasks that have remote rewards and which feel like we might not be able to solve them.

In contrast, a game will start with easy problems and provide a smooth and gradual path to the hard ones, keeping you suitably challenged all the time. The classic game structure starts off with a simple level that even a beginner can beat, and then proceeds to ever-more complicated and challenging levels, while regularly giving the player access to new kinds of powers once they’ve mastered the old ones. At the same time, it constantly keeps the difficulty exactly right, so that even as the player gets better in the game, it always feels just challenging enough to be interesting but not so challenging as to be frustrating. (By tradition, there should be a reference to flow theory here, though the challenges may occasionally also be tough enough to make one drop out of flow.)

The math courses that I’ve enjoyed the most, and also felt like I’ve learned the most from, have been ones that employed a similar methodology. They broke the problems down into small pieces and made them easy to approach, and then gradually ramped up the difficulty. This is not just anecdote, either – math courses where the exercises have been similarly broken down into smaller subproblems have seen better results. Hautala et al. (2012) provide the example of a large linear algebra exercise that has been split into four parts, and comment:

Now the set of tasks guides the students through the solving process. This is important especially when the topic is introduced for the first time. If the students encounter difficulties, they are able to ask detailed questions instead of just saying “I don’t understand a thing.”

As an aside, this is part of the reason why I don’t like the term ”gamification” very much – because games are still generally considered something trivial and frivolous, and gamification makes it sound like one is adding pointless and frivolous things on top of education in order to trick students to study. But although there are exceptions, one could say that a game is good if it also employs good educational techniques to constantly keep the player challenged and engaged… and once you take that perspective, “gamification of education” really is just the same thing as good education.

So part of the reason why we keep getting sidetracked by various addictions is that the other things that we should or could be doing in life don’t feel as rewarding. And that is because those other traditional activities haven’t been designed to be properly motivational, and they have always been flawed in this way!

Previously, the best ways of motivating people haven’t been as clear, and there also hasn’t been as much pressure to make our traditional activities more interesting, because there weren’t that many addictive alternatives. The addictive alternatives that did exist were things like drugs and sex, which weren’t as strongly intertwined with useful things and which one could either “just say no” to, or split them off from the periods when one was supposed to be working. So we could just grit our teeth and force ourselves to do things, even if they weren’t much fun. Today, that is becoming harder and harder, so we will become forced to also make our work and studies just as fun. And all the reasons for why things have been becoming more addictive also let us make things like education and work more addictive.

Of course, there are tradeoffs between things such as educational value and entertainment. A game that can just focus on being as entertaining as possible might always be more fun than a game that also has to be educational. An educational game has to teach things that are true, an entertainment game can just make up whatever seems most fun. And in general, the things that are being optimized for pure addictiveness might be able to do that more effectively than the things that are also trying to be useful.

But although it would be great if the best educational games were as fun as the best entertainment games, they don’t necessarily need to be. If they are just fun enough that we can enjoy them without constantly getting the feeling that we’d rather be doing something else, that could be enough. There have been plenty of silly little games that were nowhere as fun as the best entertainment games, but still enjoyable enough that I kept playing them once I’d gotten started. And if I knew that a game was actually teaching me useful content, that would give me an extra incentive to keep playing.

The same principle should generalize to things beyond just games: as long as we make the activities that we engage in rewarding enough, we have an easier time resisting our addictions. And while the pure addictions can provide us with plenty of short-term rewards, ultimately it is the long-term rewards which feel the most fulfilling. If we can make ourselves addicted to those, we can break the grip of the useless addictions.

So. Let us take all the psychological tricks and intricate reward systems and other things that various groups are developing in order to addict us. And then let’s turn our attention to the things that we currently have to force ourselves to do, and make those just as compelling and intrinsically enjoyable, so that we can achieve the things that we really want to achieve.


As I have presented it, my argument suffers from one major problem.

“Let’s use these techniques to make everything more compelling” works fine as long as we are dealing with well-understood problems that can be broken down into small chunks with a clear and logical progression. But not every problem is like that. Somebody who is tackling a truly novel problem often feels at a complete loss, having no idea of whether they are even on the right track. Someone who is facing a tough problem that is known to be solvable, but which nobody has yet turned into an addictive one, might feel similarly. If we motivate people to work by giving them frequent external rewards, does that train them to become even more impatient and eager to quit in cases where no such rewards are forthcoming?

Apparently, Western cultures are already doing badly with this. According to this NPR piece, American first-graders who were given an impossible math problem to work on tended to give up within less than 30 seconds. Japanese students, on the other hand, spent a whole hour trying to solve it, stopping only when the researchers told them to. We Westerners have already been trained to desire instant gratification, and it might not be a good idea to turn society even more in that direction.

I am not at all sure that we have a choice, however. It is all well and good to say that we should stop being so focused on instant gratification and train ourselves to work on problems for longer before giving up. But how are we going to do it in a society that keeps becoming ever more addictive? Most people have big projects and noble goals that they say they intend to accomplish, one day – and then they never do, because there are easier rewards available. “We should train ourselves to have a longer tolerance for uncertainty and delayed rewards” is exactly the kind of goal that provides an uncertain reward sometime late in the future… and is thus likely to be put aside in favor of easier goals.

Unless we find a way to make that goal, too, more rewarding. I actually think that the techniques for making things more compelling could even be used for helping us do things which aren’t as immediately compelling. We just have to figure out how, and I’ve personally been experimenting with some such tricks.

Previously, each new Facebook notification created in me an urge to click it. Now I have been training myself to instead take pleasure in seeing the number of unread notifications go up. Accumulated 27 notifications over three days? Yay! Good job at not having clicked on those! Maybe I could get that number up to 28 if I waited for a bit longer!

When I started meditating, I used an IRC channel to train myself to meditate for longer at a time: after each meditation session, I reported the duration of my session to that channel. It was pleasing to gradually make the number grow higher.

Similarly, we can train ourselves to get rewards from the process rather than the result. Those Japanese kids were so persistent partially because they had been taught that hard work is a virtue, so not giving up was already a reward by itself. We could learn to reward ourselves for working without interruptions for some set amount of time, regardless of how much got done. Arguably, the reason why many people find the Pomodoro Technique so effective might be because it does exactly this.

Ultimately, the rise of addictiveness is because we are learning to more reliably modify our behavior. That knowledge can be used for bad ends, but it can also be used for good ends – even for increasing our resistance to becoming addicted.


I experiment with various techniques for improving my own effectiveness and breaking the hold of harmful addictions. I make public commitments, such as declaring on Facebook that I will not be making any Facebook posts for a week. I employ structured procrastination. When I notice that I’m having trouble working from home, I try to find a job where I’m actually surrounded by other people. I ease on my perfectionism and my desire to write big essays in one go, and encourage myself to write down my ideas even if I only have energy or material for a couple of paragraphs.

I look for ways of motivating myself by briefly playing non-addictive games that are related to what I want to do. I tell myself that shipping is what differentiates a generalist from a dabbler. I employ different instant motivation techniques. I practice meditation, both to increase my tolerance to delay, and to develop a better awareness of my mind to help me notice motivating feelings better.

Some of these work better. Some of them work worse. But I’m slowly making progress. This essay gets finished by the “write a few paragraphs at a time” technique; I find the motivation to finally provide some people with feedback on their philosophy paper by playing Socrates Jones: Pro Philosopher and arguing with John Stuart Mill in-game. I also write this essay partially to motivate myself to continue working on the topic of educational games.

It is a struggle, but there is yet hope.


Edited to add: Technology is Heroin is a great essay that makes many similar points, though it’s more focused on the doom and gloom side.

Originally published at Kaj Sotala. You can comment here or there.

(1 echo left behind | Leave an echo)

Monday, August 12th, 2013
8:39 pm - What we’ve got here is a failure to communicate scientific evidence

One of the biggest ways by which scientists and science-minded people fail to communicate with laypeople, I think, is with the phrase “there’s no scientific evidence for X”.

Suppose that a layperson comes to a scientist and explains excitedly that he just heard of a new way to [raise children/lose weight/cure illnesses] that works great, and everyone should adopt that method at once!

The scientist replies that she has heard of that method too, but so far there’s no scientific evidence for it.

By that, the scientist means something roughly like “Well, there might be something to it, but the successes that you mention could very well just be coincidence, and it’s really really hard to figure out whether that thing actually works, even if we do lots of careful experiments. So although the thing that you mention could be worth looking into, we really don’t know whether it works yet, and most things like that actually turn out not to work when you do the experiments, so for now we should assume that it won’t work.”

What the layperson actually hears is “I’m going to be all stuck up and holier-than-thou and insist that we do this time-consuming and expensive examination so that the thing could first pass my utterly unrealistic burden of proof, all the while the evidence of it working is right before our eyes and we could finally [raise children/lose weight/cure illnesses] right if we just acted on it.”

And then the layperson thinks, if only you came out of your ivory tower, you would realize that in the real world you do things when you see that they work, not when you’ve wasted years addressing the one in a million chance that they might somehow not work after all.

And then both of them end up marching angrily away and thinking that the other person is a complete idiot.

(See also: Expecting Short Inferential Distances.)

Originally published at Kaj Sotala. You can comment here or there.

(Leave an echo)

> previous 20 entries
> top of page