Constraints are a blessing

I frequently encounter resource constraints when going about my research; sometimes limited human capacity in the form of working hours, but more prominently limited compute in the form of GPU hours (because they’re expensive, in demand, and academia is not OpenAI). Typically, the severity of these constraints fluctuate over time; for example, when trying to crank out results for a paper you might no longer benefit from serial incremental experiments on a single GPU, but rather desire dense grids of results which would gain from having many GPUs running in parallel, but which comes at a much higher daily cost.

When the constraints begin to constrict your work and you feel your plans getting derailed by cruel realities, it can be demotivating. My response to such a crushing constraint is I think akin to grief as I’m forced to reckon with the loss of my goal: an initial dose of denial, wherein I carry on as I were making molluscan progress towards my beyond-yonder objectives; anger at the world and fate for condemning me to this Oliver Twisting existence; bargaining as I desperately look for reasons why the constraint may not be as final as I fear; depression as I give up; and then acceptance when I realise that research goes on and progress demands we look and push forward, staring truth and reality down.

Frankly, that whole process is a little draining and a bit of a waste of time. The result is the same, we move on and push forward with a new, modified, goal, that is adapted to our new constraints. Moreover, experience is starting to show me that perhaps a constraint-induced pivot is not the loss it can first feel like.

Recall what Chomsky said about biological and cognitive constraints not being barriers to intellectual freedom, but rather being vital for structured thought and human creativity. I think there’s something along these lines at work when we do research and are required to work within certain bounds and constraints. Furthermore, in a base sense, it might actually be the throwing of ourselves against the walls of limitation that give the work a part of its meaning.

Far from each constraint of resources leading to a weakening of a research work, I’ve rather noticed that ruthlessly prioritising my objectives and desired results, in order to acknowledge real constraints, and pivoting to new objectives based off of that, is really important for moving research forward. And, after all, forward momentum is the key to progress.

The joke of it all, and the best part in my opinion, is that this ruthless prioritisation is damn difficult to do well. It’s the kind of thing that gets thrown around a lot - “you’ll be 10x more productive if you just prioritise the one task most important to you and ignore everything else” etc. etc. Brilliant. The issue of course is that it’s often very unclear what the most important tasks are in any given situation. Part of that problem is that there are likely a whole bunch of options that you have to choose between (generating those options is step zero, a whole problem unto itself, but I think also made easier by constraints - see the example below). One way to get around that is to have a well-developed sense of taste about these things - easier said than done. The simpler way to make this problem of prioritisation easier is to have resource constraints and to let them kill off as many options as possible.

A big obstacle to embracing this mindset is just not realising that there are other ways of prioritising results and that alternative objectives exist, and that, moreover, it’s unlikely the current plan is even ideal. In theoretical physics, we often present research as a narrative. The space of possible valid narratives is perhaps quite large. Some are certainly better and worse, but it’s very non-obvious to me that we should feel especially attached to the narrative we first settle into, or that we are committed to at any given time. Rather, it seems more likely that it’s suboptimal. Hell, a recurring concern I have, even when a project is going well, is whether I’m presenting the best story, or if I’m missing out on a far more powerful, pedagogical, or general story due to tunnel vision. This social aspect of the research project is significant, and I think it’s just one of the dimensions along which we should assume the space of possible result-framings/objectives is unexplored and where we can likely find a better option if we only look, even factoring in any new resource constraint.

A lack of actual brainstorming, of seriously considering other possibilities, is why we overestimate the quality of our current plans. We can’t be blamed though. It’s seriously hard to just conjure alternative visions of a research project out of thin air. I think it can be done though if you really focus on it, and a resource constraint forces you to focus in this way. Perhaps the stress response also gets the creative juices flowing… Short brainstorms are great for this, but also discussions with collaborators; though there it can be a struggle to have to overcome other peoples cemented ideas as well as your own. This gets easier though when resource constraints are accepted! At that point every party will un-cement themselves and the discussions should be much more fertile. So, you see, resource constraints really are a blessing!

To give an example, I was once working on a project in which I was hoping to produce a dense two-axis figure with many datapoints as the main result. I was emotionally attached to the idea of this being the core of the project, and I’d bound my vision for the project to that result. After some time, it became clear that each individual datapoint was slower to generate than I had naively expected. At first, during the denial stage, I just kept on glacially cranking out these datapoints. A back of the envelope calculation revealed that, at that rate, accounting for inevitable issues I’d run into because planning is hard, it’d likely take me at least six months to get the figure. I slipped into a brief period of frustration - life is so unfair, blablabla. Eventually I accepted the reality of the situation, and the fact that, while I could technically wait six months to get the results, I had no wish to do so and it would be a poor application of resources. At this point, I actually relaxed. I’d accepted the situation and could now begin looking for alternatives. The hunt for an alternative is also a creative endeavour, which is something I enjoy, so I knew that I’d have fun with it. With two-axis figures ruled out, this also shrunk the space of possible results by a ton. From then on I only had to choose between a variety of different one-axis line figures to present. I realised there were a bunch of great candidates still, but perhaps we should be more ambitious and exploratory if we’re going to present lines e.g. focusing on extremal cases, the wackiest examples, a small number of points that are experimentally relevant. By the end of this process, I’d come to view my original two-axis plan as relatively boring. The constraint had forced me to seek alternative objectives and prioritise them, which forced me to both restrict my search and also explore more, which is a handy combination. In fact, without wanting to cast a wide net on this, I wonder if a research project in which one cannot reasonably constrain and explore further is very ripe at all. In that case, perhaps a ruthless re-evaluation of your objectives is a good idea anyway.

Of course, if resource constraints are causing a lot of stress or an inability to get anything done, and you suspect you’re just being an idiot and not utilising all your options, then you should first look for more resources. Often there’s more options than you think. However, if you’re not going to whip anything up, then it’s better to quickly accept the reality and prioritise around that. Very rarely is the constraint endgame; you just need to actually think deliberately about how to adapt to it. Often the resulting plan is better than the initial one. The initial one was born out of complacency and following a path of least resistance, the new one is built out of struggle-induced creativity and conveniently restricted decision making.

As a final comment, turning instead to the human constraint of working hours, I’ll just make an observation. I think it’s a common experience in creative work that ones least fruitful periods are characterised by long hours, while those weeks in which one is most brazen with their alarm clock and their time sheet end up being the most pivotal.