A motivating toy problem: deciding whether two things are the same

Consider a very simple task: you see two shapes on a screen and must decide whether they are the same or different.

At first glance, this looks trivial. But it turns out to be an excellent test case for thinking about representations.

• A symbolic model might represent the shapes as discrete descriptions (e.g., shape = square, color = red) and apply an explicit rule: IF description(A) = description(B), THEN respond SAME.
• An analog model might represent each shape as a point in a continuous feature space (size, orientation, curvature) and compute a distance between them.
• A connectionist model might take raw pixel inputs, learn internal representations through exposure, and produce a SAME or DIFFERENT response without ever explicitly representing the rule “compare features.”

All three approaches can solve the task, but they do so by making very different assumptions about what is represented and how.

The classical view: cognition as symbolic computation

Early cognitive science, emerging in the 1950s and 1960s, was strongly shaped by developments in logic, computer science, and artificial intelligence. Influential figures such as Allen Newell and Herbert A. Simon proposed that thinking is best understood as symbol manipulation.

In this view:

• Mental representations are explicit symbols with internal structure.
• These symbols express propositional content (e.g., IF goal is X AND condition Y holds, THEN do Z).
• Cognition consists of rule-based operations over these symbols.

Production systems, formal models built from rules that fire when their conditions are met, became a central formalism for this approach. Importantly, these theories were not meant as loose metaphors. They were intended as literal computational accounts of mental processes.

Hybrids and pragmatism: symbolic systems in practice

As cognitive modeling matured, it became clear that purely symbolic systems struggled with issues like learning, noise, timing, and graded behavior. Human cognition is slow, error prone, and sensitive to frequency and recency, properties that are awkward to capture with all or none rules.

In response, many symbolic architectures incorporated continuous elements.

A prominent example is ACT-R, which retains symbolic chunks and production rules but augments them with:

• activation levels,
• continuous decay and learning functions,
• noise and utility parameters.

The connectionist alternative: representations as patterns

Where is knowledge stored: weights or activations?

A common confusion concerns where “knowledge” resides in a neural network. The answer is: in both weights and activations, but in different ways.

• Weights encode persistent knowledge. They determine the space of possible representations and transformations the network can produce. One helpful way to think about weights is as storing dispositions or potentials, what the system is capable of representing.
• Activations encode momentary representations. At any given moment, the current pattern of activity across units reflects what the system is representing or processing right now.

An analogy: weights define the grammar of a language; activations are the sentences currently being spoken.

So knowledge is not “only in the weights,” nor are activations meaningless without them. Representation arises from their interaction.

What do different layers represent?

Although details vary across architectures, it is useful to distinguish roles typically played by different layers:

• Input units represent task relevant information from the environment. These might correspond to sensory features, stimulus dimensions, or structured encodings supplied by the modeler.
• Hidden units develop internal representations shaped by learning. These are often the most theoretically interesting part of the network. They may reflect abstract dimensions, latent variables, or compressed structure in the task.
• Output units represent the system’s response, choices, classifications, actions, predictions.

A crucial unifying insight

Modern neural networks are universal function approximators. In principle, they can implement symbolic computations just as well as continuous mappings.

What distinguishes connectionist models is not what they can compute, but how the computations are specified:

• Symbolic models rely on hand crafted representations and rules supplied by the theorist.
• Connectionist models rely on learning, allowing representations and transformations to be discovered rather than prescribed.

This shifts the theoretical focus. Instead of asking What is the right representational format?, cognitive science increasingly asks:

How do structured representations arise from learning, constraints, and interaction with the environment?

That question, rather than a strict symbolic vs. analog dichotomy, captures much of what is at stake in contemporary theories of cognition.

Why this distinction matters

At an introductory level, it might seem like this debate is mostly philosophical. But it has very practical consequences.

• It affects how we design experiments. Are we testing explicit rules, similarity spaces, or learned representations?
• It affects how we interpret behavior. Is an error a failure of a rule, a noisy comparison, or a learned bias?
• It affects how we think about artificial intelligence. Modern AI systems succeed largely by learning representations rather than being given them, which brings connectionist ideas back to center stage.

Most importantly, it affects how we think about explanation in psychology. A good cognitive theory does not just reproduce behavior. It tells us what kind of internal structure could plausibly give rise to that behavior.

Understanding the representational commitments of different models is the first step toward making sense of that structure.

Meaning without pointing

A mathematical object is what it does inside a system of relations and operations

Human language lets us form grammatically clean statements that carry no content. I can say “X is roupy” just as easily as I can say “2 is even.” But unless “roupy” connects to something, I haven’t said anything.

If I define “roupy” as “paternally upy,” I haven’t fixed the problem. I’ve just stepped one move deeper into a dictionary loop. The words fit together, but nothing constrains anything.

With everyday words, we eventually escape this loop by pointing to the world. Some terms get grounded in perception and action. Even if definitions remain fuzzy, there’s friction: you can be wrong about what “red” applies to, and the world pushes back.

Mathematics doesn’t get this kind of grounding, at least not in the same direct way. The mathematician isn’t allowed to point at “two-ness” floating in the air. The only thing left is this: a property of X becomes meaningful only insofar as it connects X to other objects through formal operations and relations. In other words: if the definition doesn’t place X in a structure, it’s not doing anything.

The symbols 1, 2, 3, … only mean something, i.e., have properties, once we define operations and rules: Peano’s axioms, or Church numerals, or a set-theoretic construction. Only within such a system does the statement “6 is divisible by 2” have content, because “divisible by 2” can be unpacked and checked under the representation you’re using. It defines a relation between objects of the same class (6 and 2) and the outcome of an operation involving them (like “remainder 0”).

The point is not that “definitions matter.” That’s trivial. The point is that mathematical meaning is operational and relational. It lives in what the symbol lets you do and what it forces to be true.

A seemingly silly object called a bazz

To make this feel less like a slogan, let’s take a deliberately silly example.

Imagine an object called a bazz. It interacts with pins. There are three operations:

• kuua takes a pin and a bazz and returns a bazz
• ouz takes a bazz and returns a bazz plus a pin
• heff takes a bazz and returns a pin

Also:

• you can always fish a bazz from the void
• you can ask a bazz whether it is a jazz, and it must answer by nodding or shaking

And bazzes satisfy these rules:

1. If you kuua a bazz with a pin and then immediately ouz it, you get the original bazz and pin back.

2. If you heff a bazz after kuua-ing it with a pin, you get back that pin – and if you then ouz the bazz, you mysteriously get another free copy of the same pin and the original bazz.

3. A freshly fished void bazz always nods when asked if it is a jazz. After kuaa-ing it with a pin it shakes, and once it shakes, it keeps shaking.

It sounds like nonsense (it is), but it is also just a stack, an Abstract Data Type, in disguise: push(), pop(), peek(), create(), isEmpty():

While the third column of this table clearly articulates the intended effect of the operation, it is nothing more than a “design specification”: what we want it to do and mean. It is the equational rules that endow the system with these behaviors:

1. ouz(kuua(p, b)) = (b, p) — If you push a pin onto a bazz and immediately pop, you get back the original bazz and the pin

2. heff(kuua(p, b)) = p — If you peek at a bazz after pushing a pin, you see that pin

3. jazz(fish()) = true — A freshly created bazz is empty

4. jazz(kuua(p, b)) = false — Any bazz with at least one pin is not empty

That’s it - the system is fully specified, regardless of the symbols we use, and the helpful descriptions we attach them and to the rules that govern them. Notice that nothing above ever specifies anything about the internal mechanism of any operation; what is more, it doesn’t even specify the outcome of individual operations, except for what type of object it returns. It only specifies how the behavior of some rules affects others.

What I like about this example is how it separates three things that we often blur together:

1. The interface: the operations you’re allowed to do.

2. The axioms: the laws those operations must satisfy.

3. The interpretation: the story we tell ourselves (“it stores things,” “it has an inside,” “it grows,” etc.).

The axioms never say “last-in-first-out.”, “inside.”, nor “storage.” And yet if you take the rules seriously, you can prove an emergent structural truth, a consequence that wasn’t spelled out:

If you kuua ten pins in a row and then repeatedly ouz, you recover the pins in reverse order.

At the same time, some questions you’re tempted to ask are not merely unanswered; they’re not even well-formed in the language of the system. For example: “what happens to the pins inside the bazz?” There is no “inside” predicate. That question sneaks in an external metaphor (containers, interiors) that the abstract description never granted you.

This is a genuine constraint on cognition: we naturally reach for metaphors that feel meaningful, and those metaphors can be helpful, but they can also generate pseudo-questions that the formal structure does not support. And I think this distinction between what the system actually says, what it implies, and what our metaphors tempt us to ask is one of the most useful lenses I’ve found for thinking about mathematics.

The integers as a deceptively “thin” interface

Now we can return to the question that started all of this.

If mathematical objects are defined by their role in a web of relations, and if that web contains emergent structure that may be deeply non-obvious, then we can ask:

Is there some deep structural relation among the integers that we have simply not yet uncovered? A structure that plays little to no role in most of our everyday experience with numbers, a structure we cannot currently “see,” but that must be there – because the truth of these edge-case statements depends on it.

Think about how many of the hardest problems in mathematics are absurdly simple statements about the most elementary object imaginable: the counting numbers.

One apple, two apples, three apples. Each of my three kids can have one. If the dog steals one, I either have to cut them or only give apples to two kids. Or my husband and I can enjoy them ourselves. This is the mental world in which the natural numbers first appear: a thin, practical sequence. Knots on a rope.

Then you learn that primes are numbers divisible only by themselves and 1. Simple definition. A few steps from axioms to an algorithm. Not useful for most of human history; then suddenly the foundation of modern cryptography. And Euclid proves, 2300 years ago, that you’ll never find the biggest prime. There will always be another. Infinitely many.

But then you hit questions that look just as simple and fall off a cliff:

• Are there infinitely many twin primes? Unknown.
• Is every even number the sum of two primes? Unknown.
• Does the rule “if (x) is even, divide by 2; otherwise multiply by 3 and add 1” always reach 1? Unknown.

Before you take these questions seriously, it feels like there is almost no structure to whole numbers at all. They feel like the least interesting object: a mere successor relation repeated forever. And yet the true structure runs deep, as deep as mathematical structures go, and contains relations that feel completely disconnected from the “one apple, two apples” story.

Quadratic reciprocity. p-adics. Modular forms. The list goes on.

And this is the part that always hits me with the same weird emotion: these consequences were “there” long before we had names for them, as certainly in 100,000 BC when the first humans were learning to count as it is today in number theory textbooks. We just didn’t know it. It still feels like an utter miracle that such complexity was hiding there as an inevitable consequence of logic and formal rules.

We know for sure that even extremely sophisticated facts about integers (like Fermat’s Last Theorem) ultimately derive from whatever basic axioms you accept. They have to. There is no other source of truth available. Yet finding that path took centuries and some of the best minds in history.

It seems that so much of the complexity arises because the interaction between addition (linear structure) and multiplication (multiplicative/prime structure) is surprisingly “thick.” Most of the deep mysteries in number theory, live exactly in the friction between these two operations. And still, unless you really think about it, on the surface there appears to be nothing particularly interesting about this interaction. Multiplication distributes over addition. They feel separable, almost independent, and yet that feeling is completely misleading.

What Collatz does to my mind

This is where Collatz comes back in, not as the main topic, but as a trigger.

The thing I find psychologically striking is that the Collatz rule has a certain interface-thinness to it. It’s made of the simplest arithmetic operations imaginable (parity, division by 2, multiplication, addition). And yet whatever global behavior it has is not something I can see directly from those operations in the way I can see, say, why a stack is last-in-first-out once I have the right lens.

When I tried to think seriously about it last year, the recurring experience was this:

• I’d have an idea that felt like it captured the behavior
• it would feel obviously promising for a few minutes
• and then, as soon as I tried to formalize it, the idea would dissolve into fog

The dissolving is the interesting part. It’s like my mind keeps proposing consequences or candidate invariants that are almost_meaningful. And then I discover they’re not stable under the operations, or not expressible in the right language, or they fail on some annoying corner case that I can’t rule out. Or most commonly, just lead me to a gaping hole between the properties of numbers I think are relevant to the problem, and how the numbers acted in this system.

This feels, to me, like a lesson about cognition as much as about mathematics. And it also makes me appreciate what fundamental mathematical progress often is: not “more cleverness,” but deep and novel structural insight - discovering the right language, the right predicates, invariants, decompositions, and abstractions in which the problem becomes legible.

Final notes

I wasted a month of my life on Collatz last year, despite reading all the jokes and warnings. Never again, at least not in that mode.

But I don’t regret the detour entirely, because it pushed me to think about the gap between a formal specification and the space of meanings we think we’re entitled to attach to it.

The bazz story is a toy version of this. A few equational laws define an interface. From that interface, real structure follows by logical consequence. At the same time, many of the questions that feel most natural (“what’s inside?” “where do the pins go?”) are not deep mysteries, but category errors.

Something similar happens, I think, whenever we look at an austere set of rules like arithmetic, a dynamical map, a formal grammar, rewriting rules, celluar automata, and then try to reason about it with mental imagery that isn’t actually tied to the operations. We drift toward stories, we drift toward pictures, we drift toward “it must behave like…” And sometimes those stories guide discovery. But sometimes they generate pseudo-problems and pseudo-solutions: things that feel meaningful only because we imported them.

So for me the interesting thing about Collatz is not the conjecture itself. It’s what it highlights about mathematical meaning. If meaning is relational, constituted by inferential role inside a structure, then “understanding” is about having the right conceptual vocabulary. The vocabulary in which statements become connected to consequences you can control.

And that vocabulary is not guaranteed to sit near the surface of the original definition. Sometimes it’s ridiculously far away. The integers are the canonical example: successor and induction look like a thin interface, and yet the emergent structure is vast, deep and even today poorly understood. The fact that we needed centuries to discover much of that structure for one of the most fundamental everyday concepts is evidence that the inferential consequences of even a small axiomatic core can be profoundly non-obvious.

Widespread glitches

On October 11th, 2025, the Center for Open Science (COS) announced that the Open Science Framework — its flagship platform for preprints, data, and code — had received a major redesign. The update promised “to make managing research projects easier, faster, and more intuitive.”

Within days, many discovered that the new redesign ironically came with slower loading times, but more troubling, with a host of bugs and glitches:

PSA: seems like the update broke the ability to directly download files with a link, like osf.io/XXXX/download this lead to an error in one of my R package vignettes, and presumably will break lots of code.

[image or embed]

— Ruben C. Arslan (@ruben.the100.ci) October 12, 2025 at 10:34 AM

Thanks. I think this is urgent. I suspect it also broke Google Scholar listing PDFs from OSF, at least I get broken links for some, no links for others and generally OSF seems to be massively downranked in the list of sources. Started getting email requests for PDFs again.

— Ruben C. Arslan (@ruben.the100.ci) October 16, 2025 at 11:43 AM

@cos.io Why cant I read in rds files from my repo? I was trying to reproduce my manuscript and now I cant read in rds files from the site. I am going to assume it has to do with the new UI?

— jgeller1phd.bsky.social (@jgeller1phd.bsky.social) October 11, 2025 at 5:57 PM

Is anyone else experiencing significant issues (lags, not loading) with OSF (@cos.io) since the interface update?

— Charlotte Pennington 🌻 (@drcpennington.bsky.social) October 22, 2025 at 7:51 PM

🪲 OSF Update: Ongoing Fixes Following the OSF redesign, some issues have been identified:

• Some custom registry templates may not showall fields
• OSF shows a max. of 10 contributors
• Some pages load slowly or seem unresponsive

These are on our critical fix list & should be resolved soon. (1/2)

— Center for Open Science (@cos.io) October 17, 2025 at 7:46 PM

Also, - Login screen doesn’t work on Safari - Preregistrations impossible (got many error messages & after I was finally able to submit got an email saying trouble archiving my preregistration so it wasn’t completed)… this means I can’t actually run my study :( - Symbols not rendering (e.g., &, <)

— Cassandra Chapman (@cassandrachapman.bsky.social) October 22, 2025 at 11:38 PM

I do not mention complaints about design, as those are subjective. But it was worrisome that a redesign would break existing download link patterns without a redirect - this should not happen to an archival service. To their credit, COS fixed that particular problem quickly, though it should never needed fixing in the first place. The performance issues remain, and the irony that this performance update made performance worse is not lost to anyone. What seemed like transitional friction soon turned into something stranger.

The Disappearance

On November 2, I realized that none of my OSF preprints would open. Every single one — roughly twenty projects spanning eight years — returned either a blank page with a red banner reading “Not found” or a JSON message declaring the resource “deleted by the user.”

At first, I assumed it was a global outage consistent with the widespread performance issues mentioned above. But when I double checked, the pattern was unmistakable:

• Other authors’ preprints worked fine.
• The error appeared for every preprint and dataset linked to my name — including one uploaded by a colleague just four days earlier.
• Even Google Scholar links that had functioned for years now led to “Resource deleted” errors:

{"message_short": "Resource deleted", "message_long": "User has deleted this content. If this should not have occurred and the issue persists, please report it to <a href=\"mailto:support@osf.io\">support@osf.io<\/a>.", "code": 410, "referrer": null}

Most worrisome of all - the “blackout” included files linked in published journal articles and materials for projects currently under  peer review.

I reported the issue to OSF support with screenshots and links, expecting acknowledgment that something had gone wrong with the migration:

<On Nov 2, 2025 at 23:40 +0100, Ven Popov, wrote:>

I am experiencing a serious issue with my preprints on OSF. At first I thought this was a global OSF issue, but it seems like it is affecting all and only my preprints, which makes me very concerned.

When I try to access any of the preprints on which I am a co-author, I get an error.

If I try to access the preprint via and OSF url, I get a blank page with a red banner “Not found” (see attached screenshot). Examples:

• https://osf.io/yr9xb
• https://osf.io/gwd4s
• https://osf.io/hmu9r
• https://osf.io/dsx6y
• https://osf.io/vj8pn (this one was uploaded only 4 days ago by a colleague!)

If I try to access the PDF links on Google Scholar (e.g. this link), I get the following JSON error message:

[…]

I tested this with all ~20 preprints listed on my profile and I get the error on all of them. When I try to open any other preprint that is not authored by me, I can open it just fine. I’d appreciate any assistance with resolving this issue.

Instead, I was asked to fill out a form for “accounts flagged as spam.”

< On Mon, Nov 3, 2025 at 7:30 AM EST, Blaine Support support@osf.io wrote:>

If you received this email, it means that we need your assistance in investigating your account. Please help us by providing more information through this form: Account Disabled or Falsely Flagged as SPAM reporting. If we don’t receive your form submission, we won’t be able to investigate the issue. Your input is invaluable as we work to improve our spam detection systems.

The Support Loop

I filled out the spam-flag form, though I knew my account was active and clean. I was logged into my account and could change my account information. I could also see all my project listings, though not the details. I replied back:

< On Mon, Nov 3, 2025 at 7:47 AM EST, Ven Popov, wrote:>

I filled out the form, but my account is neither deactivated nor flagged as SPAM. I simply cannot access any of the 20+ preprints that I or my colleagues have uploaded over the last 8 years! In the form I supplied a link to one preprint, but all of my preprints are inaccessible.

A few hours later I received a message from a new support representative, asking me again to fill out the same form:

< On Nov 3, 2025 at 16:04 +0100, Michaela Support support@osf.io, wrote:>

Please fill out the form: Account Disabled or Falsely Flagged as SPAM reporting. Your preprint have been flagged as spam. This is an important step in tracking how effective (or not) our spam filters are, and to review and resolve the issue.

Frustrated, I copied Brian Nosek on a detailed escalation email explaining that this was not a single-ticket problem but a catastrophic integrity failure: twenty preprints, hundreds of files, dozens of co-authors, and years of published work were all inaccessible. It was not one “preprint flagged as spam.” Anything I — or anyone associated with me — had uploaded since 2017 was completely inaccessible, and OSF was effectively communicating to the world that this work no longer existed.

<On Nov 3, 2025 at 16:45 +0100, Ven Popov, wrote:>

As I replied in my previous email, I already filled out the form.

Furthermore, it is not one preprint that is inaccessible. All of the 20+ preprints that I have published on OSF since 2017 are unavailable. Neither the OSF links from my profile, nor the Google Scholar links that have worked for years can be opened.

Please escalate this issue. This is a serious misstep on OSF’s part. I have had an account with OSF for 8 years, I have published about 20 preprints during this time, and none of them are available. For some of them, the OSF links might be the only record online of these published works, and it is a serious issue concerning the longevity of scholarship that an archival platform like OSF cannot allow to happen.

@Brian, can you please get your support team to take this seriously? I’m getting repeated emails asking me to do something I already did and overlooking the extent of the problem. Please see the message history for the detailed report and links to my profile, a subset of broken preprint links, and an API error from Google Scholar links.

Within hours, OSF staff identified the cause: my account had been automatically flagged as spam by their filters. Once the flag was removed, everything reappeared.

The Explanation

Here’s the message I received:

< On Nov 3, 2025 at 19:31 +0100, Michaela Support support@osf.io, wrote>

Thank you for reaching out to the OSF support team! You are correct it looks like your project was caught in our spam filter. As a free open access platform we are frequently the target of spammers, and therefore we must use multiple different spam filters that take a plethora of different factors into account when flagging spam, everything from odd titles to suspicious activity from a nearby IP address could get flagged. Truthfully it’s hard to tell why your content got its attention. I’m sorry for the inconvenience. I’ve removed the flag from material and your content should be active again!

It was an honest reply — but also a disturbing one.

OSF’s filters had quarantined everything I or any of my colleagues had ever uploaded, without warning, without a visible flag on my dashboard, and without preserving any metadata or landing pages. Public DOIs and Google Scholar entries now pointed to HTTP 410 “Resource deleted” responses — a code that explicitly signals permanent removal to search engines.

In other words, for several days, OSF’s infrastructure told the world that our work no longer existed.

The Deeper Problem

This incident isn’t about data loss — most of my materials were mirrored elsewhere. The issue is trust.

Researchers use OSF because it promises persistence. It issues DOIs, integrates with journals, and serves as part of the scholarly record. That promise was violated — not through malice, but through a structural design choice: an automated filter was allowed to silently deindex archival content.

Spam detection is necessary. But a true archive must never delete first and explain later. An archive is defined by its immutability and accountability.

The design flaw wasn’t that a spam flag existed — it’s that its activation could unpublish a decade’s worth of DOI-assigned material without human review, notification, or even the retention of metadata.

This episode highlights a broader fragility in the open science ecosystem. The infrastructure of modern scholarship — repositories, indexing services, DOI registries — runs on trust. When that infrastructure silently fails, the damage is epistemic: we lose not just access to data, but confidence in the permanence of the scientific record.

An archival platform must therefore:

• Retain metadata and landing pages even when content is quarantined.
• Notify users when materials are flagged or hidden
• Require human verification before suppressing DOI-linked material.
• Use non-destructive HTTP codes for temporary removals.

These are not minor UX fixes — they are the foundations of reliability in digital scholarship. I made this much clear in an email reply:

<On Mon, Nov 3, 2025 at 2:26 PM Ven Popov wrote:>

Thank you for resolving the issue and restoring access to my materials. I understand that OSF, as a free and open platform, must protect itself against spam.

However, the way your spam-filtering system currently operates is unacceptable and incompatible with OSF’s stated mission as an archival repository. Automatically blocking access to more than twenty DOI-registered preprints, datasets, and supplementary materials spanning nearly a decade — including work co-authored by dozens of researchers and cited in published papers — is not a minor glitch. It constitutes a violation of the very FAIR principles OSF promotes and undermines trust in your platform as part of the scholarly record.

Worse, the system suppressed all metadata and landing pages, returning an HTTP 410 “Resource deleted” response. That code explicitly signals to search engines that a resource has been permanently removed and should be de-indexed. This means the issue not only disrupted access but actively damaged the discoverability and citation continuity of legitimate research outputs.

I fully support the need for robust spam protection. But the current implementation — silent removal of legitimate archival content, without notice or human verification, and with a response that signals permanent deletion — is deeply inconsistent with COS’s commitments to openness, transparency, and persistence.

I urge COS to review this policy and implement safeguards such as:

• Human review before suppressing any DOI-assigned or public scholarly record.
• Retention of metadata and landing pages for any quarantined content.
• Clear notifications to affected users.
• Use of non-destructive HTTP status codes (e.g., 451 or 503) for temporary suppression.

These are not just usability issues; they concern the credibility of OSF as a reliable component of the scientific infrastructure.

I am deeply troubled by this incident, and I do not say this lightly. I have been a staunch supporter of COS and its mission for years. But this event exposes an infrastructural weakness that seriously undermines trust in the OSF platform.

I would appreciate a response from COS leadership that demonstrates an understanding of the gravity of this failure and provides a clear plan to prevent similar occurrences in the future.

I intend to publish this correspondence publicly, and I hope that I can do so alongside a response that acknowledges these concerns and reaffirms COS’s commitment to openness, transparency, and durability in scientific communication.

Aftermath

After I sent my detailed critique, Brian Nosek responded personally and with grace. He acknowledged that this case was unprecedented, thanked me for the thorough documentation, and assured me that it would help the team identify whether others had been similarly affected.

< On Nov 3, 2025 at 22:05 +0100, Brian Nosek, wrote:>

I see from the thread that your issue is resolved, but I am sorry that you had to go through that!

I really appreciate the detailed notes that you provided to help identify the source of the problem, and even more the solutioning that you offered to improve user experience and more gracefully manage the identification and mitigation of spam. I have not heard of a case like this, so while it was a painful experience that you had to deal with, hopefully the exposure of the problem with your diligent reporting will help us to identify the extent to which it is occurring more widely and develop effective solutions.

Thanks again for giving us such useful reporting on your experience with this to help us continue to improve the service.

Despite this response, I was still deeply troubled. I briefly considered whether I’m overreacting. After consulting several colleagues, they were unanimous: the severity is not that it happened to me, but that it could happen at all — silently and retroactively. As one colleague put it:

I agree, I think the issue is more severe than it sounds like. The effect is crazy to me. Not only that they retrospectively delete the stuff but also how many people can be affected by it. It’s already severe in your case with a large network of collaborators. But imagine Klaus’s account gets flagged and he doesn’t realize it. This would probably take hundreds of preprints offline, which are primary the work of others [junior researchers]. It’s bad enough that they take the stuff down that you put on OSF but it also deletes the work where someone else just linked your name with? This also can’t be right…

After some reflection, I followed up with Brian Nosek with a longer message explaining why the issue mattered so deeply: not because of personal inconvenience, but because silent retroactive suppression of long-standing records is incompatible with the role of an archive.

< On Nov 4, 2025 at 13:49 +0100, Ven Popov wrote:>

Thank you for the thoughtful response. I appreciate that you will take this case seriously in developing future solutions.

I just really want to stress why I am so concerned about this having happened. It’s not about my work and that it affected me personally, although that clearly made me more motivated to get to the bottom of it. My work is also backed up on other platforms, including ResearchGate, a University of Zurich repository and my personal website. The real problem is that something like this should not be possible to happen in the first place, let alone as the result of an automated filter.

Whatever the reason my account was flagged, an automated filter should never have the authority to retroactively block access to the work of hundreds of researchers simply because I’m listed as a co-author. These were not new uploads but materials that had been part of the scholarly record for nearly a decade.

I was lucky to notice the disappearance within days; others might not. Under different circumstances, months could have passed during which all these digital objects appeared effectively erased. The consequences for more prominent researchers with hundreds of linked projects - and for junior collaborators relying on those links - could be far more severe.

Your own guidelines stress that even authors themselves cannot delete accepted preprints - that preprints can only be withdrawn, and that the associated metadata and reasons for withdrawn will always remain part of the record.

I considered whether I might be overreacting, but after discussing the issue with several colleagues, everyone agrees that this is a critical infrastructure-level bug. This requires more than a patch for this particular case, but systems in place to ensure that nothing can in principle cause something like it to happen in the future.

I am writing all this out in detail not to attack COS or its work on maintaining OSF. It’s exactly the opposite - I share all COS values and view it as a fundamentally important institution. The traditional academic publishing system is rotten to core, but in order for any reform to succeed long-term, people must trust the infrastructure and systems in place. I want to see COS and OSF continue to succeed and continue to play the vital role that it has attained. This is why I am being so adamant about this incident being taken so seriously.

His reply was thoughtful and for the moment I am satisfied that this incident will be taken seriously:

<On Nov 4, 2025 at 14:45 +0100, Brian Nosek, wrote:>

Thanks for the follow-up. There is nothing inappropriate at all about your comments or approach. It is very clear that your comments are made in good faith to help us improve the services and meet the aspirations of reliable, persistent open scholarship. We truly appreciate your willingness to put time into documenting and sharing the challenges you experienced here.

That, to me, felt like the right place to stop. Ideally, I’d still like an official statement from COS about what they plan to do to ensure that something like this doesn’t happen again.

Data vs. Trust

No data were ultimately lost. But something far more important was at stake: the continuity of the scholarly record.

If this episode serves any purpose, let it be a reminder that openness is not just about making information accessible — it’s about ensuring it stays accessible. Scientific infrastructure must be engineered for trustworthiness, not just availability.

As we argued in a recent perspective paper, the traditional publishing system is rotten to the core. But for any reform to succeed long-term, people must trust the infrastructure and systems in place.

I also don’t know if this incident is related to the OSF redesign - I described it here within that context - or whether it would have occurred with the previous platform. Ultimately it doesn’t matter.

Trust is slow to earn and quick to erode. I hope OSF uses this experience to reinforce that trust — not only by fixing the bug, but by institutionalizing the safeguards that make open scholarship durable.

A case of bait and switch

Here’s a simple case which recently led to a head-scratcher of a bug in our bmm package. Consider a simple a function which gives a different greeting depending on a language parameter:

greeting <- function(name, language) {
  if (language == "chinese") {
    out <- paste0("嗨 ", name, "! 多么美好的一天")
  } else if (language == "german") {
    out <- paste0("Hallo ", name, "! Was für ein toller Tag!")
  } else if (language == "bulgarian") {
    out <- paste0("Здравей ", name, "! Какъв страхотен ден!")
  }
  cat(out, "\n")
}

Simple enough, right?

greeting("Ven", "german")
#> Hallo Ven! Was für ein toller Tag!

Currently, you will get weird errors with non-character input if you don’t properly check that the language argument is a character or that it is a valid choice. But at least the user will get an error:

greeting("Ven", 2)
#> Error in greeting("Ven", 2): object 'out' not found
greeting("Ven", "japanese")
#> Error in greeting("Ven", "japanese"): object 'out' not found

Ok, maybe a little confusing for the user who doesn’t know how the function is implemented (“object ‘out’ not found? What is object ‘out’?”), but fine. You can and should go the extra mile and validate the input, but let’s keep going. Now, if you had many different language options, you might decide to use a more efficient switch statement instead of an if/else structure:

greeting2 <- function(name, language) {
  out <- switch(language,
    chinese = paste0("嗨 ", name, "! 多么美好的一天"),
    german = paste0("Hallo ", name, "! Was für ein toller Tag!"),
    bulgarian = paste0("Здравей ", name, "! Какъв страхотен ден!")
  )
  cat(out, "\n")
}

Our switch handles 3 explicitly defined cases, just like the if/else version, so unless you read the documentation, or have been already burnt by this, you might expect that this should fail:

greeting2("Ven", 2)
#> Hallo Ven! Was für ein toller Tag!

Oh, it runs! Ok, so even though we have explicitly defined the cases by name, switch tries to be clever and interpret integer input as a case selection. Not only that, but it will try really hard to make the input an integer if it can. The following makes no sense as a language selector, but R happily forces 2.8 into an integer sleeve and greets us again:

greeting2("Ven", 2.8)
#> Hallo Ven! Was für ein toller Tag!

Fine, it’s strange, but who in their right mind will try to pass numeric values for a language variable? Maybe no one on purpose, but some things in R are (sometimes) secretly integers. Namely, factors. Consider this - we have some data.frame with names and language preferences of users, the latter of which is coded as a factor variable.

users <- data.frame(
  name = c("Ven", "Gidon", "Chenyu"),
  language = factor(c("bulgarian", "german", "chinese"))
)

users
#>     name  language
#> 1    Ven bulgarian
#> 2  Gidon    german
#> 3 Chenyu   chinese

If we were to use our if/else-based greeting function to greet each user, everything works as we would expect:

purrr::pwalk(users, greeting)
#> Здравей Ven! Какъв страхотен ден! 
#> Hallo Gidon! Was für ein toller Tag! 
#> 嗨 Chenyu! 多么美好的一天

What about the greeting2, which uses switch to determine which greeting to use?

purrr::pwalk(users, greeting2)
#> 嗨 Ven! 多么美好的一天
#> Здравей Gidon! Какъв страхотен ден!
#> Hallo Chenyu! Was für ein toller Tag!

What the hell? Well, I cheated a bit and hid the warning R helpfully gave us (naughty!). Here is the output again without suppressing the warning:

purrr::pwalk(users, greeting2)
#> Warning in switch(language, chinese = paste0("嗨 ", name, "! 多么美好的一天"), : EXPR is a "factor", treated as integer.
#>  Consider using 'switch(as.character( * ), ...)' instead.
#> 嗨 Ven! 多么美好的一天
#> Warning in switch(language, chinese = paste0("嗨 ", name, "! 多么美好的一天"), : EXPR is a "factor", treated as integer.
#>  Consider using 'switch(as.character( * ), ...)' instead.
#> Здравей Gidon! Какъв страхотен ден!
#> Warning in switch(language, chinese = paste0("嗨 ", name, "! 多么美好的一天"), : EXPR is a "factor", treated as integer.
#>  Consider using 'switch(as.character( * ), ...)' instead.
#> Hallo Chenyu! Was für ein toller Tag!

Ah, that explains it (although doesn’t excuse it). When a factor variable is passed to a switch statement, the variable is treated as an integer. By default factor levels are ordered alphabetically, which we can see if we examine our language factor structure:

users$language
#> [1] bulgarian german    chinese  
#> Levels: bulgarian chinese german
str(users$language)
#>  Factor w/ 3 levels "bulgarian","chinese",..: 1 3 2

So Gidon gets greeted in Bulgarian, because his “german” language is coded as 3 in the factor variable, and the third check in switch corresponds to “bulgarian”. The documentation (?switch) helpfully explains that

switch works in two distinct ways depending whether the first argument evaluates to a character string or a number.

If the value of EXPR is not a character string it is coerced to integer. Note that this also happens for factors, with a warning, as typically the character level is meant. If the integer is between 1 and nargs()-1 then the corresponding element of ... is evaluated and the result returned: thus if the first argument is 3 then the fourth argument is evaluated and returned.

If EXPR evaluates to a character string then that string is matched (exactly) to the names of the elements in ...…

Wow, ok, I never knew this, or if I did I have completely forgotten about it.

So what, we do get a warning don’t we?

We sure do, but warnings can be suppressed, just like I did above. It’s common to suppress output of functions when running many iterations of a chatty function in an analysis script and problems can easily go unnoticed. That’s exactly what happened recently in our Bayesian measurement modeling R package when a colleage reported a weird bug. We have one computational model that can apply different forms of Luce’s choice decision rule - a standard version and a version passed through a softmax normalization. Due to this weird way that switch treats factors, the wrong normalization was applied. This is a recent and not yet officially released model, so we are yet to write all input validations. This could have easily gone unnoticed and led to incorrect model specification that nevertheless lets the model run.

Warnings are not a reliable way to signal undesired behavior. Especially when the documentation of switch’s warning itself notes that “typically the character level is meant”. Well, if typically a character level is meant, why is the default EXACTLY THE OPPOSITE?!?

Fine, but you tell users “language” should be a character… right?

We do, but here’s the kicker - R loves to turn character vectors into factors. So much so that disabling such behavior was one of the main motivation behind the development of tibbles. Until R4.0.0, whenever you used a function like read.csv() to read a file as a data.frame, R by default converted character columns to factors. Thankfully now this default has been reversed, but here’s the deal - NOT EVERYWHERE!

One place which I never knew R created factors out of character vectors is expand.grid. Expand.grid is a commonly used function to get a data.frame with all combinations of several variables, which is useful for running models with orthogonality manipulated conditions. E.g.:

conditions <- expand.grid(
  value = c(1, 100),
  version = c("cs","ss"), 
  choice_rule = c("simple", "softmax")
)
conditions
#>   value version choice_rule
#> 1     1      cs      simple
#> 2   100      cs      simple
#> 3     1      ss      simple
#> 4   100      ss      simple
#> 5     1      cs     softmax
#> 6   100      cs     softmax
#> 7     1      ss     softmax
#> 8   100      ss     softmax

Can you tell that version and choice_rule are factors? I’ve used expand.grid for years without knowing that default behavior of expand.grid, and it’s partly that standard data.frame print method does not differentiate character and factor columns in any way. You can see that indeed we have factors underneath by using str for more details:

str(conditions, give.attr = FALSE)
#> 'data.frame':    8 obs. of  3 variables:
#>  $ value      : num  1 100 1 100 1 100 1 100
#>  $ version    : Factor w/ 2 levels "cs","ss": 1 1 2 2 1 1 2 2
#>  $ choice_rule: Factor w/ 2 levels "simple","softmax": 1 1 1 1 2 2 2 2

You can examine the documentation, or directly check with formals that expand.grid, just like read.csv has a argument stringsAsFactors, which defaults to TRUE:

formals(expand.grid)
#> $...
#> 
#> 
#> $KEEP.OUT.ATTRS
#> [1] TRUE
#> 
#> $stringsAsFactors
#> [1] TRUE

Wait, but didn’t I just write that R4.0.0 solves the problem? As of R4.4.2, it only does that for read.table and data.frame, but not expand.grid

version$version.string
#> [1] "R version 4.4.2 (2024-10-31)"
formals(read.table)["stringsAsFactors"]
#> $stringsAsFactors
#> [1] FALSE
formals(data.frame)["stringsAsFactors"]
#> $stringsAsFactors
#> [1] FALSE
formals(expand.grid)["stringsAsFactors"]
#> $stringsAsFactors
#> [1] TRUE

You can of course change that by being explicit about not wanting factors, or use tidyr::expand_grid alternative instead:

expand.grid(
  value = c(1, 100),
  version = c("cs","ss"), 
  choice_rule = c("simple", "softmax"),
  stringsAsFactors = FALSE
) |> str(give.attr = FALSE)
#> 'data.frame':    8 obs. of  3 variables:
#>  $ value      : num  1 100 1 100 1 100 1 100
#>  $ version    : chr  "cs" "cs" "ss" "ss" ...
#>  $ choice_rule: chr  "simple" "simple" "simple" "simple" ...

tidyr::expand_grid(
  value = c(1, 100),
  version = c("cs","ss"), 
  choice_rule = c("simple", "softmax")
) |> str()
#> tibble [8 × 3] (S3: tbl_df/tbl/data.frame)
#>  $ value      : num [1:8] 1 1 1 1 100 100 100 100
#>  $ version    : chr [1:8] "cs" "cs" "ss" "ss" ...
#>  $ choice_rule: chr [1:8] "simple" "softmax" "simple" "softmax" ...

Consistency is important

Consistency is a loaded term. Whether two things are consistent or not necessarily depends on context and goals. One could go on and on about consistent naming styles, consistent argument order, consistent default behaviors, etc. At its core the problem with inconsistent design choices in a programming language is that it makes it very difficult, if not impossible, to build an accurate mental model of how the language works. It’s more than a bit ironic when programming languages, which are supposed to be the precise counterpart of natural languages, become a similar tangled mess of exceptions you need to learn by heart, just like irregular verbs in English.

The main example of this post concerns the concept of control flow in programming. Both if/else statements and case/switch statements aim to accomplish the same goal - to execute different parts of a program depending on a prespecified condition. One would expect that logical comparisons would work the same between those two constructs, but as this example has illustrated, they do not in R. Sometimes complex examples obscure the core problems, so let’s present it at it’s simplest form:

x <- factor("Peace", levels = c("Violence", "Peace"))

x == "Peace"
#> [1] TRUE
x == 2
#> [1] FALSE
x == "2"
#> [1] FALSE

switch(x, Peace = "I choose peace", Violence = "I choose violence")
#> Warning in switch(x, Peace = "I choose peace", Violence = "I choose violence"): EXPR is a "factor", treated as integer.
#>  Consider using 'switch(as.character( * ), ...)' instead.
#> [1] "I choose violence"

Logical comparisons treat factors as character vectors, switch comparisons treat them as integer values. This is not ok. Sure, you can learn that this is the case, but you shouldn’t have to.

These inconsistencies in base R have led to various attempts at reform, most notably through the tidyverse ecosystem. While the tidyverse strongly emphasizes consistency, it presents its own challenges. Breaking changes are frequent, forcing regular code updates and relearning of functionality. While useful for analysis code, the packages’ interdependence and heavy footprint make them difficult to justify in lightweight, stable package development. Many tidyverse functions on the surface appear to simply wrap basic R operations, unnecessarily expanding the language’s complexity. The tidyverse’s impact on the R community remains contentious, sparking numerous debates about its influence (see discussions here, here, here, here, here). My own perspective has evolved from enthusiasm to skepticism, though I still use tidyverse tools when appropriate. I don’t want to throw away the baby along with the bathwater though - despite its drawbacks, the tidyverse gets one thing right: function APIs should be consistent and avoid “magic” behavior. R along with its predecessor S, is an old language, and a lot can be forgiven due to its legacy.

This situation mirrors broader patterns in programming language evolution. Consider C++: despite being a modern, widely-used language, it carries significant legacy baggage due to its strict backwards compatibility requirements. Newer languages like Rust, free from such constraints, can implement better defaults and more consistent behavior from the start. The tidyverse represents a curious case of attempting to fix a language’s shortcomings ‘from within’ rather than through dialect separation or replacement. While this approach maintains ecosystem cohesion, it creates a complex divide within the R community.

Python offers an instructive contrast in language evolution. Through semantic versioning, Python made major breaking changes between versions 2 and 3, prioritizing language improvement over backwards compatibility. While the transition wasn’t painless—Python 2 still claimed 25% usage a decade after Python 3’s 2008 release—by 2023, Python 2 usage had dropped to just 6% (and the transition plans were announced well in advance - PEP 3000 was published in 2006). R, conversely, maintains strong API stability between major versions. Many of the tidyverse’s consistency-oriented features could serve as excellent candidates for a base R rewrite if the community embraced the possibility of meaningful breaking changes in major versions. Yes, such transitions can be challenging—Python’s experience demonstrates this—but they also show that systematic language improvement is achievable with proper planning and community support.”

What now?

First, of course, is for me to sit down and do the annoying grunt work of going through all user-facing functions and ensuring that we test and validate every input. There’s a ton of ways to do it, either with vanilla R and custom functions (stopifnot and match.arg are your friends), or with the help of some validation packages I listed earlier. Then add more unit tests about edge cases to make sure things like this don’t happen. And so on… We have done this for a lot of our existing code, but as this example taught me, it’s easy to forget - and sometimes easy to not know.

Long-term, however, I’m becoming more and more interested in programming languages that use a strong static type system. I’ve long taught that static typing is simply annoying - as a self-taught programmer, I haven’t had the benefit of learning things “the right way”. Over the last few months I’ve started digging deeper into programming as a core skill, exploring various languages and resources. I’m growing more and more attuned to the virtues of good type systems, for many other reasons. I’m sure another post on this topic is incoming at some time. In the meantime:

if (language == "R") {
  stopifnot(inputs_are_carefully_validated())
}

A bash script for automating post creation

The problem was that aside from some basic commands, my bash knowledge is quite limited. So I figured this was an opportunity to put Github Copilot to the test. I use Visual Studio Code as my IDE, and have setup my website repository as a workspace. I already had a markdown document that describes in plain language the steps I need to do when creating new blog posts². So could the built-in Copilot in VSCode write me a bash script just by being told to implement the steps I’ve described in my little instruction manual?

It worked flawlessly, and this code was the result after a few iterations:

#!/bin/bash

# Prompt for post details
read -p "Enter post title: " title
read -p "Enter post subtitle: " subtitle
read -p "Enter post categories (comma-separated): " categories

# Convert title to a slug for the folder name
slug=$(echo "$title" | tr '[:upper:]' '[:lower:]' | tr -cd '[:alnum:] ' | tr ' ' '-')

# Create the new post directory
year=$(date +%Y)
post_dir="$HOME/venpopov.com/posts/$year/$slug"

# Check if the directory already exists
if [ -d "$post_dir" ]; then
  echo "Error: The directory $post_dir already exists."
  exit 1
fi

mkdir -p "$post_dir"

# Get today's date
date=$(date +%Y-%m-%d)

# Append the current year to the categories list
categories="$categories, $year"

# Create the .qmd file with the provided metadata
cat <<EOF > "$post_dir/index.qmd"
---
title: "$title"
subtitle: "$subtitle"
categories: [$(echo "$categories" | sed 's/,/, /g')]
date: "$date"
---
EOF

echo "New post created at $post_dir/index.qmd"

# Open the new post in the default editor
code "$post_dir/index.qmd"

I saved this code in a file new_blog_post and Copilot instructed me that I can make it executable from the command line by first setting file permission via:

chmod +x new_blog_post

I saved this file in a directory on my path, so now when I want to create a new blog post, I simply open a terminal and type

new_blog_post

This prompts me to enter a title for the post, a subtitle and some tags, then creates all the necessary boilerplate and opens the file in VSCode for editing. In fact, I used it to make this very post, and for example, here is the terminal output from it

Neat. Will this help me post more often? Time will tell.

Postscript: AI and learning to code

I tell my students that they can use AI tools when working on projects, but I urge them that it is still important to learn how to code for many reasons. They need to understand what the code is doing, at the very least, and know how to fix it. Did I follow my own advice here? Ugh, maybe not, depending on the perspective. I still have no idea about the core bash syntax and how to write a script from scratch. But I do have a “lifetime” of experience with programming, and I could understand what the script was doing, and I could modify it to suit my needs. I think that’s the important part. I don’t need to know everything about bash scripting, but I need to know enough to be able to use it effectively. In any case, I have a tendency to fall into rabbit holes when learning new things, and this is the last thing I want to do right now while trying to optimize my workflow! So, I have a working script, I asked Copilot to add some reasonable checks that occurred to me, and I know what each line does, even if I don’t know how to write it from scratch. I think that’s good enough for now.

The Comfort of Old Habits

For many years I had relatively stable workflows for project organization and version control of my research projects. They were never perfect, and given the state of reproducibility in academic psychology, likely much better than the status quo. Since 2014, I’ve been using systematic project template folders, project-based organization, programmatic data wrangling, version control with Git and GitHub, and I’ve dabbled in literate programming tools such as Jupyter Notebooks and R Markdown. I’ve taugth these tools in graduate courses. I knew my way around a command line, and on the rare occasions I needed it, I could run demanding analyses on the Carnegie Mellon computing cluster. I was able to get things done efficiently while maintaining a somewhat decent level of reproducibility.

My comfortable routine started changing last year, gradually at first, as I began working on my first R package. What began as a side project quickly evolved into something that preoccupied me for months. This was partly because I had a fantastic collaborator, working with whom was an exhilarating, idea-bootstrapping experience. But collaborating also meant I had to up my game — I couldn’t rely on the shortcuts and hacks I’d developed over the years. For the first time, I had to learn about proper documentation, testing, and collaborative workflows.

When you’re working on a package, you’re not just writing code for yourself—you’re writing code for others to use. This means writing clear and concise documentation, creating tests to ensure your code works as expected, and developing code that is modular and easy to understand. And you have to do it in a way that makes collaboration less painful.

The Pitfalls of Self-Taught Coding

The painful truth is that proper computational skills are rarely taught in academic programs (at least in psychology). Many of us are self-taught, each with our own quirky ways of doing things. Tools and processes that are standard fare in software development are often foreign to us. So, we muddle through, doing the best we can with what we have. My use of GitHub was a glorified Dropbox, and my coding practices, if you could call them that, were a mishmash of concepts I picked up from various tutorials and blog posts. It mostly worked, but even now I am barely able to reproduce my own work from a few years ago. Broken package dependencies, uncertain code order, and the utter lack of systematic documentation have made my old projects a nightmare to revisit.

Maybe I’m exaggerating a bit, but the reality is that as my career has progressed, my projects have grown in number and complexity, and it’s become more and more frustrating to keep track of everything. My existing workflow sat squarely in the middle of “the reproducibility iceberg” - better than most, but I was starting to feel cold.

Thankfully, while formal education in this area still lags behind, the online landscape is now rich with resources. Over the last year, I’ve been trying to incorporate better coding practices, Quarto websites reporting for projects, and renv for package management. It’s been an uphill battle, consuming a lot of time and energy.

The Tension Between Efficiency and Reproducibility

The hardest part is not learning new tools but unlearning old habits and deconstructing my mental models of code, data, and reporting. One of the reasons I love coding in R is the incredibly quick iteration cycle and feedback loop. The ability to have an idea and, within minutes, simulate, visualize, and analyze it often feels like a superpower.

The problem is that this same superpower also makes it so easy to be sloppy. Who has time and patience for carefully curating a reproducible workflow when that puts a delay between your idea and its realization?

The thing is, I am no longer a grad student chasing down any odd idea that comes my way. As fun as the wild west of coding can be, it’s not sustainable for a long-term research program, especially when other people depend on you. And let’s be honest, I also need to be kinder to my future self.

What a Mess

Case in point: I just resumed work on a simulation project that I last touched in April. Despite all my best intentions, I was shocked to find that I couldn’t reproduce a set of figures I had sent to my collaborator at the time. Even worse, it took me a full day to even figure out what I was doing back then.

Why did this happen? There are many culprits, but part of it is that while I coded all the simulation scripts locally, I ran the simulations on a cluster because they were very computationally demanding. At the time, I had no established workflow for sharing intermediate data objects between my remote and local codebases. My attempts to reconstruct what had happened have driven me to the brink of madness. Trust me when I say that I’ve spent more time trying to figure out what I did than it would have taken me to redo everything from scratch.

To avoid scenarios exactly like this, in the last six months I’ve been experimenting with the targets package. Targets is a pipeline toolkit for R that helps you manage the dependencies between your scripts and data objects. It’s a bit like make for R, but with a lot of bells and whistles. I even implemented it for a couple of other projects.

When combined with renv for package management, and Quarto for reporting, it comes close to what I imagine as a nearly ideal scenario: A self-contained research website, with all the code, data, and results in one place, and a clear, reproducible workflow that can be run on any machine. A modern reporting pipeline that is both transparent and efficient.

The Complexity Conundrum

When it works. Sigh.

Maybe I’m just getting old, but putting together all these pieces has proven to be a lot more challenging than I anticipated. The learning curve is steep. These are all fantastic tools, with good documentation, and in some respects I feel lucky to be working in a time when such tools are becoming widely available, and there are so many people voluntarily developing them. The open source community is truly a marvel. But the sheer number of moving parts, and the complexity of the interactions between them, is overwhelming.

Git + Github + renv + Quarto + targets + crew + deployment + credential management + testing + documentation + collaboration + teaching + writing + thinking + living. It’s a lot.

Ok, I got carried away with the list, but even when just considering the computational parts, each of them comes with lengthy manuals, quirks, and a dizzying amount of options and configurations, and they don’t always play nice with each other. I feel like I’m constantly fighting fires, and my self-help tutorials are getting longer and longer. These past 6 months, I’ve spent more time debugging my workflow than actually doing research.

I’m not giving up, though. This is a path worth walking. I just can’t help but feel there must be a better way. I’ve been trying to code up a personal library of tools to automate some of that complexity, but other things keep getting in the way. I just can’t help but compare this state of affairs to the relatively much smoother package development workflows, largely thanks to devtools and usethis packages, as well as the consistent framework around it. Package development is not any less complex in scope, but the community has managed to converge on an integrated consistent workflow that just works.

Part of this is the nature of the beast - research projects are by definition more diverse and less predictable than package development. Every project is a new adventure, with its own unique challenges and requirements, and formats vary so widely between disciplines or even subfields. Part of it is also that these tools are still relatively new.

The Signal-to-Noise Ratio

But even if all the quirks were ironed out, and the process streamlined, another issue, that I rarely see discussed, is just how much more “irrelevant” artefacts are produced in the codebase. When I look at my best attempts to implement the full workflow, I am a bit paralized by the sheer amount of files and code in my repositories that is not directly related to the substance of the research. There are layers upon layers of abstraction, scaffolding, configuration files, and helper functions that are just necessary for the workflow to function.

Targets itself, for all its truly wonderful functionality, suffers (or at least I do!) from 1) an incredible level of syntax verbosity and 2) too many ways to do the same thing, both of which make it hard to read and understand the code. Flexibility is a double-edged sword.

Even in an ideal world in which I finally learn all the ins and outs of these packages, I can’t help but wonder: does the improved reproducibility and transparency pay off if barely anyone else can understand what’s going on? If I share the code with expert colleagues, I doubt many of them would be able to make sense of it given the signal-to-noise ratio in the codebase. It might as well be written in a different programming language given the layers of abstraction. This perhaps is a temporary problem, as the tools mature and the community converges on best practices, but it’s a real one.

This is not a critique of the tools themselves, but rather a reflection on the complexity of the research process. The tools are a reflection of the complexity of the problem they are trying to solve. But there has to be a better way.

PS: This post took an unexpected direction. I was planning to write a short introduction to my frustrations, and a detailed guide to targets as I figured out a how to apply it to a new project. I learn best by writing and teaching, and I was hoping that by writing a tutorial I would solidify my understanding of the package. As often happens in writing though, our thoughts, especially those bottled up frustrations, tend to take a life of their own. I guess I might still write the more technical post I was imagining later

What is bmm for?

bmm provides a simple and intuitive interface for fitting measurement models in psychology using Bayesian methods. It is designed to be accessible to researchers with little to no experience with Bayesian statistics or cognitive modeling. If you know how to fit a mixed-effects regression model in R, you already know nearly everything you need to fit a measurement model with bmm.

But first things first: what are measurement models and why do we need them?

If you are a meteorologist, you can measure atmospheric temperature with a thermometer. And while thermometers are technically indirect measures, the relationship between the volume of a liquid and its temperature is so strong and so well understood that we take it for granted.

If you are a psychologist, things get a bit more complicated.

Behavioral and psychological data are messy and only rough proxies for what they are supposed to measure. We rarely care about how much time it takes someone to press a button after a flashing light, how precisely they can remember a specific shade of blue, or how similar they judge two badly drawn alien animals to be. Such data are only interesting insofar as they tell us something about the underlying psychological processes that govern attention, perception, decision making, memory and categorization.

To make matters even more difficult, usually multiple distinct cognitive processes contribute to behavior. Combined, these issues often make it difficult to draw clear conclusions from behavioral data alone. Running t.tests on averaged raw behavioral data can only get you so far. Measurement models are an important tool to bridge the gap between latent psychological constructs and observable behavioral data.

Such measurement models are nowadays used in many different areas of psychology. Some of the more popular models include drift diffusion models in decision making, signal detection theory models in perception and memory, and mixture models in visual working memory. The basic idea uniting these approaches is that we can decompose one or more observed measures (e.g., reaction times, accuracy rates, angular deviation, confidence ratings) into distinct theoretically meaningful parameters that reflect latent psychological processes (e.g., decision thresholds, different memory strength signals, the quality or precision of representations). These derived parameters can then be used to test hypotheses about the underlying cognitive processes that govern behavior.

The challenges of measurement modeling

Traditionally, to fit a measurement model in psychology, you had to build it yourself. The simple reality is, however, that most researchers don’t have the time, resources, or expertise to build and fit these models from scratch. This is doubly true for Bayesian hierarchical implementations, which require a different set of tools and skills than traditional maximum likelihood estimation. And even for those who do have the skills, the process can be time-consuming and error-prone.

This is why we started working on the bmm package in the first place - we were tired of doing the same thing over and over again for every new project. What started as a personal project to make our own daily work easier has now turned into a fully-fledged R package. What we have built is a package that allows you to fit a wide range of measurement models with just a few lines of code.

You no longer need to:

• copy-paste custom Jags or Stan code from one project to the next (or worse, try to decipher someone else’s code!) and painstakingly adjust it for your new experiment
• worry about whether you have correctly specified your priors or likelihoods
• worry about whether you have correctly implemented your model
• worry about how to adjust the script for your new experimental design and to spend hours debugging it

The bmm package takes care of all of that for you.

While there exist tools for some measurement models, such as for drift diffusion models (via the wiener distribution in brms for R or via the HDDM package for Python), this is not the case for the vast majority of measurement models used in psychology. The bmm package aims to fill this gap by providing a general framework that can be extended and continuously improved by the community.

Design principles

The bmm package is built on a few key design principles, some of which were strongly inspired by the brms package for Bayesian regression modeling:

Currently supported models

We currently have the following models implemented:

Visual working memory

• Interference measurement model by Oberauer and Lin (2017).
• Two-parameter mixture model by Zhang and Luck (2008).
• Three-parameter mixture model by Bays et al (2009).
• Signal Discrimination Model (SDM) by Oberauer (2023)

However, the setup of the bmm package provides the foundation for the implementation of a broad range of cognitive measurement models. In fact, we are already working on implementing additional models, such as:

• Signal-Detection Models
• Evidence Accumulation Models
• Memory Models for categorical response

If you have suggestions for models that should be added to the package, feel free to create an issue on GitHub. Ideally this should describe the model, point towards literature that gives details on the model, and if possible link to code that has already implemented the model.

Given the dynamic nature the bmm package is currently in, you can always view the latest list of supported models by running:

bmm::supported_models()

So stay tuned for updates and new models! We hope you will find the bmm package useful and will try fitting one of the already available models to your data. We appreciate all feedback and hope that the bmm package will make the use of measurement models easier for everybody.

The Solution

The solution depends on the state of the file.

echo "<file>" >> .git/info/exclude

git update-index --skip-worktree <file>

git update-index --no-skip-worktree <file>

Define an alias for easy access

I find that I use this command often enough to warrant an alias. You can run the following commands to add an alias to your git configuration:

git config --global alias.ignore 'update-index --skip-worktree'
git config --global alias.unignore 'update-index --no-skip-worktree'
git config --global alias.ignored 'git ls-files -v | grep "^S"'

and then you can use the following commands to ignore and unignore files:

git ignore <file>
git unignore <file>

Putting it all together (an example)

Let’s say I want to contribute code to an R package which is developed on GitHub. I can fork the repo and clone it to my local machine. The package has an .Rprofile file which overwrites my user configuration. I have a bunch of convenience configurations in my user .Rprofile which I want to use when working on this project. I can add the following lines to the project .Rprofile to source my user .Rprofile:

try(rprofile::load())

I can then add the project .Rprofile to the local .git/info/exclude file:

echo ".Rprofile" >> .git/info/exclude

and finally tell git to ignore the file locally (assuming I already have the alias defined):

git ignore .Rprofile

Base R

The way to work with different package versions in base R requires you to manually specify a folder in which to install each version. Let’s assume your current version of the package stringr is 1.4.0, and you want to install separately the latest version (as of the time of this writing, 1.5.1). The following will install stringr in a folder stringr-new in your user home folder:

dir.create('~/stringr-new')
install.packages('stringr', lib = "~/stringr-new")

Afterwards, you can load the default package with

library(stringr)

and the new version with

library(stringr, lib.loc = "~/stringr-new")

A few issues with this approach:

• You have to manually specify the folder for each version. And remember what it was when you want to use it again.
• You have to remember to specify the lib.loc argument every time you want to use the new version.
• You have to remember to detach the old version before loading the new one.
• You cannot install a specific version of a package from CRAN. You have to download the tarball from CRAN, extract it, and install it from the extracted folder.
• You have to do this one by one for each package you want to test.

remotes::install_version

We can augment the base R approach with the remotes package, which provides the install_version function. This function allows you to install a specific version of a package from CRAN. The following will install stringr version 1.5.1 in a folder stringr-new in your user home folder:

dir.create('~/stringr-new')
remotes::install_version('stringr', version = '1.5.1', lib = "~/stringr-new")

Loading the packages is the same as before. This approach solves the issue of having to download and install a specific version of a package from CRAN. However, it does not solve the other issues.

renv

The renv package is a package manager for R. It allows you to create a project-specific library, and to specify the versions of packages you want to use in a renv.lock file. It allows for a completely reproducible environment, and is the best solution for that purpose. However, it can be an overkill if you just want to test a few versions of a package. For an introduction to renv, see this blog post.

Introducing Vmisc: pkg_vload()

I was dissatisfied with the existing solutions, so I wrote a package to do that. The Vmisc package provides the pkg_vload function, which allows you to load a specific version of a package, or to install it if it is not already installed. You can start by installing and loading the package:

install.packages('Vmisc', repos = c('https://popov-lab.r-universe.dev'))
library(Vmisc)

The function pkg_vload combines the functionality of library(), remotes::install_version(), and dir.create(), and it also allows you to list as many packages as you want. The simplest option, for everyday use, is to specify just the package names, as you would with library():

pkg_vload(stringr, dplyr, ggplot2)

If you already have the package installed, it will load the default version. If you don’t, it will install the latest version from CRAN in the default library. This use case is identical to xfun::pkg_load(), but there is some added functionality for handling different versions of a package.

To load a specific version, you can specify the version argument:

pkg_vload(stringr('1.5.1'), dplyr, ggplot2)

The function expects a call to the package name, followed by the version in parentheses. It will also recognize if the version you specified is already installed. For example, if you already have stringr version 1.5.1 installed the good old way, it will load it from the default library. And you will see the following output:

#> Loading required package: stringr
#> Loading required package: dplyr
#> Attaching package: ‘dplyr’
#> Attaching package: ‘ggplot2’

But let’s say we want to install version 1.0.0 of stringr. We can do that with the following:

pkg_vload(stringr('1.0.0'))

which results in

#> Downloading package from url: https://cran.rstudio.com//src/contrib/Archive/stringr/stringr_1.0.0.tar.gz
#> * installing *source* package 'stringr' ...
#> ** package 'stringr' successfully unpacked and MD5 sums checked
#> ** using staged installation
#> ** ## output omitted ##
#> * DONE (stringr)
#> Loading required package: stringr

pkg_vload has created a folder stringr-1.0.0 in the default library path, installed the package there, and loaded it from there. The following two folders now coexist:

dirs = list.dirs(.libPaths(), recursive = FALSE)
dirs[grepl('stringr', dirs)]
#> [1] "C:/Users/vepopo/AppData/Local/R/win-library/4.3/stringr"
#> [2] "C:/Users/vepopo/AppData/Local/R/win-library/4.3/stringr-1.0.0"

pkg_vload(stringr('1.0.0')) not only installed the package but also loaded it. If you restart your session, you can load either versions by simply using pkg_vload(stringr) or pkg_vload(stringr('1.0.0')).

pkg_vload(stringr('1.0.0')) # for version 1.0.0
pkg_vload(stringr) # for the default version

Benefits of using pkg_vload:

• vectorized: you can load multiple packages at once
• if a package exists, it will be loaded, otherwise it will be installed and loaded
• you can specify the version of the package you want to load/install, without having to specify the library path (although you can)
• you can install as many versions of a package as you want, and they will coexist in the same library path
• you can switch which version will be the default with another function from the package, pkg_switch_default

Switching the default version of a package

The Vmisc package also provides a function to switch the default version of a package. For example, if you have versions 1.0.0 and 1.5.1 of stringr installed, and you want to make 1.0.0 the default, you can do that with the following:

pkg_switch_default('stringr', '1.0.0')
#> The default version of stringr has been switched to 1.0.0. The previous default version has been renamed to stringr-1.5.1
#> Please restart R to complete the process.

What this will do is rename the folder stringr to stringr-1.5.1 and stringr-1.0.0 to stringr. After you restart your session, pkg_vload(stringr) or even just library(stringr) will load version 1.0.0. You can also switch back to the default version with:

pkg_switch_default('stringr', '1.5.1')
#> The default version of stringr has been switched to 1.5.1. The previous default version has been renamed to stringr-1.0.0
#> Please restart R to complete the process.

Conclusion

Although not a replacement for renv for reproducible environments, Vmisc provides a simple way to work with multiple versions of an R package. It is especially useful for testing and comparing different versions of a package. The package is available on my R-universe and on GitHub. It was inspired by the xfun package, and contains other functions that I found useful in my everyday work. I hope you find it useful too.

Bonus: finding all global options used by a package

The Vmisc package also provides a function to find all global options used by a package. These are options you can set via options(), but they are rarely well documented in the package documentation. The function packageOptions will list all global options used by a package, and their default values. For example, to find all global options used by the brms package, you can use the following:

packageOptions('brms')

#> Package brms current options:
#> 
#> brms.save_pars       :  NULL 
#> mc.cores             :  1 
#> brms.threads         :  NULL 
#> brms.opencl          :  NULL 
#> brms.normalize       :  TRUE 
#> brms.algorithm       :  "sampling" 
#> brms.backend         :  "rstan" 
#> future               :  FALSE 
#> brms.file_refit      :  "never" 
#> wiener_backend       :  "Rwiener" 
#> brms.verbose         :  FALSE 
#> shinystan.rstudio    :  FALSE 
#> brms.plot_points     :  FALSE 
#> brms.plot_rug        :  FALSE 
#> brms.short_summary   :  FALSE 
#> .brmsfit_version     :  NULL 

The function is experimental - it scrapes the source code of the package to find mentions of getOption(‘something’, default = something). It also does not provide documentation. But I found it useful for reminding myself of the options I can set for a package, and their default values.