Adding in-person event value to AI-assisted research

The Impetus

Zachary Yang and I recently ran a LLM simulation workshop at the Conference On Language Modelling (COLM) in San Francisco earlier this month.

We are trying to test uses of modern tools like structured knowledge bases for the social web: we made all submissions cards in a Workshop specific Semble collection. No brainer and easy with agents. It even lead to Semble now loading metadata from OpenReview urls. Win-win!

Beyond that? What about breakout sessions? These are periods in a workshop day during which the audience disperses into groups to have focussed discussions on one of a predetermined set of topics. Breakouts serve as a way to enrich the in-person component of these events. As Zach and I considered this option, we realized that AI-assisted research was a discussion topic in its own right, but nothing in the event acknowledged its increasing role in research.

Aside from the obvious networking benefits of in-person events (starting collaborations, hearing the word on the street etc.), how could we leverage the in-person nature of the event to advance the state of the field in how we are using AI?

As the generator-verifier gap disappears, the human researcher's value is increasingly as verifier. However, individual experts can be biased from limited scope, simple error, and other sources. Group verification naturally addresses these limitations: integrating more samples can reduce this noise (c.f. wisdom of the crowds); working in groups activates social learning benefits; and increasing social bonds that lead to more coherent researcher networks going forward. When the verification is logged as data, it can also provide valuable feedback for AI assistant research systems to improve, e.g. their research taste.

This all boiled down to the question: What can you do in an hour with a room full of researchers on a particular topic?

All votes are public

Ideally, we would do all these things at the same time!

Our Contribution: AI Red Penning

AI red-penning is a new workshop breakout session concept: do a group critique of AI-generated proposals for bleeding-edge research on topics relevant to the workshop theme.

A natural, data-driven way for AI research tools to do this is to take the corpus of accepted papers presented at the workshop as the space from which proposals should be sampled.

Zach and I knew that TipTreeSystems and co-authors had already developed exactly such a method (just recently accepted to NeurIPS 2026).

Alien Science: Sampling Coherent but Cognitively Unavailable Research Directions from Idea Atoms
Large language models are adept at synthesizing and recombining familiar material, yet they often fail at a specific kind of creativity that matters most in research: producing ideas that are both coherent and non-obvious to the current community. We formalize this gap through cognitive availability, the likelihood that a research direction would be naturally proposed by a typical researcher given what they have worked on. We introduce a pipeline that (i) decomposes papers into granular conceptual units, (ii) clusters recurring units into a shared vocabulary of idea atoms, and (iii) learns two complementary models: a coherence model that scores whether a set of atoms constitutes a viable direction, and an availability model that scores how likely that direction is to be generated by researchers drawn from the community. We then sample “alien” directions that score high on coherence but low on availability. On a corpus of ∼\sim7,500 recent LLM papers from NeurIPS, ICLR and ICML, we validate that (a) conceptual units preserve paper content under reconstruction, (b) idea atoms generalize across papers rather than memorizing paper-specific phrasing, and (c) the Alien sampler produces research directions that are more diverse than LLM baselines while maintaining coherence.
https://arxiv.org/html/2603.01092v1

They call it sampling alien science. In simple terms, it samples proposals from the "good gaps" in a body of provided literature. Good here means the combination of high coherence in ideas (good ideas) and low availability (not likely be proposed by authors of existing papers in the corpus).

Our trial

In an hour that we had set aside in the afternoon, we presented the task to the workshop participants: a set of 10 of these proposals, one for each of the top 10 gaps surfaced by the method. We printed out copies, piled in stacks around the room and told participants to go to the stack associated with the topic they were most interested in.

The results were instructive. Three take-aways:

  1. 1.

    Low uptake. Only 3 groups congealed (something like 15% or participants). Those that did, however stayed until the end. We think most of this low uptake was the absence of a point person for each proposal. We had instructed participants that they were to select a facilitator from those that had self-organized into the group. In these highly social activities, such a person is an anchor without which people stand around unmoored and diffuse away because social-proof signals are too weak. A simple fix for next time: assign group facilitators beforehand, ideally the event speakers themselves, since participants enjoy getting face time with speakers.

  2. 2.

    The proposals were hard to read. Around 3-pages each, and walls of LLM-ish text, it was just a hard task. A workshop room with distractions is an unconventional reading environment even with easy-to-parse documents. Without visual parsing cues and clear text, it was pretty hard to extract, hold, and process the main essence of each proposal. I presume these are easy fixes for TipTreeSystems, since they already have live figure generation in their research assistant.

  3. 3.

    The wrap-up phase was high-value scientific discussion. We had a wrap-up phase where each group presented their review: strengths, weaknesses, suggestions for improvement. This went very well. The outcomes were heterogeneous: some proposals seemed really cool, others not so much. The self-selection of people willing to stick it out apparently correlated well with people who understood and were actually really good at the critiquing assignment.

Overall, we got a lot out of this trial and now we (both us as organizers and TipTreeSystems as research tool developers) have some clear ideas for next time (a AAAI workshop in February), where we will run it again and I am hopeful for wider adoption and further design improvements.

If we are going to have to develop new norms for research, making the most use of this in-person time to test new ideas is going to be crucial.

Do you have ideas for how we can do AI-assisted research? Share them!

Thanks again to TipTreeSystems for doing this with us!