Marcin Abram

Selected projects

A selection of completed and current projects, grouped by subject.

Verification and evaluation

Verifying the work of an AI agent

FirstPrinciples · 2026

FirstPrinciples builds AI agents that assist with research in physics. An agent of this kind reads the literature, plans a study, derives results, writes and runs code, and reports back.

With several colleagues, I built a hybrid sub-system that verifies what the agent produces. It combines symbolic (non-LLM-based), numerical, and conceptual checks.

Layer How the verdict is reached What it catches
Symbolic Each step is checked symbolically and deterministically. No language model takes part in the verdict. A dropped factor, a wrong sign, units that do not match.
Numerical A language model proposes numerical tests, such as a known limit or an independent recomputation. The tests are executed and the result decides. Code that runs without complaint and returns a plausible, wrong number.
Conceptual Agents built on models from different vendors review the substance of the work. Correct mathematics on a wrong premise. An approximation used outside its range.

Test suite

Cases with a known answer, work with one or many planted errors, twins (the same work with and without a planted error), public and private benchmarks, adversarial inputs, etc.

The work also includes a custom ledger, translation layers, deep integration with our scientific platform, a watchdog system, repair loops, and a research toolkit for ablation studies.

Measuring where frontier models fail

FirstPrinciples · 2026

In my work, I have investigated how frontier language models and agents fail on expert-level scientific work, and how we can detect those failures. Among other things, I have developed experiments to test:

  • how a model chooses between competing descriptions of the same problem, and whether that choice is calibrated;
  • whether a valid objection survives several turns of conversation with a user who disagrees;
  • whether an error made in one step of an agent is repaired in the next step or carried forward;
  • how far LLM judges can be trusted, alone and in panels.

The findings of those experiments showed us how to build a robust verification harness and make our product more reliable. Some of the general findings that I can share are written up in the notes section.

Evaluation beyond one accuracy number

IBM · 2024 – 2026

At IBM, my main role was to assist in the development of non-standard MVPs that went beyond the scope of a typical engagement. Another frequent task was to evaluate a generative-AI system that did not behave as expected. In one such engagement, I built an evaluation framework that covers multi-turn use as well as perturbed and adversarial inputs. This substantially reduced the time to market of the MVPs.

Retrieval and context

A retrieval system that kept inventing answers

IBM · 2024 – 2025

In one of my engagements, I assisted a leading European car maker in assessing its question-answering system. The system, built on standard retrieval-augmented generation (RAG) pipelines, kept hallucinating answers. The root cause was the structure of the client's documents and their domain language. I documented the failure and designed an agentic RAG system that reduced hallucinations by over 80% in the client's trial. This saved the engagement, which the client was happy to extend substantially. In addition, we made the framework I proposed reusable across a few engagements, and I passed it to another team to turn it into an official asset that streamlines similar work.

When a more relevant context does not help

University of Southern California · 2023 – 2024

With a group of students, I measured how the answers of a language model change with the relevance of the context it is given. It turned out that the quality of the answer does not grow monotonically with the measured relevance. This means that a context closely aligned with the topic does not always help more than a loosely related one, especially when the question is open-ended or novel. In addition, the structure of the context might matter more than what the context contains, especially for easy and moderately hard questions.

arXiv:2312.07069, arXiv:2407.02028

Multi-agent systems and trust

A roadmap for the next generation of agentic AI

IBM · 2025

I was a member of a small cross-division group, led by IBM's chief scientist, that helped create the company's roadmap for the next-gen agentic AI development cycle. I contributed an analysis of where current multi-agent systems fall short, and research on self-adapting and self-improving agent loops.

Training a model together without sharing data

University of Southern California · 2022, and Fetch.ai · 2020

The same question came up at two employers: how can parties that do not trust each other train a model together? At Fetch.ai, we designed a system in which groups of nodes train models jointly without sharing their data and coordinate through a ledger. This concept was novel enough to earn us a patent. At USC, with colleagues from the Information Sciences Institute, I worked on privacy-preserving federated training protocols and added security mechanisms that make them more robust to data poisoning attacks.

US Patent 12,749,011, arXiv:2205.01184

A protocol redesigned around its attacks

Fetch.ai · 2018 – 2020

Fetch.ai was building a network in which autonomous software agents trade and coordinate without a central operator. I joined with no background in blockchains. Three months later I had found a critical flaw in the design of the main protocol, and I was asked to lead the work on a solution.

Finding myself in that new role, I first systematically catalogued the ways in which a participant could attack or game the protocol. This included attack vectors that had not been described before. I then designed mechanisms that stay correct when some of the participants misbehave. Next, I built the test suites, benchmarks, and attack simulations, and then experimental implementations and proofs of concept. Finally, I led the redesign of the main protocol. We proposed a novel protocol, based on a swarm of agents communicating through a DAG-style ledger, in which no single participant decides the content of a block.

arXiv:2006.05390, presented at Tokenomics 2019

Machine learning for physics

Finding transitions in pattern-forming processes without labels

University of Southern California, with Sandia National Laboratories · 2021 – 2022

The microstructure of a material depends on how it was made, and it can change character when a process parameter crosses some value. Usually there are no labels that say where one regime ends and the next begins. I proposed a self-supervised method that finds those boundaries from images alone, without human annotations or any other labels: a network learns to predict the process parameters from an image of the microstructure, and the uncertainty of its prediction changes where the microstructure changes. We tested it on two simulated processes: spinodal decomposition of a two-phase mixture, and the growth of thin alloy films by physical vapor deposition.

uncertainty of the prediction share of the dark phase

Top: spinodal decomposition at five compositions. Each picture is a separate simulation that I ran for this page with the Cahn–Hilliard equation. Bottom: a sketch of what the method returns, not data. The pictures only illustrate the idea; the paper has its own simulations and results for both processes.

npj Computational Materials 8, 205 (2022) (Nature Portfolio), first author. Supported by a sponsored research award from Sandia National Laboratories.

Neural networks that emulate quantum dynamics

University of Southern California · 2020 – 2022

I directed a group of five students that trained a neural network to emulate the dynamics of a quantum wave packet, and asked how far such an emulator can be trusted outside the range it was trained on. Trained on a curriculum of simple examples, it generalizes to wave packets and potentials it has not seen.

Frontiers in Materials 9, 1060744 (2023)

Strongly correlated quantum materials

Jagiellonian University · 2011 – 2016

My PhD. In these materials the electrons interact too strongly for the standard approximations to hold, and a large part of the work is to find out where a method can still be trusted. I studied unconventional superconductivity and its coexistence with magnetism, and the phase transitions of the heavy-fermion compound UGe2, and wrote the numerical methods and the high-performance code in C++ and Python. I published seven peer-reviewed papers.

The papers

Machine learning in production

Prototypes and first products for enterprise clients

IBM · 2024 – 2026

Architectures, proofs of concept, and minimum viable products for clients in advertising and government. One of them was the core logic of an explainable-AI engine inside an analytics platform for an advertising agency. I was the technical lead of the Data and AI group in my unit, led the technical discussions with client executives, and ran multi-day workshops for client engineers.

A credit score from heterogeneous data

TypeScore · 2016 – 2018

TypeScore was a fintech startup that scored the credit risk of businesses from a wide variety of sources, structured and unstructured. Data of this kind is never complete and never consistent, and the models had to stay robust on it. I was a founding engineer in a small, lean team and wrote a large part of the codebase myself. We built the ML platform from scratch: data pipelines, NLP risk models, training and retraining, CI/CD, and deployment on Google Cloud. I owned the pipeline in production. The models were behind the company's main product, the first free credit score for businesses in the United Kingdom.