RFDiffusion-AA (All Atom) was recently (finally) released, a little over a year after the original RFDiffusion release. This blog post was started months ago, when I had first started to play with RFDiffusion, but abandoned due to Life happening. Now that the second generation model has dropped, I thought it’d be fun to return to this post to study how RFDiffusion has fared over the past year. I’ll do a short recap on aspects of RFDiffusion, and then dive into a mostly unstructured set of observations I've had of the model. The observations come from a mixture of my own work, my coworkers, and posts from forums. I'll update this post as I come across any other interesting insights.
First, an introduction.
In November 2022, the Baker Lab at the University of Washington released a protein design model called RFDiffusion, which demonstrated remarkable capabilities in generating monomers, higher-order oligomers, binders, and motif scaffolds while being symmetry aware. The model's speed, ease of use, and accuracy massively surpassed previous methods, making it a potential game-changer in the field of protein design, akin to the impact of AlphaFold2 on protein folding. As someone who works in a protein design company, this was especially exciting for me personally.
RFDiffusion is built upon a prior structure prediction model, RosettaFold, released by the same lab a year earlier. During the training process, RFDiffusion receives a masked input sequence, noised coordinates of the structure, and a prediction from the prior step (known as 'self-conditioning' or providing a template). Over 200 denoising steps, RosettaFold rearranges the noised coordinates by adding its own 'noise,' with the goal of matching the true structure. Through fine-tuning across many proteins, RosettaFold evolves into RFDiffusion, a powerful protein generation model. There's well over a dozen pages of extra supplementary material defining exactly how one can use this process to conditionally generate an extraordinary wide range of structures, but none of it is super relevant to this post, so I'll skip it.
Now, as someone who has extensively used RFDiffusion, I have a fair bit of compiled thoughts about it. Absolutely none of it is research-grade in quality, and shouldn't be trusted upon by anybody or anyone. But! It might be useful reading by curious laymen or other confused people in ML x biology.
Peptide binder generation success rates are dramatically lower for most users than the papers results.
This is anecdotal, purely based on internet forum claims. But as far as I can tell, whereas the paper claims RFDiffusion hit-rates (defined as high-affinity binders) at around 1-5%. This is quite good given how useful even a single novel binder is, but the true success rate has proven to likely be much much lower; likely by an order of magnitude (.1%).
To note, I am not claiming RFDiffusion misreported their results. But...it is quite possible that the binder design tasks they used (creating binders for PD-L1, TRKA, IL7R-α, INSR, and HA) were a bit...easy? All of these are proteins with known peptide binders and had well-established crystal structures used as input and these crystal structures were already in complex with something it bound to (maybe the most important part!!!). This is not the norm for a lot of more general binder design problems, and likely where the discrepancy arises from.
There are still zero real-world uses of RFDiffusion in non-computational settings.
Although the RFDiffusion Nature paper has been cited 218 times since its release 8 months ago, most of these citations are from review articles or less well-known protein design papers. Surprisingly, there is a lack of experimental papers that directly utilize RFDiffusion for creating proteins and using them in in-vitro or in-vivo settings to gain insights. In fact, I have yet to find a single experimental paper that straightforwardly says ‘we used RFDiffusion for creating [X] and used [X] in an in-vitro/in-vivo setting and it gave us X insight’. Which is shocking for a model with as much fanfare as RFDiffusion had! Will cross this out the second I find one...
Alphafold2, on the other hand, have a bevy of papers that treat Alphafold2 as just a tool to obtain the actual, more interesting experimental result, such as this or this. Of course, Alphafold2 has been out for a little over 3 years at this point and has 20,000 citations, so comparing the two may be premature....
Nobody understands ideal filters to use w/ RFDiffusion outputs. There is still no study that closely analyzes this beyond the original paper, which is limited in scope.
RFDiffusion, like many other life-science models, does not claim to be perfect in its generation process, more-so aiming to increase the 'hit rate' of its generated proteins. The original paper suggests generating a large number of proteins (e.g., 10,000) with desired requirements and using provided metrics (RMSD, iPAE, iPTM, and pLDDT) to filter them down (e.g 2<RMSD, >.8 pLDDT, etc).
However, the thresholds for these metrics vary dramatically depending on the generation problem and are based on a limited set of wet-lab experiments described in the paper. People on forums (primarily Discord) often come up with dramatically different ideal filters for their problem. There is a need for further research to analyze the impact of these filters on RFDiffusion's generated outputs.
Recapitulating known motifs using RFDiffusion seems challenging, if even possible.
Not much to say about this. Every time I've seen someone try to generate something using vanilla RFDiffusion and they expect some well-known motif (in that structure class) to arise, it never does. Maybe this isn't an RFDiffusion problem, more a ProteinMPNN one, given that RFDiffusion does not design sequences...or maybe RFDiffusion gives bad structure predictions for these short motif segments. Either one is possible.
RFDiffusion seems successful in creating synthesizable proteins.
I've heard of relatively few cases in which RFDiffusion popped out something that simply could not express at all. This doesn't feel particularly impressive, given that its base model (RosettaFold) had seen a fairly significant fraction of all protein structures that exist, but it's worth noting.
I write essays about biology and ml. You can find all my posts at https://www.abhishaike.com/