Performance difference issue

[Jeffery-MIC]

Hi, thank you so much for this astonishing work. I really appreciate your efforts.
I have a question regarding the performance difference.
After installing the environment by running

pip install -r requirements/dev_3.11.txt

I successfully finish running the rxrx3_core_benchmarks_openphenom.ipynb without doing any modification.

However, I find the performance of gene-gene interaction recall shown in the notebook as

But the official paper reported

I wonder what step did I miss or I should do some corresponding modification to the evaluation steps to obtain the identical performance from OpenPhenome S/16.

Thank you for your time reading this.

[safiyecelik]

Hi Jeffery,

Thanks for reaching out!
Am I understanding correctly that you are seeing ~.02 higher recall in your own runs than what was reported in the paper?

[Jeffery-MIC]

Yes, I found around .02 gap between the performance of the released weight and the reported paper in terms of OpenPhenom S/16. I didn't do any modification to the code so I wonder how can I obtain identical performance or which number should I be following. Thank you so much for your prompt response.

[safiyecelik]

Hi @Jeffery-MIC ,

Looks like we did not get back to you on this (sorry about that!), but we had tested it again back then and had not seen any performance difference.
Can you please let us know whether this was resolved on your end so we can close the issue? I'd be curious to see where the discrepancy was coming from!