Robustness Testing Reproducing #21206
-
|
Hello! I need some help with the robustness testing. For example, with issue #17529 I am trying to reproduce it using the command from the README.md. I might not understand everything correctly, but should the issue always be reproducible, or is it flaky? Currently, I get "Failed to reproduce" at the end, even though the test passes 100 times. Is it possible to reproduce it consistently? TLDR: Can I somehow reproduce the issues documented in README.md consistently? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
It's very hard to provide 100% reproducibility of issues, some have 10% reproducibility while other <1%. That's why we are running 100 tries to reproduce. The overall goal is to keep it as high as possible to ensure we are not losing previous functionality while developing robustness. However reality is it's very hard to maintain a soft guarantee across the time, as it's pretty hard to automate and maintain. My latest attempt to improve this was documenting the process. As part of that I tracked and documented which was the last commit I was able to reproduce the particular bug. I did that running the reproduction script and if it didn't work I bisected the commit history to find last commit. My goal is that we could use track down which change has broken the reproduction, and fix all reproductions over time. For #17529 specifically I was not able to reproduce it on main branch for some time. Most recent reproduction was on c272ade from May 30, 2025. |
Beta Was this translation helpful? Give feedback.
-
|
Okay, thank you for the quick reply! |
Beta Was this translation helpful? Give feedback.
It's very hard to provide 100% reproducibility of issues, some have 10% reproducibility while other <1%. That's why we are running 100 tries to reproduce. The overall goal is to keep it as high as possible to ensure we are not losing previous functionality while developing robustness. However reality is it's very hard to maintain a soft guarantee across the time, as it's pretty hard to automate and maintain.
My latest attempt to improve this was documenting the process. As part of that I tracked and documented which was the last commit I was able to reproduce the particular bug. I did that running the reproduction script and if it didn't work I bisected the commit history to find last com…