Around 10-20% of the time, when I generate the plots for posterior contraction and z-score, I see unusual sampling like the one below. What would be the reason for this? Does this mean that the neural network model is showing some sensitivity? Or is this some randomness generated by GPU?
FYI, the simulation model is a simple, toy, random walk model. Also, this is done on the latest version of BayesFlow (I think 2.10.1) that’s compatible with CUDA on Windows.
I appreciate any thoughts or feedback.
Thank you,
Ali
these data sets look like nasty outliers at first glance.
Some questions that come to mind to better understand what’s happening:
What’s your concrete random walk simulator?
Does the simulator have some heavy-tailed components (e.g., non-Gaussian error terms)?
How many data sets do you use for training (i.e., what’s the simulation budget)? → If the budget is small and you simulate many test data sets, it’s quite likely that you’ll get some test data sets that the neural network hasn’t really seen during training, even though they are not per se extreme under the joint model.
Can you plot the actual observations (i.e., the random walk) for the data set that you highlighted with a circle? Does that data set look abnormal in some way?
What inference network are you using? Normalizing flows can be a bit eccentric when the data come from the tails, and flow matching might go less off-rails here.
These are great points helping me to narrow down the scope. We’re using a third-party software for running the simulations before feeding the data to BayesFlow. That software could also play a role. I will look into it and will update you.
Once again, thank you so much for these very helpful guidelines.
Best,
Ali
