# Cannot generate posterior samples after training **URL:** \ **Category:** General\ **Created:** [February 23, 2024, 9:13am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68 "2024-02-23T09:13:53Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![wyy](https://avatars.discourse-cdn.com/v4/letter/w/bc8723/32.png) [@wyy](https://discuss.bayesflow.org/u/wyy)\ **Post date:** [February 23, 2024, 9:13am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/1 "2024-02-23T09:13:53Z")
Recently I used the R package to generate some data as summary statistics and put it in Bayesflow for training. The data is in high dimension. I try a small size training, epochs=1, iterations\_per\_epoch=100, batch\_size=32, validation\_sims=20. I find that val\_losses cannot be plotted out. The most important thing is it cannot generate some posterior samples after training. So the diagnostic plot is empty also. At first, I was considering whether the R data(generated from the R package, I use rpy2) format is incorrect. because I wrote the simulator\_fun using the r package, but when I saw the test data generated successfully from the model, I thought the R package already worked. But I don’t know why “val\_losses” has no plot, and also I put the test data into the model, and the outcome is nan. I am not sure whether the training is not enough (but I try iteration=10000, same situation) or there is any other error in the model. But I already tried it on a simple toy example, the outcome is ok. May I ask do you know why the model looks like that, Did the data from R not go into the training successfully? or is there any other error? Thanks so much. ---
**Author:** ![elseml](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/elseml/32/19_2.png) [@elseml](https://discuss.bayesflow.org/u/elseml)\ **Post date:** [February 23, 2024, 9:23am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/2 "2024-02-23T09:23:53Z")
Hello wyy, receiving nan outcomes sounds like a breaking error rather than insufficient training. Even if you did not train your network at all, running data through a random initialized network should give you some (random) output. Did you format your training data in the required format (e.g., for offline training: a simulations\_dict with multidimensional sim\_data and prior\_draws numpy arrays)? Does your training data possess some nans that you have to filter out before passing to the network / fix during the simulation? Cheers, Lasse ---
**Author:** ![ali](https://avatars.discourse-cdn.com/v4/letter/a/8c91f0/32.png) [@ali](https://discuss.bayesflow.org/u/ali)\ **Post date:** [February 23, 2024, 7:42pm UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/3 "2024-02-23T19:42:58Z")
I’m not sure if it’s related, but coincidentally, this also happened to me a few hours ago. I was doing a round-based training, and it went on for more than 20 hours. In my experience, it happens when the data becomes so large because the ‘nan’ issue does not occur when I set a lower value for the number of rounds. On the other hand, if I reduce the number of simulations (data points) in each round, then I can increase the number of rounds. This was round 7 with 280,000 simulated data (40K each round). I hope it helps. ![Screenshot 2024-02-23 at 1.51.35 PM](https://canada1.discourse-cdn.com/flex007/uploads/bayesflow/original/1X/7eb1044cd376f97ca51cea7d4f85a52a27225133.png) ---
**Author:** ![wyy](https://avatars.discourse-cdn.com/v4/letter/w/bc8723/32.png) [@wyy](https://discuss.bayesflow.org/u/wyy)\ **Post date:** [February 25, 2024, 7:33am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/4 "2024-02-25T07:33:52Z")
Hi Lasse Thanks for the reply, yes I have checked no nans in training data, and the simulations\_dict with multidimensional sim\_data and prior\_draws are all numpy arrays. By the way， may i ask another question: is the following example offline training? I thought this example was for online training. [https://bayesflow.org/\_examples/Intro\_Amortized\_Posterior\_Estimation.html](https://bayesflow.org/_examples/Intro_Amortized_Posterior_Estimation.html) Thank you wyy ---
**Author:** ![wyy](https://avatars.discourse-cdn.com/v4/letter/w/bc8723/32.png) [@wyy](https://discuss.bayesflow.org/u/wyy)\ **Post date:** [February 25, 2024, 7:36am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/5 "2024-02-25T07:36:08Z")
Thanks so much, is that means to reduce the batch size and iteration and increase the epoch? Best wyy ---
**Author:** ![KLDivergence](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/kldivergence/32/15_2.png) [@KLDivergence](https://discuss.bayesflow.org/u/KLDivergence)\ **Post date:** [February 25, 2024, 5:22pm UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/6 "2024-02-25T17:22:04Z")
Correct, this example tackles online training! ---
**Author:** ![KLDivergence](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/kldivergence/32/15_2.png) [@KLDivergence](https://discuss.bayesflow.org/u/KLDivergence)\ **Post date:** [February 25, 2024, 5:26pm UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/7 "2024-02-25T17:26:52Z")
In general, the only cases where I have seen nans in the loss functions are: 1. When training diverges (e.g., exploding gradients) 2. When there is something terribly wrong with the data (e.g., nans, inf, etc) Number one can be easily inspected. Number two requires more attention. Do the data or parameters contain super large numbers? If so, standardization is needed, as in any deep learning application. Seeing the network setup will also help. ---
**Author:** ![ali](https://avatars.discourse-cdn.com/v4/letter/a/8c91f0/32.png) [@ali](https://discuss.bayesflow.org/u/ali)\ **Post date:** [February 26, 2024, 1:57am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/8 "2024-02-26T01:57:32Z")
Stefan is addressing the problem systematically, so I would suggest providing more information as he requested. But from my limited experience in offline/round-based training, reducing the number of rounds or the data you provide might be helpful. For example, if you are introducing 100,000 data points, try with lower numbers (like 1000 data points) to see if the problem exists, or you can reduce the number of epochs or rounds. At least, this is how I can resolve the issue with my code, but there is a chance that something weird is going on with my data, too, when I generate a large number of data points. ---
**Author:** ![wyy](https://avatars.discourse-cdn.com/v4/letter/w/bc8723/32.png) [@wyy](https://discuss.bayesflow.org/u/wyy)\ **Post date:** [February 26, 2024, 3:24am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/9 "2024-02-26T03:24:24Z")
Thanks so much for your reply!!! ---
**Author:** ![wyy](https://avatars.discourse-cdn.com/v4/letter/w/bc8723/32.png) [@wyy](https://discuss.bayesflow.org/u/wyy)\ **Post date:** [February 26, 2024, 3:24am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/10 "2024-02-26T03:24:56Z")
Do we have offline training examples? ---
**Author:** ![marvinschmitt](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/marvinschmitt/32/98_2.png) [@marvinschmitt](https://discuss.bayesflow.org/u/marvinschmitt)\ **Post date:** [February 26, 2024, 5:50am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/11 "2024-02-26T05:50:32Z")
Here’s an example that uses offline training: > **[Google Colaboratory](https://colab.research.google.com/drive/1ub9SivzBI5fMbSTwVM1pABsMlRupgqRb?usp=sharing)** ---
**Author:** ![wyy](https://avatars.discourse-cdn.com/v4/letter/w/bc8723/32.png) [@wyy](https://discuss.bayesflow.org/u/wyy)\ **Post date:** [February 26, 2024, 8:12am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/12 "2024-02-26T08:12:47Z")
Thanks so much, i think I use online training, so I generated some samples to check the data, i think it is not a bigdata, and I haven’t seen nans or inf. I find a weird thing is my code sometimes can generate the loss and posterior samples with values, and sometimes the outcome is all nans. I will try to check all of the possible errors. ---
**Author:** ![wyy](https://avatars.discourse-cdn.com/v4/letter/w/bc8723/32.png) [@wyy](https://discuss.bayesflow.org/u/wyy)\ **Post date:** [February 26, 2024, 8:14am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/13 "2024-02-26T08:14:11Z")
thanks so much!!! ---
**Author:** ![elseml](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/elseml/32/19_2.png) [@elseml](https://discuss.bayesflow.org/u/elseml)\ **Post date:** [February 26, 2024, 9:18am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/14 "2024-02-26T09:18:45Z")
@wyy Here is also another example of offline training: [https://bayesflow.org/\_examples/TwoMoons\_Bimodal\_Posterior.html](https://bayesflow.org/_examples/TwoMoons_Bimodal_Posterior.html) ---
**Author:** ![wyy](https://avatars.discourse-cdn.com/v4/letter/w/bc8723/32.png) [@wyy](https://discuss.bayesflow.org/u/wyy)\ **Post date:** [February 29, 2024, 8:03am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/15 "2024-02-29T08:03:21Z")
may i ask one more question. I try a very small dataset only 50 samples, i have check no nan or inf, i use offline training so that can control the data, i find the loss value is nan, and i can plot the graph for history[“train\_losses”], but the graph for history[“val\_losses”] is empty. if we can plot history[“train\_losses”], does that means the training is successfully, but no value in history[“val\_losses”]. Also i try the standardization, same situation. Thanks. ---
**Author:** ![marvinschmitt](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/marvinschmitt/32/98_2.png) [@marvinschmitt](https://discuss.bayesflow.org/u/marvinschmitt)\ **Post date:** [February 29, 2024, 8:21am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/16 "2024-02-29T08:21:42Z")
Could you please post a minimal reproducible example? I would need to investigate the concrete case in more detail to help here. Thanks! ---
**Author:** ![wyy](https://avatars.discourse-cdn.com/v4/letter/w/bc8723/32.png) [@wyy](https://discuss.bayesflow.org/u/wyy)\ **Post date:** [March 4, 2024, 3:32am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/17 "2024-03-04T03:32:27Z")
so sorry for the late reply, these days, I try my best to solve the problems, but I find that sometimes if I change to another dataset, then the loss has values. I guess my coding is right, there may be some problems with the data generated. Thanks so much for your help!!! ---
**Author:** ![marvinschmitt](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/marvinschmitt/32/98_2.png) [@marvinschmitt](https://discuss.bayesflow.org/u/marvinschmitt)\ **Post date:** [March 4, 2024, 9:19am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/18 "2024-03-04T09:19:18Z")
Ok, thanks for the update. If you run into any similar issues again, I’m happy to help with a minimal reproducible example at hand. Cheers, Marvin ---
**Author:** ![wyy](https://avatars.discourse-cdn.com/v4/letter/w/bc8723/32.png) [@wyy](https://discuss.bayesflow.org/u/wyy)\ **Post date:** [March 6, 2024, 8:07am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/19 "2024-03-06T08:07:49Z")
Thanks so much, during my coding, I found use the same code, sometimes the loss is nan, and if I rerun it again it has values, do you know why? Also, sometimes I see the same problem with Ali, using a smaller dataset will solve the problems, but why a larger dataset are easier to see nan in loss? Best wyy ---
**Author:** ![wyy](https://avatars.discourse-cdn.com/v4/letter/w/bc8723/32.png) [@wyy](https://discuss.bayesflow.org/u/wyy)\ **Post date:** [March 7, 2024, 9:04am UTC](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68/20 "2024-03-07T09:04:34Z")
![loss](https://canada1.discourse-cdn.com/flex007/uploads/bayesflow/original/1X/6c6e6e921a365710b23ae6929baa586d83de33a1.png) Hi~~May I ask a question, in the training, I saw we have loss and Avg loss value (for example Loss: -2.036,W.Decay: 0.056,Avg.Loss: -1.990), but why in the pink area, the loss is nan? I have checked my data do not contain any inf or nan. and also I think if we have avg.loss value, that means the training is successful right? Best wyy [Next page](https://discuss.bayesflow.org/t/cannot-generate-posterior-samples-after-training/68.md?page=2)