# Training data with additive noise **URL:** \ **Category:** General\ **Created:** [March 13, 2024, 10:12pm UTC](https://discuss.bayesflow.org/t/training-data-with-additive-noise/82 "2024-03-13T22:12:14Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Jice](https://avatars.discourse-cdn.com/v4/letter/j/c77e96/32.png) [@Jice](https://discuss.bayesflow.org/u/Jice)\ **Post date:** [March 13, 2024, 10:12pm UTC](https://discuss.bayesflow.org/t/training-data-with-additive-noise/82/1 "2024-03-13T22:12:14Z")
Hi, all, I added noise to training data in BayesFlow, such as Gaussian noise, but I found the the model performance after training is not as good as those without noise. I tried to tune the model for a while, the issue is still there. I read some papers and wonder whether it results from the model misspecification or not. The real data contains noise at different level, considering noise in training process is quite necessary. Does anyone suggest some methods to improve model performance in the case of additive noise? Thanks! Jice ---
**Author:** ![ali](https://avatars.discourse-cdn.com/v4/letter/a/8c91f0/32.png) [@ali](https://discuss.bayesflow.org/u/ali)\ **Post date:** [March 14, 2024, 11:37pm UTC](https://discuss.bayesflow.org/t/training-data-with-additive-noise/82/2 "2024-03-14T23:37:27Z")
Others should have better and more systematic solutions, but from my experience, introducing more data and reducing the batch size (e.g., batch\_size=16) would be helpful. More data would increase the chance of the neural network being exposed to more distinct modes of behavior, enabling it to understand the role of the noise parameter. A smaller batch size would possibly let the network escape local minima, but it can increase the training time significantly. ---
**Author:** ![elseml](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/elseml/32/19_2.png) [@elseml](https://discuss.bayesflow.org/u/elseml)\ **Post date:** [March 15, 2024, 9:37am UTC](https://discuss.bayesflow.org/t/training-data-with-additive-noise/82/3 "2024-03-15T09:37:19Z")
Hi Jice, Are you referring to performance on simulated data or real-world data? For diagnostics on simulated data, model misspecification should not be an issue since your test data is simulated from the same model that you train your network on. Adding noise makes the inverse task that the network learns harder, so I would try making the network more expressive (= bigger), letting it train longer while monitoring potential overfitting, and tuning the learning rate. For real-world data, we recently proposed [using the variation between the predictions of a neural network ensemble to measure the impact of model misspecification](https://arxiv.org/abs/2310.11122). It would be interesting to see whether adding noise to your model reduces the sensitivity induced by misspecification by improving the reliability of the networks. ---
**Author:** ![Jice](https://avatars.discourse-cdn.com/v4/letter/j/c77e96/32.png) [@Jice](https://discuss.bayesflow.org/u/Jice)\ **Post date:** [March 18, 2024, 6:49am UTC](https://discuss.bayesflow.org/t/training-data-with-additive-noise/82/4 "2024-03-18T06:49:24Z")
Hi Elseml, Thanks for your reply. It is true that adding the noise to the observations makes the inference more difficult. My training data is (x, y), x are parameters, y are observations. The neural network performs well for posterior approximate given the new y\*. However, if I add the noise to y\*, such as y\*+noise, since the true observations always contain noise, the performance degenerates. I guess it is caused by that the training data does not include any noise, or we call it model misspecification, as y in training data differs from (y\*+noise). I found one paper "Noise-Net: Determining physical properties of Hii regions reflecting observational uncertainties’. This paper implements noise training that considers noise during cINN training based on Soft Flow. I found the BayesFlow framework has the option of soft flow for inference network. But the cINN with soft flow works on perturbing x not observations, such as x+noise and concatenate y and noise [y, noise]. Do I understand right? If I am right, the part of soft flow in BayesFlow should be modified as below if I want to train noise-net like the paper: # Perturb data with noise (will broadcast to all dimensions) (_we should not perturb x_) ``` if len(shape_scale) == 2 and len(target_shape) == 3: targets += tf.expand_dims(noise_scale, axis=1) * 0 else: targets += noise_scale *0 ``` # Augment condition with noise scale variate (_we should perturb the condition with noise_) ``` condition +=noise_scale * tf.random.normal(shape=condition_shape) ``` Thanks for your discussion with me! Jice ---
**Author:** ![marvinschmitt](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/marvinschmitt/32/98_2.png) [@marvinschmitt](https://discuss.bayesflow.org/u/marvinschmitt)\ **Post date:** [March 18, 2024, 7:17am UTC](https://discuss.bayesflow.org/t/training-data-with-additive-noise/82/5 "2024-03-18T07:17:55Z")
Hi Jice, Thanks for kicking off that interesting discussion! ### re misspecification/robustness: - Your observation that posterior inference gets worse when we encounter unexpected noise is in line with findings from our paper on model misspecification in amortized Bayesian inference: [https://www.dagm-gcpr.de/fileadmin/dagm-gcpr/pictures/2023\_Heidelberg/Paper\_MainTrack/030.pdf](https://www.dagm-gcpr.de/fileadmin/dagm-gcpr/pictures/2023_Heidelberg/Paper_MainTrack/030.pdf) - Here’s a paper that also adds noise during the training to achieve more robust inference in noisy real-data settings: [[2307.13918] Simulation-based Inference for Cardiovascular Models](https://arxiv.org/abs/2307.13918) ### re softflow \<\> noise: I don’t think you need softflow in this case. As you said, softflow acts on the parameter domain and not on the data domain. Adding noise during training is independent of softflow. ### re noise during workflow: Where in the data flow are you currently adding the noise? My naïve take would be to add the noise in the configurator. Cheers, Marvin ---
**Author:** ![Jice](https://avatars.discourse-cdn.com/v4/letter/j/c77e96/32.png) [@Jice](https://discuss.bayesflow.org/u/Jice)\ **Post date:** [March 18, 2024, 4:53pm UTC](https://discuss.bayesflow.org/t/training-data-with-additive-noise/82/6 "2024-03-18T16:53:23Z")
Hi Marvin, Thank you very much for the suggestions. You are right, the soft flow perturbs parameter domain by adding noise to X, not acting on observation Y. I have tried to directly add the noise to training data (all training data are available on hand), then trained BayesFlow on those noisy training data (as you said, adding noise to configurator). But the model performance is bad. What I want to do is we can perturb the Y similar as soft flow, as the paper does ([https://academic.oup.com/mnras/article/520/4/4981/6982917](https://academic.oup.com/mnras/article/520/4/4981/6982917)). So I modify the code of soft flow part in BayesFlow as: # not perturb x in soft flow ``` if len(shape_scale) == 2 and len(target_shape) == 3: targets += tf.expand_dims(noise_scale, axis=1) * 0 else: targets += noise_scale *0 ``` # perturb the condition with noise ``` condition +=noise_scale * tf.random.normal(shape=condition_shape) ``` How do you think? Is it feasible? Thanks! Jice ---
**Author:** ![marvinschmitt](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/marvinschmitt/32/98_2.png) [@marvinschmitt](https://discuss.bayesflow.org/u/marvinschmitt)\ **Post date:** [March 20, 2024, 9:05am UTC](https://discuss.bayesflow.org/t/training-data-with-additive-noise/82/7 "2024-03-20T09:05:30Z")
Hi Jice, Thanks for clarifying. Where exactly are you doing the ` condition +=noise_scale * tf.random.normal(shape=condition_shape)` part? I don’t see why you would need to modify “the softflow part”. Cheers, Marvin ---
**Author:** ![Jice](https://avatars.discourse-cdn.com/v4/letter/j/c77e96/32.png) [@Jice](https://discuss.bayesflow.org/u/Jice)\ **Post date:** [March 22, 2024, 2:15am UTC](https://discuss.bayesflow.org/t/training-data-with-additive-noise/82/8 "2024-03-22T02:15:04Z")
Yes, you are right. Actually, it is nothing about softflow part. I just sampled error from uniform distribution, then added error to the observation. This process iterates during the training. I trained a model like this, the model now perfoms very stably under different noise level. ---
**Author:** ![elseml](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/elseml/32/19_2.png) [@elseml](https://discuss.bayesflow.org/u/elseml)\ **Post date:** [March 25, 2024, 11:21am UTC](https://discuss.bayesflow.org/t/training-data-with-additive-noise/82/9 "2024-03-25T11:21:12Z")
That’s great to hear!