# Setting up TimeSeriesTransformer

**URL:** <https://discuss.bayesflow.org/t/setting-up-timeseriestransformer/61>\
**Category:** General\
**Created:** [February 12, 2024, 4:57pm UTC](https://discuss.bayesflow.org/t/setting-up-timeseriestransformer/61 "2024-02-12T16:57:21Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![hazhir](https://avatars.discourse-cdn.com/v4/letter/h/edb3f5/32.png) [@hazhir](https://discuss.bayesflow.org/u/hazhir)\
**Post date:** [February 12, 2024, 4:57pm UTC](https://discuss.bayesflow.org/t/setting-up-timeseriestransformer/61/1 "2024-02-12T16:57:21Z")

</div>

I am trying to compare the performance of SequenceNetwork and TimeSeriesTransformer for summary networks for time series data (e.g. the simulation model generates 3 time series of 300 time points each), but have a hard time setting up the TimeSeriesTransformer correctly and can’t find any examples online. Any tips or examples on how to set up this network structure effectively? Thanks!

---

<div class="post-metadata">

**Author:** ![marvinschmitt](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/marvinschmitt/32/98_2.png) [@marvinschmitt](https://discuss.bayesflow.org/u/marvinschmitt)\
**Post date:** [February 12, 2024, 7:48pm UTC](https://discuss.bayesflow.org/t/setting-up-timeseriestransformer/61/2 "2024-02-12T19:48:35Z")

</div>

Hi,

What’s the exact output shape of your simulator? If I understand you correctly, I would assume (batch\_size, 300, 3).

Do the 300 time steps of all three time series correspond to the same points in time? In other words: Does the 50th time step of all time series happen simulateneously in the real world?

---

<div class="post-metadata">

**Author:** ![hazhir](https://avatars.discourse-cdn.com/v4/letter/h/edb3f5/32.png) [@hazhir](https://discuss.bayesflow.org/u/hazhir)\
**Post date:** [February 12, 2024, 9:19pm UTC](https://discuss.bayesflow.org/t/setting-up-timeseriestransformer/61/3 "2024-02-12T21:19:17Z")

</div>

Yes, exactly as you guessed, think about data on times 1…300 for three different variables in the model, e.g. cases, deaths, and hospitalizations in an epidemic. Thanks Marvin!

---

<div class="post-metadata">

**Author:** ![marvinschmitt](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/marvinschmitt/32/98_2.png) [@marvinschmitt](https://discuss.bayesflow.org/u/marvinschmitt)\
**Post date:** [February 12, 2024, 9:53pm UTC](https://discuss.bayesflow.org/t/setting-up-timeseriestransformer/61/4 "2024-02-12T21:53:53Z")

</div>

Thanks for the info! That means you just have to add a (linear) time embedding to the last axis of the data, so that you get (batch\_size, 300, 4).

You can do that in the configurator so that your generative model remains a pure implementation of your scientific model.

I’ll send you a code snippet, probably by tomorrow.

---

<div class="post-metadata">

**Author:** ![marvinschmitt](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/marvinschmitt/32/98_2.png) [@marvinschmitt](https://discuss.bayesflow.org/u/marvinschmitt)\
**Post date:** [February 13, 2024, 10:31am UTC](https://discuss.bayesflow.org/t/setting-up-timeseriestransformer/61/5 "2024-02-13T10:31:17Z")

</div>

```auto
def configurator(input_dict):
   out_dict = {}

   # prior draws are the parameters we want to estimate
   theta = input_dict['prior_draws']
   out_dict['parameters'] = theta

   x = input_dict['sim_data']
 
   # add time encoding to the data x
   batch_size, num_timesteps, data_dim = x.shape
   time_encoding = np.linspace(0, 1, num_timesteps)
   time_encoding_batched = np.tile(time_encoding[np.newaxis, :, np.newaxis], (batch_size, 1, 1))

   out_dict['summary_conditions'] = np.concatenate((x, time_encoding_batched), axis=-1)

   return out_dict

```

---

<div class="post-metadata">

**Author:** ![hazhir](https://avatars.discourse-cdn.com/v4/letter/h/edb3f5/32.png) [@hazhir](https://discuss.bayesflow.org/u/hazhir)\
**Post date:** [February 13, 2024, 12:04pm UTC](https://discuss.bayesflow.org/t/setting-up-timeseriestransformer/61/6 "2024-02-13T12:04:27Z")

</div>

Thanks so much Marvin; I will try this. Also the TimeSeriesTransformer seems to have a couple of optional settings (attention\_settings and dense\_settings); do you typically find value in customizing/fine-tuning those?

---

<div class="post-metadata">

**Author:** ![KLDivergence](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/kldivergence/32/15_2.png) [@KLDivergence](https://discuss.bayesflow.org/u/KLDivergence)\
**Post date:** [February 13, 2024, 1:34pm UTC](https://discuss.bayesflow.org/t/setting-up-timeseriestransformer/61/7 "2024-02-13T13:34:29Z")

</div>

You can try tuning those if default performance is not satisfying. The most useful hyperparameters in my mind would be the dimensionality of the template `template_dim`, the actual number of summary statistics, `summary_dim`, and the number of heads for Multihead Attention.

---

<div class="post-metadata">

**Author:** ![hazhir](https://avatars.discourse-cdn.com/v4/letter/h/edb3f5/32.png) [@hazhir](https://discuss.bayesflow.org/u/hazhir)\
**Post date:** [February 14, 2024, 2:26pm UTC](https://discuss.bayesflow.org/t/setting-up-timeseriestransformer/61/8 "2024-02-14T14:26:50Z")

</div>

Thanks; I will experiment with those. In the meantime, there seems to be some inconsistency in the TimeSeriesTransformer when setting ‘bidirectional’ flag to True, leading to an error about mismatch of vector sizes somewhere in the network setup.

---

<div class="post-metadata">

**Author:** ![KLDivergence](https://yyz1.discourse-cdn.com/flex007/user_avatar/discuss.bayesflow.org/kldivergence/32/15_2.png) [@KLDivergence](https://discuss.bayesflow.org/u/KLDivergence)\
**Post date:** [February 14, 2024, 10:46pm UTC](https://discuss.bayesflow.org/t/setting-up-timeseriestransformer/61/9 "2024-02-14T22:46:54Z")

</div>

Nice spot! This has been fixed in the Development branch, but hasn’t made it into the official new release yet. You can run the following to get the development version without cracking up the already installed dependencies:

```auto
pip uninstall bayesflow
pip install git+https://github.com/stefanradev93/bayesflow@Development --no-deps

```