Cross attention for conditioning image generation

General
1

Hi,

I have been working in a project in which I have to predict an image from an input with shape (timesteps, quantity). “timesteps” has a size of about 2000, while “quantity” about 500.
In order to compress the input, I calculated the root mean square along the first entry, getting a single vector with shape ~500.
I have seen that in the “inference network” page in the Bayesflow documentation it is written that “Image-generation subnets concatenate the condition channel-wise with the target image”.
What I could do I think is to compress the “quantity” vector with, for example, a MLP, then reshape it for broadcasting to the image shape.
However, I was wondering if cross-attention conditioning (used for example in Stable Diffusion) can be used in Bayesflow.

Thank you

2

Hi Corte, this can work, but it will require a custom subnet backbone that combines the time series with the diffused image in the particular way you want. You can definitely reuse the existing attention machinery for it. We also have a diffusion transformer, but that one does not operate on images out of the box.