IntenseWebs/openvino
3
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

update-notebooks (#19337)

Browse Source
This commit is contained in:
Sebastian Golebiewski authored and GitHub committed 2023-08-22 15:37:37 +02:00
1 parent de65abc6b3
commit 6cd70af7c2
9 files changed
+1743 -3
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/notebooks/250-music-generation-with-output.rst
+574
View File
@@ -0,0 +1,574 @@
Controllable Music Generation with MusicGen and OpenVINO
========================================================
.. _top:
MusicGen is a single-stage auto-regressive Transformer model capable of
generating high-quality music samples conditioned on text descriptions
or audio prompts. The text prompt is passed to a text encoder model (T5)
to obtain a sequence of hidden-state representations. These hidden
states are fed to MusicGen, which predicts discrete audio tokens (audio
codes). Finally, audio tokens are then decoded using an audio
compression model (EnCodec) to recover the audio waveform.
.. figure:: https://user-images.githubusercontent.com/76463150/260439306-81c81c8d-1f9c-41d0-b881-9491766def8e.png
:alt: pipeline
pipeline
`The MusicGen model <https://arxiv.org/abs/2306.05284>`__ does not
require a self-supervised semantic representation of the text/audio
prompts; it operates over several streams of compressed discrete music
representation with efficient token interleaving patterns, thus
eliminating the need to cascade multiple models to predict a set of
codebooks (e.g. hierarchically or upsampling). Unlike prior models
addressing music generation, it is able to generate all the codebooks in
a single forward pass.
In this tutorial, we consider how to run the MusicGen model using
OpenVINO.
We will use a model implementation from the `Hugging Face
Transformers <https://huggingface.co/docs/transformers/index>`__
library.
**Table of contents**:
- `Requirements and Imports <#prerequisites>`__
- `Original Pipeline Inference <#musicgen-in-hf-transformers>`__
- `Converting the Models to OpenVINO IR <#convert-models-to-openvino-intermediate-representation-ir-format>`__
- `Convert Text Encoder <#convert-text-encoder>`__
- `Convert MusicGen Language Model <#convert-musicgen-language-model>`__
- `Convert Audio Decoder <#convert-audio-decoder>`__
- `Embedding the Converted Models into the Pipeline <#embedding-the-converted-models-into-the-original-pipeline>`__
- `Run Gradio App <#try-out-the-converted-pipeline>`__
Prerequisites `⇑ <#top>`__
###############################################################################################################################
Install requirements `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
!pip install -q "openvino==2023.1.0.dev20230811"
!pip install -q torch onnx gradio
!pip install -q transformers
Imports `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
from collections import namedtuple
import gc
from pathlib import Path
from typing import Optional, Tuple
import warnings
from IPython.display import Audio
from openvino import Core, convert_model, PartialShape, save_model, Type
import numpy as np
import torch
from torch.jit import TracerWarning
from transformers import AutoProcessor, MusicgenForConditionalGeneration
from transformers.modeling_outputs import BaseModelOutputWithPastAndCrossAttentions, CausalLMOutputWithCrossAttentions
# Ignore tracing warnings
warnings.filterwarnings("ignore", category=TracerWarning)
MusicGen in HF Transformers `⇑ <#top>`__
###############################################################################################################################
To work with
`MusicGen <https://huggingface.co/facebook/musicgen-small>`__ by Meta
AI, we will use `Hugging Face Transformers
package <https://github.com/huggingface/transformers>`__. Transformers
package exposes the ``MusicgenForConditionalGeneration`` class,
simplifying the model instantiation and weights loading. The code below
demonstrates how to create a ``MusicgenForConditionalGeneration`` and
generate a text-conditioned music sample.
.. code:: ipython3
# Load the pipeline
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small", torchscript=True, return_dict=False)
In the cell below user is free to change PyTorch model inference device
and the desired music sample length.
.. code:: ipython3
device = "cpu"
sample_length = 8 # seconds
n_tokens = sample_length * model.config.audio_encoder.frame_rate + 3
sampling_rate = model.config.audio_encoder.sampling_rate
print('Sampling rate is', sampling_rate, 'Hz')
model.to(device)
model.eval();
.. parsed-literal::
Sampling rate is 32000 Hz
Original Pipeline Inference `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Text Preprocessing prepares the text prompt to be fed into the model,
the ``processor`` object abstracts this step for us. Text tokenization
is performed under the hood, it assigning tokens or IDs to the words; in
other words, token IDs are just indices of the words in the model
vocabulary. It helps the model understand the context of a sentence.
.. code:: ipython3
processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
inputs = processor(
text=["80s pop track with bassy drums and synth"],
return_tensors="pt",
)
audio_values = model.generate(**inputs.to(device), do_sample=True, guidance_scale=3, max_new_tokens=n_tokens)
Audio(audio_values[0].cpu().numpy(), rate=sampling_rate)
.. raw:: html
<audio controls="controls" >
<source src="data:audio/wav;base64,UklGRiTQBwBXQVZFZm10IBAAAAABAAEAAH0AAAD6AAACABAAZGF0YQDQBwA1+gX4WQK5AEEA4AS5+ALwxvsR/4wRhw8l8gHg2dZN/sn2xOOwAJsO7PrV7gj8/+1q+Sj4N/kgETcJywgbFWT7wgSgGSb9YvDM89z3OfzsA9Xnls5X3LEf7iDp/mn5Z+FQAAgF/O2L6nT8JAjrAYoEK+046Br5Q+kO/uEVEe4N6ffyifIK9kHzPgc1Du3meuZX6CcLJivrBb0HPg0R9uHxfxMCBA/V7fs1CRzv/fCY+jYMUfh55LPh+AczDnrsNe+28/v7APhwD00jlQmu5ZH0wAbL/aIA2vqB9/wBHhUSGmoUigseEFITZgJy8U/6gRrYIuwnDwxNDRwn/w35FJ4fuxadDpMNzRBxEZQkRRteAY8XghyDFdQeqi1cLBgOeRR6NkMjhSTHOuselhepA8P8BxbhOG8gavo2HaoPnw/vK1UivxJqIok1N0NwMS0L0heVDN4O2yMOE4kxKEA7GjgXFyJ2IC0g7R+nGYcTchfNIZEmYigjFw8HFhm6F1AXeCUhIAUTIA+pJ/EWVw6/HXYPxBJGGYgaWCJDGmgKrxCBE636f/KkCn8JkAy/CbACXwLXAMQEuv5IAXT4DBD1EuUCtQYi9acMSgjx5+Dxo/c3B0cKAw4RDv3qINpC6mXx5P1yAJnvxusq1lDrLw+kBMn4Bva8+RTy/uvJ7ePgmeDy45Ld4+Ge6njtleiq7xniyezm7vLf6e8D6DPo1N3o7E73kNuf30oBGe1Y3UTwH+AF5yjStOC7BLjultFD0obVAsYf8+cKNQw8AdDfAggs9oq2mMRM1qbtPRGC/RXUT8+o65z2ZNGi11IE2fDH6KXsLedY+sj1Ova08Aroo/lY5eLbCfWv8dHhqeEw7yLqLeeD+U4ERezz2IfdbfGJ/wwL1v2g8VP2/gFD9nPwNwbX4x/jhfZT+Fb0EOSn58L3mfszBnT4QOpX9BrwfPtQ+cXs9PH+8oDenuJzA54SxfNQ65L2nfKuGosKfevz+b8MKxQR/aD9O/wz7r/25vT95NT+ZBG487P+GBaE7fzi2/VR6Ov36Qbp9tH5E/p+/qAHH/769IL2rPuY/EMJVQdo8tH8ufw09Nn7ivDY7BD0hPd2/uECUAEz4v/XpAHFBYT2OO484fj3VQYtFkockOwJ39DqtPl39ZLg8e4q8Fjnifl77lzu6gwvDjMMdvsR5tvmhPvrAAbvmujx+OAPh/ZF8J0ApPc86hz31Ag++Vb9Tu5+5+MFPfoD+RwIUPd++p3+UfhPD9IRyBAvEeP6/vOR/hgKlQ0SBnYEgQ4PEAsdWQ+k8E0G6gkkAlj+ThinI10DhwRPAlL7ggf0GbUS1hLZEX0HLAq6DkEa8gPj/uAoBya0G8gc5xRwBgYJpS5SO1EhzQo6ImgobCjBN3g1niUeHXYW8Be0J0cUXxg5KcMTTBRCJ2Uy6jMMIawg0BiwIqQ17jNxMvwqFCcCLtg2ySIzKYEuMB1ULcw8USJNHVkyXCmXHJAnUS3eKXg1YzMyJ4AqOikMIpwgvxwKIjMbRxlaLhczuyajGk4QdBiVIIslqyEWAh8RBB1vKUooFQxEBEn7cQ/sE2kg2DDjM8kmRQtjBKkGDBUpC4YDvgicBXwOlxX3H5waSALD9lr8UADK/Jj7bgGoCnASkAeE+/sROR6nFVIJ6wcXB/T+oQMc/Qr7tf+v9av1dvrY990KGQp6960LlAf99doAju4F7TgDLfae+KP6WPOM+6T+7/v2AA/+bt9K3Qj4SwL8/Sv46BHy+PnT1OnM7TX9kwHk7TvfBNe722LtFu0T7VYAdvr47PblzuHm6EDoseEj/F8H7O3W7Z3x+N/H5S/orOCR+OYCNPo3+5nuU+ns4snz9xOhC5v+jOX/8okJVfD07rbviunz8WL31/oo4kDi4/ND/OkGl/lVAh3vK+M9AukICu+j6gcDWgbt+c700AZlABUC8xQi96Hr4wbvEQr76N6o4QnwbvusAy0Dzgo0EPLzJOPJ5AHpyv/AE7X+QfMb+YPsQPoH91HrzfQ47hDkC/rr+dvjVvk190n26fx37U7vPOUT7k78YO8h8JTv8ucy8T728+oa5XDwCfLU8QoKDgWP+fXyoe/X7I7fGe2d9Aj2nvBP71DrO+eH8Pflcd1/2+LonvIY9jMCWO3j3UjXedt04//h7uo48RTvzup95nnwwfaD3lvmqd4g3pPthtkq9BYFUPnw6svZ7+Zu9P7qQd1237HgmONt2TLWL+9e6rnmC/qP+PT9NwMP+xH59O0O76L06+v+8rQB0wOe/AT9IPXX+Ov4yvMn+73wmvdaCC4LKvxvAmv4vN+sCWQd/QnIDjQMk/7rBYIJjAExBfwHSghkCloNRgvuCxoFMv5NEe4UJBFXFuYO/wX8EPsgQxsEE/QQmwq+FvwdGg/KHKUcBA5EFskKXRKlGlwR3BILDQAl8in3BnP/yh23IVoQFBAjEvccFAgL/qMMQBd7FmYF+hSUIt4ePCkyH34XWRi4CkQHFgl7FswdCR2dE8YGOgw0CLMOhh1yCOv4OgODDgwWbCJRGJzzg/stAfv4/f8DDc0RiQWrCHQKHhC//MDzKQ3uAvACPghnAY8Ccwc+Bbj2C/rbBgESSQnnB3AHOv1b/8/0zPuFCFH9ufVw/vUClACm/O/5HgDWAn3zU+yZ9m373Oxe3rnpeO3p4p7ml/NCBuD/GvGB6bLoqvOp5hzq/fmq8TDvTPbe+In2M/Ck+Vz97/TZ7LrrPPR0+Hfv6Off9LjwJ+NW38HpaPBe7Qnzk+6H8+rtIeVvAHQBauzj7S/vvvEN/vH3Ru037C/nZvb4/jX+Rf6a7ar5mfrw4Zvk1u0V+1fweOoB7Xjm+Px//kH0tvXB7sDwu/Z/51LlHfao8ujtvvY68SfsqfCv3+/fHuLO7N0LaQcUBHH8tu5D/Fn3mfOcBKgDK/wK9P/uVPUj+xb8dv22+SP5D/Rj8Ff8+Pw+/If4Ue+f86D6n/xO/sv0h+1n6zX34fpf/tsA6u1L+I4Btffx/uMDKfFd+bIFg/+8A4f55PslCb/toOjF65frmgg2DZcCXvI35EDkFPAkBL70E+tvBwT+i+0X9ML4cP6aBxMFXvof+/n30PTJ9GEDYQeq8zbxj/rs/R38l+ni5iP2XP2FAZH1tPFv+9AA1vd2B9wUn/Iu5GcF4BVJ/vT0Pviu98gAxwIL+ZoB0PT/7pb91wCRDtQHI/6SB6kCAwXeAQD61QizB9/2qvWnB60KzAlaCwkT5yHxF/4JRAwxCgsQawtcD7UVtAy7GnIXKxIpG+sYNgtm/nQEfxBVGKkTMxamGoMPZx9eJFkY+RmqEHIZLh7nH1kgzBOHG+Ia/RoBKDwuOS3TJBsicRplFHkocCxOJmw1bjKWI4kZ/RyHNWMtrSeSNcArDyfiIbYg1yyYJ+wgxCFCLskxiSDmGxskSyYGHQAv4zlmNE8uKSC9LCUw9ij4KRYaWSL4JM0QPhwKIDocJhZlFA8iZyaSHrMWaxj0EawVRyKoFX0VfiRNHqUZQhXZBpQJiBRpHnkWnwVTBgYOihPuFDsXqhUbC4wINxR/E10RNRRqCqgH3AvFDIgRehI/FTMQxA7rF28Pcf0CAVwN4QyaEr4PMf9C+ZAGpQqlBaYCewCLCS0A+PGnCy8d2AzfAbn7sQWSDO4A8vOh+kkJ0PpT/+EEg/brBzMGEO966t/xR/93/3D8RgVg90XrB/9PAiD9ePQ1+Sn6rvbT/rv59/I588MB1/pm7uX1VPdmACwFVPXb8G32hfEh96v72usw6ofzTvon/GP7d/bF8Or2wfNz6EH2CQBk6c/ot+iC6yn1Ru67AKD6Be5L/H8DkgTb5pfkdvgt7M7m2O2a/UT9JPG3+p/6cPhk9iLvtPju+s745va88h7ulfG/7DPngPND7zXxX/Bl9vj9Ju7K8U7tcO2r+yLyRO5B7E7mseJl7SruKeaN9TTwpOy687LnderP6T3jgesN72jpYucr8JTtv/a59yXnwO266sPmous38u7yOOSS8dfsrd/b4S/fnvDo6TzlLent6U/xtO668tfvJO+i7aTkQOom62Xoge5w72brLulj7BfrCueH4iflZOaQ6JvrJ+gz5ZTe/ehs5zbe6+EX4Kjqo+t75h/tkua13WXbxeCn7CTx6uwa5WXo7e+p53TpLN6x1k3uKe/06lXeENRW367tJe1u73jzJef189n0HvOq9q/x9fqx8oj3nfzJ9DL5VfJUAO4HN/x09WzvVv8ECukAVALu+1/78ALJ/Y0HzQUuBpwLOAsuA2z+RwoG/y4ElAqhAbYHXwiyDMoSuw4sBxQJWRA1FtwajhgaHpEfExFsERYXTRX5IWMdxhL7E8USCB7BIU8dqRUUF6oYFxt+FskJNhCBFTIhVx4dF8wYbxgAGRoK2Q4yHr4UaxTSD4ABtQ9pGYkVkxTBEEwTxhVfEfYOFhjwHPUW6Q4BDz4SgRjnE6YIRw3eBPIHRBIaCGAFNQaqBQcINgLf/rUENwT/BEUGzgZzA6n7efes/NcIHgUYB3YLMwVlB0sCYf98AwsB//1H/TsGMRFLD9QDPfxB8h30O/4LADP5De298H70UP8FABPwDvYA8WfyL/sy8FbxC/Bu7aT52/YI9Nvttt/v6Ff3cfFR7x30Z+r09jnyruWw8sno9O3N+TTsM/Bf99PrW+tQ9iz2mvAU9AP0Ounx7N70Ie6Q8O3wge8Q9ALyXe4i9G35tPQ094nz0eeq7rTzSvQI9bHxHvUIBqUC0OtF7b3tLfUJ9wjt6u907avpLegP7jH8J/xh/Fz1Red48Eb21PmM9P7tJ/PR6w7wVvS77s3xOu8m82j4cPBi8ELzm+8h93j00+xI9UDxFvyvAk/0pvmA927xvPk27xTutvZrLine truncated
Your browser does not support the audio element.
</audio>
Convert models to OpenVINO Intermediate representation (IR) format `⇑ <#top>`__
###############################################################################################################################
Model conversion API enables direct conversion of PyTorch models. We
will utilize the ``openvino.convert_model`` method to acquire OpenVINO
IR versions of the models. The method requires a model object and
example input for model tracing. Under the hood, the converter will use
the PyTorch JIT compiler, to build a frozen model graph.
The pipeline consists of three important parts:
- The `T5 text encoder <https://huggingface.co/google/flan-t5-base>`__
that translates user prompts into vectors in the latent space that
the next model - the MusicGen decoder can utilize.
- The `MusicGen Language
Model <https://huggingface.co/docs/transformers/model_doc/musicgen#transformers.MusicgenForCausalLM>`__
that auto-regressively generates audio tokens (codes).
- The `EnCodec model <https://huggingface.co/facebook/encodec_24khz>`__
(we will use only the decoder part of it) is used to decode the audio
waveform from the audio tokens predicted by the MusicGen Language
Model.
Let us convert each model step by step.
0. Set Up Variables `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.. code:: ipython3
models_dir = Path("./models")
t5_ir_path = models_dir / "t5.xml"
musicgen_0_ir_path = models_dir / "mg_0.xml"
musicgen_ir_path = models_dir / "mg.xml"
audio_decoder_onnx_path = models_dir / "encodec.onnx"
audio_decoder_ir_path = models_dir / "encodec.xml"
1. Convert Text Encoder `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The text encoder is responsible for converting the input prompt, such as
“90s rock song with loud guitars and heavy drums” into an embedding
space that can be fed to the next model. Typically, it is a
transformer-based encoder that maps a sequence of input tokens to a
sequence of text embeddings.
The input for the text encoder consists of a tensor ``input_ids``, which
contains token indices from the text processed by the tokenizer and
``attention_mask`` that we will ignore as we will process one prompt at
a time and this vector will just consist of ones.
We use OpenVINO Converter (OVC) below to convert the PyTorch model to
the OpenVINO Intermediate Representation format (IR), which you can
infer later with `OpenVINO
runtime <https://docs.openvino.ai/latest/openvino_docs_OV_UG_OV_Runtime_User_Guide.html#doxid-openvino-docs-o-v-u-g-o-v-runtime-user-guide>`__
.. code:: ipython3
if not t5_ir_path.exists():
t5_ov = convert_model(model.text_encoder, example_input={'input_ids': inputs['input_ids']})
save_model(t5_ov, t5_ir_path)
del t5_ov
gc.collect()
2. Convert MusicGen Language Model `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
This model is the central part of the whole pipeline, it takes the
embedded text representation and generates audio codes that can be then
decoded into actual music. The model outputs several streams of audio
codes - tokens sampled from the pre-trained codebooks representing music
efficiently with a lower frame rate. The model employs innovative codes
intervaling strategy, that makes single-stage generation possible.
On the 0th generation step the model accepts ``input_ids`` representing
the indices of audio codes, ``encoder_hidden_states`` and
``encoder_attention_mask`` that were provided by the text encoder.
.. code:: ipython3
# Set model config `torchscript` to True, so the model returns a tuple as output
model.decoder.config.torchscript = True
if not musicgen_0_ir_path.exists():
decoder_input = {
'input_ids': torch.ones(8, 1, dtype=torch.int64),
'encoder_hidden_states': torch.ones(2, 12, 1024, dtype=torch.float32),
'encoder_attention_mask': torch.ones(2, 12, dtype=torch.int64),
}
mg_ov_0_step = convert_model(model.decoder, example_input=decoder_input)
save_model(mg_ov_0_step, musicgen_0_ir_path)
del mg_ov_0_step
gc.collect()
On further iterations, the model is also provided with a
``past_key_values`` argument that contains previous outputs of the
attention block, it allows us to save on computations. But for us, it
means that the signature of the model’s ``forward`` method changed.
Models in OpenVINO IR have frozen calculation graphs and do not allow
optional arguments, that is why the MusicGen model must be converted a
second time, with an increased number of inputs.
.. code:: ipython3
# Add additional argument to the example_input dict
if not musicgen_ir_path.exists():
# Add `past_key_values` to the converted model signature
decoder_input['past_key_values'] = tuple(
[(
torch.ones(2, 16, 1, 64, dtype=torch.float32),
torch.ones(2, 16, 1, 64, dtype=torch.float32),
torch.ones(2, 16, 12, 64, dtype=torch.float32),
torch.ones(2, 16, 12, 64, dtype=torch.float32),
)] * 24
)
mg_ov = convert_model(model.decoder, example_input=decoder_input)
Moreover, the ``past_key_values`` argument is passed as
``Tuple[Tuple[torch.tensor]]`` which is a hard call for the converter.
The converter flattens these tuples, but have hard time identifying the
correct shapes for the tensors. The code below goes over raveled
``past_key_values``-related model inputs and sets the correct shape and
type for them.
.. code:: ipython3
if not musicgen_ir_path.exists():
for input in mg_ov.inputs[3:]:
input.get_node().set_partial_shape(PartialShape([-1, 16, -1, 64]))
input.get_node().set_element_type(Type.f32)
mg_ov.validate_nodes_and_infer_types()
save_model(mg_ov, musicgen_ir_path)
del mg_ov
gc.collect()
3. Convert Audio Decoder `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The audio decoder which is a part of the EnCodec model is used to
recover the audio waveform from the audio tokens predicted by the
MusicGen decoder. To learn more about the model please refer to the
corresponding `OpenVINO
example <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/234-encodec-audio-compression>`__.
The audio decoder computation graph contains an LSTM sub-network which
currently should be converted through ONNX. To do this we create a
wrapper class with its ``forward()`` method calling
``encodec.decode()``.
.. code:: ipython3
if not audio_decoder_onnx_path.exists():
class AudioDecoder(torch.nn.Module):
def __init__(self, model):
super().__init__()
self.model = model
def forward(self, output_ids):
return self.model.decode(output_ids, [None])
audio_decoder_input = {'output_ids': torch.ones(1, 1, 4, n_tokens - 3, dtype=torch.int64),}
with torch.no_grad():
torch.onnx.export(
model=AudioDecoder(model.audio_encoder),
args=audio_decoder_input,
f=audio_decoder_onnx_path,
input_names=['output_ids',],
output_names=['decoded_audio'],
dynamic_axes={
'output_ids': {3: 'sequence_length'},
'decoded_audio': {2: 'audio_values'}
}
)
.. parsed-literal::
/home/idavidyu/.virtualenvs/test/lib/python3.10/site-packages/torch/onnx/symbolic_opset9.py:4476: UserWarning: Exporting a model to ONNX with a batch_size other than 1, with a variable length with LSTM can cause an error when running the ONNX model with a different batch size. Make sure to save the model with a batch size of 1, or define the initial states (h0/c0) as inputs of the model.
warnings.warn(
.. parsed-literal::
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================
Now we can convert the frozen ONNX computation graph to OpenVINO IR.
.. code:: ipython3
# Now we can convert the model to OpenVINO IR
if not audio_decoder_ir_path.exists():
audio_decoder_ov = convert_model(str(audio_decoder_onnx_path))
save_model(audio_decoder_ov, audio_decoder_ir_path)
del audio_decoder_ov
gc.collect()
Embedding the converted models into the original pipeline `⇑ <#top>`__
###############################################################################################################################
OpenVINO™ Runtime Python API is used to compile the model in OpenVINO IR
format. The
`Core <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.Core.html>`__
class provides access to the OpenVINO Runtime API. The ``core`` object,
which is an instance of the ``Core`` class represents the API and it is
used to compile the model.
.. code:: ipython3
core = Core()
Select inference device `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Select device that will be used to do models inference using OpenVINO
from the dropdown list:
.. code:: ipython3
import ipywidgets as widgets
DEVICE = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value='AUTO',
description='Device:',
disabled=False,
)
DEVICE
.. parsed-literal::
Dropdown(description='Device:', index=2, options=('CPU', 'GPU', 'AUTO'), value='AUTO')
Adapt OpenVINO models to the original pipeline `⇑ <#top>`__
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Here we create wrapper classes for all three OpenVINO models that we
want to embed in the original inference pipeline. Here are some of the
things to consider when adapting an OV model: - Make sure that
parameters passed by the original pipeline are forwarded to the compiled
OV model properly; sometimes the OV model uses only a portion of the
input arguments and some are ignored, sometimes you need to convert the
argument to another data type or unwrap some data structures such as
tuples or dictionaries. - Guarantee that the wrapper class returns
results to the pipeline in an expected format. In the example below you
can see how we pack OV model outputs into special classes declared in
the HF repo. - Pay attention to the model method used in the original
pipeline for calling the model - it may be not the ``forward`` method!
Refer to the ``AudioDecoderWrapper`` to see how we wrap OV model
inference into the ``decode`` method.
.. code:: ipython3
class TextEncoderWrapper(torch.nn.Module):
def __init__(self, encoder_ir, config):
super().__init__()
self.encoder = core.compile_model(encoder_ir, DEVICE.value)
self.config = config
def forward(self, input_ids, **kwargs):
last_hidden_state = self.encoder(input_ids)[self.encoder.outputs[0]]
last_hidden_state = torch.tensor(last_hidden_state)
return BaseModelOutputWithPastAndCrossAttentions(last_hidden_state=last_hidden_state)
class MusicGenWrapper(torch.nn.Module):
def __init__(self, music_gen_lm_0_ir, music_gen_lm_ir, config, num_codebooks, build_delay_pattern_mask,
apply_delay_pattern_mask):
super().__init__()
self.music_gen_lm_0 = core.compile_model(music_gen_lm_0_ir, DEVICE.value)
self.music_gen_lm = core.compile_model(music_gen_lm_ir, DEVICE.value)
self.config = config
self.num_codebooks = num_codebooks
self.build_delay_pattern_mask = build_delay_pattern_mask
self.apply_delay_pattern_mask = apply_delay_pattern_mask
def forward(
self,
input_ids: torch.LongTensor = None,
encoder_hidden_states: torch.FloatTensor = None,
encoder_attention_mask: torch.LongTensor = None,
past_key_values: Optional[Tuple[torch.FloatTensor]] = None,
**kwargs
):
if past_key_values is None:
model = self.music_gen_lm_0
arguments = (input_ids, encoder_hidden_states, encoder_attention_mask)
else:
model = self.music_gen_lm
arguments = (input_ids, encoder_hidden_states, encoder_attention_mask, *past_key_values)
output = model(arguments)
return CausalLMOutputWithCrossAttentions(
logits=torch.tensor(output[model.outputs[0]]),
past_key_values=tuple([output[model.outputs[i]] for i in range(1, 97)]),
)
class AudioDecoderWrapper(torch.nn.Module):
def __init__(self, decoder_ir, config):
super().__init__()
self.decoder = core.compile_model(decoder_ir, DEVICE.value)
self.config = config
self.output_type = namedtuple("AudioDecoderOutput", ["audio_values"])
def decode(self, output_ids, audio_scales):
output = self.decoder(output_ids)[self.decoder.outputs[0]]
return self.output_type(audio_values=torch.tensor(output))
Now we initialize the wrapper objects and load them to the HF pipeline
.. code:: ipython3
text_encode_ov = TextEncoderWrapper(t5_ir_path, model.text_encoder.config)
musicgen_decoder_ov = MusicGenWrapper(
musicgen_0_ir_path,
musicgen_ir_path,
model.decoder.config,
model.decoder.num_codebooks,
model.decoder.build_delay_pattern_mask,
model.decoder.apply_delay_pattern_mask
)
audio_encoder_ov = AudioDecoderWrapper(audio_decoder_ir_path, model.audio_encoder.config)
del model.text_encoder
del model.decoder
del model.audio_encoder
gc.collect()
model.text_encoder = text_encode_ov
model.decoder = musicgen_decoder_ov
model.audio_encoder = audio_encoder_ov
We can now infer the pipeline backed by OpenVINO models.
.. code:: ipython3
processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
inputs = processor(
text=["80s pop track with bassy drums and synth"],
return_tensors="pt",
)
audio_values = model.generate(**inputs.to(device), do_sample=True, guidance_scale=3, max_new_tokens=n_tokens)
Audio(audio_values[0].cpu().numpy(), rate=sampling_rate)
.. raw:: html
<audio controls="controls" >
<source src="data:audio/wav;base64,UklGRiTQBwBXQVZFZm10IBAAAAABAAEAAH0AAAD6AAACABAAZGF0YQDQBwCi/BH8RQAHAF//0gCW/Kr4jP02AJgIJQik/CL1efO/BBEApfdiAs0I/gAW+/X/7/p//hr9bf3VBUYD/wIxBhL+lAMYClQAw/w6/O8BfQawBvL7hfP59zAR/hTJBmcCi/osBjIG2v6B/uIDEAheBoIIQgCA/VkB3v3nB5QOzgG6/pcCYwLTAPMChQuQD4EAffnO+tYIchKkB4sHhwksAmMA3glABKD3IwQPBwj/5wFRBu0LHAJ3/Lv9aAgqDIX+CAG4AuD/IwBwCB8QxgUj96L9MQY6BBwB4Ptb+2MBFghvBGADEwWqCCsHbPr98ur1PwJvBjcFif3AAgAH0fweAQz/Xfw0AA0BUwFLAeEDUQCf+OX+6AEE/eL8qgFn/Wr19PjgAf3/j/5gBKr+9foU8QXvOvcZAfj8QfEq+n/4OPdM+uH5z/Pd9zYBjQZTAebypfMj8NPyfvmy9EP/iQSZ+Hb3GPhQ+eL8xP3390PzWvew/iQARfwH+l/3A/rC99j5fP8p/974XvRkA979tPuMAE/3kfi3AMEDLQYEBEP8EAIABJr3ePG3/NMCsgMUA84E3ABgAZABTP1M/tL7GwbOCeUAhwCvAK8GAAa2+Gr+tAVCCFIHqQhcBbL8oPwUALQE9gxlDcz9VvjE/Z8HjA0lDboIrwopBnAAZwWcAxMDaQYwBl8DSQVVCawJgwVRBWwFagXCAD8EFAtSCXoFP//1BqcN6gNG/Y8M8AyuB7cID/8uA10BZwV5Dy0LugR0ABT5iPc0BeAFlQaLA8r8cwpWCCgBagLbACr8FAFcACD1mflaAKUAp/stAnoE/voj+RH8iQGHBIMALP1w++X1jfhA+VD6gQE1/d373AFb+471Z/qM/63+wve09mD+QwO+/x78lvoPATH/pf5s+t75IANp+j75qv/2+RD2h/bz+aYCHgStAl8AsvxP90L02vbLAHwEoAfM/k3v6P6zCXgBcvJU+Sr+bfV+A7YC8QEP/ZT5Rv5R/g8Def0B98H5BAXPAtT5bP2kAKwBggAs+2z8lPgQ+ZsFjQb8/sb8tf+y/c36Uf/0//n7//6JBaEIqAagBcsE1AOJALb95wNcCYICh/jT/pv9VPgMBOoQKg+C/uH4/gULCjQKHAM6/noElf+SBBIDyQBSDTwHTvws+wcArwphChQC7Pqu+lEHUgt5BjABCgDdA+v9hfnS/MP/UP32BP8IgwBF/4j/zAPu/SD9KP6lAfILvQH59zX8hgh8Ak74l/qgAWEAdf8oCUgC5/7b/zP88vjT/D4DzQK/CrgLnwC+9gb54vwc+t/3w/sSBKMEvgDP/Un99P80AYX/dvxG9UHypfN2+50GiANK/y4DaPoh9lH3kgCYBOL7Bv/y9WfzQfnM/0oCAv3YAXICKPpO8njx8/Wn/z4ESwW7BWX6efZZ+wj7Ovuk/OIA3PeA+eIG1/6T/Rz+1v24BYwFh/v//KkA5PosApIJCwGTADoAF/wJA+0K5gf0/2sCrACuA8IDM/zY/9T/HAPnD+0O6v7++okFhAnKCZMIlAT4/Jr8pQK//Sv6bgKsDbYGKAJM/BT6jv6pAb0OAAQ//sT9VAISCqwF9P/E+qABSQTN/yz80Po0/k0H5g5eCOr9BvmH9/n/Cf2Z/HcFCP7jApYFrP4HBaUH+AK5+0L+GwUyAHX61P1L//cBBQK4+s8AuQUH/XX8Kv4o/mYH0gUyA6cF2P6M+NX0qwCZBE0BLwgwCxUEw/qi/kb+cfpk9u/7qv7z9oD7bQYiCGsBkfsz+SP+x/6EAZcAUP+o/PX62vrE9YL5eP/TCysKNf6F/Wv9E/mz+if9Kvts/zYEUgFV+rn4agFjA/oBUgJq/nADP//b9L79GART+9v76f0k/wIDcP8A/Jj56v1++rb1UwH3CjEFiPwPAcb+2/bH/8cH6QNwA1z5mfJDAGwGTwQL/Wf11Pt0BuMHCgPf/d36RwIRAw4BcAKV+E71Rf0iBI8GmAVb+pb2VwWhCkIEoQVYCtgHtQOP/eT5/PxDBPQHdgD1AD8CAv38/McAtgmCDEQCPfkLAzoKgwNPBOoBhAFIBEgBiwPEAvf/7/4eAlAGJwRWAIIG4AOb+fv3mfxoAswECweu/D4AKgSG+379XAEj/537pft3BEoGyACN+/b3mvy+AIoCQP/8/nUBzADc+QLyye8Z+gkFcgt+Bdv9IvvE9QT81Pwv+xf96fqZ/UL9e/q//Nn9VgIBBmQHUwBs+W/9mvut+bP9ywJGAtMHzwCZ+4gI4AZMBzT+qPdV+zr/GwnXBc/7VPw5Ba0EFgRvBLAAKgaMB4ABlwB1CI0BDPa//Az++P5H/zX/6/+7BQ4HagC//ej5vQCaAD79uP+jBEsECP0b/jf/SACA+1j8ygFzBcYC/fhX/MT8uvnUAZEHjADH9ZfyVAOvAKb8kgW/BgIDwfr8/wUB2ADB/pv/qAF+AJH/0vhE+rcD6AKu/WMBbgP3AjYGxwLX/t36OPXAAcIGegFRBooG/wT3+VX08focAnQG8QBO/kUB7QHO/sz8TwCXBYf/WwH4Bn4CoAFcAdD+nvvT+V/6nQPbB10AYP6KAogIOgfF/NT5bvuN+Uf7QAJDACH/IALvBnsB3/jI/DH9o/59+7wB7AN/AF0BLACJCzoKHP+A+rT0k/cVAeEFIAGx+dv9CggfAQ7+6wL6/W36ffuIAn8A2/t5/XYAaAU0BlYBZvsS/H8Cp/5F+Gf65velAHsH6/z+/DX9p/ka//r/9/3M/fcFbQhq/Mr1sv0+BNr4cvh/AH0GFA02BXf28ffjALkCWwZMBsf+Svtp/AP7wf5k/u782AemByT9lwBXASX/2/1K/BcC/QQEBbQB3AO6BNL+a/1NAAABCf/3/wMHfgeqATcBEP//+h3//gYKAWv/7QCz+zYB1gMcBqwAAvvL/j/3V/xBBIH+R/s3/Cv/9wAfAmH+iv3v/lD7of08BGUGKQBY+qL2C/iS+j78WfzQ+4AClAJEANX+Pv+sAecHXQNh+Kv56fyzARQCGgH0Ajz/qvmn+27/AwOWBfsFowEE+lf8x/5MAx4LWwVcAOQDEgTrAOL/GQIJBPsD9QlvBQf6nv3ZAkgHYAb6/B35WftiANkHUAMq/YoAcAcbCJgAbAEnBJ//VP3d+lb35Pia/5QEZQH4/bkAyAVtAUMBagSx/xT52v5MCC0D9ftD/YEBBfk3+WoCUgLRBMcAmgDiBPQE4gT6/Cv5r/opAfIEjQZDA3n9IPra+UD8Ev+MBnEFtv/L+C8B7gR9/Xr/JgEf/4L9//9pA0UClgELAK/5LPgU/RsDCQN///v6pfhTAvwG7AQWAgv6Bf+hAXr+U/+s/gX9rfsLAMkC1wA7AFv+kPoa/24LNwY3AgwIYAFd+gT6KvtV/Ef8tf96A3cIPAgeAFj8YvvvAnkJnwbA/VX5pgFzAVD4r/oPAxkGBgWVALT4w/i3/En+EgN6BeEB3fss/1j+3/v0BecIgwQE/2T69PtJ/E36NAJrB5UDZ/2I+YD7ZgDkBAAE2gFFAVv5Lvge+mb4j/5KBMEE9QAG/9b51/n5/00EugIs/nT9IgAjACYCFP2i+aUBff3Y/1IDbQHCBgMGiQA9/k7/dgH7/yr88vuFA1kE7v///2sCZwKoBNMFlv6R/dMAiwkrBaj7yv1lAD8CW//A/OT+0QfDBCH7Sf4sBEYDe/1d/M798AGjAkD+qABf/iz6c/yLAx4Guf5F/tP/zf/sA9P/6faN9939/wDi/fH9wgCj/74CzALW/8T9F/ed/RUEdwGNArQCbADF/H//Tv+r+9H7sf+wAZf7Kv4vAez/Dvt1+gQEGAhGBusCyQBX/3H9evse/y//KfzXABsAugDsAz0DHgbwA8j+GfwgAJT9k/o7A0b/rPuI/XkAGARaA1oCBv0n+nT6I/ox/g0BVgBtBcIE1QIgAdv7Av4YBWsFc/yN/bwBbgSJA4P9afqU+6cCXAUC/xD8oP0N/YgDcQdwAXT+jQNOBvj/Z/x3/Wn8IP+VAW0EggTT/6L/rvjR+woF9AHN/4QCEAB1+rT/JwYsB9kCiP5ZAUkCf/sz+Cb9hgH+/a/+iAAZ/Zv+agGSAOP/5/ye/Nz+tgECBB0Ay/3b/Sz8jPcO+uD+FQH5AQ4CcQJbAwECuvpJ+Zb82f1S/sECRAOI/xn/Lvqr+voBzwNEAl0AyAAiAlIABQRgAlAA4wJlAHUCIQM6/Wn7g/89/jMB4AKJANUCWwY2BxQBQf9y/Lz86QQkBLr/EQHiAT0Atvy0+jwCngVY/3T5k/z7AaADlQAd+SP6FQBpA7gCd/+a/eMAwANG/tv/JwAD9vX4cf/o/GX91gIQBb0DywAnAC8BJgBi/kEByQRz/qj9rvyF+mf/1QBK/vv7if2D/Zf+df7vAPoGugMX/oP/ZQJgA1MFNwUrAVb+8QCGAbr/AwCv/Bj+xAKjBXYGSP/r/JL9swCqBEsAaQHmBOwGSgST/ab4pfhU/jv+gP/IANj9wf9eBBkGyQRaAMT5cPkYABMEFAGJ/rb+9fyy/QgBvALqBDkDsf0L/6H/KACDBJQCv/3H/8QBEf2s/LT/LAOd//z+WAR1AaD/l/4a/C/9qACIBp8HcP0C+Hb7ngNLAyQCIQO9/UYBRgQkBJ38mvsfBZUBuPka+a4AoAOXAZ8BOwBQ/lIBYgE5/hT/iv2X/wABtgA0A4YBBf7E/Ir+LALcAjABWv/+/jAAzf5I+kX8GAOKA47/FwGfA9T/AAJL/8j7s/uG/hUFJgS5ANP+zv4B/qsDTwaiAs4AaP7LADIFvAF3ANgDGQCQ/wv/1fxv/qkAHwGn/7QCKgS1AhECQwJbATMBbgKC/Yb6avxx/54CWABc+jr4Tfx6/g4C8gPlASr/kflaLine truncated
Your browser does not support the audio element.
</audio>
Try out the converted pipeline `⇑ <#top>`__
###############################################################################################################################
The demo app below is created using `Gradio
package <https://www.gradio.app/docs/interface>`__
.. code:: ipython3
def _generate(prompt):
inputs = processor(
text=[prompt,],
padding=True,
return_tensors="pt",
)
audio_values = model.generate(**inputs.to(device), do_sample=True, guidance_scale=3, max_new_tokens=n_tokens)
waveform = audio_values[0].cpu().squeeze() * 2**15
return (sampling_rate, waveform.numpy().astype(np.int16))
.. code:: ipython3
import gradio as gr
demo = gr.Interface(
_generate,
inputs=[
gr.Textbox(label="Text Prompt"),
],
outputs=[
"audio"
],
examples=[
["80s pop track with bassy drums and synth"],
["Earthy tones, environmentally conscious, ukulele-infused, harmonic, breezy, easygoing, organic instrumentation, gentle grooves"],
["90s rock song with loud guitars and heavy drums"],
["Heartful EDM with beautiful synths and chords"],
],
)
try:
demo.launch(debug=True)
except Exception:
demo.launch(share=True, debug=True)
# If you are launching remotely, specify server_name and server_port
# EXAMPLE: `demo.launch(server_name='your server name', server_port='server port in int')`
# To learn more please refer to the Gradio docs: https://gradio.app/docs/
docs/notebooks/251-tiny-sd-image-generation-with-output.rst
+1136
View File
File diff suppressed because it is too large. Load diff
docs/notebooks/251-tiny-sd-image-generation-with-output_files/251-tiny-sd-image-generation_33_1.png
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5d4313551a427c44d4bba572717be2134c99e0e8785a8f11d5fd2993ccfb10da
size 470354
docs/notebooks/251-tiny-sd-image-generation-with-output_files/251-tiny-sd-image-generation_37_1.png
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e97e1e206d64979d8fc9aaa6e46786172062c9a9eb1817a351af2a38557a012f
size 835162
docs/notebooks/251-tiny-sd-image-generation-with-output_files/251-tiny-sd-image-generation_39_1.png
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7e4fb5a7c284be4de90e648badd409935c29b561b30f728ff905f522dd5a99f9
size 699062
docs/notebooks/251-tiny-sd-image-generation-with-output_files/index.html
View File
Whitespace-only changes.
docs/notebooks/notebooks_with_binder_buttons.txt
+1
View File
@@ -30,6 +30,7 @@
229-distilbert-sequence-classification
243-tflite-selfie-segmentation
247-code-language-id
250-music-generation
401-object-detection
402-pose-estimation
403-action-recognition-webcam
docs/notebooks/notebooks_with_colab_buttons.txt
+2
View File
@@ -21,6 +21,8 @@
232-clip-language-saliency-map
243-tflite-selfie-segmentation
244-named-entity-recognition
250-music-generation
251-tiny-sd-image-generation
305-tensorflow-quantization-aware-training
401-object-detection
404-style-transfer
docs/tutorials.md
+21 -3
View File
@@ -257,6 +257,10 @@ Demos that demonstrate inference on a particular model.
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
| `249-oneformer-segmentation <notebooks/249-oneformer-segmentation-with-output.html>`__ | Universal segmentation with OneFormer and OpenVINO™. | |n249-img1| |
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
| `250-music-generation <notebooks/250-music-generation-with-output.html>`__ |br| |n250| |br| |c250| | Controllable Music Generation with MusicGen and OpenVINO™. | |n250-img1| |
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
| `251-tiny-sd-image-generation <notebooks/251-tiny-sd-image-generation-with-output.html>`__ |br| |c251| | Image Generation with Tiny-SD and OpenVINO™. | |n251-img1| |
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
Model Training
@@ -318,7 +322,7 @@ The following tutorials are guaranteed to provide a great experience with infere
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
| `Whisper - Subtitles generation <notebooks/227-whisper-subtitles-generation-with-output.html>`__ |br| |c227| | Generate subtitles for video with OpenAI Whisper and OpenVINO. | |n227-img1| |
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
| `CLIP - zero-shot-image-classification <notebooks/228-clip-zero-shot-image-classification-with-output.html>`__ | Perform Zero-shot image classification with CLIP and OpenVINO. | |n228-img1| |
| `CLIP - zero-shot-image-classification <notebooks/228-clip-zero-shot-convert-with-output.html>`__ | Perform Zero-shot image classification with CLIP and OpenVINO. | |n228-img1| |
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
| `BLIP - Visual-language-processing <notebooks/233-blip-visual-language-processing-with-output.html>`__ | Visual question answering and image captioning using BLIP and OpenVINO™. | |n233-img1| |
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
@@ -326,11 +330,15 @@ The following tutorials are guaranteed to provide a great experience with infere
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
| `DeepFloyd IF - Text-to-Image generation <notebooks/238-deep-floyd-if-with-output.html>`__ | Text-to-image generation with DeepFloyd IF and OpenVINO™. | |n238-img1| |
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
| `ImageBind <notebooks/239-image-bind-with-output.html>`__ | Binding multimodal data, using ImageBind and OpenVINO™. | |n239-img1| |
| `ImageBind <notebooks/239-image-bind-convert-with-output.html>`__ | Binding multimodal data, using ImageBind and OpenVINO™. | |n239-img1| |
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
| `Dolly v2 <notebooks/240-dolly-2-instruction-following-with-output.html>`__ | Instruction following using Databricks Dolly 2.0 and OpenVINO™. | |n240-img1| |
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
| `248-stable-diffusion-xl <notebooks/248-stable-diffusion-xl-with-output.html>`__ | Image generation with Stable Diffusion XL and OpenVINO™. | |n248-img1| |
| `Stable Diffusion XL <notebooks/248-stable-diffusion-xl-with-output.html>`__ | Image generation with Stable Diffusion XL and OpenVINO™. | |n248-img1| |
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
| `MusicGen <notebooks/250-music-generation-with-output.html>`__ |br| |n250| |br| |c250| | Controllable Music Generation with MusicGen and OpenVINO™. | |n250-img1| |
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
| `Tiny SD <notebooks/251-tiny-sd-image-generation-with-output.html>`__ |br| |c251| | Image Generation with Tiny-SD and OpenVINO™. | |n251-img1| |
+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------+
@@ -487,6 +495,10 @@ Made with `contributors-img <https://contrib.rocks>`__.
:target: https://user-images.githubusercontent.com/29454499/258651862-28b63016-c5ff-4263-9da8-73ca31100165.jpeg
.. |n249-img1| image:: https://camo.githubusercontent.com/f46c3642d3266e9d56d8ea8a943e67825597de3ff51698703ea2ddcb1086e541/68747470733a2f2f6769746875622d70726f64756374696f6e2d757365722d61737365742d3632313064662e73332e616d617a6f6e6177732e636f6d2f37363136313235362f3235383634303731332d66383031626430392d653932372d346162642d616132662d3939393064653463616638642e676966
:target: https://camo.githubusercontent.com/f46c3642d3266e9d56d8ea8a943e67825597de3ff51698703ea2ddcb1086e541/68747470733a2f2f6769746875622d70726f64756374696f6e2d757365722d61737365742d3632313064662e73332e616d617a6f6e6177732e636f6d2f37363136313235362f3235383634303731332d66383031626430392d653932372d346162642d616132662d3939393064653463616638642e676966
.. |n250-img1| image:: https://user-images.githubusercontent.com/76463150/260439306-81c81c8d-1f9c-41d0-b881-9491766def8e.png
:target: https://user-images.githubusercontent.com/76463150/260439306-81c81c8d-1f9c-41d0-b881-9491766def8e.png
.. |n251-img1| image:: https://user-images.githubusercontent.com/29454499/260904650-274fc2f9-24d2-46a3-ac3d-d660ec3c9a19.png
:target: https://user-images.githubusercontent.com/29454499/260904650-274fc2f9-24d2-46a3-ac3d-d660ec3c9a19.png
.. |n301-img1| image:: https://user-images.githubusercontent.com/15709723/127779607-8fa34947-1c35-4260-8d04-981c41a2a2cc.png
:target: https://user-images.githubusercontent.com/15709723/127779607-8fa34947-1c35-4260-8d04-981c41a2a2cc.png
.. |n401-img1| image:: https://user-images.githubusercontent.com/4547501/141471665-82b28c86-cf64-4bfe-98b3-c314658f2d96.gif
@@ -647,6 +659,12 @@ Made with `contributors-img <https://contrib.rocks>`__.
:target: https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/244-named-entity-recognition/244-named-entity-recognition.ipynb
.. |n247| image:: https://mybinder.org/badge_logo.svg
:target: https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F247-code-language-id%2F247-code-language-id.ipynb
.. |n250| image:: https://mybinder.org/badge_logo.svg
:target: https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F250-music-generation%2F250-music-generation.ipynb
.. |c250| image:: https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667
:target: https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/250-music-generation/250-music-generation.ipynb
.. |c251| image:: https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667
:target: https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/251-tiny-sd-image-generation/251-tiny-sd-image-generation.ipynb
.. |c305| image:: https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667
:width: 109
:target: https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/305-tensorflow-quantization-aware-training/305-tensorflow-quantization-aware-training.ipynb