PyTorch Vulkan Backend Person Workflow — PyTorch Tutorials 1.13.1+cu117 documentation

Introduction¶

PyTorch 1.7 helps the power to run mannequin inference on GPUs that assist the Vulkan graphics and compute API. The first goal units are cell GPUs on Android units. The Vulkan backend will also be used on Linux, Mac, and Home windows desktop builds to make use of Vulkan units like Intel built-in GPUs. This characteristic is within the prototype stage and is topic to alter.

cd $VULKAN_SDK_ROOT
supply setup-env.sh
sudo python install_vulkan.py

cd PYTORCH_ROOT
USE_VULKAN=1 USE_VULKAN_SHADERC_RUNTIME=1 USE_VULKAN_WRAPPER=0 python setup.py set up

cd PYTORCH_ROOT
USE_VULKAN=1 USE_VULKAN_SHADERC_RUNTIME=1 USE_VULKAN_WRAPPER=0 MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py set up

import torch
print(torch.__version__)

cd PYTORCH_ROOT
ANDROID_ABI=arm64-v8a USE_VULKAN=1 sh ./scripts/build_android.sh

cd $PYTORCH_ROOT
USE_VULKAN=1 sh ./scripts/build_pytorch_android.sh

Mannequin preparation¶

Set up torchvision, get the default pretrained float mannequin.

Python script to avoid wasting pretrained mobilenet_v2 to a file:

import torch
import torchvision

mannequin = torchvision.fashions.mobilenet_v2(pretrained=True)
mannequin.eval()
script_model = torch.jit.script(mannequin)
torch.jit.save(script_model, "mobilenet2.pt")

PyTorch 1.7 Vulkan backend helps solely float 32bit operators. The default mannequin wants extra step that may optimize operators fusing

from torch.utils.mobile_optimizer import optimize_for_mobile
script_model_vulkan = optimize_for_mobile(script_model, backend='vulkan')
torch.jit.save(script_model_vulkan, "mobilenet2-vulkan.pt")

The outcome mannequin can be utilized solely on Vulkan backend because it comprises particular to the Vulkan backend operators.

By default, optimize_for_mobile with backend='vulkan' rewrites the graph so that inputs are transferred to the Vulkan backend, and outputs are transferred to the CPU backend, due to this fact, the mannequin will be run on CPU inputs and produce CPU outputs. To disable this, add the argument optimization_blocklist={MobileOptimizerType.VULKAN_AUTOMATIC_GPU_TRANSFER} to optimize_for_mobile. (MobileOptimizerType will be imported from torch.utils.mobile_optimizer)

For extra info, see the torch.utils.mobile_optimizer API documentation.

Utilizing Vulkan backend in code¶

C++ API¶

at::is_vulkan_available()
auto tensor = at::rand({1, 2, 2, 3}, at::system(at::kCPU).dtype(at::kFloat));
auto tensor_vulkan = t.vulkan();
auto module = torch::jit::load("$PATH");
auto tensor_output_vulkan = module.ahead(inputs).toTensor();
auto tensor_output = tensor_output.cpu();

at::is_vulkan_available() perform tries to initialize Vulkan backend and if Vulkan system is efficiently discovered and context is created – it’ll return true, false in any other case.

.vulkan() perform referred to as on Tensor will copy tensor to Vulkan system, and for operators referred to as with this tensor as enter – the operator will run on Vulkan system, and its output shall be on the Vulkan system.

.cpu() perform referred to as on Vulkan tensor will copy its information to CPU tensor (default)

Operators referred to as with a tensor on a Vulkan system as an enter shall be executed on a Vulkan system. If an operator shouldn’t be supported for the Vulkan backend the exception shall be thrown.

See Also

Roosevelt Pursues the Boat Thieves

Listing of supported operators:

_adaptive_avg_pool2d
_cat
add.Scalar
add.Tensor
add_.Tensor
addmm
avg_pool2d
clamp
convolution
empty.memory_format
empty_strided
hardtanh_
max_pool2d
imply.dim
mm
mul.Scalar
relu_
reshape
choose.int
slice.Tensor
transpose.int
transpose_
unsqueeze
upsample_nearest2d
view

These operators permit to make use of torchvision fashions for picture classification on Vulkan backend.

Python API¶

torch.is_vulkan_available() is uncovered to Python API.

tensor.to(system="vulkan") works as .vulkan() shifting tensor to the Vulkan system.

.vulkan() in the mean time of writing of this tutorial shouldn’t be uncovered to Python API, however it’s deliberate to be there.

Android Java API¶

For Android API to run mannequin on Vulkan backend we’ve got to specify this throughout mannequin loading:

import org.pytorch.System;
Module module = Module.load("$PATH", System.VULKAN)
FloatBuffer buffer = Tensor.allocateFloatBuffer(1 * 3 * 224 * 224);
Tensor inputTensor = Tensor.fromBlob(buffer, new int[]{1, 3, 224, 224});
Tensor outputTensor = mModule.ahead(IValue.from(inputTensor)).toTensor();

On this case, all inputs shall be transparently copied from CPU to the Vulkan system, and mannequin shall be run on Vulkan system, the output shall be copied transparently to CPU.

The instance of utilizing Vulkan backend will be present in take a look at utility inside the PyTorch repository:
https://github.com/pytorch/pytorch/blob/master/android/test_app/app/src/main/java/org/pytorch/testapp/MainActivity.java#L133

Constructing android take a look at app with Vulkan¶

cd $PYTORCH_ROOT
USE_VULKAN=1 sh ./scripts/build_pytorch_android.sh

Or if you happen to want solely particular abi you possibly can set it as an argument:

cd $PYTORCH_ROOT
USE_VULKAN=1 sh ./scripts/build_pytorch_android.sh $ANDROID_ABI

Add ready mannequin mobilenet2-vulkan.pt to check applocation property:

cp mobilenet2-vulkan.pt $PYTORCH_ROOT/android/test_app/app/src/most important/property/

cd $PYTORCH_ROOT
gradle -p android test_app:installMbvulkanLocalBaseDebug

After profitable set up, the applying with the title ‘MBQ’ will be launched on the system.

Testing fashions with out importing to android system¶

Software program implementations of Vulkan (e.g. https://swiftshader.googlesource.com/SwiftShader ) can be utilized to check if a mannequin will be run utilizing PyTorch Vulkan Backend (e.g. test if all mannequin operators are supported).