文档中心 > 店铺动态卡片-开发指引

店铺动态卡片-开发指引

推理引擎（MNN）

更新时间：2023/01/29 访问次数：6234

一、中文版本

1.概览


MNN小程序插件利用MNN，加速小程序中的深度学习推理运算。插件主要由用于加载模型的loader和用于推理计算的builder构成。


2.权限申请


使用MNN小程序插件需申请权限，请联系小程序运营审批。审批后，需要在小程序的app.json文件中，添加插件相关信息，如下：

{
  "pages": [  "pages/index/index"  ],
  "plugins": {
      "MNN": {
              "version": "REPLACE_WITH_CURRENT_VERSION",  
              "provider": "3000000002046703"  
       }  
   } 
}


3.Loader


当前，插件仅支持加载MNN模型。MNN模型转换请参考文档。


1）算子列表


使用supportedOperators方法获取当前支持的MNN算子列表。示例如下：

var MNN = requirePlugin('MNN'); 
var builder = MNN.native_builder; 
console.log((new builder).supportedOperators()); 
// Array (36) 
// 0 "Const" 
// 1 "Input" 
// 2 "Convolution" 
// ...


若模型包含未在列表中的算子，请联系小程序运营评估添加。


2）加载MNN模型


使用loadMNN方法加载MNN模型。加载时，需要传入模型数据和用于构建计算图的builder。示例如下：

var MNN = requirePlugin('MNN'); 
var loader = MNN.loader; 
var builder = MNN.native_builder; 
var data = await getMyModelData(); // 请自行实现模型数据获取方法 
var map = await loader.loadMNN(data, [new builder])
【可选】如果数据格式读取出来为 ArrayBuffer，请通过 Uint8Array 转变为 byte 流
data= new Uint8Array(data); 
var map = await loader.loadMNN(data, [new builder])


若使用过程中，出现unsupported op: opname, type: optype的错误日志，则表明模型中存在MNN未支持的算子。


4.Builder


当前，插件仅支持通过应用内集成的MNN加速推理。将builder作为参数传入加载函数，完成计算图构造。


1）Variable访问


通过张量名访问代表张量的variable。示例如下：

var map = await loader.loadMNN(data, [new builder]) 
var input1 = map["net/input1"]; 
var input2 = map["net/input2"]; 
var output1 = map["net/prob1"]; 
var output2 = map["net/prob2"];


2）输入输出


variable提供了setData、getData方法用于设置输入、获取输出，也可以利用原型上提供的infer方法一次性完成输入输出。

setData需要传入4个参数：数据数据、数据类型、维度信息和数据布局。数据类型上，当前支持float32、uint8、int8。数据类型上，当前支持NCHW、NHWC和NC4HW4。示例如下：

// setData = async function(data, type, dims, format) { ... } 
var data = new Float32Array(3 * 144 * 256); 
var buffer = data.buffer.slice(data.byteOffset, data.byteLength); 
await input1.setData(buffer, "float32", [1, 256, 144, 3], "NHWC");


getData需要传入数据布局，示例如下：

// getData = async function(format) { ... } 
var score = await output1.getData("NHWC"); 
var data = score.data; 
var type = score.type; 
var dimensions = score.dims;


infer可以简化多个输入或输出的调用，示例如下。

// infer = async function(outVarArr, outFmtArr, inVarArr, inFmtArr, inTypeArr, inDimsArr, inDataArr) 
var input1Buffer = prepareData(); 
var input2Buffer = prepareData(); 
var net = output1.__proto__; // notice, infer is defined on __proto__ 
var results = await net.infer(  
// outputs  
  [output1, output2], ["NC4HW4", "NHWC"],  
// inputs  
  [input1, input2], ["NHWC", "NC4HW4"], ["float32", "float32"], [[1, 3, 64, 64], [1, 3, 64, 64]], [input1Buffer, input2Buffer] ); 
var result1 = results["net/prob1"]; 
var result2 = results["net/prob2"];


3）预处理


插件提供了对输入数据的预处理。预处理作为setData和infer接口的可选参数传入，示例如下：

let process = {
  "type": "image",
  "sourceFormat": "rgba",
  "destFormat": "bgr", 
}; 
await input1.setData(rgba.buffer, "float32", [1, 256, 144, 3], "NHWC", process);

或

let process = {
  "type": "image",
  "sourceFormat": "rgba",
  "destFormat": "bgr", 
}; 
let results = await net.infer(  outputs, formats,  [input], ["NHWC"], ["uint8"], [[1, 256, 144, 4]], [rgba.buffer], [process] );


当模型包含多个输入时，如果某输入无需预处理，可以传入undefined占位。

下面，具体介绍当前支持的预处理。


图形预处理


图形预处理与MNN的ImageProcess对应，因此，也可以参考文档使用，示例如下：

let image = { width: 640, height: 320 }; 
let input = { width: 320, height: 160 }; 
let offset = { x: (image.width - input.width) / 2, y: (image.height - input.height) / 2 }; 
let process = {
  "type": "image",
  "sourceFormat": "rgba",
  "destFormat": "bgr",
  "filterType": "nearest",
  "wrapType": "clamp_to_edge",
  "means": [102.9801, 115.9465, 122.7717, 0.0],
  "norms": [0.017, 0.017, 0.017, 0.0],
  "cropAndResize": [[offset.x, offset.y, input.width, input.height], [0, 0, input.width, input.height]],  
   // "matrix": [
       // input.width / image.width, 0, 0,   
      // 0, input.height / image.height, 0,   
     // 0, 0, 1  // 
      ] 
};


其中：

type为预处理类型，必选参数，值为image；

sourceFormat为图像颜色空间，可选参数，可选值包括：bgr、bgra、gray、rgb、rgba、yuv_nv12、yuv_nv21，默认rgba；

destFormat为目标颜色空间，可选参数，可选值包括：bgr、bgra、gray、rgb、rgba、yuv_nv12、yuv_nv21，默认rgba；

filterType为插值方法，可选参数，可选值包括：nearest、bilinear、bicubic，默认nearest；

wrapType为边缘填充，可选参数，可选值包括：clamp_to_edge、zero、repeat，默认clamp_to_edge；

means和norms为归一化参数，可选参数，计算规则为(value - mean[x]) * norms[x]，默认值分别为0和1；

cropAndResize为裁切缩放，可选参数，不能与matrix同时生效，值为由两组[x, y, width, height]定义的源图片区域和目标图片区域；

matrix为变换矩阵，可选参数，不能与cropAndResize同时生效，值为由9个number构成的变换矩阵。


4）资源释放


使用完成后，需要调用release方法释放构图创建的资源。示例如下：

Object.values(map).forEach(v => { v.release(); });


二、English Version

1.Overview


The MNN plugin uses MNN to speed up deep learning inference in the Mini App. The plugin is mainly composed of loader for loading models and builder for inferential calculation.


2.Access Request


You need to apply for permission to use the MNN plugin. Please contact the Mini App staffs. After approval, you need to add plugin-related information in your app.json file, as follows:

{
  "pages": [  "pages/index/index"  ],
  "plugins": {
      "MNN": {
              "version": "REPLACE_WITH_CURRENT_VERSION",  
              "provider": "3000000002046703"  
       }  
   } 
}


3.Loader


Currently, the plugin only supports loading MNN models. Please refer to documentation for MNN model conversion.


1）Operator list

Use the supportedOperators method to get the list of currently supported MNN operators. Examples are as follows:

var MNN = requirePlugin('MNN'); 
var builder = MNN.native_builder; 
console.log((new builder).supportedOperators()); 
// Array (36) 
// 0 "Const" 
// 1 "Input" 
// 2 "Convolution" 
// ...


If your model contains operators that are not in the list, please contact the Mini App staffs. *


2）Load MNN Model


Load the MNN model using the loadMNN method. When loading, you need to pass in the model data and the builder used to build the calculation graph. An example is as follows:

var MNN = requirePlugin('MNN'); 
var loader = MNN.loader; 
var builder = MNN.native_builder; 
var data = await getMyModelData(); // 请自行实现模型数据获取方法 
var map = await loader.loadMNN(data, [new builder])


If message like unsupported op: opname, type: optype appears in error logs, there is an operator not supported by MNN in the model. *


4.Builder


Currently, plugins only support accelerated inference through MNNs integrated within the app. Pass builder as a parameter to the loader to construct the calculation graph. Access variable representing each tensor by tensor's name. An example is as follows:


1）Access Variable


Access variable by tensor's name as follows:

var map = await loader.loadMNN(data, [new builder]) 
var input1 = map["net/input1"]; 
var input2 = map["net/input2"]; 
var output1 = map["net/prob1"]; 
var output2 = map["net/prob2"];


2）Input and Output


variable provides the setData, getData methods for setting inputs, getting outputs, or using the infer method provided on the prototype to do the input and output at one time.


setData needs to pass in 4 parameters: data, data type, dimension information and data layout. On the data type, float32, uint8, int8 are currently supported. On the data type, NCHW, NHWC and NC4HW4 are currently supported. An example is as follows:

// setData = async function(data, type, dims, format) { ... } 
var data = new Float32Array(3 * 144 * 256); 
var buffer = data.buffer.slice(data.byteOffset, data.byteLength); 
await input1.setData(buffer, "float32", [1, 256, 144, 3], "NHWC");
getData needs to pass in the data layout, examples are as follows:
// setData = async function(data, type, dims, format) { ... } 
var data = new Float32Array(3 * 144 * 256); 
var buffer = data.buffer.slice(data.byteOffset, data.byteLength); 
await input1.setData(buffer, "float32", [1, 256, 144, 3], "NHWC");


infer can simplify multiple input or output calls, examples are as follows:

// infer = async function(outVarArr, outFmtArr, inVarArr, inFmtArr, inTypeArr, inDimsArr, inDataArr) 
var input1Buffer = prepareData(); 
var input2Buffer = prepareData(); 
var net = output1.__proto__; // notice, infer is defined on __proto__ 
var results = await net.infer(  
// outputs  
  [output1, output2], ["NC4HW4", "NHWC"],  
// inputs  
  [input1, input2], ["NHWC", "NC4HW4"], ["float32", "float32"], [[1, 3, 64, 64], [1, 3, 64, 64]], [input1Buffer, input2Buffer] ); 
var result1 = results["net/prob1"]; 
var result2 = results["net/prob2"];


3）Input Preprocess


The plugin provides pre-processing for input data. Preprocessing is passed to the setData and infer function as an optional parameter, for example:

let process = {
  "type": "image",
  "sourceFormat": "rgba",
  "destFormat": "bgr", 
}; 
await input1.setData(rgba.buffer, "float32", [1, 256, 144, 3], "NHWC", process);
or
let process = {
  "type": "image",
  "sourceFormat": "rgba",
  "destFormat": "bgr", 
}; 
let results = await net.infer(  outputs, formats,  [input], ["NHWC"], ["uint8"], [[1, 256, 144, 4]], [rgba.buffer], [process] );


When the model contains multiple inputs, if an input does not require preprocessing, you can pass in undefined as placeholder.

Supported preprocessings are described in detail below.


Image preprocessing


The image preprocessing corresponds to the ImageProcess of MNN. Therefore, you can also refer to [Document] (https://www.yuque.com/mnn/cn/input#sTOZw) and use it as follows:

let image = { width: 640, height: 320 }; 
let input = { width: 320, height: 160 }; 
let offset = { x: (image.width - input.width) / 2, y: (image.height - input.height) / 2 }; 
let process = {
  "type": "image",
  "sourceFormat": "rgba",
  "destFormat": "bgr",
  "filterType": "nearest",
  "wrapType": "clamp_to_edge",
  "means": [102.9801, 115.9465, 122.7717, 0.0],
  "norms": [0.017, 0.017, 0.017, 0.0],
  "cropAndResize": [[offset.x, offset.y, input.width, input.height], [0, 0, input.width, input.height]],  
   // "matrix": [
       // input.width / image.width, 0, 0,   
      // 0, input.height / image.height, 0,   
     // 0, 0, 1  // 
      ] 
};


among them:

type is preprocessing type, a required parameter, and its value should be image;

sourceFormat is the image colorspace, an optional parameter, available values: bgr, bgra,gray, rgb,rgba, yuv_nv12,yuv_nv21, defaults rgba;

destFormat is the target colorspace, an optional parameter, available values: bgr, bgra,gray, rgb,rgba, yuv_nv12,yuv_nv21, defaults rgba;

filterType is the interpolation method, an optional parameter, available values: nearest, bilinear,bicubic, defaults nearest;

wrapType is edge wrap type, an optional parameter, available values: clamp_to_edge, zero,repeat, defaults clamp_to_edge;

means and norms are normalized parameters, optional parameters, the calculation rule is (value - mean[x]) * norms[x], the default values are 0 and1 respectively;

cropAndResize is cropping and scaling, an optional parameter, and an alternative to matrix, its value is source image rect and target image rect defined by two [x, y, width, height] arrays;

matrix is transformation matrix, an optional parameter, and an alternative to cropAndResize, its value is a transformation matrix composed of 9 number.


4）Resource release


After use, you need to call the release method to release resources occupied by variables. An example is as follows:

Object.values(map).forEach(v => { v.release(); });



FAQ

关于此文档暂时还没有FAQ
有用(0) 我要提问
返回
顶部
阿里巴巴集团 | 淘宝网 | 天猫 | 聚划算 | 全球速卖通 | 阿里巴巴国际交易市场 | 1688 | 阿里妈妈 | 阿里旅行·去啊 | 阿里云计算 | YunOS | 阿里通信 | 万网 | 高德 | 优视 | 友盟 | 酷盘 | 虾米 | 天天动听 | 来往 | 钉钉 | 11Main | 支付宝
关于淘宝 合作伙伴 营销中心 联系客服 开放平台 诚征英才 联系我们 网站地图 法律声明 © 2016 Taobao.com 版权所有
网络文化经营许可证：文网文[2010]040号 | 增值电信业务经营许可证：浙B2-20080224-1 | 信息网络传播视听节目许可证：1109364号