You can find extensive documentation on the official homepage, there is the GitHub page, some courses on Coursera and many other resources. But how to start as fast as possible without a need to study all of the materials? One of the possible ways can be found in the following paragraphs.

No Need to Install, Use Docker

OpenVINO has a couple of dependencies which need to be present on your computer. Additionally, to install some of them, you need to have root/admin rights. This might not be desirable. Using Docker represents much cleaner way. Especially when there is an image prepared for you on Docker Hub.

If you are not familiar with Docker, it might seem like a complicated piece of software, but we highly recommend you to try it and learn the basics. It is not hard and worth the effort. Docker is an important part of today’s SW development world. You will find installation instructions here.

Running the Docker Image

Containers are stateless. That means that next time you start your container, all the changes made will be gone. (Yes, this is a feature.) If you want to persist some files, just prepare a directory on your filesystem and we will bind it as a shared volume to a running container. (To /home/openvino directory).

We will run our container in interactive mode (-it) so that you can easily execute commands inside it and it will be terminated after you close the command prompt (--rm). With the following command, the image will be downloaded (pulled), if it was not already done, and it will be run as a container.

docker run -it --rm -v __YOUR_DIRECTORY__:/home/openvino openvino/ubuntu18_dev:latest

To be able to use all the tools, OpenVINO environment needs to be initialized. For some reason, this is not done automatically. (At least not for a normal user, if you start docker as a root, -u 0, setup script is run.)

source /opt/intel/openvino/bin/setupvars.sh

Conformation is then printed out.

[setupvars.sh] OpenVINO environment initialized

TensorFlow 2 Dependencies

TensorFlow 2 is not present inside the container by default. We could very easily create our own image based on the original one with TensorFlow 2 installed. This is the best way in production. With this being said, we will show you another way using the original container and installing the missing packages to an virtual environment into the shared directory (volume). This way, we can create as many such environments as we want. Or easily modify this environment. In addition, we will be still able to try the older TensorFlow 1 models. We prefer this approach during initial development.

Following code needs to be executed only once, after you first start your container.

mkdir ~/env
python3 -m venv ~/env/tensorflow2 --system-site-packages
source ~/env/tensorflow2/bin/activate
pip3 install --upgrade pip
pip3 install -r /opt/intel/openvino/deployment_tools/model_optimizer/requirements_tf2.txt

When you close your container and open it again, this is the only part you will need to repeat.

source ~/env/tensorflow2/bin/activate

Converting TensorFlow SavedModel

Let’s say you have a trained model in SavedFormat. For the sake of this tutorial, we can take a pretrained MobileNet. Execute python3 and run following couple of lines to download and save the model.

import tensorflow as tf
model = tf.keras.applications.MobileNetV2(input_shape=(224,224,3))
model.save("/home/openvino/models/tf2")

Conversion is a matter of one command. However, there are few important parameters we need to use and which will be described below. Complete list can be found in the documentation. Exit python interpreter and run following command in a bash.

/opt/intel/openvino/deployment_tools/model_optimizer/mo_tf.py --saved_model_dir ~/models/tf2 --output_dir ~/models/converted --batch 1 --reverse_input_channels --mean_values [127.5,127.5,127.5] --scale_values [127.5,127.5,127.5]

• --batch 1 sets batch size to 1. OpenVINO cannot work with undefined input dimensions. I out case, the input shape for MobileNetV2 of is (-1, 224, 224, 3).
• --reverse_input_channels tells inference engine to convert image from it’s BGR format to RGB used by MobileNetV2.
• --mean_values [127.5,127.5,127.5] --scale_values [127.5,127.5,127.5] finally performs the necessary preprocessing so that image values are between -1 and 1. In TensorFlow, we would call preprocess_input.

Be careful, if you use pretrained models, different preprocessing and channel order can be used. If you try to use a neural network with input preprocessed in wrong way, you will of course get the wrong result.

You don’t need to include the preprocessing inside the converted model. The other option is to preprocess every image inside your own code before passing it to converted model. However, we use some OpenVINO inference tool which expects correct input.

At this point, we also need to mentioned that you might get slightly different values from SavedModel in TensorFlow and converted OpenVINO model. But from my experience on classification models, the difference is quite similar as when you use different ways to downscale your image to a proper input size.

Run Inference on Converted Model

First, we will get a testing picture which belongs to one of the 1000 ImageNet classes. We chose zebra, class index 340. (For TensorFlow, Google keeps the class indexes here.)

Let’s download it to our home directory We saved the small version of the image on our server so you can get it from there.

 curl -o ~/zebra.jpg -s https://images.ximilar.com/tutorials/openvino/zebra.jpg

There is a script you can use for testing the prediction with no need to write any code.

python3 /opt/intel/openvino/deployment_tools/inference_engine/samples/python/classification_sample_async/classification_sample_async.py -i ~/zebra.jpg -m ~/models/converted/saved_model.xml -d CPU

We get some info lines and top 10 results at the end. Since the numbers are pretty clear, we will show only the first three.

classid probability
------- -----------
  340    0.9556126
  396    0.0032325
  721    0.0008250

Cool, our model is really sure about what is on the picture!

Using OpenVINO Inference in Python

That was easy, right? But you probably need to run the inference inside your own Python code. You can take a look inside the script. It is pretty straightforward, but for the sake of completeness, we will copy some of the code here. We will also add a code for running an inference on the original model so that you can compare it easily. If you please, run python3 and we can start.

We need one basic import from OpenVINO inference engine. Also, OpenCV and NumPy are needed for opening and preprocessing the image. If you prefer, TensorFlow could be used here as well of course. But since it is not needed for running the inference at all, we will not use it.

import cv2
import numpy as np
from openvino.inference_engine import IECore

As for the preprocessing, part of it is already present inside the converted model (scaling, changing mean, inverting input channels width and height), but that is not all. We need to make sure the image has a proper size (224 pixels both sides) and the dimensions are correct – batch, channel, width, height.

img = cv2.imread("/home/openvino/zebra.jpg")
img = cv2.resize(img, (224, 224))
img = np.expand_dims(img, 0)
img_openvino = img.transpose((0,3, 1, 2))

Now, we can try a simple OpenVINO prediction. We will use one synchronous request.

ie = IECore()
net = ie.read_network(model="/home/openvino/models/converted/saved_model.xml", weights="/home/openvino/models/converted/saved_model.bin")
input_name = next(iter(net.input_info))
output_name = next(iter(net.outputs))
net.batch_size = 1
# number of request can be specified by parameter num_requests, default 1
exec_net = ie.load_network(network=net, device_name="CPU")
# we have one request only, see num_requests above
request = exec_net.requests[0]
# infer() waits for the result
# for asynchronous processing check async_infer() and wait()
request.infer({input_name: img_openvino})
# read the result
prediction_openvino_blob = request.output_blobs[output_name]
prediction_openvino = prediction_openvino_blob.buffer

Our result, prediction_openvino, is an ordinary NumPy array with the shape (batch dimension, number of classes) = (1, 1000). To print the top 3 values as before, we can use following couple of lines.

ids = np.argsort(prediction_openvino)[0][::-1][:3]
probabilities = np.sort(prediction_openvino)[0][::-1][:3]
for id, prob in zip(ids, probabilities):
    print(f"{id}\t{prob}")

We get exactly the same results as before. Our code works!

Comparing Results with Original TensorFlow Model

Now, let’s do the same with TensorFlow model. Do not forget to preprocess the image first. Prepared function preprocess_input can be used for that.

import tensorflow as tf
img_tf =  tf.keras.applications.mobilenet_v2.preprocess_input(img)
model = tf.keras.models.load_model("/home/openvino/models/tf2")
prediction_tf = model.predict(img_tf)

The results are almost the same, the difference is so small that we can ignore it. The top result from this prediction has probability 0.95860416, compared to 0.9556126 we had before. The order of the other predictions might be slightly different, because the values are so tiny.

By the way, there is a build-in function decode_predictions, which will not only give you the top results, but also class names and codes instead of just ids. Top 3 TensorFlow predictions.

from tensorflow.keras.applications.mobilenet_v2 import decode_predictions
top3 = decode_predictions(prediction_tf, top=3)

Here is the result:

[[('n02391049', 'zebra', 0.95860416), ('n02643566', 'lionfish', 0.0032956717), ('n01968897', 'chambered_nautilus', 0.0008273276)]]

Benchmarking

We should mention that there is also a tool for benchmarking OpenVINO models called Benchmark Python Tool. It offers synchronous (latency-oriented) and asynchronous (throughput-oriented) measuring modes. Unfortunately, it does not work for other models (like TensorFlow) and cannot be used for direct comparison.

How OpenVINO Helped Us

Enough of code, at the end of the article we will add some numbers. In Ximilar, we use often recognition models with resolution of 224 or 512 pixels. In a batch of 1 or 10. We used TensorFlow Lite format as often as possible, because it is very fast to load. (See a comparison here.) Because of a fast loading, it is not necessary to have the model in the cache all the time. To make running TF Lite models faster, we enhance the performance with XNNPACK.

Below you can see a chart with results for MobileNet V2. For the batches, we show prediction time in seconds per single image. Tests were done on our production workers.

Summary

In this article, we briefly introduced some of the basic functionality of OpenVINO. Of course, there is much more to try. We hope that this article has motivated you to try it yourself and maybe continue to explore all the possibilities through more advanced resources.

The post OpenVINO: Start Optimizing Your TensorFlow 2 Models for Intel CPUs with Docker appeared first on Ximilar: Visual AI for Business.

Object Detection — Customized

For some of our customers, we already had the opportunity to train custom detection models. However, we decided to integrate this into the popular app.ximilar.com and not only keep it in our Annotate tool.

The biggest news is that we re-implemented a new architecture for Object Detection (called CenterNet), using TensorFlow 2+. And made it open-source for you guys out there. The system is more scalable and faster than before, and we still have multiple ideas for further improvements. Creating detection models has so far a long and painful process, so we believe that you will love the new service as it significantly speeds up the workflow. We’ve also cloned favourite features from Image Recognition, so you can now configure your custom image augmentation settings, model versioning, download models for offline usage, evaluation on an independent test dataset, connect it to the flow, and more. It was a ride! More in the video ↓

Flows

Flows are a service that simplifies the process of building computer vision systems.

We spent a lot of man-hours on this feature. Ximilar is the only system in the market that can connect visual AI models to complex workflow without any coding. Before we had Flows, it was really painful to connect individual recognition or detection models into one API service. Now it is effortless, and we again have more features coming.

Flows are incredibly powerful with endless possibilities. Saving the costs of expensive machine learning development. You should see the happy faces of Flows users.

Imagine that you are creating a car monitoring system over the parking place. You can create detection of parking spots and then analyze individual spots with image recognition models and decide if it is used or not. With Flows, you just connect several actions and the system is automatically deployed for you! This is something that takes a long time for a team of engineers to develop, but with Ximilar Flows, the entire process can be done in just minutes.

Fashion Tagging and Search Improved

With custom object detection and Flows, we were able to build one of the best systems for visual apparel analysis (Clothing, Underwear, Footwear, Jewelry, Watches, Accessories, Bags, Hats, Glasses, etc.). The entire system consists of a hundred models integrated into one flow. The system is also able to detect individual clothing items and can tell you more information about the background and particular view or detail. Ximilar is ready to create a custom profile for you with the customization of tag names. And not only that.

Because some of our customers have all of our three services for Fashion: Tagging, Similarity, and Visual Search, we have also created a full-featured Fashion Search which includes all of those 3. This could save you a lot of money and provide the fastest solution for your e-commerce outlet or mobile app.

Some use cases require image analysis on a deeper level. We call such a service META image analysis. That is great for automated decisions such as:

• is there a person in the photo?
• is the product dominant in the image?
• what kind of background colour is there?
• are there additional details of the item to see?
• is this a front or a side view of the object in the photo?

Building a high-quality image tagging system for any other field is very effective on the Ximilar platform (real estate, stock photos, product categorization, etc). As an example, we have pre-built visual AI for home decor products right in the app.

New: Custom Image Similarity

Would you like to build a visual search engine for example for skincare products, but you don’t know how? Check out the web of our customer Skintory. We have taken this challenge to create Visual Search/Image Similarity as a service. So you can train custom similarity models and create visual product searches on click.

This could really change how we interact and search on retail sites. For most of the use cases, the generic similarity model trained on stock photos is not working as well as it could. So we took the opportunity to work on a service for the training of customized image similarity models with an integrated search engine.

That feature is still in BETA, and our AI team is working hard to deliver the best experience soon. If you would like to test this service, please contact us directly.

Multiple App Improvements

We have not abandoned all our now older services, such as Image Recognition. There are new features and improvements. The most visible additions are these:

• complex management of your image data
• new services in the Ximilar App and one-click login with social networks
• evaluation of models on the independent test dataset
• improved advanced options for better image augmentation
• moving to TensorFlow 2, adding export of offline models on edge devices, and more

Annotate

It is our internal image annotation system, which was released to the public at the beginning of the year 2020. It offers a workflow for teams (annotation jobs), multiple verifications of your image data, and more. If you are working on a large machine learning project that requires a balanced and high-quality dataset → ask us to show you Annotate. We are constantly working on improvements, so your team can deliver your final product faster with higher accuracy. Read about image annotation for teams.

Cooperations & Partnership

Thanks to Intel, our prediction system is running faster than ever. Our AI and Backend engineers work together with the Intel team (thank you, Ellen and Vishnu) on the optimization of machine learning models on the x86 architecture with OpenVINO. We changed the backend of our models from TF2 to OpenVINO and got a massive speed-up of performance. Now our entire fashion system is not only the most accurate but also the fastest one. Of course, you can build any other visual inspection application on top of the Ximilar Platform. We have published an in-depth behind the scenes.

In the middle of 2020, we also became a member of the NVIDIA Inception Program, which supports cutting-edge AI startups that are revolutionizing industries. We are looking forward to being active in the area, thanks to the Brno.AI platform, which supports companies and universities in Czechia.

Open Source Projects

Our tech team was active quite a lot in the open-source community. Implementing and publishing machine learning projects:

• xCenterNet — fast object detection model
• tf-image — image augmentation for TensorFlow that makes your model more robust
• TF-metric-learning — distance metric learning library

Tutorial videos

Lastly, Ximilar released video tutorials for the Platform. With the support of JIC Brno. A first series of educational videos.

And more in 2021!

In the year of 2020, we have grown on all levels. We have more happy customers and a bigger team. It was not easy to manage everything during the early Covid-19 era. Some of us spent a lot of time at home office. Luckily, we have a great team that supports our idea of making machine learning and computer vision a pleasant and creative process.

We are working on improving stable services as well as creating new projects. There are many interesting topics that we plan to probe such as Custom similarity, Explainability/Interpretability, Regression from image, a combination of image and text data, a model for background removal, image super-resolution service, and many more. Stay tuned & healthy for next year at least!

If you have any ideas that you would love to have on the platform, then please let us know.

The post The Year 2020 at Ximilar appeared first on Ximilar: Visual AI for Business.

This brought of course a great number of challenges, but also many opportunities. We would like to share with you some of the latest accomplishments. Mainly, we are proud to announce that we have become a part of Intel AI Builders Program. Having direct access to technical resources and support from Intel helps us in using our hardware in the best way possible.

Running the entire MLaaS (Machine learning as a service) platform on our cloud infrastructure as efficiently as possible and delivering high-quality results is essential for us.

Making The Most of It

Knowing our servers enables us to optimize used models to be very efficient. It is not hard to get incredible amounts of computational power today, but we still believe that it is worth using them to their fullest potential. We love saving energy, and resources, and at the same time providing our services for reasonable prices.

Machine learning computations are performed both by GPUs (graphics processing units) and CPUs (central processing units). During training, the CPU prepares the images, and the GPU optimizes the artificial neural network. For prediction, both of them can be used. But generally, we use GPU only when the model is huge, or we need the results very fast. In other cases, the CPU is sufficient. Recently, we focused on optimizing our prediction on our Intel Xeon CPUs to make them run faster.

We use the TensorFlow library for your machine-learning models. It was developed by Google and released as open-source in 2015. Google also came up with their own hardware for machine learning – Tensor Processing Units, TPUs in short. Nevertheless, they are still interested in other hardware platforms as well, and they provide XNNPACK, an optimized library of floating-point neural network inference operators.

OpenVINO Toolkit

However, we achieved the best results by using the OpenVINO Toolkit from Intel. It consists of two basic parts. A model optimizer, which is run after a model is trained, and an inference engine for predictions. We are able to speed up some of our tasks by 5 times. The predictions run especially efficiently on larger images and larger batches.

So far, we are using OpenVINO for our image recognition service and various recognition tasks in different fashion services. For example, fashion tagging or visual search runs many models on a single image and every improvement in efficiency is even more noticeable. But we are not forgetting the rest of our portfolio, and we are working hard to extend it to all the different services we provide.

Last but not least, we would like to thank Intel for their support, and we are looking forward to our cooperation in the next years!

Do you want to know more? Read a blog on the Intel AI website.

The post Behind The Scene: Ximilar and Intel AI Builders Program appeared first on Ximilar: Visual AI for Business.