Amount of AI deployment in IoT Edge segment is growing quickly.  IoT Edge, Near Edge and  Embedded Systems are sharing similar constraints on available computing for application which is native to Edge.  

Above listed items appears to be using AI and providing successful services in various segment. All of them embedded, but cloud connected applications.  May be car and satellites will have less dependency on cloud side support  and carry out most of required inference on board.

Most of the above listed items, using GPU based massive parallel computing on board for real time inference.  Trained Deep learning networks deployed on each system for inference and GPU based CUDA cores are used for parallel computing requirements of Deep learning networks.

FPGA and GPU are competing  devices in massive parallel computing requirement.  It appears that Jetson series ( Nano, TX2, Xavier, Orin etc) had opened up huge advantage for IoT edge design and development teams across the Globe.  Where Jetson series devices are from Nvidia with software support via CUDA SDK and associated libraries.   

FPGA also playing critical role in embedded devices where input reading is necessary and input reading via JESD204B or  Serial LVDS.  In the case of Jetson series, input reading  ( high throughput ) is possible via PCI gen 4 .   

May be soon, there will be a standard for high throughput reading from sensors for real time inference.

Above infrastructure is used in Deploying Deep Learning Networks.  For example, at device side there options that include GPU and FPGA. There are devices trying to use existing DSP or CPU or both for inference work, but quickly that trend is changing and most of them moving towards GPU based devices for Real time inference. 

On top programming languages that are used to design and develop applications for Real time inference in Embedded devices.  Mentioned list include, Python, C++, Curl Java etc. Where TensorFlow and PyTorch are used to model Deep Learning Networks. 

Android Drevices and Linux based Jetson series devices are used in IoT Edge segment. It appears that mostly ARM is used as CPU in embedded device, but this trend appears to be dominating while IBM had introduced Open Power and released Power 9 IC  information such that many other silicon vendors can create Power 9 Silicons for  Embedded segment. May be Base station segment will use these Power 9 silicons as a embedded processors instead of ARM.

DSP is coming back because huge number of AI applications in Speech and Audio segment. Where DSPs can be low power compared to FPGA and GPU. Thus there is place for DSP as well in embedded segment.

Deployment of trained model on to Edge requires lot more care and hard work. Following sub systems configuration is given for IoT Edge. In this, Android phone working as IoT device, It will work well for small size model. Deploying huge model on to Android phone might create challenges and may not work well if reduce model size to fit in Android phone . 

DLtrain  is a platform which is used to deploy Deep Learning Networks in above devices. Illustrating examples are shown for Android phone and Jetson Nano series. 

Training platform is different from deployment platform. Same provides obstruction to carry trained network in to limited capability deployment edge. Mostly there is a  need to cut down Model size or optimise weights of each node in Model. This mentioned optimisation of Model size of node weight may not be there if deployment happens in cloud side of infrastructure.

DLtrain  is shown in the above picture.  DLtrain is used for NN / CNN  traning and also deploying NN/CNN on target devices. Above picture shows Jetson Nano as Target device which is Edge AI computer. Where Jetson nano is connected to IoT node via SPI interface of Jetson nano. Sensor data is read by using SPI interface and IoT node is having interface with Sensor.  Jetson Nano also having connection with IBM Watson IoT service.  User can create their applications in Cloud by using Node-Red and IBM Watson IoT platform. DLtrain is designed to work in the above configuration and also support Deployment engineers to deploy Deep Learning Networks in IoT Edge devices.

DLtrain is used handle following AI Inference  workflow  at IoT Edge.


Above  mentioned applications  ported onto  Android phones.  Android NDK is used to make above computing methods from Java. And also for Mobile phone section,  J7app is an example application  in Android phone.  

Above  mentioned applications  ported onto many Edge AI computers that are using Jetson series. 

DLtrain platform is used in the above and  development  took place from fresh thinking and innovation by using know-how on embedded system requirements.   DLtrain is designed to  make user silicon EDGE computing ready for AI work load. DLtrain is a perfect tool to handle issues in porting trained AI models in Edge AI computers with ease.

Case Study : Deploy AI application in IoT Edge computer

Step 1:  Run MQTT client in IoT Edge

In this case Jetson Nano is used. 

Local support will be given  to run python version of MQTT client to work with IBM Watson IoT platform. 

Step 2:  Run TCP/IP socket Server to connect with IoT Nodes 

Embedded board  is working as  IoT node.  ARM CPU is used in IoT Node.  Local support will be given in the form of Sample code  for IoT Node.

Step 3:  Run DC motor Control app in IoT Node

Embedded board with ARM cpu is used as  a IoT node. 

Local support will be given in the form of Sample code  for IoT Node.

Step 4:  Create IoT device in IBM watson IoT platform 

Local support will be given  to create IoT device in ibmcloud.  But user need to have account in , in case not then new account will be open during event for each user in

Step 5:  Create App in  Node-Red

 Local support will be given  to create app in Node-Red in ibmcloud 

Step 6:  AI Application in IoT Edge

Run Image Classification app in IoT edge  and run a motor if given image is Banana.  And also update MQTT broker in IBM watson iot platform such that MQTT broker can publish this news on Banana to app in Node-Red