Amount of AI deployment in IoT Edge segment is growing quickly. IoT Edge, Near Edge and Embedded Systems are sharing similar constraints on available computing for application which is native to Edge.
Above listed items appears to be using AI and providing successful services in various segment. All of them embedded, but cloud connected applications. May be car and satellites will have less dependency on cloud side support and carry out most of required inference on board.
Most of the above listed items, using GPU based massive parallel computing on board for real time inference. Trained Deep learning networks deployed on each system for inference and GPU based CUDA cores are used for parallel computing requirements of Deep learning networks.
FPGA and GPU are competing devices in massive parallel computing requirement. It appears that Jetson series ( Nano, TX2, Xavier, Orin etc) had opened up huge advantage for IoT edge design and development teams across the Globe. Where Jetson series devices are from Nvidia with software support via CUDA SDK and associated libraries.
FPGA also playing critical role in embedded devices where input reading is necessary and input reading via JESD204B or Serial LVDS. In the case of Jetson series, input reading ( high throughput ) is possible via PCI gen 4 .
May be soon, there will be a standard for high throughput reading from sensors for real time inference.
Above infrastructure is used in Deploying Deep Learning Networks. For example, at device side there options that include GPU and FPGA. There are devices trying to use existing DSP or CPU or both for inference work, but quickly that trend is changing and most of them moving towards GPU based devices for Real time inference.
On top programming languages that are used to design and develop applications for Real time inference in Embedded devices. Mentioned list include, Python, C++, Curl Java etc. Where TensorFlow and PyTorch are used to model Deep Learning Networks.
Android Drevices and Linux based Jetson series devices are used in IoT Edge segment. It appears that mostly ARM is used as CPU in embedded device, but this trend appears to be dominating while IBM had introduced Open Power and released Power 9 IC information such that many other silicon vendors can create Power 9 Silicons for Embedded segment. May be Base station segment will use these Power 9 silicons as a embedded processors instead of ARM.
DSP is coming back because huge number of AI applications in Speech and Audio segment. Where DSPs can be low power compared to FPGA and GPU. Thus there is place for DSP as well in embedded segment.
Deployment of trained model on to Edge requires lot more care and hard work. Following sub systems configuration is given for IoT Edge. In this, Android phone working as IoT device, It will work well for small size model. Deploying huge model on to Android phone might create challenges and may not work well if reduce model size to fit in Android phone .
DLtrain is a platform which is used to deploy Deep Learning Networks in above devices. Illustrating examples are shown for Android phone and Jetson Nano series.
Training platform is different from deployment platform. Same provides obstruction to carry trained network in to limited capability deployment edge. Mostly there is a need to cut down Model size or optimise weights of each node in Model. This mentioned optimisation of Model size of node weight may not be there if deployment happens in cloud side of infrastructure.
DLtrain is shown in the above picture. DLtrain is used for NN / CNN traning and also deploying NN/CNN on target devices. Above picture shows Jetson Nano as Target device which is Edge AI computer. Where Jetson nano is connected to IoT node via SPI interface of Jetson nano. Sensor data is read by using SPI interface and IoT node is having interface with Sensor. Jetson Nano also having connection with IBM Watson IoT service. User can create their applications in Cloud by using Node-Red and IBM Watson IoT platform. DLtrain is designed to work in the above configuration and also support Deployment engineers to deploy Deep Learning Networks in IoT Edge devices.
DLtrain is used handle following AI Inference workflow at IoT Edge.
Inferencing by using NN model.
Inferencing by using g CNN model
Above mentioned applications ported onto Android phones. Android NDK is used to make above computing methods from Java. And also for Mobile phone section, J7app is an example application in Android phone.
Above mentioned applications ported onto many Edge AI computers that are using Jetson series.
DLtrain platform is used in the above and development took place from fresh thinking and innovation by using know-how on embedded system requirements. DLtrain is designed to make user silicon EDGE computing ready for AI work load. DLtrain is a perfect tool to handle issues in porting trained AI models in Edge AI computers with ease.
Case Study : Deploy AI application in IoT Edge computer
Step 1: Run MQTT client in IoT Edge
In this case Jetson Nano is used.
Local support will be given to run python version of MQTT client to work with IBM Watson IoT platform.
Step 2: Run TCP/IP socket Server to connect with IoT Nodes
Embedded board is working as IoT node. ARM CPU is used in IoT Node. Local support will be given in the form of Sample code for IoT Node.
Step 3: Run DC motor Control app in IoT Node
Embedded board with ARM cpu is used as a IoT node.
Local support will be given in the form of Sample code for IoT Node.
Step 4: Create IoT device in IBM watson IoT platform
Local support will be given to create IoT device in ibmcloud. But user need to have account in ibmcloud.com , in case not then new account will be open during event for each user in https://cloud.ibm.com/login
Step 5: Create App in Node-Red
Local support will be given to create app in Node-Red in ibmcloud
Step 6: AI Application in IoT Edge
Run Image Classification app in IoT edge and run a motor if given image is Banana. And also update MQTT broker in IBM watson iot platform such that MQTT broker can publish this news on Banana to app in Node-Red
