All the project updates are in this blog. The process of developing a prototype and software is slow. However, this blog will be updated with the latest updates regarding this project.
One small thing every day and at the end of 100 days you will have 2 features and 99 bugs.
Hardware Updates
All the hardware updates are listed below. All the purchases are made with personal funds and as a result of that specific hardware is delayed because the decision to buy is hasn't been made. Ideally, we would never buy anything that isn't useful. But prototyping is experimentation which is never ideal but we try our best and do a lot of research into a product before buying it.
13th August 2020
The first step in prototyping is to build a project NorthStar Headset. Since it is an open-source project, the hardware BOM (bill of materials) is available for free but often sourcing individual parts is very expensive. Luckily the developers behind that project have created a kit for other devs to buy. Smart prototyping sells kits and parts for NorthStar headsets. For our project, we bought Kit A as the other kits were very expensive and 3D printing can be done locally. Also, the 3D printed parts are under constant development and it is easier and cheaper to print them locally until a final prototype version is available.
This kit arrived in the first week of August. But since the 3D printer is not available until the end of August, the prototype build hasn't started. It is expected to begin in the first week of September. More updates will be posted here.
Once the hardware is ready, a computer needs to run the software. Of course, the headset can be connected to a PC but who likes to be tied down with a leash? So for portability, we decided to find a small SBC (Single Board Computer) that can handle AR/VR. But it turns out that buying a machine like that does not come cheap. Now I know what you are thinking, "Raspberry Pi" well, people have tried VR on the Raspberry pi with Unity and they report somewhere about 9-14 frames per second. Since a user's impression of a product is highly dependent on the UI, and users of Infinity NorthStar will wear a headset, the experience of laggy UI will be physically discomforting. Imagine the world around you starts lagging. On some research, it turns out that anything under 90fps is discomforting to the user.
Sigh, we need something quite powerful. Also, it needs to be carried on the user so it cannot be too big either. And these two requirements are what make the SBC expensive. After considering 15 SBCs the research proved that the UDOO Bolt v8 is the best option. It rocks a Ryzen Embedded 1606B processor which is a quad-core, eight-thread CPU running at 2.0GHz (3.6 Boost) and an AMD Radeon Vega 8 GPU. Yes, that is very powerful. It also has a very small footprint 12cm x 12cm. But it costs $418.00. A very difficult decision indeed. The plan is to buy this board as it is the cheapest 1606 board and that is what the project needs. But due to some financial restrictions, this will either be done in September or before November (hopefully).
24th August 2020
With most of the electronics hardware on its way (scheduled to arrive in a week's time) development is finally resuming (after a 2-week stall). I now have access to a Prusa Mk3 3D printer. This week that will be mended and upgraded for better performance and accuracy. This blog will be updated when I start printing the NorthStar headset parts.
With the wheel beginning to spin again, I hope to start printing the NorthStar parts by the weekend starting 29th Aug 2020.
25th August - Printing in progress
Well, I managed to fix the printer and get some of the parts started already! That is 4 days ahead of a target? Not quite, although the NorthStar parts are being printed right now, the printer is not tested thoroughly and with roughly 35 parts to print, we will see some inconsistencies and a lot of redos. Some parts are actually very difficult or almost impossible to print. The primary aim is to get most of the parts printed with the bigger parts being printed first and the smaller ones later.
It is very expensive to print the headset from any online store. But if I can get most of the parts printed on this printer, there is hope to only buy the remaining ones online. Let's see how this goes.
New parts are being printed all day! Some that do not fit the printer volume (200mmx200mmx180mm) or that take longer than 12 hours to print is ordered from wenext.
I have ordered 5 parts from the NorthStar headset because one part was estimated to take 20 Hours! And some others were hard to print without support or did not fit the print volume. Since the aim of this project isn't to spend weeks on printing and frustratingly keep trying over failed long prints, I choose to bite the dust and hand over my wallet :( By the time these parts arrive, the others will have finished printing. The headset should be ready in the weekend starting 4th September 2020.
31st August to - 06th September
3D printing Update - All of the headset parts have either been printed or ordered online. The calibration stand is now printing and is expected to be done by Wednesday. The online parts should be dispatched by Tuesday but there aren't any updates on that yet.
The printing is almost done. The last parts (Calibration stand) are will be printed today and WeNext have dispatched the remaining parts from Hong Kong today as well. It is expected to arrive on the 7th of September.
The parts arrived earlier and over the weekend, I managed to make the prototype! Check out the pictures.
There are some issues with the hardware. It is almost impossible to wear. This is due to 2 main reasons: 1. Some of the 3D printed parts are weak. And when a user wears the headset these parts come out loose. This can be solved by replacing these parts by better manufactured 3D printed parts. But this costs money and until that is sorted, the problem will remain.
2. The cables from the headset are of varying lengths and too many. This won't be a problem when the compute board will be mounted on the user's body. Until then, the plan is to use a USB hub and reduce the number of wires by half.
Other than the problems stated above, there aren't any other issues. The headset is fully functional and now the task is to focus more on software development. Here is a picture of a fully assembled headset:
The replacement parts for the headset are on their way! Also, more cables are on their way. This should reduce the number of cables from the headset to the computer and have longer cables to make it more potable during development.
14th to - 18th September
On the hardware front, there isn't much planned this week. Still waiting for the parts from WeNext, they are expected to arrive this week. Once that happens, I will install them and test the integrity of the head mounting mechanism again.
Software Updates
Software development is what takes a lot of time. The development tools do not cost anything but it takes time to learn and become efficient at using them. We are using Unity 3D Engine to develop a UI.
13th August 2020
The software is divided into two different areas: UI design, calibration using Unity and designing a communications interface between Unity and "the outside world". What does that mean? well, the communications interface will be able to send and receive data to industrial controllers.
This is the progress so far:
The second area is a bit more undecided for now, so the primary focus is to fully develop a usable UI and calibrate the headset using Unity and C#. Once that is done we will need the UDOO Bolt to work on the second part. Since windows require a license for all commercial products, it is generally not preferred for HMIs. It is also generally slow as the kernel takes its sweet time to send messages between its layers. Linux on the other hand is a lot more nimble, efficient and free! Now that does come with its own challenges, we are ready to face them. But this part will be developed later.
24 - 29th August 2020
We will be running Ubuntu 19.04 or 20.04 (the decision will be made next week starting 31st August) on the Udoo Bolt V8. The bolt will be configured with a Transcend 2206 128GB SSD paired with an 8GB Ram module and an Intel 3165AC Wireless Wi-Fi + Bluetooth Module. This is a fairly standard configuration for the Bolt. Later in development, the RAM and Storage will be monitored and possibly changed to get the best 'bang for a buck'.
Udoo has good support for Ubuntu 19.04 so that is the primary candidate for OS but we do not want to run on an older OS as Ubuntu constantly gets more feature and performance updates. Most probably we will start out development in 19.04 and once the first part of the project is done we will look at upgrading to 20.04 or even 21.04 beta (depends on the timeline).
While the 3D printer buzzes by me, the software development continues. Until the headset is ready, I can't really start working on the UI, I can learn and develop the back-end and basics till then. And that is being done. It is difficult to test without a headset and only sensors but expect to see some cool clips here!
The Bolt is here. Exciting times! Some of its components are on their way, I expect to get this bad boy up and running sometime around 2nd/3rd September.
This is a little peek into UI development. Still getting used to Unity and some of the assets but we can now grab windows and move them in the workspace! When the headset is ready everything will be to scale, right now the panels might be a bit big.
31 - 05th August 2020
UDOO Bolt Update:
Two parts were ordered wrong for the Bolt, the Wi-Fi Antenna and the Power supply. The Antenna connector was the wrong type and on a lot of research, it appears that UDOO designed the antenna connector with a very niche connector to save space, meaning only UDOO sell the antenna. Since I couldn't buy the antenna only, I had to buy the M.2 board as well. Same goes for the power supply, the connector is not a standard DC connector but UDOO says that the board can also be powered by the USB-C connector. Not wanting to experiment on an expensive board, I also ordered the PSU from UDOO. Both of these things are due to arrive on Tuesday.
The UDOO Bolt is good to go. All the parts necessary to run it are here and I will set up the Bolt sometime before Friday.
Bolt Update:
The Bolt has been powered on and runs successfully for the first time! It is running Ubuntu 19.04 Disco dingo. However, there is a slight hurdle......
Since it is a powerful board, it gets warm and eventually, runs hot when doing anything useful. And this unit does not have a case yet, meaning, I cannot use it until I have something to keep this in. It is very unlikely that a case will be ready and printed in the next week so looks like we need to stop working on it (just as I started) and start once this has a case. :(
UI Update:
Machine UI is going well. There is a problem which is expected to be fixed this week along with the addition of more windows.
7th - 11th September
This week's aim is to calibrate the headset and test of all the UI developed until now, works as expected. Once calibration is done, the UI windows and elements will be resized to be functional and good looking.
The calibration is almost done. Currently, it works well, but due to the lack of the head mounting mechanism, it is almost impossible to calibrate perfectly. Hopefully, with the new parts, this will be fixed. Currently, there is no ETA on the new parts but I expect them to arrive sometime by the end of next week. Same for the cables. Till then, I will be working on a 'confidential' UI for an application that cannot be made public.
I shall alternate between the new UI development and TCP Client implementation.
Unfortunately, due to lack of access to a monitor, the Udoo Bolt development is haled for 3 weeks.
An interesting development regarding Microsoft Azure got me a bit excited. I have set up an Azure account, it is slightly more difficult to integrate it into unity but once it's done, it should work flawlessly. This project could act as a gateway into IIoT for controllers that do not have such functionality. Also, this supports the original multi-device support plan and moves the server onto the Azure network so the same scene can be recreated anywhere.
14th - 18th September
This week's plan is to try to add Telnet to Unity and hopefully communicate with a virtual controller to test. I have found a couple of good telnet libraries and tested one on the command line in Windows 10. It's just a matter of including and adding some code to a C# script into unity.
'IF' I manage the do the telnet integration into unity, I will continue developing the UI.
Update (14th): Well that was quick. One of the most productive Mondays till date.
We can now communicate with a MODBUS TCP Master!
Notes: This only works with Windows for now. It should work regardless if the IP is local or not. I will clean up the files and wait to set up on Linux. Don't want to go too deep into windows and end up having to change the target OS.
Throughout the week, I created a storage container for and some UI panels for a *confidential?* project. Can't disclose what it is until its complete. Mostly because it won't make sense if I try to put it in words.
21st - 25th September
This week's aim is to fix the mechanical parts so that the headset is wearable again. It's my first time building a NorthStar headset so I never expected it to be smooth. The only problem now is a small part which is held together by 3 m2x8 screws. The problem is the hole size is exactly 2 and so the screws don't fit well. After talking to the designers they say the problem is because the part was accurately printed, they saw printers over-extruding so that 2mm hole is actually smaller and the screws fit well. They recommended me to reduce the hole size so I did that.....but the extruder on my printer clogged. So in trying not to spend too much time fixing it, I shall try the ultimate thing to fix the parts, SUPER GLUE. Yes yes, I know its dirty but I really need to wear the headset to further develop it. Besides I have already altered the part for smaller holes and before the extruder clogged, I was able to almost print a part and it came out well. So considering this part fixed, I will try to 'make it work' for now.
On the software front, I spent most of last week trying to learn how to use Microsoft's Azure service to upload data and to keep sync between multiple headsets. This is getting deep into the territory and so there wasn't any example code or help available, I will still keep going at it and hopefully find a breakthrough. Hopefully next week I can continue UI development again.
ooh and I almost forgot
We have a sponsor now!
I have a long(almost 2 years working I think) work history with Trio and so they were happy to provide some financial aid to help me finish the project. Two weeks ago, I mentioned that the project costs are getting too high for me to bear alone and that is when Trio stepped in. With their help I got the broken 3D printed parts ordered along with some cables that will help reduce the number of cables from the headset. As I said before, developing this prototype is expensive and I am thankful to Trio. Go check out their website! You will now see the 'sponsor' name on the Infinity NorthStar page and the 'About Page'.
28th September - 2nd October
Things are getting back to normal this week. I will work more on the Udoo Bolt since I have access to a TV/Monitor now. Everything is installed on it today and From tomorrow I will start to load the project file to see how/if my current projects work. At the end of the week, I will change some of the cables on the Headset so that fewer cables are attached from the computer to the headset.
Unfortunately, I do not have any images/videos for the last couple weeks, hopefully, I will have some at the end of this one :)
5th - 9th October
This week's aim was to so tests on the prototype and see if the device is comfortable, durable and most importantly, works as expected. It is somewhat comfortable. Wearing a large device on your head will always take some "getting used to". The weight of the device is not an issue, the cables pulling from the back offset the front heavyweight of the headset. I have tried tightening up the headset to the max and it is still comfortable. Only time will tell how durable it is. I will keep a record of any weak/broken parts over time. And finally, it works better than I expected! Maybe it was the amazement if wearing AR glasses for the first time, but the interaction feels very natural. One thing I realized was that the FPS (frames per second) needs to be quite high. When a user shakes his/her head, the UI panels are translated perfectly into space, giving the illusion that they are an object in the physical space. But the text on the elements does looks burry and that kind of breaks the illusion. A higher FPS would easily solve the issue. This is a video from initial testing.
12th - 16th October
There isn't much progress this week. The main focus was to get the Unity project running on the Udoo Bolt to see how well it can run. There were several issues while importing the Leap Motion multi-beta service. Some libraries like 'NUnit' weren't present and adding a GitHub repos of that caused even more compilation errors. In short, it was a mess and I wasn't able to sort it out this week. Another way to get the project running on the Bolt is to build a Linux application from my Windows 10 Machine. And that did not work either. The build returned a compellation error which made no sense. According to Unity's forums, that particular error was fixed a year ago and so any workarounds from back then do not work now. Another way to get it running is to build a Windows application and then run it on a VM on Linux. I know that is weird but that did not work either. Unity hands and uses 0% CPU resources when I try to build a windows application. I did resize some of the windows in the UI to make them less comical in size and seem more natural. Next week, I will give the Linux thing another go!
19th - 30th October
Since I am also doing a full-time masters degree, unfortunately, there will be a couple of weeks where I have to focus on my assignments and university work rather than this project. The last 2 weeks have been exactly that.
Having said that, there is still some progress.
This is the Universal Robot UR5 that my university owns. Thanks to the pandemic, this year, no one is doing any projects on this little guy! So I teamed up with a friend to work on a group project. My part is to integrate the headset with the robot controller. There arent any specific goals as of now. My plan is just to be able to show different errors, tools (maybe), target points, etc.
2nd - 6th November
This is what I have been able to do so far. At this point, it is critical to get this project running on Linux because it is restricting movement as it's connected to my laptop (which isn't pocketable). So for the next 2 weeks, I will focus mainly on getting this thing to run on the UDOO Bolt. If it is too difficult, I will have to switch to windows in desperation. Let's hope that day never arrives.
10-20th November -- The switch to windows
After tying to get the project working on Ubuntu, I found out that Leap Motion does not support their multi-beta service on Linux. Shame. But, we keep moving, developing! I have switched the Udoo Bolt to windows 10! And with all the installations underway this week, the project will continue development on win 10 until Leap Motion decide to work on their drivers.
On the up-side, I wont have to struggle trying to get drivers installed on Linux, it gets hairy real quick. I will find out a way to get the case for the board printed.
The plan is to get the board running the project in the coming week and find a battery pack for it to run on. Also, UI and comms development will be back on track, now that I don't have to focus on the OS so much.
This is the initial case design for the project. Once I confirm all the details are perfect, it will be 3D printed, hopefully in the same colors you see here.
A friend designed it for me. Since I will need to use the bolt soon, I am planning to get this printed.
23-27th November -- A step towards portability
The project has been tested on windows 10 running on the Udoo Bolt. The Bolt does not have a Display Port. I will be testing a a few adapters to see which one works, to convert DP to USB-C.
Speaking of USB-C, the board can be powered on with the USB-C port. The plan is to buy a power bank to power the board. When both of these goals are successful, the headset will be completely portable.
The UI development has come to a halt. The focus is still on getting the communications working. Once that works well, I can decide what data I can and cannot read from the controller. The project is expected to be completely portable by mid-December.
28th November - 9th December -- The final stretch for hardware
These two weeks have been focused on being portable.
There are a couple things in here that are new: Battery and 3D printed case.
The batter is a 60W power bank that powers the board though USB-C. The board requires 20V at 3A to work and the battery can provide that. I will be running tests on the board in the coming weeks to measure the battery life. From the first test, the battery powered the board for about 75 minutes. The battery was not full charged, it was at about 65-70% and the board was not running a application, it was running the Unity project which is expected to take more resources.
This is the first prototype of the case. There are a couple issues with this which have been fixed and the second prototype is being a manufactured. It is expected to arrive next week.
December - March
The UI has been developed further during the last few months. As England went into lockdown again, my access to University labs was cut off, restricting me from working on the real robot controller to develop the communication module. Having said that, the development is still underway. I have gotten more comfortable using Unity3D, understanding more features.
December: The new 3D printed case arrived. Here are some pictures of it.
While this prototype was better than the last one, the tolerances on it were too tight. So the design has been modified to add more than sufficient tolerances to the case. But another prototype hasn't been made due to a lack of funds and closing of university labs.
I can now also generate .exe application files from Unity. This will make the application run faster as it does not have to run through unity.
January:
This month didn't see any development as I was busy with university assignments.
February:
During December I fixed errors that were hindering good Unity builds so the project can be run as in independent application. After generating a few builds, it was clear that the UI isn't visible in builds but it is during test runs from Unity. After a lot of searching on the internet, it found to be a screen scaling issue. There are two ways of fixing this: fix the scaling for the screen resolution or make it a 3D object.
I have exhausted the first option and it seems to make UI development a massive mess. The plan for now is to continue using Unity. Meanwhile, I plan on trying to make this into a 3D object.
March:
The development is slowing down this month again as university assignments demand my time. But, the plan is to integrate python scripts to get at least basic communication with the UR5 robot controller running. We have a script that controlled the UI on the real controller from a python script run from a raspberry pi 3b+ right on the last working day (before the lockdown).
April and future plans ?
My focus remains on doing well at university. It is expected to finish in June. I will keep working on this project, be it at a slower pace till then. The aim is to work on integrating all the various gestures the leap motion drivers can recognize. For example, you can make a gesture to move a window towards you.
The UI will also be developed further to support pining of windows to the user, so that the window follows the user's head, like google glass does, this will be helpful to see status of the machine.
After June/July, I will take a deeper look into updating this project into the newest Unity release, Leap motion SDKs and next windows releases. This requires a lot more work than just developing UI and I have held this off until I have more free time. This update will also take advantage of all the new features from the new Leap motion drivers.
Experiments
Read this section for entertainment or a peek into different technologies. Do not expect any of this to be a part of any current or future projects.
This section is full of experimental things we are trying out to see if any of the different technologies would make the main project experience better. It is always fun to play with new tech and this is a small distraction in the hope of finding something new.
Keep in mind that this is highly experimental technology and it might not be useful or possible to integrate into the project. It does give us some experience with the technology for involvement in current or future project if possible.
Cubemos Skeletal Tracking
Cubemos is a Skeleton Tracking SDK allows tracking of 18 joints simultaneously. AI algorithms help in tracking up to 5 people in a scene.
This one is very interesting and immediately caught my eye when I saw it on Intel RealSense website. Since project NorthStar has several cameras used for different purposes, the idea was to add this SDK to the mix and then think of a way to use it. Wouldn't it be cool if you saw UI panels over people with your headset?
We can't just track people but also their actions. What if we can use the high FOV cameras on the headset and be on the lookout for any humans entering the robot's workspace? That is just one of the applications I can think of but since we are tracking people, the possibilities are endless.
What's the catch?
Well, there are a few. First, Cubemos only works with Intel 6th to 10th gen CPU and Intel GPU. And since this is an Intel product there is no hope for support with AMD processors.
Second, it will need a fairly capable CPU and GPU. So the software might be cheap but it needs some powerful hardware to run on. And last, it is not currently compatible with the T265 tracking camera. Weirdly it is compatible with all other RealSense products. I guess something about the fisheye lens makes the algorithm go bananas. But it does work with the Leap Motion controller.
So, is there hope to see this be used with Infinity NorthStar one day? No. It will take me a while to learn and understand this technology. And since this is not a part of the original plan, it will not be a part of development. Maybe one day everything works perfectly and I have time to work on this model. One can only hope.
Depth mapping with T265
This experiment tries to use the stereo cameras from the Intel RealSense t265 to build a depth map. Theoretically it sounds like a good idea. So then why would Intel make the D-series and not provide depth mapping in the T265? Are they greedy? Well, no. This is the quality of depth maps the T265 can provide.
Although it looks salvageable, its simply not good enough. The stereo cameras are Monochrome, meaning no color information. And light sources mess up the depth maps. This is the reason the D-series cameras exist, they also have an IR sensor that can measure depth directly. And it can do position tracking too! Does that mean I should have gone for that one instead of the inferior T265? Well, no again. Its the same problem, but reversed. The D-Series cameras cannot do position tracking that well. Since tracking is a central part of the project, the T265 is a good fit for now. Maybe later on, I can try using one of the D-Series cameras and see what happens.