Who Monitors the Monitron?
I do, right here
Posted by William Fulton - 3 December 20216 minute read
This is a guest post by William Fulton from CCL, who has been doing some awesome stuff with the AWS Monitron. This blog post will be published on the CCL blog in the coming weeks. - Chris
The Innovation Practice at CCL is probably the most exciting place in New Zealand to be. It’s like a theme park but instead of roller coasters all the rides are the nearly unsolvable problems at the edge of the economy. Each of them with the same ups and downs, spirals and recoveries hurtling into the future.
To keep up with all this I must keep faith that all potential solutions have value, and all problems can be understood. More important than that is to appreciate how much others know, so to start this story we really need to go back to before I had even seen the Monitron.
Figure 1: The Monitron, in all its glory
Like a drum from over the horizon, I heard people talking about the Monitron, first in company meetings, “we’re really excited to receive the Monitron”. Then I started hearing about it in Team meetings, “it’s going to be the first Monitron in New Zealand”. It really started to sink in when, Josh Hobbs, the CCL Innovation Ventures Lead, started using my name and Monitron in the same sentence.
With this imminent responsibility I first had to learn what I was dealing with, quickly cleared up that it was not the MonOtron, but was instead the multi-piece MonItron, AWS’s cutting edge machine learning driven monitoring device. Six yellow trapezoid polyhedrons, one large, five small. The small ones contain temperature and velocity sensors and low power Bluetooth to send temperature and vibration information to the large ‘gateway’.
With some solid intel I wasn’t as shocked to find that little brown box smiling at me from my desk one morning. I had a great first impression due to the solid build quality of each of the components, the work had clearly been put in to have these be comfortable around heavy industry and the documentation seemed robust and clear for set-up, the team behind this is clearly passionate about what they are doing.
Figure 2: It all started with that damn smile
The plan for the Monitron was to set it up on some vulnerable machinery. This would later be decided to be on two circulation pumps used by a client to stabilise the temperature of plant. At this time the aim was to understand how the Monitron system works from the technician’s point of view.
It was at this point that the pandemic sent everyone back home, so, by this fact, my apartment became the vital testing apparatus for the Monitron. Firstly, Chris McKinnel very kindly set me up with an AWS account, in turn this unveiled the second tool that I needed. The Monitron’s information output is only accessible through a phone app which is only available on android. Given I’m still clinging onto my iTunes account, I won’t be abandoning Apple for a while. Josh Hobbs came to the rescue with an old Samsung phone. With this I was able to connect and to then read their outputs via the app.
Sitting alone in my home office I was struck with a problem. I must test a system designed to monitor industrial plant, and the closest thing I have to industrial plant is a vacuum cleaner, worse than that, I couldn’t imagine the fight it would cause to leave that running 24/7. For the sake of testing, the Monitron sensor sat up on the windowsill recording the temperature throughout the day along with the vibration. Fortunately, there was no great earthquake to measure so the only vibrations measured would be caused by stiff wind or the occasional truck rolling by.
Figure 3: Sun hitting the Monitron just
after noon causing a spike, temperature in celsius.
(No Sensor as the sensor has been disabled)
Figure 4: Root mean squared velocity of my windowsill, highest record during setup.
I then had an idea for an integrated plant monitoring and security system utilising the Monitron alongside the Amazon DeepLens (an amazon enabled camera with edge AI functionality). The plan was to have the Monitron on the given piece of plant, with the DeepLens pointed at the general area. Using the Monitron’s alert system alongside the DeepLens, running a human detection algorithm, I could create a system that records to cloud storage when either case was true, i.e., record the footage only in the situation where a sensor claims that a failure was occurring or when people are within frame, the two major cases to film.
Well, I say “I could create”, by which I mean I immediately asked for the help of the ever-supportive Thomas Claridge who might know more AWS than the documentation. Working with Thomas revealed that unfortunately the Monitron service is currently not accessible through other AWS services like Lambda. This was confirmed by Amazon Support, so for now the camera integrated monitoring system for the pump is a pipe dream.
The results from my windowsill had come in, and what I had thought had been a slow sample rate from the sensor, once an hour, had revealed itself to be a very good long-term method of showing the experience of the sensor through time. It even showed the interesting effect when the sun came through the window in the late afternoon rapidly increasing the temperature of the sensor above the ambient temperature of the room (Figure 3).
With all this in mind, and back in the office, I realised it was time to pull back and just test the limits of the Monitron as a pure system. On site, the distance between the office and the pump was estimated to be about five meters with a wall and an iron fence in between. To test the viability of the Monitron’s sensor range ahead of time I placed the sensor on the break room fridge ~25m away through a wall. The results kept coming in fine, and the system really started to shine as a robust communication system.
After all this, I felt comfortable working with the Monitron, and I was confident that it would be able to function on the coolant pump with the Gateway placed through the wall inside the office.
Figure 5: Pump with sensor positions
The pumping system includes two centrifugal pumps, one which acts as the pump, the other kept for redundancy. The two pumps are alternated between, week on, week off. The pumping system cycles water into coolers in the plant from a heat exchanger which cools the cycled water with groundwater. (Figure 5).
One sensor has been placed on the exit flow pipe above where the two pump egress pipes converge. On the pumps themselves, one sensor has been placed on each of the impeller units, (Figure 5 Blue Dot) while the last two sensors were placed on each of the rigid adaptors as that was the closest somewhat flat surface to the bearing (Figure 5 Red Dot). These three places give us a way to monitor the performance of the pumping system, to monitor the overall temperature, the pumps performance, for both vibration and temperature inconsistencies, and the impeller where the valuable activity of the pumping system takes place.
Figure 6: How the Monitron sensor is attached
Because each of these surfaces have a slight curve, a thick epoxy adhesive was required to attach the Monitron sensors correctly. The sensors require a direct mechanical interface along the majority of the bottom of the sensor. So, epoxy was added to make up the distance between the surface and the sensor (Figure 6).
The sensors paired to the gateway through the app using NFC from the phone. After they are set up, they will be able to communicate with the cloud, and will reconnect if the gateway is turned off, or disconnects, for some time. The sensors do not need any active maintenance when set up, and the app communicates any errors in the situation of a failure. However, after two years the sensors will need to be replaced as there is no way to charge them or replace the battery. With the sensors firmly attached and paired I was ready to receive the data.
After a while of watching the data come through, I finally understood the value of the Monitron. I originally had a mindset of active testing. If there’s a problem with a system I typically want to get in there and poke around, get some measurements, have ideas, test a bit more etc, until I can finally diagnose a solution to the problem. Then I want to get in there, apply the fix, fire off a party popper and not worry about it again until it breaks. I think a lot of people have this mindset to only deal with problems when it’s clear there’s a problem. Surely the important things already have monitoring systems, so any other failure must just be a black swan event, unpredictable. Right?
I realised that I might have approached this ‘examination of the Monitron’ with this mindset. I would come in, slap it on a system, the Monitron would say that its broken, and then I would get to write a big assessment pointing out how the Monitron had heroically saved the day. If you’re hoping for me to do that then I’m going to have to apologise, because that isn’t this story.
Figure 7: Vibration of left compressor at 4mm/s at an ISO warning level, worth looking into
Figure 7: Right compressor, 3mm/s vibration below the ISO warning level. Note the switch of pump on 11/11
What I’ve learnt, and what makes the Monitron a visionary device, is that it opens the domain of retroactive passive testing. I can see, in high detail, the performance of both two pumps, compressors and how they are performing with respect for one another. For example, I can see that the left compressor (Figure 7) vibrates significantly more than the right compressor (Figure 8) when the respective pump is running. But both still stay mostly between ISO guidelines. Could this have caused the incident with the leak that fixed itself? Could the air temperature have influenced the vibration of the machine? Might the changes cause the leak to return?
The value here is that I now have exposure to the metrics of the pump like never before. The natural response is to begin looking for patterns in the data, how does heat change vibration? Is the average more important than the extremes? What effect does the running pump have on the other? In effect this data naturally makes one pre-empt possible problems and act to solve them before they manifest in any meaningful way. That is to say, with the Monitron, there never really is a day to save.
Figure 9: Outlet temperature slowly rising from ~14 to ~16 degrees through November, note that the 11/11 change does not affect the outlet temperature
What makes it so exciting is that it can be easily placed on any currently active machine and could even be considered as a simple monitoring method for a machine in the design phase. After seeing the data coming from the Monitron, other machines seem like they are running naked and on luck.
Using the device has made me aware of some concerns, and some hopes for further development in the Monitron as an ongoing project. The biggest concern that must be addressed is the lack of AWS interface to other AWS services. For substantially large machines or factories, the user needs to be able to filter the information and see it alongside other data, this could be done with access to AWS Lambda.
My other major concern is in the disposability of the sensors. In an increasingly wasteful world, it is a bit disappointing to not have the ability of charge or change the battery in the individual sensors. Perhaps if there was a service offered to swap them out for a fee then I would have less of an issue.
I would be very excited to see future iterations of the service offering different sensors on top of the velocity and temperature sensors it currently contains. Air pressure, off-face temperature, noise level, sensors like this might give a more wholistic interpretation of the machine’s experience and might assist in identifying more patterns or issues with the wider system.
In conclusion I have to say that the Monitron is a brilliant device with a great future. It’s easy to forgive the few small teething issues of a new service when the value proposition is this strong. I am so excited to monitor the Monitron’s future. The ability to retroactively install passive sensing to just about any machine opens a huge opportunity to improve machine maintenance the world over. The design team for the Monitron should be very proud, and anybody in factory maintenance would be short sighted not consider adding this device to their arsenal.