Project on Aisler: AISLER - Quick and affordable manufacturing for your Electronic Project
Page on our website: DDoS
Source project files: Die DOsen Stapler / Evergreen / PCBs / VaCANt · GitLab
We’re a robotics team that competes annually in Eurobot.
This year, after our long drive to the competition location, we plugged a battery into our robot, and the magic smoke
came out. Yes, we destroyed our robot the day before the competition.
How did this happen?
Our best analysis points to the root of the problem being a bad ground connection between our central computer and our satellites.
The positive wire was present, so without a ground connection, the return current flowed through our CAN bus signal
wires. This destroyed our Raspberry Pi and our vacuum controllers we wrote about previously Developing composable, reusable CAN modules for Eurobot.
Luckily, our motor controllers, which are also on the CAN bus, were not damaged.
These are quite expensive and we had no spares.
Losing the Pi stung given the price increases over the last years, but thankfully we had a spare.
However, we did not have any extra pre-populated vacuum control boards.
We tried to replace the blown components on the boards, but they would never work reliably.
Even after exchanging the eFuse and some other damaged components,
the boards would work and communicate over CAN bus, but they would often get stuck in the “sucking” state.
When the robot should put down some game element, it would instead keep holding onto it.
Not only does this not score any points, but it can also damage the robot.
If it tries to pick up another block when there’s already one attached, it will push one block into the other,
damaging the servo motor that controls the arm.
We eventually had to give up on making the boards work as originally intended.
We soldered wires from a microcontroller development board and the solid-state relays that turn the pumps on and off.
This microcontroller was then plugged into the main computer over USB.
The vacuum controller boards then only provided power through the repaired eFuse and provided a mounting location for
the relay. Nothing else from the board worked.
This resulted in a mess of wires that looks bad and is likely to get snagged on something and ripped off the board,
especially given that the robot’s arms have to move through this area.
Indeed, after bringing the robot home from the competition, we noticed that one wire already came off.
Having our boards destroyed took away our valuable time in the last day before the competition, when we should have been
putting the final touches on our strategy. It was also discouraging that the CAN modules we put so much time into were
wrecked so quickly.
What are we going to do about it?
-
Check connections: The most obvious and simplest to implement solution is to always check the connections before powering up the bot. However, this is prone to human error (and human laziness). It’s also not good engineering to have a module so fragile it relies on manual checks to not fry itself. Therefore, we will keep this as an additional check, but not the only line of defense.
-
Have more spares ready: We should have more pre-assembled boards ready as spares. It’s more work up front, but it
would have saved us a lot of time to have boards ready to go. Since these are supposed to be reusable modules,
any extra effort put towards them gets amortised over multiple competitions. -
Don’t solder the pump: Having the pump soldered directly to the PCB through-hole was very inconvenient for
rework. We had to replace chips on the board multiple times when examining the extend of the damage. Each time we had
to unsolder the pump. Since both terminals are connected through a rigid body, we had to heat one up at a time and
wiggle it out. Then to reattach it, we had to clean the holes completely and push the terminals back in. We ended up
ripping some of the pads off after repeat this so many times. Next time, we will attach the pump with wires until we
are 100% sure the board is rock solid. -
Isolated CAN bus: We have redesigned the board to use isolated CAN bus. This means that the zero reference of the
CAN bus and the CAN signals are galvanically isolated from the rest of the module.
I will focus on the last point as it’s the most interesting.
We redesigned the board to use a ISO1044 isolated CAN transceiver.
The galvanic isolation this provides prevents any unwanted current from flowing through the CAN signals, among other
protection. This means that for our specific case of a bad ground connection between nodes, the CAN signal wires would
not serve as the return path for the entire board.
For the isolated transceiver to work, it needs an isolated power domain. If the ground and power pins of the transceiver
are connected to the main power domain, it would completely defeat the purpose of the isolation.
To generate this isolated domain, we use the UCC33420 isolated DC/DC power module.
To maintain a reference voltage between nodes, we now have to connect all of their isolated ground wires.
Previously, we only connected power, ground, CANL, and CANH. Now, we additionally connect the isolated ground,
so our CAN bundle grows from 2 wires to 3.
Each node is connected to some actuator or sensor, but the only wires that cross nodes are power, ground, CANL, CANH,
and isolated ground. This means that any wire could be cut without turning a signal wire into a return path.
If ground is cut, the node will float since power is connected. If power is cut, the board will simply not be powered.
If any of the CAN signals is cut, CAN will not work but it does not damage the board.
We’ve already redesigned the vacuum control board to use isolated CAN and we’re in the process of testing it.
We will soon make a followup post all about the improvements made in the new revision.
We are also planning a custom Raspberry Pi hat with isolated CAN, better power protection, and many other nice features.
We don’t want to destroy another Pi, those are expensive!
A big thank you to Aisler for their continued support of our team! We love our Beautiful Boards.
