I found out this week that sometimes goods and services purchased in China can be of low quality.
Disclaimer If you purchased a QR clock and are reading this, don't worry. I have every intention of shipping you exactly what you paid for. If your clock is found to have any issues or if you are in any way dissatisfied, don't hesitate to contact me, and I will do whatever I can to make it right.
I just spent last weekend installing the LED modules on the QR clock PCBs that I discussed a few posts ago. At the time of writing, I was really impressed with the overall quality of the soldering job on the PCBs I received from Myro, and although I wasn't exactly looking forward to the prospect of soldering for 20 to 25 hours, I was excited to finish my QR clocks and ship them to my customers.
Unfortunately, I ran into a few hiccoughs.
The QR clocks require nine 8x8 LED matrices to work, and they work best with displays with square pixels. Square pixel LED matrices are apparently a rarity, so I had to get these special ordered from a manufacturer that I found on Alibaba. You can read all about it here.
I got samples from two different vendors and was very impressed by the one I ended up choosing. Their displays were bright, evenly lit, and didn't show any defects like I found with the first vendor I picked out:
Soon after my fundraiser ended, I ordered 500 LED modules from the better manufacturer. I only needed 405 for the 45 preorders I received, but I figured it'd be a good idea to order a bunch extra to install into my extra PCBs and sell to any interested customers in the future.
I was informed shortly after placing my order that there was going to be an unexpectedly long lead time on these displays of 35 days. This wasn't a problem since I was waiting for the PCBs anyway, and sure enough, almost exactly 35 days later, I got the shipment.
They were for the most part very well packaged, and showed very little if any cosmetic damage.
I wasn't planning on trusting them entirely though. Part of the reason I decided to install the LED modules myself was that it would give me the opportunity to test all of the modules before installing them. Removing the modules isn't impossible with a hot air gun, but I'd rather just choose not to install them in the first place.
Well, about 10 modules in, I found a defective module:
No bother. Set it aside and move on...
Well, I mean, I bought some extras right?
Alright, I'll save you some scrolling:
38 of the 500 displays I ordered showed at least one pixel missing. That's a 7.6% failure rate. More importantly, these things cost me $2.67 a piece, so that's around $100 worth of product I'll be throwing in the trash.
The good news is that I still had more than enough displays to fulfill all of my preorders with around 5 complete clocks to spare (I'm still in the process of assembling the extras as of time of writing).
The bad news is that some of the modules that passed my initial test on the breadboard showed problems after they were soldered in place. It seems that some have intermittent internal connections that may have been broken through the soldering process. I'm really hoping that none of the working ones I've already installed and shipped have this problem.
I'm not certain if this level of drop out is to be expected with parts like this.
I contacted the vendor two days ago and still haven't gotten a reply. If it comes down to it, I might try to file a complaint with Alibaba, but I'm not naive enough to expect this issue to be resolved in my favor.
Not two hours after I posted this, I got this email:
I very much regret the difficulties you have been with our company.
I checked with our production Dept again, we would like to remake 30pcs goods for you. (This was in response to an email I wrote before I found the last 8)
After finished, I will text them one by one. Please don't worry. And the shipping cost, we will pay it.
Apologize for the inconvenience cause by our error. Kindly contact me if any problem!
Looks like everything's coming up Milhouse!
This is a major inconvenience for me, but at least they're taking the effort to make the situation right.
In addition to this issue, the new LEDs are not quite as bright as the samples I was sent for the prototype:
This is not as big of a deal however because at maximum brightness, the original QR clock prototype will often force a cellphone camera to increase the shutter speed to the point where you start to see refresh artifacts:
These interfere with the ability to scan the display. I tried mitigating this effect in the firmware by increasing the refresh rate, but even after maxing out around 240Hz, the problem still persists.
Basically, in any environment where the maximum brightness of the sample LEDs is required for contrast, the clock couldn't be scanned anyway because of refresh artifacts. Besides, the new LEDs are plenty bright for indoor use anyway.
Still would have been nice if the samples matched the product though.
In my previous article where I discussed ordering these QR clock PCBs, I raved about how good they looked. After soldering on the displays and installing firmware however, I was less excited:
Something like 50% of the assembled clocks had at least one row of LEDs blacked out.
Because all of these LEDs were on the same row of the same display module, I had reason to believe that the problem existed outside the displays themselves. The LEDs in the display are addressed by row and column, so an entirely dead row of LEDs tells me that there's something wrong with the components addressing them.
A quick visual inspection of the parts on the backside showed no immediate issues. I ran some more solder over the IC to clear up any potential cold solder joints, but it didn't fix the problem. Only after adding some solder to the current limiting resistors did I see something I've never seen before.
The resistor split in half.
Seriously, right down the center.
Apparently, a large portion of these resistors had very very small hairline fractures through them which are invisible to the naked eye.
Let's run through one such case. A resistance measurement across this 62 resistor shows a resistance of:
A (very very) close visual inspection shows a very faint crack:
And any attempt to rework this resistor renders it in twain:
This is one of the stranger things I've ever seen happen to a PCB. The good news is that it's very easy to fix by replacing the broken resistors with fresh new ones. The bad news is that it can be difficult to find. Some of the resistors were held together so well that the display appeared to be working flawlessly until it was jostled and the halves of the resistor very slightly separated.
As a precaution, I manually flexed the PCB of every clock I shipped out to locate broken resistors. I believe I found all of them (over 30), but there's still a possibility that more will pop up after shipment.
Though the problem was easy to resolve, it still doesn't answer the question of what the heck happened here. I contacted Myro to ask such a question. I went through the typical round of back and forth and finger pointing:
Do you think somewhere is shorting caused it or resistors problem? Or current is big?
Until I finally got a fairly straight answer:
I just checked your order. There was a information about V-cut. Dring separated boards , they felt V-cut too thin, which v-cut was not enoug for separated boards. Maybe they did press too much and broke the resistors. We promise do not happen again. We will control the V-cut for a panel. Or Use machine to cut to single board.
Translated to English, when PCBs are manufactured, they are often made in panels. These panels are large and easier to deal with than multiple smaller PCBs. The components are soldered on in this panel state, and then the panels are separated. One of the methods of separation or "depanelization" is "V-scoring" where channels are cut in the PCB to make it weaker at the separating edges. Once panelization is complete, the boards need to simply be flexed, and they should break along the v-scored edges.
In this case, the v-score channel wasn't deep enough, so the PCBs had to be bent exceptionally far in order to separate the boards. I imagine it went down something like this:
I suspect the edges were bent downwards away from the soldered on components. Because the resistors were on the outside of this curve, they were literally pulled end from end and split in two.
As of writing, I am very displeased with my service from Myro. They have not yet offered any kind of compensation for the time that I've wasted cleaning up their mistakes, and I will not be using them or promoting them again in my future projects.
Besides, in addition to these broken resistors, I also saw a few of these:
and some cases where resistors were missing entirely.
Really shoddy work.
After some more negotiation, I got this:
Thanks. We refunded $30 to your account. You can use it for your future orders.
The margins on their assembly services are very tight, so they couldn't afford to pay me for my time, but it's still something. I would have rather gotten a cash refund, but since I've had so few problems with Myro produced PCBs, I think I will continue to use their PCB fab services and avoid their assembly especially when I have pre-orders to worry about.
To be clear, I'm not disparaging outsourcing manufacturing in general. The system works, and a lot of really high quality products get manufactured in China. I just ran into some bad luck with the vendors I chose who sent me poor quality work after I had already verified the quality of some of their work earlier.
Despite these setbacks, the QR clocks were all extensively tested and shown to work and have already begun to ship. If any problems do crop up in the field, I'm ready to pounce on customer complaints and resolve them professionally. Besides, I never planned to turn a profit with this project but instead to learn as much as I could about the whole process.
I'll elaborate on this with a later article, but if you were to work out the math, I will probably be making less than a dollar an hour on this whole arrangement. The lessons learned, however are invaluable.
Still, I would have liked to have learned these lessons when the quantities were smaller and stakes were lower.