Vendor Bender

I found out this week that sometimes goods and services purchased in China can be of low quality.

Disclaimer If you purchased a QR clock and are reading this, don't worry.  I have every intention of shipping you exactly what you paid for.  If your clock is found to have any issues or if you are in any way dissatisfied, don't hesitate to contact me, and I will do whatever I can to make it right.

I just spent last weekend installing the LED modules on the QR clock PCBs that I discussed a few posts ago.  At the time of writing, I was really impressed with the overall quality of the soldering job on the PCBs I received from Myro, and although I wasn't exactly looking forward to the prospect of soldering for 20 to 25 hours, I was excited to finish my QR clocks and ship them to my customers.

Unfortunately, I ran into a few hiccoughs.

Displays

The QR clocks require nine 8x8 LED matrices to work, and they work best with displays with square pixels.  Square pixel LED matrices are apparently a rarity, so I had to get these special ordered from a manufacturer that I found on Alibaba.  You can read all about it here.

I got samples from two different vendors and was very impressed by the one I ended up choosing.  Their displays were bright, evenly lit, and didn't show any defects like I found with the first vendor I picked out:

Soon after my fundraiser ended, I ordered 500 LED modules from the better manufacturer.  I only needed 405 for the 45 preorders I received, but I figured it'd be a good idea to order a bunch extra to install into my extra PCBs and sell to any interested customers in the future.

I was informed shortly after placing my order that there was going to be an unexpectedly long lead time on these displays of 35 days.  This wasn't a problem since I was waiting for the PCBs anyway, and sure enough, almost exactly 35 days later, I got the shipment.

They were for the most part very well packaged, and showed very little if any cosmetic damage.

I wasn't planning on trusting them entirely though.  Part of the reason I decided to install the LED modules myself was that it would give me the opportunity to test all of the modules before installing them.  Removing the modules isn't impossible with a hot air gun, but I'd rather just choose not to install them in the first place.

Well, about 10 modules in, I found a defective module:

No bother. Set it aside and move on...

Huh...

Well, I mean, I bought some extras right?

Alright, I'll save you some scrolling:

38 of the 500 displays I ordered showed at least one pixel missing.  That's a 7.6% failure rate.  More importantly, these things cost me $2.67 a piece, so that's around $100 worth of product I'll be throwing in the trash.

The good news is that I still had more than enough displays to fulfill all of my preorders with around 5 complete clocks to spare (I'm still in the process of assembling the extras as of time of writing).

The bad news is that some of the modules that passed my initial test on the breadboard showed problems after they were soldered in place.  It seems that some have intermittent internal connections that may have been broken through the soldering process.  I'm really hoping that none of the working ones I've already installed and shipped have this problem.

I'm not certain if this level of drop out is to be expected with parts like this.  I contacted the vendor two days ago and still haven't gotten a reply.  If it comes down to it, I might try to file a complaint with Alibaba, but I'm not naive enough to expect this issue to be resolved in my favor.

Update
Not two hours after I posted this, I got this email:
I very much regret the difficulties you have been with our company. 
I checked with our production Dept again, we would like to remake 30pcs goods for you. (This was in response to an email I wrote before I found the last 8)
After finished, I will text them one by one. Please don't worry. And the shipping cost, we will pay it.
Apologize for the inconvenience cause by our error. Kindly contact me if any problem!
Looks like everything's coming up Milhouse!

This is a major inconvenience for me, but at least they're taking the effort to make the situation right.

In addition to this issue, the new LEDs are not quite as bright as the samples I was sent for the prototype:

This is not as big of a deal however because at maximum brightness, the original QR clock prototype will often force a cellphone camera to increase the shutter speed to the point where you start to see refresh artifacts:

These interfere with the ability to scan the display.  I tried mitigating this effect in the firmware by increasing the refresh rate, but even after maxing out around 240Hz, the problem still persists.

Basically, in any environment where the maximum brightness of the sample LEDs is required for contrast, the clock couldn't be scanned anyway because of refresh artifacts. Besides, the new LEDs are plenty bright for indoor use anyway.

Still would have been nice if the samples matched the product though.

PCBs

In my previous article where I discussed ordering these QR clock PCBs, I raved about how good they looked.  After soldering on the displays and installing firmware however, I was less excited:

Something like 50% of the assembled clocks had at least one row of LEDs blacked out.

Because all of these LEDs were on the same row of the same display module, I had reason to believe that the problem existed outside the displays themselves.  The LEDs in the display are addressed by row and column, so an entirely dead row of LEDs tells me that there's something wrong with the components addressing them.

A quick visual inspection of the parts on the backside showed no immediate issues.  I ran some more solder over the IC to clear up any potential cold solder joints, but it didn't fix the problem.  Only after adding some solder to the current limiting resistors did I see something I've never seen before.

The resistor split in half.

Seriously, right down the center.

Apparently, a large portion of these resistors had very very small hairline fractures through them which are invisible to the naked eye.

Let's run through one such case.  A resistance measurement across this 62\Omega resistor shows a resistance of:

Infinity.

A (very very) close visual inspection shows a very faint crack:

And any attempt to rework this resistor renders it in twain:

This is one of the stranger things I've ever seen happen to a PCB.  The good news is that it's very easy to fix by replacing the broken resistors with fresh new ones.  The bad news is that it can be difficult to find.  Some of the resistors were held together so well that the display appeared to be working flawlessly until it was jostled and the halves of the resistor very slightly separated.

As a precaution, I manually flexed the PCB of every clock I shipped out to locate broken resistors.  I believe I found all of them (over 30), but there's still a possibility that more will pop up after shipment.

Though the problem was easy to resolve, it still doesn't answer the question of what the heck happened here.  I contacted Myro to ask such a question.  I went through the typical round of back and forth and finger pointing:

Do you think somewhere is shorting caused it or resistors problem? Or current is big?

Until I finally got a fairly straight answer:

I just checked your order. There was a information about V-cut. Dring separated boards , they felt V-cut too thin, which v-cut was not enoug for separated boards. Maybe they did press too much and broke the resistors. We promise do not happen again. We will control the V-cut for a panel. Or Use machine to cut to single board.

Translated to English, when PCBs are manufactured, they are often made in panels.  These panels are large and easier to deal with than multiple smaller PCBs.  The components are soldered on in this panel state, and then the panels are separated.  One of the methods of separation or "depanelization" is "V-scoring" where channels are cut in the PCB to make it weaker at the separating edges.  Once panelization is complete, the boards need to simply be flexed, and they should break along the v-scored edges.

In this case, the v-score channel wasn't deep enough, so the PCBs had to be bent exceptionally far in order to separate the boards.  I imagine it went down something like this:

I suspect the edges were bent downwards away from the soldered on components.  Because the resistors were on the outside of this curve, they were literally pulled end from end and split in two.

As of writing, I am very displeased with my service from Myro.  They have not yet offered any kind of compensation for the time that I've wasted cleaning up their mistakes, and I will not be using them or promoting them again in my future projects.

Besides, in addition to these broken resistors, I also saw a few of these:

and some cases where resistors were missing entirely.

Really shoddy work.

Update

After some more negotiation, I got this:

Thanks. We refunded $30 to your account. You can use it for your future orders.

The margins on their assembly services are very tight, so they couldn't afford to pay me for my time, but it's still something.  I would have rather gotten a cash refund, but since I've had so few problems with Myro produced PCBs, I think I will continue to use their PCB fab services and avoid their assembly especially when I have pre-orders to worry about.

Conclusion

To be clear, I'm not disparaging outsourcing manufacturing in general.  The system works, and a lot of really high quality products get manufactured in China.  I just ran into some bad luck with the vendors I chose who sent me poor quality work after I had already verified the quality of some of their work earlier.

Despite these setbacks, the QR clocks were all extensively tested and shown to work and have already begun to ship.  If any problems do crop up in the field, I'm ready to pounce on customer complaints and resolve them professionally.  Besides, I never planned to turn a profit with this project but instead to learn as much as I could about the whole process.

I'll elaborate on this with a later article, but if you were to work out the math, I will probably be making less than a dollar an hour on this whole arrangement.  The lessons learned, however are invaluable.

Still, I would have liked to have learned these lessons when the quantities were smaller and stakes were lower.

20 thoughts on “Vendor Bender

  1. The information you share on the process of the clocks creation is truly invaluable! It really sheds light on what does it mean to work with manufacturers from China.

  2. Great article, but you need a title that is less apt to induce forehead slapping. Of course goods from China are low quality - that's what we all get when we shop to the lowest bidder! It shouldn't take a project like this to realize that quality control is not something you see in products sourced overseas. It's just a shame you couldn't find a source outside the PRC for those display modules.

    • Not all goods from China are low quality. For example, nearly every Apple product is made in China, and they are famous for the quality of their products. I've done product test manufacturing work in some of these facilities, and as long as you have a tight enough product spec and thorough testing protocols, you can produce some really awesome stuff. Besides, I didn't order from the lowest bidder. I immediately tossed out the first manufacturer I talked to (which were half the price) because of the poor quality of their products.

      The subtitle was meant to poke fun at the typical China stereotype. Some stuff in China is very cheap, but some of it can be very high quality if you take the necessary precautions. I went through several steps to vet my vendors and only ordered from those who I had a preexisting relationship with, but still got screwed by what I'm going to attribute to bad luck.

      The point of this post wasn't to disparage China as a place to get things manufactured, it was to document the particular problems I came across, and how I dealt with them.

      • I agree with Ch00f about Chinese quality. I have experienced many of the same things that he's gone through outsourcing to China. My Company has a factory in Shenzhen that produces outstanding quality work, so in fact not all Chinese vendors produce poorly made products and it is very stereotypical indeed that everything or most is inferior from China. With PCB design and manufacturing your "one off" for mass production and bringing it to market is a dynamic nightmare in itself. Copy and pasting your single eagle drawing to make a large board to ease cost with V-Grooving along with several other methods and tricks takes a lot of trial and error to gain the experience to make perfect production runs on the first try. Knowing the other dynamics involved with parts, like components breaking by scoring the v-groove, plus waste while manufacturing and so many things I could mention can and will save you huge amounts of lead time, time to produce on the machine and a whole lot of money. I would like to share a link to an awesome video in 2 parts that I guarantee will educate anyone from a very experienced point of view about larger scale PCB design for manufacture. I thought I knew mostly what I needed to know from experience, but this video I caught a couple of years ago has really given me the edge to do it perfectly the first time. It is very informative and entertaining I promise. Now many will already know several things mentioned already, but I promise there's a ton of knowledge you'll find in there that will enlighten anyone on this subject experienced or not. I hope this helps you Ch00f as well as others as much as it has helped me. Daves 2 part video can be found here: http://www.youtube.com/watch?v=VXE_dh38HjU

        • I have no personal experience with Chinese manufacturing, but both anecdotes sound very familiar- samples from third party vendors are good to fantastic, but when it comes to the production run the quality seems to inevitably drop a fair bit. It is also well known that quality out of China can be fantastic, but when you scratch under the surface these top quality products are produced in China in a factory either entirely owned and controlled by the manufacturer or it is more of a 'first party' arrangement where the manufacturer and vendor have very close ties (read Apple and Foxconn etc).

  3. Thanks for the interesting read!

    I'm glad you took the time to explain the 'V-cut', that one had me scratching my head hahaha

  4. Interesting article. You are actually very lucky that the vendor is willing to work with you at all- in many cases, the overseas supplier will just cut their losses and move on.

    Best of luck!

  5. Great post.

    This is the biggest problem with outsourcing to China and buying from Alibaba.

    Outside of the ones you already mentioned, do you recommend any other PCB manufacturers and sellers on Alibaba? Thanks!

  6. Pingback: Vendor Bender « adafruit industries blog

  7. You can get high-quality products out of China, but (and I'm sure you know this) companies like Apple and Amazon have staff over there, watching over the manufacturing process, to make sure that proper procedure is followed, and no corners are cut.

    For readers of this blog who would like more info on the subject of outsourcing to China, there's a great book:
    "Poorly Made in China: An Insider's Account of the China Production Game" by Paul Midler. It's about non-electronic products but will give insight on how the economy works as a whole.

  8. Pingback: Making 55 of Something Without Lifting a Soldering Iron | ch00ftech Industries

  9. Pingback: In Praise Of Slow Hardware | Elexonic.com | Breaking News

  10. Pingback: iPhoneNation.com: Apple News and Technology Insiders – In Praise Of Slow Hardware

  11. Pingback: Kleinburd News In Praise Of Slow Hardware | Kleinburd News

  12. What a nightmare - you have unreliable LED modules you may or may not be able to catch in test, and you have unreliable assembly that you may or may not be able to catch in test. Prepare for a high return rate and watch your profits dwindle.

    Sorry - I feel your pain.

  13. Yeah, i ordered a console for fireworks off alibaba a while back. The problems seem to be soldering, i.e. they solder in chain really really fast. add shipping, vibrations and whatnot and i guess one or two can come loose.

    Its not that big of a deal, as you can repair it yourself....but the fact that they dont check the product at all or very little once it is done is of some concern. How fucking hard can it be to power it and check all buttons and functions? Apparently it seems checking product for quality is more expensive than refunding/replacing/etc.

  14. Pingback: The Slowest $380 I’ll Ever Make | ch00ftech Industries

  15. Pingback: Getting Itty Bitty With The Ice Breaker 2.0 | ch00ftech Industries

  16. "Something like 50% of the assembled clocks had at least one row of LEDs blacked out."

    50% of the assembled boards had manufacturing defects, and the best Myro could do was offer you a $30 credit for a future order?! That's crazy. Is there no guarantee on their assembly work? What's their incentive to assemble things properly, if they can have a 50% failure rate but still charge you for the entire production run?

  17. Pingback: In Praise Of Slow Hardware | BaciNews

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>