These Metrics Make No Sense

[LeiShen]

Let’s Start with the first one: Accept Rate

According to this graph, I have a current accept rate of 41.18%, right? However the “top 10%” of 3DHubs have and accept rate of only 25%. So the top rated hubs reject more orders than I do? So in order for my hub to get into the top 10%, I need to reject more orders? What exactly is this graph trying to tell me? What target am I trying to achieve? It looks to me that if I were to accept 100% of orders, I would actually be performing far worse that the top 10% of 3D Hubs…

Now, for second confusing graph: Accept/Decline Time

Again, my hubs is accepting/declining order within 1h 57m on average. However, the top 10% of 3D Hubs are doing so in only slightly less than 12 hours. So I need to sit on my thumbs for another 10 hours in order to equal the accept/decline time of the top 10% of the hubs? What exactly are you basing your ‘top 10%’ rating on then? Do you really want slow/lazy responses from your hubs?

Third Metric: Order Comment Time

Again, I reply with a comment to a new order within 32m of it coming in (a difficult feat considering your SMS messaging doesn’t work in China and I have to wait for snail email to arrive). But I digress. 6h 35m is the average time the top 10% of 3D Hubs take to respond to a new order. Again: so I need to take a nap and respond much later in order to get into this top 10% spot?

I ask again: What exactly are you trying to tell us with these crappy metrics? Be slower? Accept Less? Sleep more? I sure hope not, but that’s what these graphics and metrics sure seem to be saying: “The 10% of 3DHubs are rated highly by their very slow response times and high rejection rate”

What exactly constitutes a “Top 10%” hub?

You need to rethink your rating of hubs and your metrics:

Top performing hubs should have the highest acceptance rate:

1) based on highest production quality

2) shortest response times

3) highest acceptance rate (excluding unprintable object, customer non-responsiveness, mis-matched material requests, etc)

4) Length of continuous service (excluding minor outages due to printer maintenance, downtime, vacations, sleep)

If you wish to keep this weird graphs, then they need to change to provide clear, actionable targets for the hubs:

Accept/Decline Time: This target should be the top 10% Accept/Decline times of all hubs, not Accept/Decline times of the top 10% ‘rated’ hubs. In other words, short times are better, not worse

Accept Rate: This target should be the top 10% Accept Rate of all hubs (excluding exceptions noted above), not the Accept Rate of the top 10% ‘rated’ hubs. In other words, a higher accept rate is better, not worse

Order Comment Time: This target should be the top 10% comment time of all hubs (something I’ll never get because 3DHubs v. China), and not the target of the top 10% rated hubs. In other words, shorter comment times are better, not worse.

Personally, I think all I’ve stated here is pretty obvious and logical, but given the length of time these messed up graphs and metrics have been up on 3DHubs and not corrected, I guess not.

Perhaps these will be acted upon soon, but given that absolutely none of my other posts, emails and rants on errors in 3DHubs have been correct over the past 6 months, I hold little hope:

Districts listed as cities

Non working SMS messaging

Non-vetting of location of customers

Non-vetting of print material on orders

Poor vetting of printability of objects on orders

Horrible geographic search results around the world

New One: Free shipping can be set on orders over set totals, but none of that information is visible to a customer.

Thank you for your time. I’m sure this will be files in the circular bin shortly.

[cobnut]

Hi @LeiShen I’m pretty sure the top 10% refers to Hub performance overall, not the top 10% of those particular metrics. So the figure for Accept/Decline is not the top 10% of “best” results for Accept/Decline, but the average accept rate of the top performing Hubs overall. How can the best performing Hubs have what look like poor figures? Well…

Short Accept/Decline times are not necessarily better, it’s not a race to see who can hit the Accept button first. I often enter into a dialogue with my customers before accepting (or declining) an order and those conversations via comments usually take hours and sometimes days as the customer considers their options or even remodels their order.

The same applies to Accept rate. Hubs are not necessarily “better” because they accept a higher percentage of orders. If you accept an order that you cannot realistically print to a sufficient quality, you’re letting the customer down and giving yourself (and 3DHubs) a bad reputation. A good Hub should decline orders that they know they cannot print and if that means a relatively low accept rate, so be it.

I’d agree that Hubs should respond to customer comments promptly and the figure for the top 10% does seem high, but if you ship internationally it’s possible a lot of your orders arrive in the middle of the night and again, there’s a big difference between simply posting any response to get your time down and responding once you have all the information you need and can compose a considered answer.

The comparison figures for the top 10% are not “targets” they’re merely there to provide some kind of benchmark. Bear in mind the dashboard is brand new and still being worked on by 3DHubs so what you’re seeing now is subject to development and change.

[wirlybird]

It is like everything else the hub has been doing … it is meaningless. Someone took a class in some bizarre statistics method and then has been sitting around with buddies at the bar dreaming up these pointless metrics which seem to be based on nothing. I don’t know if they would even be meaningful if they actually were accurate.

I don’t really care about most of them and I am sure the customer probably doesn’t either.

3Dhubs needs to stop worrying about so many irrelevant statistics, that they don’t even measure correctly, and focus on improving the working environment for the hubs and customers.

[LeiShen]

So spitting out meaningless gibberish is supposed to be better than nothing?

From your Hub Dashboard: “See how your Hub stacks up against the top 10% of service providers on 3D Hubs. Improve these key metrics to increase your ranking and grow your orders.”

Sounds like a Target to me…

[LeiShen]

“3Dhubs needs to stop worrying about so many irrelevant statistics, that they don’t even measure correctly, and focus on improving the working environment for the hubs and customers.”

Totally agree!!!

An effort that is apparently beyond their capabilities given the amount of time this metric drivel has gone on and grown…

[keebie81]

I feel like you are trying to make a big stink out of something that really isn’t a big deal.

If you look at your dashboard you will see a big button next to hub performance that says **BETA.**So problems with the data are too be expected. Then under each graph it explains it. As long as you are better then those numbers then you are doing good.

I have noticed in the last few days the numbers have been changing often. I think they are still working on it and since its in beta that is to be expected. Previously the accept rate of top 10% was around 75% i think. Hub rating was around 4.8.And i think accept/decline time was 7hrs or so.

Also if you ever have questions on how they get the data support is generally helpful and will explain it. An example is I asked how the accept rate is calculated and was told it is your last 25 orders.

​

[LeiShen]

First, their comparison of what is better is Not better.

Second, its may be Beta, but 3DHubs is ranking us on total B.S: “See how your Hub stacks up against the top 10% of service providers on 3D Hubs. Improve these key metrics to increase your ranking and grow your orders.”

So we are to increase our rankings based on broken Beta metrics to grow our orders.

There was a saying in our I.T. dept. (Telephone Industry) that management always followed but never admitted to: “There is never enough time to do it right, but always enough time to do it over”. We were rushed to get crap out the door, even if we told them it wasn’t ready, but if it was wrong, our heads rolled. And we ended up spending more time trying to fix crap than had it actually been done correctly the first time.

And , YES, I am making a big deal out of this because 3DHubs is!!!