Taking up a Lifeless IoT Firm

Now, 5 years after the corporate collapsed, I acquired one in all their indicators to research why the corporate failed. Alongside the way in which I ended up taking up the corporate’s signal management area and writing an exploit to get full management of any indicators nonetheless within the area.

Someday in 2021 I discovered somebody on reddit that was promoting a NYCTrainSign. They have been one of many few who had acquired product from the corporate and have been trying to eliminate it.

Amazingly, the unique proprietor saved the unique packaging.

The signal itself is housed in a wood case that the corporate handmade. I imagine the corporate had employed a woodworker out of faculty to make the instances.

As somebody with little to no woodworking abilities, the case appears to be like fairly good. There are some unhealthy corners nevertheless it wouldn’t look misplaced in any dwelling.

The signal internally was comprised of

• 2 LED Matrix Panels
• 1 Raspberry Pi 3
• 1 4GB MicroSD Card
• 1 LED Matrix HAT from Adafruit
• Wiring for the facility provide connection
• Wiring for the LED Matrix Panels
• A small button wired to GPIO on the Pi

Whereas the case is ok, internally it shouldn’t take a lot to inform that the signal just isn’t that properly made.

The Pi was solely half screwed on by two free screws. The button (supposed for reset) was type of simply hanging round. There’s additionally an enormous gap for some unknown motive. The ability provide connection looks like it may break with an excessive amount of motion and it was really unplugged after I first received the signal.

A invoice of supplies or BOM is a listing of the uncooked elements that go right into a product. Most electronics initiatives, particularly critical ones, can have an in depth BOM that describes the merchandise and worth that it goes for.

Usually the ultimate value of the BOM shall be a lot decrease than the retail worth due to the price of delivery, R&D, revenue margin, and so forth. Small modifications in BOM worth can have a big effect on the ultimate value of an merchandise. It’s not unusual to change distributors or components to save lots of simply cents on the BOM.

One trick I take advantage of is that multiplying the BOM value by 4 will usually get you the retail worth.

With entry to an indication we are able to put collectively a hypothetical BOM:

• Raspberry Pi 3 – $35
• Adafruit LED Matrix Hat – $25
• LED Matrix * 2 – $60 ($30 every not less than)
• 5V 2A Energy Provide – $5 (Greatest Guess)
  • Suspiciously this isn’t sufficient amperage to energy every little thing at full energy draw. Most guides suggest 4 to 10 amps.
• 4GB MicroSD Card – $7 (Greatest Guess)
• Wooden Case – $15 (Greatest Guess)
• Miscellaneous Gadgets
  • Wiring, Buttons, Cabling, Screws, Packaging (the signal had some cardboard, a sheet of paper, and a few bubble wrap and foam) – $3 (Greatest Guess)

So roughly the BOM value was $150. If we have been to use our pricing trick, the instructed retail worth must be not less than $600.

Based mostly on archived webpages the NYCTrainSign workforce was promoting indicators for $600 however there are articles mentioning prices like $300. Some individuals point out purchases for around $200 and even only paying $100.

It appears there have been additionally plans to lease the indicators at $30/month as properly.

$300 is clearly too low of a worth primarily based on the BOM however but $600 might be an excessive amount of. It’s only a signal in spite of everything.

Renting may have been an attention-grabbing concept however I feel it could be very troublesome to recoup the preliminary money funding. {Hardware} companies usually must have an preliminary money injection to construct stock after which need to recoup that money and collect revenue as rapidly as doable by promoting their product.

There’s a saying that “throughout a gold rush, you must promote shovels”.

In our story, Adafruit offered the shovels. By utilizing the Adafruit LED Matrix HAT, the BOM value will get inflated by $25 or about 20%. The HAT can be totally avoidable with a bit of engineering work.

It is because:

1. The HAT just isn’t completely crucial. The rpi-rgb-led-matrix library utilized by the signal consists of direct wiring instructions and designs for a passive adapter board. These directions & boards existed back in 2017.
2. A cheaper HAT additionally existed again in 2017. At $2.10 it could have been considerably cheaper than the $25 Adafruit HAT. Additionally regardless of utilizing the dearer Adafruit HAT the signal for some motive glints continuously.
3. More often than not Adafruit components are used for prototyping however are changed with one thing cheaper when going to manufacturing. That is doubtless factored in Adafruit’s pricing scheme.  

Now you may also assume that The Raspberry Pi Basis can be a shovel vendor. The $35 value is so much to abdomen and it’s the costliest single merchandise within the BOM.

Having thought of it, whereas it could appear to be overkill and I would not suggest it, I feel utilizing a Pi wasn’t a nasty choice in comparison with utilizing an Arduino or ESP32. It gave the corporate free marketing and a simple growth atmosphere.

For my part, if the NYCTrainSign workforce didn’t need to make investments upfront engineering time in utilizing a microcontroller just like the ESP32, it could make sense to begin with the Raspberry Pi 3, then swap to the Pi Zero W when it was launched.

Finally they need to intention to change to the ESP32 or comparable and decrease their BOM value additional nevertheless it doesn’t appear to be beginning with the Pi was overtly fallacious so long as the long run plan could be to finally exchange it.

Because the product is Raspberry Pi primarily based, it’s easy to make a backup of the MicroSD card in order that I can discover the filesystem and make modifications as wanted.

The signal’s codebase consists of some customized made Python & NodeJS code in addition to numerous open supply components.

There are 2 major customized elements operating on the Pi:

• The Python server (LED Server)
• The NodeJS server (Config Server)

LED Server

The LED Server written in Python is accountable for drawing to the LED Matrix and getting prepare information from the corporate’s API. The LED server communicates with the Config Server to find out what settings the consumer has configured after which points frequent HTTP calls to a distant server to get information like prepare arrivals and climate.  

With the prepare information, the LED Server will generate a picture or textual content domestically after which render that on the LED Matrix.

Config Server

The Config Server written in NodeJS is accountable for storing consumer configuration in a JSON file and receiving requests to retrieve and replace that file. At boot time, the Config Server will pull the most recent configuration from an HTTP server. As well as, the Config Server will connect with an AWS IoT Core endpoint to obtain actual time config updates from an MQTT server.

Different Parts

• On first startup, Wifi is configured with raspberry-wifi-conf which is an open supply software that can have the Pi create a wi-fi community that the consumer is meant to connect with after which present precise WiFi connection particulars to.
• The Reset button is managed by a small Python script that’s run within the background. When the button is pressed the script deletes wifi settings, resets the hostname, deletes some distant monitoring performance, and reboots the server.
• It appears that evidently the corporate may remotely connect with a terminal on each signal. My signal appeared to have distant management software program from https://www.dataplicity.com/ put in.

Code High quality

Whereas wanting by way of the code, I seen numerous code high quality points:

• Their transit API did not appear to think about {that a} station can have a number of prepare traces. Two traces on the similar station would present as ending on the similar stops.
• No discernable firmware replace course of. Maybe the replace course of would have simply been to have prospects flash the MicroSD card with a brand new Pi picture.
• Numerous the Python code merely makes use of system calls to make modifications with the underlying system
• The Python LED server communicates with the NodeJS Config server to retailer/retrieve configuration. I think this was completed as a result of it was simpler for the workforce to work together with AWS IoT from NodeJS however simpler to work together with the show from Python.
• Frequent mixing of tabs and areas as a result of misconfigured code editors
• Complete git historical past was saved on the MicroSD card
• Bash historical past saved on the MicroSD card
• Little or no code reuse

Getting a Shell

It’s comparatively easy to get a root shell on most Raspberry Pi’s as usually there’s no encryption on the MicroSD card. We will merely boot into single consumer mode and reset the password for the pi consumer.

Nevertheless, whereas now we have a shell now and we are able to mess around, none of it actually issues as a result of the corporate’s API doesn’t exist anymore. The signal was programmed to make use of hard-coded native information when it has no web entry so any information that’s being proven is ineffective.

Recreating the Server

We may in fact replace the area that the signal talks to however fortunately the area that the signal communicates to by default was accessible for buy.

So I purchased it.

With full management of the area, we are able to create a brand new API primarily based on what the signal is anticipating and revive the entire indicators which can be out within the area. Doubtlessly, we may additionally carry out some form of replace to replace indicators to extra trendy software program.

I reconstructed the endpoints for the prepare arrival instances (not less than for NYC) and climate information. For prepare instances I made a decision to make use of the API behind https://wheresthefuckingtrain.com/. For climate information I used the OpenMeteo API.

Getting Signal Management

Like most IoT units, the signal makes lots of system calls. One name instantly concatenates the signal ID right into a shell command. With management of the server in idea we are able to instantly get distant management of any prepare signal.

Since we management the area, in idea, we are able to feed any signal a malicious signal ID and run any arbitrary command. We will then use this to register the signal to our new management server and provides individuals management of their indicators once more.

After handwaving away among the boring information wrangling particulars, our exploit appears to be like like the next:

1. The signal is turned on and it makes an attempt to retrieve configuration. This may loop endlessly till the signal retrieves one thing.
2. The signal will finally ship a request for a picture emblem. This request will include two ID’s distinctive to every signal. We retailer these IDs and create an exploit signal config.
3. On the signal’s subsequent config request we serve our exploit to the signal.
4. We instruct the consumer to restart their signal and our exploit is run on restart
5. The exploit updates any code that’s wanted to pair it with our new server

Here’s a video of the exploit operating:

With the power to remotely serve arbitrary signal information, we are able to formally say that our signal has been restored!

With full management of the area, we at the moment are the brand new NYCTrainSign captains.

Too Many Reductions

I feel the core concern is having a excessive BOM value in addition to promoting lots of indicators at a reduction.

Even throughout a beta I actually don’t assume you may make a product for $150 after which promote it for $117 with none enterprise capital backing.

As we mentioned earlier, even at $300 the product is just too low-cost. The signal ought to have doubtless been promoting at $600 from the very starting.

The product had some concepts round serving adverts however an LED signal isn’t actually a “get big fast” form of firm. So promoting a number of indicators at a loss actually simply serves as advertising.

Too Many Adverts

The NYCTrainSign firm at one level had a Chief Advertising Officer in addition to a Social Media Supervisor and a Social Media Assistant.

This appears extreme for a burgeoning startup.

On the time, the MTA was experiencing frequent delays which the media was closely masking. It looks like the corporate received lots of free curiosity in a short time. It doubtless would have been adequate to coast off of free advertising.

Mixed with the extra adverts and advertising that the corporate was buying, there was lots of demand for the product.

Not Sufficient Product

Nevertheless regardless of the demand, the workforce couldn’t produce the volumes wanted. To not point out that each signal was being manually in-built Brooklyn which is clearly unsustainable with a low retail worth. It additionally appears unlikely that the workforce was conserving sufficient stock available so that they have been in all probability additionally affected by the lead time on sourcing components.

Regardless of having no potential to satisfy orders, they continued to take them in; doubtless in a Ponzi-like try and get funds to satisfy the earlier discounted orders.

See Also

Report Variety of Tornadoes within the NWS Chicago Forecast Space in 2023

At this level, the workforce advised prospects they have been anticipating a 6 month delay whereas they moved their manufacturing to China. Clients who didn’t need to wait may contact their bank card firm to concern chargebacks.

Transferring to China looks like a pipe dream however by no means manifested. Shortly after their announcement the corporate was shut down and their office shuttered.

Too Many Cooks

Based mostly on Linkedin data there have been initially 4 founders. Over time it grew to 11 people after which sooner or later 15 people concerned with the corporate altogether.

At even simply 60k per founder, with the revenue per signal offered being possibly $400, you’ll in all probability must promote ~600 indicators at full worth per 12 months to create sufficient income to run payroll.

One of many founders sooner or later launched a screenshot of their gross sales exercise they usually had reached $250k in income in about two months.

Nevertheless it’s unclear how a lot of that was revenue because it looks like many indicators have been offered at a steep low cost. The $600 worth level appears extreme. Actually the $200 or $300 worth level makes extra sense.

I think about that in a metropolis of about 8.5 million it shouldn’t be too onerous to promote 600 indicators in not less than the primary 12 months. However subsequent 12 months you’d need to promote ~600 indicators over again. And this isn’t together with the fee for every other workers, contractors, and so forth.

For my part, with no higher technique for having a decrease BOM value from the start or some form of recurring income, the corporate would have rapidly grow to be unsustainable. Which it clearly did.

Good Concept, Good Timing, Unhealthy Workforce, Unhealthy Product

After the corporate was lifeless within the water the founders & workers tried to make some sort of consulting firm for some motive.

The founders have by no means come clear about what actually occurred and why so little product was shipped and the place the cash went. No less than one founder says that they personally by no means acquired any cash however the cash needed to have gone someplace.  

This lack of transparency could be what turned the mini darling firm into one thing of a meme.

In the present day the corporate is totally gone and three out of 4 founders have publicly moved on. Lots of the former workers nonetheless record the corporate on their LinkedIn. One founder (the CEO) retains a low profile and may’t readily be discovered on the web.

On the finish of the day, the workforce had no discernible plan about what to do with their product and it actually simply looks like individuals who jumped into making an organization with out considering fastidiously about what they’d do at each step. Whilst you can possibly try this with a software program firm, it’s troublesome to try this within the {hardware} house.

What’s actually aggravating is that I really imagine that if the NYCTrainSign workforce spoke to somebody with a background in electronics they might have been extra profitable. As a substitute it looks like their major advisor was their faculty laptop science professor.

It looks like NYCTrainSign simply took a undertaking that the CEO created in his spare time after which tried to promote it for $300 to $600 with out productionizing it or eager about what may occur afterwards.

In abstract: good concept, good timing, unhealthy workforce, unhealthy product.

The founders & workforce for essentially the most half do appear to be trustworthy individuals. I don’t assume that they had any ailing will.

They merely have been some pals who didn’t have the expertise wanted to construct a {hardware} enterprise and have been caught off-guard by the success of their advertising.

Nevertheless, the CEO, Timothy Woo, ought to come clear about what occurred with all the client funds and what went fallacious.

Whereas I’m undecided if there’s a authorized requirement to return funds, I do imagine there’s an ethical obligation to do proper by your buyer. Particularly if you happen to don’t find yourself giving them their buy. On the very least I feel an evidence is so as.

Once I initially launched into this journey I had desires of constructing my very own signal after which promoting it as a product.

I prototyped a really helpful signal utilizing an ESP32 that I nonetheless use to this present day.

The G and R within the prime proper nook point out rubbish and recycling pickup days. I really discover this to be its most helpful function.

Nevertheless, the extra I thought of it, the extra I felt that I wasn’t the proper particular person to convey this to market. My strengths are in software program and possibly enterprise, not in electrical engineering or woodworking.

To not point out, LED signage is a crowded house with loads of current options. Not solely are there loads of LED indicators on Amazon, however there are additionally extra fancy indicators like Tidbyt or this one I found on Etsy.

I’ve determined that as a substitute of coming into right into a crowded house, I’ll proceed engaged on my ESP32 signal principally as a private studying undertaking.

For the neighborhood I’ll open source the underlying sign code for the NYCTrainSign in addition to the reconstructed API server that includes the exploit code.

I will even keep the brand new NYCTrainSign server as long as the internet hosting prices are pretty low. I don’t intend on including any new options to the present NYCTrainSign however I do have some concepts for an improved signal firmware.

So in case you have an indication from NYCTrainSign, check out the site that I created to manage the signs remotely.

If the directions don’t work, file an issue. However if you happen to don’t have an NYCTrainSign, positively don’t purchase one.

Due to Sharan for placing up with me after I was overly on this signal, and due to Linda, Kai, and Soly for reviewing and modifying this put up.