Reverse-engineering an encrypted IoT protocol

##
TL;DR

• I reverse-engineered the encrypted protocol GoodWe sensible meters and photo voltaic inverters use to ship metrics to the cloud.
• I used this analysis to construct a prometheus exporter.

##
The Solar: so scorching proper now

I bought a photo voltaic PV system put in in my home in mid 2023.
I did the naked minimal of analysis beforehand – simply talked to a few completely different installers about pricing, sizing and the economics of a battery.

One factor I actually did not do is any analysis into manufacturers and their relative hackability or safety deserves.
I simply specified that I needed to observe the gadgets and see some metrics.
The installer informed me that this required a wise meter and a cell app.
Truthfully I assumed that each one manufacturers could be equally horrific IoT junk, so I simply went with the advice of the installer.
At the very least that approach the electrical performance needed to be affordable, proper?

The results of my fortunate dip was a GoodWe DNS G3 Inverter and a GoodWe HomeKit 1000 Smart Meter.
These gadgets look fairly slick, and so does the web site.
They’re additionally standard right here in Australia, so my hopes had been excessive that it might be simple to arrange native monitoring, as a result of absolutely another person had found out the way to do it.

##
Publish-install setup

###
Metrics? You want to be on-line.

Proper after bodily set up the system is producing energy, however the metrics aren’t seen anyplace.
The documented technique to see metrics is to attach the machine to GoodWe’s cloud, after which use their net UI or cell app.

The gadgets act in simultaneous wi-fi AP and STA modes, and setup works like so:

1. Connect with the machine’s WLAN, which can be named Photo voltaic-WiFiXXXXXXXX, the place the Xs are the serial variety of the machine. The password is, naturally, admin.
2. Go to the machine’s net UI on 10.10.100.253.
3. Log in (utilizing credentials admin / admin, after all!).
4. Within the net UI, choose the WLAN that you really want the machine to make use of to hook up with the Web.

Now the gadgets are linked to GoodWe’s cloud.
However you continue to can’t see any metrics.

###
SEMS Portal account required

The following step is to go to GoodWe’s SEMS Portal and create an account.
Then let the installer know that the gadgets are linked, and the e-mail you used to create an account on the SEMS Portal.
Then the installer will electronic mail GoodWe (!?) to inform them to assocate your account with the serial variety of the gadgets, and in some unspecified time in the future GoodWe will motion that request (I used to be assured they checked their inbox commonly).

Lastly after a day or so the machine’s metrics are seen within the SEMS Portal.

Based on this flyer, it appears that evidently the installer would have a portfolio of “energy crops”, and so they can use the SEMS Portal to carry out “Fault self-analysis & troubleshooting”.

SEMS features a vary of capabilities and options to make sure dependable operation and to ship exact data to operators on the press of a button. It’s accessible by a number of accounts with completely different ranges of entry for homeowners, installers and EPC corporations

##
Publish-setup state of play

So now these two gadgets had been bodily put in, and linked to GoodWe’s cloud over the web through my remoted IoT VLAN.
However I had questions:

1. I needed to scrape metrics regionally, dammit! Why ought to I’ve to make use of the crappy cloud UI or equally dangerous cell app?
2. What else can GoodWe do with this connection? E.g. can they remotely administer the gadgets? If that’s the case, can I disable this “function”?

It seems that the inverter is powered by the photo voltaic panels, not by the grid.
So it loses energy and goes offline as quickly because the solar goes down.
And since I largely have time to hack on these items after darkish, I focused on the sensible meter.

There may be fairly a cottage business on-line documenting the way to extract information from GoodWe inverters.
They reply to Modbus queries, an Operational Technology commonplace.
There are numerous Github repositories with helpful details about the GoodWe Modbus protocol, resembling:

Sadly, my Homekit 1000 sensible meter will not be supported by any of those libraries.

##
Hacking the Homekit 1000

I’m presenting the method I adopted in chronological order.
So if you wish to discover out what truly labored, skip to the tip.

###
nmap

The very first thing I did was hearth up nmap, and level it on the HK1000.
It confirmed listening TCP port 23 – good outdated Telnet!
Connecting to this port and attempting Username: admin, Password: admin gave me a command immediate!

$ nc 192.168.18.17 23
Login as:admin
Password:admin
CMD>?
cfg         web         os          mft         
CMD>

Poking round this immediate quickly confirmed that it was fairly restricted^{, and it appeared to be a improvement interface that was left enabled.
I couldn’t get any metrics out of it.}

I additionally ran nmap in aggressive mode and was rewarded with a tough crash within the net server, and the machine resetting again to manufacturing facility settings.

###
Packet seize

Sniffing the site visitors from the machine confirmed that it was connecting out to tcp.goodwe-power.com:200001, and sending packets at common intervals.
Nevertheless a fast take a look at the site visitors revealed that whereas the serial variety of my inverter was seen, the primary physique of the payload was a high-entropy blob.
So the metrics information I used to be after appeared to be encrypted.

I additionally discovered a Github comment which got here to the identical conclusion.

###
Modbus

There’s a GoodWe Modbus protocol spec sheet and register map floating across the web which was invaluable in understanding how GoodWe encodes metrics from their inverters.
From this documentation I constructed a Modbus scanner that merely queried each register.
The deal with is barely 2 bytes large, so there are ~65k attainable addresses.

Sadly the HK1000 solely returns a worth for a single register deal with.
I overlook which register it was, however it was one thing ineffective like Firmware Model.

###
AA55 protocol

GoodWe gadgets additionally help one other (older?) protocol referred to as the AA55 protocol.
I couldn’t discover a lot data about it aside from another old spec sheet.

I constructed a scanner for this too, however the HK1000 didn’t reply to any queries.

###
ZZ/5A5A protocol (cell app)

The SEMS portal cell app has an fascinating perform the place you possibly can connect with the SOLAR-Wifixxxx community, and configure the machine utilizing the app however with out any authentication.

Sniffing this site visitors (because of airodump-ng and Wireshark’s WPA2 decrypt support) exhibits that the machine could be configured with out authentication by sending plaintext UDP packets to the correct port.
After all, this port is listening on all interfaces so it additionally most likely works through whichever native wifi community you join the machine to.
Gross.

Nevertheless, this protocol appeared to solely be used for community configuration.
I didn’t discover any approach of extracting information from the machine utilizing this protocol.

###
Firmware Reverse Engineering

After no success with the question protocols, I made a decision that perhaps the community was the incorrect strategy and I ought to attempt firmware as an alternative.
I managed to dump the firmware of the machine utilizing the command immediate and a command much like this^:

echo -e 'adminnadminnspi rd 0 2097152n' | nc 192.168.18.17 23 | tee ~/obtain/hk1000.spi2.img

This hexdump is interspersed with log strains, and the bytes are transposed.
So I dumped it twice, diffed the 2 dumps to remove the log strains, and stuck the transposition manually utilizing vim.

Then I unhexlified the binary with xxd:

xxd -r -p hk1000.spi.img > hk1000.spi.bin.img

And ran binwalk over it:

binwalk -eM hk1000.spi.bin.img

This revealed that the OS was eCos RTOS on a MIPS architecture.
I spent a while attempting to reverse this binary utilizing Ghidra, however actually I simply don’t know what I’m doing in the case of binary reverse engineering.

Lastly, whereas staring on the binwalk output, these strains caught my eye:

1976456       0x1E2888        AES Inverse S-Field
1977752       0x1E2D98        AES S-Field

###
Packet Seize redux

Going again to the packet seize I lastly seen that the size of the encrypted blob part was at all times a a number of of 16, plus 2.

Wait a second… AES block measurement is 16 bytes!

##
Evaluation of the GoodWe metrics protocol

Since that is was a black-box evaluation, I needed to depend on probing through the I/O I managed: community and energy.

###
Community “glitching”

It was at this level that I discovered what could be the important thing to cracking the encryption scheme.

Again in October 2021, another person did principally all the identical work I did, and introduced it on the Melbourne Linux User’s Group.
Not solely that, however they put their presentation online!
Thanks Danny!

Anyway, Danny made a really fascinating statement: if the web connection went down, the machine would buffer messages, and ship them unexpectedly when the connection got here again up.
Crucially, for buffered frames despatched in the identical second, the primary few 16-byte blocks of ciphertext had been similar!

I used to be in a position to replicate this regionally!

###
Empathy: a strong reverse-engineering software

After I’m an issue like this, I prefer to put myself within the sneakers of the developer.
What sort of particular person are they?
What are their motivations?

On this case, we will observe:

• Telnet left on in a manufacturing firmware picture, with credentials admin:admin.
• nmap can crash the machine exhausting sufficient to manufacturing facility reset.
• Packets despatched over TCP with figuring out information (serial quantity) within the clear.
• The metrics appear to be poorly encrypted (similar part of ciphertext in consecutive frames).
• Unauthenticated configuration protocol.
• An online UI that appears prefer it was hacked collectively in a day. Inspecting the supply exhibits a number of commented out HTML blocks.

In Danny’s presentation, he used this slide after discovering the Telnet port password:

Nevertheless I feel that is extra acceptable:

What these observations inform me is that GoodWe doesn’t put quite a lot of effort into securing their gadgets, and due to this fact the builders engaged on this machine didn’t have a lot incentive to create a safe protocol.
So there’s an opportunity I can hack round their encryption.

Placing myself into the sneakers of those builders, what would I must implement a metric protocol?

• Framing: that is TCP; it’s a byte stream. So we want a header of some form to know the place frames begin.
• Size: what number of bytes after the header do we have to learn to get the total body?
• Detecting information corruption: not something malicious, simply bitflips.

Trying on the packet captures, it’s simple to see POSTGW is the body header, and the very subsequent discipline seems to be like a big-endian encoded int32 with a worth constantly three bytes shy of the size of the information earlier than the following POSTGW.
That have to be the size!

And eventually: detecting information corruption.
Within the GoodWe Modbus doc linked above, there’s a description of the CRC used to detect information corruption.
It’s a commonplace Modbus CRC-16 (two bytes), designed to successfully detect bitflips.
Once more, assuming I’m a software program developer who’s aware of Modbus however who has been tasked with sending information over the web (and didn’t actually care a lot for safety), why wouldn’t I take advantage of an algorithm or library I’m already aware of?

A fast verify proves that operating the information between the size discipline and the final two bytes by the Modbus CRC algorithm returns a worth matching the final two bytes of the body.

My finest guess for the size discipline being three bytes shy of the size of knowledge moderately than two is that it’s only a sloppy implementation with an off-by-one error, which matches my profile of the builders.

One other information level to color an image of the engineering high quality: the CRC of frames from the consumer are encoded in big-endian byte order (similar as all the opposite integers encoded within the protocol).
Nevertheless the server sends the CRC in little-endian byte order.
Why?
Possibly the server is x86 and the developer forgot to name htons()?

Now I simply had the encrypted blob to decipher.

###
Encryption scheme

I guessed that they have to be utilizing AES in CBC mode as a result of:

• The similar part of ciphertext in consecutive frames is a classic CBC failure mode when reusing IVs.
• That is an outdated mode and extensively supported in libraries, making it simple to make use of.
• Since they don’t care about safety they’re hardly more likely to be utilizing AEAD modes.

When implementing a scheme utilizing CBC, it’s critically necessary that initialization vectors are usually not reused.
In any other case similar plaintext provides you with similar ciphertext.
Metrics from a wise meter are extremely more likely to be the identical minute-to-minute, which might be why we see similar sections of ciphertext in successive frames with the identical IV!

A standard observe is to prefix the IV to the ciphertext.
This is named an specific initialization vector, and it doesn’t have to be secret – simply randomly generated in a cryptographically safe method.
Nevertheless what if you’re operating on a microcontroller with no NRBG? Or perhaps you simply don’t know or care about CBC footguns?
Then it’s a must to use another “unique-ish” worth!

The machine is designed to solely ship metrics each minute.
Due to this fact the builders could have assumed that point based mostly IVs can be distinctive sufficient, with out bearing in mind buffering on community outage.

###
Energy “glitching”

The ultimate and most tough query: what’s the encryption key?

The very first thing I checked was what occurred when the machine rebooted: was there any key alternate or handshake?
Fortuitously the online UI has a reboot button, so it was simple to substantiate that no, there is no such thing as a key alternate on startup.

So as a result of we’re assuming AES (symmetric encryption), that most likely means… mounted keys!

Because the keys are mounted, they’re seemingly hard-coded.
AES can use 16, 24, or 32 byte keys, so I began by assuming a 16-byte key.
I suspected they’d use some string like GoodWeSolarPower, and retailer it as a static string or byte array.
I poked round within the firmware a bit with Ghidra, however didn’t discover any promising strings.

However in any case, there was one other drawback.
One of many properties of AES–CBC is that you could plug any IV and secret key into it and it’ll “decrypt”.
However until the IV and key are appropriate, the output can be rubbish.
So the way to know if I handle to appropriately guess the IV and key?

At this level I made one other educated guess.
The body header and size discipline use ASCII characters and main null bytes respectively.
Assuming the plaintext metric information is equally structured, it is going to have comparatively low Shannon entropy.
One other property of AES is that it’s a secure block cipher.
That’s, the ciphertext ought to be indistinguishable from random bytes.
Due to this fact, utilizing the inaccurate key or IV ought to lead to excessive entropy rubbish.

Assuming the timestamp within the body (which is null-padded to 16 bytes) is the IV, I wrote a extremely dumb software to:

1. step by the firmware one byte at a time, taking the following 16 bytes as a key.
2. “decrypt” the encrypted blob utilizing that key, and the timestamp prefix because the IV.
3. calculate the entropy of the decrypted blob. Whether it is beneath a given threshold, print the plaintext and key.

Fortuitously though this was a really naïve brute power algorithm, one beauty of 2024 is that computer systems are quick.

Working this software over the firmware dump from my machine solely took a couple of seconds and yielded… nothing.
Huh.

Fortuitously my earlier googling efforts had found a public Google drive with comparatively current updates (early 2023) containing firmware for (all?) GoodWe inverters^{.
Working the software over a firmware picture for an additional machine yielded… nothing once more!}

Lastly on the third try, I bought a single hit:

After all!
The important thing was simply all bits set.
Why not!?

I doubt this was truly hard-coded as a key anyplace within the binary blob.
I feel I simply bought fortunate that this firmware had a run of 0xff bytes.

Lastly I had a plaintext with apparent construction, however nothing mapping fields to metrics values.
Nevertheless I did have an oracle: the SEMS Portal API!
I used to be in a position to dump metric values for my sensible meter utilizing curl on the SEMS Portal API, and observe the metrics altering each time a packet was despatched from the sensible meter.

Then by eyeballing the packets and the values (assuming commonplace two’s complement signed integer encoding) it was comparatively easy, although a bit of time consuming, to map offsets to metrics values.

This wiring diagram was useful to grasp that there have been actually solely two CT sensors and each different metric was calculated from these two numbers:

##
Prometheus & Grafana

I like Prometheus for gathering metrics.
So I constructed an exporter based mostly on the analysis described above.
It really works by conducting a man-in-the-middle attack on the protocol.
Pointing the HK1000 on the IP deal with of the exporter when it requests tcp.goodwe-power.com will trigger the HK1000 to hook up with the exporter as an alternative of the GoodWe cloud.
Then the exporter will sniff the metrics out of the frames and ahead them to the actual tcp.goodwe-power.com.

The good factor about this design is that you just nonetheless get metrics in SEMS Portal.
These metrics are seen to your installer, so when you’ve got issues it’s simple for them to troubleshoot.
I additionally added help for my inverter, which makes use of roughly the identical protocol.

As well as, the Prometheus exporter will reject any packets from the server that it doesn’t perceive.
So hopefully unsolicited firmware updates can be blocked.

Lastly, I created a dashboard in Grafana:

##
Conclusions

• This train has bolstered my prejudice that IoT gadgets are horribly insecure. Within the case of GoodWe, the place they even have authentication, they use mounted default passwords resembling admin, and depart Telnet debug interfaces listening on their manufacturing gadgets.
• Though the metrics protocol and encryption scheme are insecure, I didn’t discover something that would actually be described as a safety vulnerability versus a design determination.
• Solely the metrics had been encrypted within the information despatched to SEMS Portal over the web.
  Not the mannequin or serial quantity.
  So even with (dangerous) encryption, they’ve left probably the most delicate information unprotected.
  I assume they’re simply obfuscating the metrics?
  Or perhaps the boss requested for encryption?
  “He said encryption! Give him encryption!”.
• Conversely the {hardware} appears fairly good, capabilities nicely, and appears nice!
• I spent months tinkering on this on-and-off.
  I used to be motivated by equal components indignant anger at not having the ability to scrape metrics regionally from a tool so intimately built-in into my home and operating on my community, and morbid curiosity about what safety flaw I used to be going to uncover subsequent.
  Now I perceive what jwz means when he talks about writing software in self-defence.

##
Easy methods to safe GoodWe gadgets

Lastly, right here’s my recommendation when you’ve got a GoodWe machine:

• No matter else you do, maintain these items off the general public web!
  Ideally in your personal, firewalled IoT VLAN.
• There doesn’t appear to be a easy technique to disable the Photo voltaic-Wifixxxx WLAN after the gadgets are arrange.
  So set a robust password, as a result of the default is admin.
  You are able to do this through the online UI.
• The online server is listening on all interfaces, so it’s accessible out of your VLAN.
  Change the password for the online UI from admin to one thing a bit safer.
  Notice: not all gadgets have this feature simply accessible.
  For instance the HK1000 solely permits altering this password through the Telnet interface.

For the paranoid:

• My prometheus exporter drops incoming packets it doesn’t acknowledge. Solely metrics will stream, not e.g. firmware updates (I hope – I haven’t seen any come by but). So in idea it is going to block distant administration of the gadgets.

##
Miscellaneous notes

This part comprises a couple of notes I made that didn’t match into the narrative of the weblog submit, however are fascinating nonetheless.

###
GoodWe’s Cyber Safety claims

GoodWe has a web page on Cyber Safety on their web site with a pleasant infographic, principally confirming every little thing I’ve simply found:

So as to forestall cyber-attacks on photovoltaic techniques to the best extent, inverter producers normally deploy varied safety insurance policies on the gear facet and server facet. Taking GoodWe for instance, to make sure the safety of knowledge transmission between the inverter and the server, we use the transmission protocols of CRC+AES and TLS respectively for communication with servers with completely different capabilities.

It is a nice demonstration of how you should use safe cryptographic primitives resembling AES–CBC, and nonetheless give you an insecure encryption scheme.

###
Hello-Flying and Xinwu

The GoodWe gadgets appear to make use of an IoT platform frequent to a number of Chinese language producers, for instance Solarman.
It has a novel discovery protocol the place you broadcast a particular packet to a given port, and the machine replies with its IP, MAC, and SSID (which incorporates the machine serial).

For instance (in separate terminals):

nc -u -l -p 50123
192.168.18.17,907856FECDAB,Photo voltaic-WiFi12345678

echo -n WIFIKIT-214028-READ | nc -u -b -p 50123 192.168.18.255 48899

Based on the config dumped from the Telnet command immediate, the chip within the HK1000 is the HF-A21, from an organization referred to as Hello-Flying, based in Shanghai.
You possibly can construct your IoT machine on prime of this platform by loading your personal software onto it, whereas the included OS takes care of the {hardware}, community and so forth.

An fascinating a part of the invention protocol is the string 214028.
The place does this come from?
Nicely roughly 150km from the Hello-Flying workplace is Xinwu district, Wuxi.
Based on Wikipedia:

In 2013, the output worth of Web of Issues (IoT) core business in Wuxi New District exceeded 70 billion yuan, accounting for 38.4 % of the output worth of the entire high-tech business within the district. Wuxi New District has fashioned a cloud computing industrial distribution, that includes {hardware}, platform and software.

Xinwu’s postcode is 214028.

###
Distant administration

Based on market researchers, GoodWe was the fifth largest supplier of solar inverters worldwide in 2022.
GoodWe have full distant administration functionality on the gadgets, together with the aptitude to push firmware updates.
This looks like loads of energy for any firm, not to mention an organization headquartered in a totalitarian dictatorship, to have over nationwide energy grids.

###
Batman mode

To validate the MITM performance For enjoyable, I applied Batman mode within the prometheus exporter.
On this mode, moderately than forwarding metrics to the SEMS Portal, the exporter replaces them with the batman equation.

###
DNS updates

The GoodWe gadgets ship their metrics to tcp.goodwe-power.com:200001.
After I first began investigating the protocol in mid 2023, this resolved to an IP deal with in Alibaba Cloud.
Nevertheless late final yr this was up to date to now resolve to a pair of ELBs in AWS.

In each Alibaba Cloud and in AWS they appear to be doing DNS load balancing, as a result of whereas the SecurityTrails screenshots above present US IPs, from right here in Australia each these domains resolved to IPs in Alibaba Cloud China (beforehand), and now to AWS Sydney.