blog/markdown/Transform-a-RaspberryPi-into-a-universal-Zigbee-and-Z-Wave-bridge.md

[//]: # (title: Transform a RaspberryPi into a universal Zigbee and Z-Wave bridge)
[//]: # (description: Use the Platypush Zigbee and Z-Wave integration to create a smart home bridge to rule them all.)
[//]: # (image: /img/zigbee-zwave-1.jpg)
[//]: # (author: Fabio Manganiello <fabio@platypush.tech>)
[//]: # (published: 2020-02-25)

Home automation comes with plenty of potential to make our lives easier. But in order to succeed in its task, it often
requires you to fill your house with bridges that can connect your smart devices to your Wi-Fi network. Unless you buy a
smart device that communicates directly over Wi-Fi (like
a [TP-Link](https://www.tp-link.com/nl/home-networking/smart-plug/hs100/)
or [Belkin](https://www.belkin.com/us/p/P-F7C063/) smart plug), odds are that many of your favourite smart devices use
either Bluetooth, Zigbee or Z-Wave to communicate. These protocols solve some of the issues of Wi-Fi when it comes to
smart devices — like latency, centralised topology and relatively high power requirements — but they do require some
physical hardware in between to do the smart protocol <-> Wi-Fi translation and make the devices actually controllable
from a Wi-Fi-connected client. The bad thing is that you’ll probably need a different bridge or hub for each class of
devices you want to use. Philips Hue lights come with their own bridge, same for Lutron, same for HomeKit, same for
Belkin, same for Switchbot, and the list goes on. What’s ironic is that most of these devices actually speak the same
protocol (either Zigbee or Z-Wave) but, in most of the cases, they can only control their own devices. Try to imagine an
alternative reality where all the ethernet cards can send and receive TCP/IP packets, but you’ll need an adapter just
for HTTP traffic, one for FTP, one for SMTP, and so on: that’s more or less the reality today when it comes to smart
bridges and hubs. In order to solve the problem, many companies are throwing on the market even more hubs and bridges (
from Samsung SmartThings, to the plethora of Amazon Echo, Google and Apple devices etc.), often compatible only with a
subset of the devices out there and incompatible with one another, often incompatible by design with devices produced by
competitors: that only brings even more fragmentation to the current smart home hell.

![How standards proliferate](../img/standards.png)

Moreover, many hubs can usually talk only to their smartphone app, adding an app hell to the bridge hell.

One of the most sought goals nowadays when it comes to home automation is (or it should be) to find consistent ways to
communicate with as many devices as possible using the same tool, the same box and the same interface.

I’ve recently worked on achieving this goal in platypush. Other platforms have tackled the issue too (from HASS.io to
Openhab) but, in my opinion, they all still require a certain degree of user configuration (like specifying what each
device is and what it’s supposed to do) that in my opinion shouldn’t be required. In this article, we’ll go through what
Zigbee and Z-Wave actually are, what you need to set up a universal bridge for them using just a Raspberry Pi and a USB
dongle, and how to permanently move your existing smart bridges to the storage room.

## Zigbee vs. Z-Wave

Zigbee and Z-Wave have arguably risen to become the dominant communication protocols in the smart home and IoT
industries. They share some common features, such as:

- They are better suited for low-powered devices. Wi-Fi is a relatively power-hungry protocol, needs to deal with packet
  losses, relies on a centralised topology, and isn’t ideal for communication with devices that can be asleep for a long
  time because of its high overhead on reconnection. Zigbee and Z-Wave have instead been designed to work well with
  low-powered devices, be low-latency and optimized for sending less bytes on the network. Some Z-Wave or Zigbee devices
  can theoretically run on the same battery for a couple of years; a similar Wi-Fi device won’t usually last more than a
  couple of days.

- While Wi-Fi networks are usually configured using a star-based topology (one access point/router in the middle and all
  the devices connect to it), Zigbee and Z-Wave have been designed to support more flexible topologies. Each device can
  be both a client and a repeater on the network, each can directly connect to other devices and expand the network with
  no need to introduce further routers or access points. This allows the creation of scalable *mesh* networks that can
  include devices that would otherwise be out of range.

When it comes to the differences:

- **Number of devices**: Z-Wave networks are limited to 232 devices, while Zigbee supports up to 65k devices.

- **Maximum hops**: Z-Wave supports up to four hops between a device and the network controller, while Zigbee has
  theoretically no such limitations.

- **Operating range**: Most Zigwave devices operate around the 2.4 GHz spectrum — although some specific devices also
  use 784 MHz in China, 868 MHz in Europe and 915 MHz in the US and Australia. Z-Wave devices instead operate around the
  850–900 MHz range (at 868.42 MHz in Europe, at 908.42 MHz in North America and other frequencies in other countries
  depending on their regulations).

- **Signal range**: A thumb rule when it comes to electromagnetic waves is that, at fixed transmission power, the higher
  the frequency the lower the range. It means that Z-Wave devices, which operate at lower frequencies, have usually a
  longer range compared to Zigbee. An unobstructed Z-Wave signal can travel up to 100 meters outdoor, although a common
  guideline is 30 meters indoor for unobstructed signals and 15 meters if there are walls in between. The indoor range
  for Zigbee is instead usually limited around 12 meters. However, Zigbee networks theoretically support an unlimited
  number of hops between the nodes and the controller, so the range can be easily extended by adding more nodes to the
  network.

- **Signal reliability**: Z-Wave has a longer range than Zigbee. It means that, at a fixed distance, its signal is
  usually more reliable. Moreover, Zigbee operates in the relatively “quieter” 800–900 MHz spectrum, therefore it
  doesn’t have to share the crowded 2.4 GHz spectrum with Wi-Fi and Bluetooth devices. That overall leads to more
  reliable communication.

- **Ownership and protocol**: Z-Wave is a proprietary technology developed and maintained by Sigma Designs. Sigma (
  acquired by Silicon Labs in 2018) owns the protocol, licenses the compatible devices and runs the Z-Wave Alliance, and
  grants certifications to the devices that comply with the standard. Its selling point is, in my opinion, the strong
  enforcement of a shared protocol both on hardware and software side. Each sensor, configuration value or switch is
  defined by a variable type, a range, a read-only vs. read-write policy and a structured representation that applies to
  all the compliant devices. That makes it very easy to develop consistent interfaces that can comprehensively represent
  and control any device as long as it speaks Z-Wave, even if the implementation doesn’t know exactly which device it
  is. Zigbee, on the other hand, is an open standard maintained by the Zigbee Alliance. It also has a certification
  process in place, but that comes in two parts — one part certifies the hardware, the other certifies the software. It
  is possible to produce Zigbee certified hardware even if the software isn’t certified nor compatible with other
  clients. While this makes the protocol much more open than Z-Wave, it also makes the task of writing an all-purpose
  interfacing software much harder, as different devices may name their properties following different conventions.

That should cover most of the knowledge you need when it comes to the theory of Zigbee and Z-Wave. You’ll find many Zigbee and Z-Wave compatible smart devices around. Some examples of Zigbee devices are the Philips Hue and Ikea smart bulbs, Honeywell thermostats, Belkin smart plugs and bulbs, Bosch sensors and Osram products. Z-Wave includes around 65,000 compatible devices out there, including many garage doors, presence and temperature sensors, thermostats, dimmers, remote controls, smoke detectors and so on.

That should cover most of the knowledge you need when it comes to the theory of Zigbee and Z-Wave. You’ll find many
Zigbee and Z-Wave compatible smart devices around. Some examples of Zigbee devices are the Philips Hue and Ikea smart
bulbs, Honeywell thermostats, Belkin smart plugs and bulbs, Bosch sensors and Osram products. Z-Wave includes around
65,000 compatible devices out there, including many garage doors, presence and temperature sensors, thermostats,
dimmers, remote controls, smoke detectors and so on.

## Hardware and software

We’ll use a RaspberryPi in the following examples as a DIY bridge (any model and any distribution should work fine), and
Platypush as a home automation platform that also runs the Zigbee and Z-Wave integrations.

## Making your own Zigbee bridge

On the hardware side you’ll need:

- One or more Zigbee compatible devices, like Philips Hue, Ikea or Osram lights or Belkin switches.

- A Zigbee-to-USB adapter/sniffer.
  The [CC2531](https://www.amazon.com/Gowoops-Protocol-Analyzer-Sniffer-802-15-4/dp/B07P5LY7Z6/ref=sr_1_6?keywords=zigbee+usb+adapter&link_code=qs&qid=1582640215&sr=8-6)
  is one of the most popular options out there.

- A [Zigbee debugger+adapter cable](https://www.amazon.com/Debugger-Downloader-Programmed-Simulation-Programmer/dp/B07T13JX32/ref=pd_sbs_147_9?_encoding=UTF8&pd_rd_i=B07T13JX32&pd_rd_r=d44d5902-3147-4af9-ba8a-de7d31d4d1e0&pd_rd_w=j45M7&pd_rd_wg=TNdkl&pf_rd_p=7cd8f929-4345-4bf2-a554-7d7588b3dd5f&pf_rd_r=KRHVT3D830174F7FCB05&psc=1&refRID=KRHVT3D830174F7FCB05),
  that you’ll need in order to flash the firmware on the dongle.

- A RaspberryPi or similar device (any model and distribution should work fine), or any computer that you want to use as
  a bridge.

On the software side:

- Platypush uses [`zigbee2mqtt`](https://www.zigbee2mqtt.io/) as a backend to interact with the Zigbee dongle. Install
  the zigbee2mqtt firmware on the dongle by following the instructions (for Windows, macOS and Linux)
  on [their website](https://www.zigbee2mqtt.io/getting_started/flashing_the_cc2531.html).
  You can check a list of the compatible devices [here](https://www.zigbee2mqtt.io/information/supported_devices.html).

- Install, start and enable an MQTT instance on the local machine, if you don’t have a server already running in your
  network. If you’re running Debian/Ubuntu/Raspbian and want to use Mosquitto, for example:

```shell
[sudo] apt-get install mosquitto
[sudo] systemctl start mosquitto.service
[sudo] systemctl enable mosquitto.service
```

- Install zigbee2mqtt:

```shell
[sudo] apt-get install nodejs git make g++ gcc
git clone https://github.com/Koenkk/zigbee2mqtt
cd zigbee2mqtt
npm install

# Change mqtt.server and serial.port to respectively
# match your MQTT server and USB dongle device.
vi data/configuration

npm start
```

You can also make a systemd service out of it:

```yaml
[Unit]
Description=zigbee2mqtt
After=network.target
        
[Service]
ExecStart=/usr/bin/npm start
WorkingDirectory=/path/to/zigbee2mqtt
StandardOutput=inherit
StandardError=inherit
Restart=always
        
[Install]
WantedBy=multi-user.target
```

- Note that the zigbee2mqtt configuration file also includes a `permit_join` option. Set this to true while you’re
  pairing your Zigbee devices for the first time, and set it to false afterwards to prevent other devices from
  accidentally or malignantly join the network — you can always temporarily allow joins later.

- Once started and in `permit_join` mode, you can start pairing Zigbee devices to your new network. That is usually done
  by doing a factory reset of the device. The procedure varies with the device: Philips Hue bulbs, for example, can be
  reset either from the app (if they are paired to a bridge) or by pressing the ON/OFF buttons of a Hue dimmer
  simultaneously for 10 seconds while keeping the dimmer within 10 cm from the lightbulb. Other Zigbee devices may
  include a reset button instead.

- Once a Zigbee device joins the network, the zigbee2mqtt logs should show a trace like the following:

```
Successfully interviewed '0x00158d0001dc126a', device has successfully been paired
```

- Install Redis and Platypush with the Zigbee, HTTP and MQTT extensions:

```shell
[sudo] apt-get install redis-server
[sudo] systemctl start redis.service
[sudo] systemctl enable redis.service
pip install 'platypush[zigbee,http,mqtt]'
```

- Edit your `~/.config/platypush/config.yaml` file to enable the Zigbee and HTTP services:

```yaml
backend.http:
    port: 8008
    
zigbee.mqtt:
    host: localhost
    
backend.zigbee.mqtt:
    enabled: true
```

- Start Platypush (either by running `platypush` or starting the `platypush.service` systemd service).

- Open `http://your-raspberry:8008/` in a browser. The web panel should include the Zigbee icon in the navigation bar,
  you’ll be able to control your network from there.

![Zigbee plugin screenshot 1](../img/zigbee-zwave-2.png)

![Zigbee plugin screenshot 2](../img/zigbee-zwave-3.png)

You can send requests through
the [supported API](https://docs.platypush.tech/en/latest/platypush/plugins/zigbee.mqtt.html) over HTTP, Python
code or through whichever platypush backend you have configured:

```shell
# HTTP request
curl -XPOST \
  -H "Authorization: Bearer $PP_TOKEN" \
  -H 'Content-Type: application/json' -d '
{
  "type":"request",
  "action":"zigbee.mqtt.device_set",
  "args": {
    "device":"White Bulb",
    "property":"state",
    "value":"ON"}
}' http://localhost:8008/execute
```

```python
# Python API
from platypush.context import get_plugin

get_plugin('zigbee.mqtt').device_set(device='White Bulb', property='state', value='ON')
```

Or hook any custom logic to the [supported events](https://docs.platypush.tech/en/latest/platypush/events/zigbee.mqtt.html):

```python
from platypush.event.hook import hook
from platypush.utils import run

from platypush.message.event.zigbee.mqtt import ZigbeeMqttDevicePropertySetEvent

@hook(ZigbeeMqttDevicePropertySetEvent, device='White Bulb')
def on_white_bulb_on(event, **context):
    if event.properties.get('state') == 'ON':
        run('tts.say', text='The light went on')
```

Congratulations, you’re now ready to use your Zigbee devices and build automation without bridges!

## Making your own Z-Wave bridge

Making a DIY Z-Wave bridge is even simpler than making a Zigbee bridge, as you won’t need a debugger to flash a custom
firmware, nor an MQTT service in between.

- You’ll need a Z-Wave USB adapter dongle. I use
  [this one](https://www.amazon.com/Z-Wave-Me-Smart-Stick-Cloud2-Z-Wave/dp/B00VKEH1BQ/ref=sr_1_9?keywords=z-wave+usb&qid=1582658700&sr=8-9),
  but any compatible dongle should work. Take note of where the adapter is mapped on your system — e.g. `/dev/ttyUSB0`.

- Install Redis and Platypush with the Z-Wave and HTTP extensions:

```shell
[sudo] apt-get install redis-server
[sudo] systemctl start redis.service
[sudo] systemctl enable redis.service
[sudo] pip install 'platypush[zwave,http]'
```

- Configure your integrations in `~/.config/platypush/config.yaml`:

```yaml
backend.http:
    port: 8008

zwave:
    enabled: true

backend.zwave:
    device: /dev/ttyUSB0
```

- Start Platypush (by running `platypush` or through a systemd service) and point your browser
  to `http://your-raspberry:8008/`. You’ll see a new tab for the Z-Wave integration.

![Z-Wave plugin screenshot 1](../img/zigbee-zwave-4.png)
![Z-Wave plugin screenshot 2](../img/zigbee-zwave-5.png)
![Z-Wave plugin screenshot 3](../img/zigbee-zwave-6.png)

Each Z-Wave compatible device has its own way of pairing to a network. All you need to do is to press on the `+` button
to put the network in pairing mode and then pair your devices within one minute through the procedure referred in the
user manual. Since Z-Wave has a stricter protocol and all the compliant devices publish their values using the same
format, the Z-Wave interface is much more granular and detailed compared to Zigbee.

You can, of course, send commands to the new network through
the [available API](https://docs.platypush.tech/en/latest/platypush/plugins/zwave.html) and subscribe custom hooks
on [Z-Wave events](https://docs.platypush.tech/en/latest/platypush/events/zwave.html):

```shell
# HTTP request
curl -XPOST \
  -H "Authorization: Bearer $PP_TOKEN" \
  -H 'Content-Type: application/json' -d '
{
  "type":"request",
  "action":"zwave.get_value",
  "args": {
    "value_label":"Temperature",
    "node_name":"Kitchen Sensor"
  }
}' http://localhost:8008/execute
```

```python
# Python usage
from platypush.context import get_plugin

get_plugin('zwave').get_value(value_label='Temperature', node_name='Kitchen Sensor')
```

```python
# Example output
{
    "type": "response",
    "target": "http",
    "response": {
        "errors": [],
        "output": {
            "command_class": 49,
            "data": 26.799999237060547,
            "data_items": "Read only",
            "genre": "User",
            "index": 1,
            "is_read_only": True,
            "is_write_only": False,
            "label": "Temperature",
            "node_id": 3,
            "type": "Decimal",
            "units": "C",
            "value_id": 72057594093256722
        }
    }
}
```


```python
# Event hook example
from platypush.event.hook import hook
from platypush.utils import run

from platypush.message.event.zwave import ZwaveValueChangedEvent

@hook(ZwaveValueChangedEvent)
def on_white_bulb_on(event, **context):
  if event.node['name'] == 'Motion Sensor' and \
        event.value['label'] == 'Sensor' and \
        event.value['data'] is True:
    run('tts.say', text='Motion has been detected')
```

You should now have all the ingredients to build your custom IoT networks and ditch those bridges for good!

### April 2021 note

The native Z-Wave integration provided by Platypush will slowly get phased out in favour of the new `zwave.mqtt`
integration (see [issue #186](https://git.platypush.tech/platypush/platypush/-/issues/186)).

The reason is that the library used by the native Z-Wave integration (`python-openzwave`) is currently
[unmaintained](https://github.com/OpenZWave/python-openzwave/issues/198) and its last commit was around the
beginning of 2019. Many bugs keep stashing up on the library because of progress made on the OpenZWave framework
while the library doesn't get updated, and some Platypush methods may be broken as of now.

Instead, it is now advised to set up a [`zwavejs2mqtt`](https://github.com/zwave-js/zwavejs2mqtt) gateway to
that uses an MQTT broker to dispatch events and messages (follow the installation instructions
[here](https://zwave-js.github.io/zwavejs2mqtt/#/getting-started/quick-start)).

After installing the gateway, you can configure the `zwave.mqtt`
[plugin](https://docs.platypush.tech/platypush/plugins/zwave.mqtt.html) and
[backend](https://docs.platypush.tech/platypush/backend/zwave.mqtt.html) on Platypush side to enable the integration.
The format of the requests and responses is the same and the UI is also cross-compatible, although zwavejs2mqtt 
doesn't implement some advanced features provided by OpenZWave (such as low-level controller commands or e.g. rename
values and scenes).