cbus2zigbee

Hi all,

I've just published to Github (with a lot of inspiration from the code written by ssaunders) a couple of scripts that connect cbus via a SHAC to zigbee2mqtt, unidirectionally, with cbus as the sole source of truth for lighting.

I was previously using about 8 lights via Steve's Hue integration, but I found the bridge response time was in the seconds, and I just generally wanted to migrate away from Hue.

With zigbee2mqtt, I'm getting almost instant response to buttons on the EDLT and wall switches, which has made it much nicer to use!

I haven't documented it yet, but let me know if anyone finds this interesting, and if there's any demand to make it bidirectional, or any other suggestions. I have included a database lookup in the event script, which seems to perform well (even though it includes a LIKE), but I realise this doesn't seem to be best practice for other people.

Absolute Lua neophyte, so please feel free to weigh in with suggestions or improvements.

https://github.com/geoffwatts/cbus2zigbee/

 

@geoffwatts, oooo...

This I gotta check out. Bi-directional challenge accepted, except I very rarely use the Hue app, so what-evs. I set colour temperature then let my Samsung Galaxy put it to sleep... But I do use Home Assistant heavily as authoritative, as you would have gathered.

 

I absolutely love the reference of 0xBEEF1 in the port address, @geoffwatts.

It brings to mind 0xDEADBEEF (or maybe 0xBEEFBABE?) from days of old. 0xDEADBEEF being frequently used to indicate a software crash. I think your code is simple enough, and elegant enough to avoid that mostly, though, so good choice in 0xBEEF1.

That said, I do suspect that when your Mosquitto broker goes away (like when it's restarted, or a switch is upgraded) that 0xDEADBEEF may happen, somewhat reducing partner approval factor, unless you're like me rattling around alone in an empty nest. So an ON_DISCONNECT callback might be wise to trigger a subsequent recovery and client reconnect.

Using loop_start() I've found dents that plan (I think that's why I ditched it), so it may be wise to also switch to synchronous client:loop(0) in the main infinte loop with a lesser socket timeout of, say 0.1 seconds, or even less in your case given the light code. That way Mosquitto gets serviced very quickly, and AC CPU doesn't suffer.

Either recovery/reconnect in script, or when a disconnect is seen exit the script completely with a 'do return end' in the main loop, and set the resident to a sleep interval of ten seconds or so for reset. I've had socket sketchiness doing it that way ... so force close the socket with a "pcall(function () server:close() end)" before trying the "server = require('socket').udp()" (pcall to thumb your nose at any errors). The script will remember the "server" value on sleep reentry it would seem, and without would fail to connect because socket in use, which does not happen with a disable/enable...

Setting an availability topic with a LWT is also a good thing to do. Tells other Mosquitto clients that entities are unavailable. "Something like client:will_set('zigbee2mqtt/status', 'offline', 2, true)" after mqtt.new(). The broker will act on the last will and testament when it can't reach your script, and other broker client subscriptions of the avilability topics can then obey. (edit) Oh, and set in ON_CONNECT callback:

Code:

client:publish('zigbee2mqtt/status', 'online', 2, true)

I'm a fan of Mosquitto QoS of 2, for a "send exactly once" as well, but taste varies. I'd rather a message get there, or not. Period. Speed be damned, because nanoseconds usually, so I'll wait for the PUBCOMP to be certain.

I'd also set some sort of a 'connected' variable in ON_CONNECT and ON_DISCONNECT, and wait/timeout while calling client:connect() before proceeding. Don't succeed? Log it, do return exit, and try it all again. Disconnect? Log it, do return exit, and try it all again.

Honestly, most of programming is expecting and catching nasties. That and platform quirks like firmware differences. My code would be halved in size without, so take this as snippets of some learnings that I have learned over time, and certainly not criticism.

I am a fan of LIKE clauses, BTW, so good on you. And regular expressions in SQL. And I do SQL, Cognos and Python for a day job. LUA is for fun. Better the server sorts it out than me later, because it's running super fast compiled code, not Cognos, or Python, or LUA bytecode (which is a contentious grey area)... If I was doing that query from compiled C++ I'd probably do it without, like I've done in the past for max speed, but not from LUA. A nice job on that query.

But go on. I dare you to change the LIKE to a REGEXP, which would give you tag validation for hex string format as well to prevent yet another latent possibility for error. Query speed be damned. It only happens when turning on a light, after all.

... AND ot.tag REGEXP '^z=[xX0-9a-fA-F]+$' Much more readable than LIKE. Not.

 

Steve this is awesome, thank you for thoughtful reply and the benefit of your experience. I'll take your advice (and read your code!) and add some polish, and amp up the partner acceptance factor with some resilience.

Thanks for doing my regex homework, too.

Bidirectional seems a good idea - I've started to convince myself that HASS is probably the "right" place to centralise everything. I'll do a few more commits and ping back here in a few days or so.

Geoff

 

Hey @geoffwatts.

Your comment in event script code "We will ignore this, because cbus repeatedly calls this during ramps - maybe we figure out transitions later"...

The AC will not call the event script during ramps when "Execute during ramping" is not ticked (the default), but only with firmware <= 1.6.0, and >= 1.15.0. There is a bug in firmware > 1.6.0, and < 1.15.0 that means that event scripts will always execute during ramping regardless of this setting.

With this option off the event script will fire only once at the end of the ramp. So you may need to upgrade, or if already upgraded to 1.15.0+ then turn off this option.



Ramp/transition tracking was (is) my one greatest headache with Philips Hue integration. The Hue event stream from the bridge is unreliable of timing at best.

But I have a Tube's ZB coordinator inbound from the US that will hopefully fix all my Hue woes. My hope is to put the Philips Hue Bridge in the bin, deploy Tube's bad boy and zigbee2mqtt, and collab. with you to expand your AC zigbee2mqtt code to bidirectional goodness. It looks like the configure_reporting topic minimum_report_interval:0 may be our friend, where a device should report its status immediately. Philips may have this set to non-zero for devices, which would explain a laggy event stream. Or their bridge is lazy.

Oh, and that REGEXP would probably be better as '^z=0[xX][0-9a-fA-F]+$'

 

I have been using Zigbee2mqtt for about 3 yr and i must say it is Delightful, Intergration with 5500NAC is approached differently to how you guys have achieved it, with bi-directional custom 2 way scripting based on topic. as i dont use HA, its a bit more work, but i think it gives more flexibility (obviously this way is not always the best fit).
also i cache tagged obj at the start of the script it just means you have to rember to restart script when obj tags are updated but i think its less on the cpu than searching for something each time a value or obj address is req.

function update_cache_obj()

cached_objects = {}
local MQ_tag = grp.tag('MQ')

for i = 1 , #MQ_tag do
cached_objects[MQ_tag.address] = {['address'] = MQ_tag.address, ['name'] = MQ_tag.name, ['datatype'] = MQ_tag.datatype, ['comment'] = MQ_tag.comment, ['tags'] = MQ_tag.tagcache, ['id'] = MQ_tag.id}
end
return cached_objects
end

but anyhow i have a total of 19 zigbee devices including 5 zigbee pir sensor/s (various brand/s) tunring on/off/dim a cbus chanell (dimmer/relay) faster than an actual cbus sensor... absolutly wild.
I find zero latency even with everythng travelling through Zigbee>MQTT>NAC>Cbus and back again.

 

Nice work, @Benla. Delightful is music to my ears. Can't wait to dive in and ditch the Bridge.

I like your cache setup. For my Home Assistant MQTT integration I utilise 'create/update/delete' functions in a resident that monitors for keyword changes every five seconds. In that set up, the event scripts fire on a single keyword (MQTT), and the resident uses other keywords to configure discovery options. This is used to publish discovery topic changes for Home Assistant. At an aggresive five second check I see little AC utilisation impact, but do see some message latency around 100-150ms at times, which is mostly unnoticible. Five seconds is way too often, but my setup is a bit of a crash-test-dummy for others using the same scripts. The grp.tag() function is a little slow, but using the SQLite approach that @geoffwatts is doing is very fast.

If it were me, I'd have his resident lookup the tags on init (and maybe check for keyword changes every so often), and just send the net/app/grp alias to the resident from event, which would then be able to look up the ZigBee address.

 

I'm going to refactor to what Steve is doing with a cache at init when I get to the zigbee->cbus stuff. Having said that, if Steve's courier is faster than my motivation, more likely I'll just end up running his code!

I pushed a few changes last night, too.

My setup is currently homebridge with a really old cbus/cgate plugin, and the shac only runs the zigbee integration - so my single source of truth for lighting is the shac, and the control of zigbee devices is via cbus groups from homebridge.

I've seen the light and moving to HASS to future-proof the home automation setup, and get it closer to a backup and restore to some off-the-shelf hardware.

@Benla I'm going to ingest some sensor data too - outside temp on my edlt screens seems fun, and I have a cute zigbee temperature/humidity sensor. I'll probably end up round-tripping things like motion sensors thru HASS once I get that connected to the SHAC, just so that things like configuration of on times is really easy to administer for someone who isn't me.

 

Nice one. Running HAOS on a NUC or VM will allow installing add-ons from the HA store like a Mosquitto broker, Zigbee2mqtt, ESPHome, RTSPtoWeb, NodeRed, etc. It's a great platform. It has a huge selection of integrations for seemingly everything but CBus. My acMqtt code should help you there.

I've not mucked around with Raspberry Pi given a bad crash experience some time back. I run Supermicro servers here in a hyperconverged Proxmox three-node cluster, but that's a little fancy for most HA use cases. These servers do a load more like web serving, email, reverse proxies, DNS, LDAP.

Proxmox does allow snapshot restore and backup to secondary NAS storage, which is a godsend when I occasionally mess something up. So it might be worth investigating as a platform for HAOS even for single node use on a NUC.

 

Hey @Benla and @geoffwatts. I'm thinking about how to implement sensor/status returns from zigbee2mqtt. My biggest roadblock at the moment is waiting on a coordinator from the US, so I'm relying on documentation alone.

It would be absolutely fantastic to see in a code block the zigbee2mqtt/bridge/devices topic from one of your setups, grabbed from MQTT Explorer. And also a couple of zigbee2mqtt/FRIENDLY_NAME topics, like one from a light, and another a sensor.

 

Steve here's a sensor:

Code:

{"battery":100,"humidity":39.39,"linkquality":83,"power_outage_count":58181,"pressure":1015.6,"temperature":25.14,"voltage":3065}

Here's a light:

Code:

{"brightness":1,"color":{"h":0,"hue":0},"color_mode":"color_temp","color_temp":500,"do_not_disturb":null,"linkquality":18,"state":"OFF"}

Here's the devices block:

Code:

[{"definition":null,"disabled":false,"endpoints":{"1":{"bindings":[],"clusters":{"input":[],"output":[]},"configured_reportings":[],"scenes":[]},"10":{"bindings":[],"clusters":{"input":[],"output":[]},"configured_reportings":[],"scenes":[]},"11":{"bindings":[],"clusters":{"input":["ssIasAce","genTime"],"output":["ssIasZone","ssIasWd"]},"configured_reportings":[],"scenes":[]},"110":{"bindings":[],"clusters":{"input":[],"output":[]},"configured_reportings":[],"scenes":[]},"12":{"bindings":[],"clusters":{"input":[],"output":[]},"configured_reportings":[],"scenes":[]},"13":{"bindings":[],"clusters":{"input":["genOta"],"output":[]},"configured_reportings":[],"scenes":[]},"2":{"bindings":[],"clusters":{"input":[],"output":[]},"configured_reportings":[],"scenes":[]},"242":{"bindings":[],"clusters":{"input":[],"output":[]},"configured_reportings":[],"scenes":[]},"3":{"bindings":[],"clusters":{"input":[],"output":[]},"configured_reportings":[],"scenes":[]},"4":{"bindings":[],"clusters":{"input":[],"output":[]},"configured_reportings":[],"scenes":[]},"47":{"bindings":[],"clusters":{"input":[],"output":[]},"configured_reportings":[],"scenes":[]},"5":{"bindings":[],"clusters":{"input":[],"output":[]},"configured_reportings":[],"scenes":[]},"6":{"bindings":[],"clusters":{"input":[],"output":[]},"configured_reportings":[],"scenes":[]},"8":{"bindings":[],"clusters":{"input":[],"output":[]},"configured_reportings":[],"scenes":[]}},"friendly_name":"Coordinator","ieee_address":"0x00124b0024c2ce0e","interview_completed":true,"interviewing":false,"network_address":0,"supported":true,"type":"Coordinator"},{"date_code":"","definition":{"description":"Zigbee light","exposes":[{"features":[{"access":7,"description":"On/off state of this light","label":"State","name":"state","property":"state","type":"binary","value_off":"OFF","value_on":"ON","value_toggle":"TOGGLE"},{"access":7,"description":"Brightness of this light","label":"Brightness","name":"brightness","property":"brightness","type":"numeric","value_max":254,"value_min":0}],"type":"light"},{"access":2,"description":"Triggers an effect on the light (e.g. make light blink for a few seconds)","label":"Effect","name":"effect","property":"effect","type":"enum","values":["blink","breathe","okay","channel_change","finish_effect","stop_effect"]},{"access":3,"description":"Do not disturb mode, when enabled this function will keep the light OFF after a power outage","label":"Do not disturb","name":"do_not_disturb","property":"do_not_disturb","type":"binary","value_off":false,"value_on":true},{"access":1,"category":"diagnostic","description":"Link quality (signal strength)","label":"Linkquality","name":"linkquality","property":"linkquality","type":"numeric","unit":"lqi","value_max":255,"value_min":0}],"model":"TS0501B","options":[{"access":2,"description":"Controls the transition time (in seconds) of on/off, brightness, color temperature (if applicable) and color (if applicable) changes. Defaults to `0` (no transition).","label":"Transition","name":"transition","property":"transition","type":"numeric","value_min":0},{"access":2,"description":"State actions will also be published as 'action' when true (default false).","label":"State action","name":"state_action","property":"state_action","type":"binary","value_off":false,"value_on":true}],"supports_ota":false,"vendor":"TuYa"},"disabled":false,"endpoints":{"1":{"bindings":[],"clusters":{"input":["genIdentify","genGroups","genScenes","genOnOff","touchlink","genLevelCtrl","lightingColorCtrl","manuSpecificTuya","genBasic"],"output":["genOta","genTime"]},"configured_reportings":[],"scenes":[]},"242":{"bindings":[],"clusters":{"input":[],"output":["greenPower"]},"configured_reportings":[],"scenes":[]}},"friendly_name":"office wall strip","ieee_address":"0xa4c1389bf2e3ae5f","interview_completed":true,"interviewing":false,"manufacturer":"_TZ3210_4zinq6io","model_id":"TS0501B","network_address":39480,"power_source":"Mains (single phase)","supported":true,"type":"Router"},{"date_code":"20220727","definition":{"description":"Zigbee LED Controller W (pro)","exposes":[{"features":[{"access":7,"description":"On/off state of this light","label":"State","name":"state","property":"state","type":"binary","value_off":"OFF","value_on":"ON","value_toggle":"TOGGLE"},{"access":7,"description":"Brightness of this light","label":"Brightness","name":"brightness","property":"brightness","type":"numeric","value_max":254,"value_min":0}],"type":"light"},{"access":2,"description":"Triggers an effect on the light (e.g. make light blink for a few seconds)","label":"Effect","name":"effect","property":"effect","type":"enum","values":["blink","breathe","okay","channel_change","finish_effect","stop_effect"]},{"access":7,"category":"config","description":"Controls the behavior when the device is powered on after power loss. If you get an `UNSUPPORTED_ATTRIBUTE` error, the device does not support it.","label":"Power-on behavior","name":"power_on_behavior","property":"power_on_behavior","type":"enum","values":["off","on","toggle","previous"]},{"access":1,"category":"diagnostic","description":"Link quality (signal strength)","label":"Linkquality","name":"linkquality","property":"linkquality","type":"numeric","unit":"lqi","value_max":255,"value_min":0}],"model":"GL-C-009P","options":[{"access":2,"description":"Controls the transition time (in seconds) of on/off, brightness, color temperature (if applicable) and color (if applicable) changes. Defaults to `0` (no transition).","label":"Transition","name":"transition","property":"transition","type":"numeric","value_min":0},{"access":2,"description":"State actions will also be published as 'action' when true (default false).","label":"State action","name":"state_action","property":"state_action","type":"binary","value_off":false,"value_on":true}],"supports_ota":true,"vendor":"Gledopto"},"disabled":false,"endpoints":{"11":{"bindings":[],"clusters":{"input":["genBasic","genIdentify","genGroups","genScenes","genOnOff","genLevelCtrl","lightingColorCtrl","touchlink"],"output":["genOta"]},"configured_reportings":[],"scenes":[]},"242":{"bindings":[],"clusters":{"input":[],"output":["greenPower"]},"configured_reportings":[],"scenes":[]}},"friendly_name":"office desk strip","ieee_address":"0xa4c1388a474eac0f","interview_completed":true,"interviewing":false,"manufacturer":"GLEDOPTO","model_id":"GL-C-009P","network_address":50900,"power_source":"Mains (single phase)","software_build_id":"10651203","supported":true,"type":"Router"}]

 

@ssaunders the other consideration is zigbee relays with >1 switch output (there are switch plates and 3-way relays on the market), and they expose different entities. I was writing about this in the comments on your PR. Here's one example:

Code:

{"indicator_mode":null,"linkquality":134,"power_on_behavior":null,"power_on_behavior_l1":"previous","state_l1":"OFF","state_l2":"OFF","state_l3":"OFF","switch_type":"toggle"}

I was considering either another tag (probably need one for switch vs dimmer), or perhaps a decorator for the zigbee address with the name of the state to toggle.

Bonus human presence detector:

Code:

{
    "fading_time": 10,
    "illuminance": 1534,
    "indicator": "OFF",
    "large_motion_detection_distance": 6.5,
    "large_motion_detection_sensitivity": 5,
    "linkquality": 142,
    "medium_motion_detection_distance": 4.5,
    "medium_motion_detection_sensitivity": 5,
    "motion_state": "large",
    "presence": true,
    "small_detection_distance": 4.5,
    "small_detection_sensitivity": 5
}

 

Thanks for that, @geoffwatts.

In the documentation, under MQTT topics and messages, zigbee2mqtt/FRIENDLY_NAME,
"The FRIENDLY_NAME is the IEEE-address or, if defined, the friendly_name of a device or group."

Could I please get a screen shot of an expanded zigbee2mqtt/ topic showing the sub-topic naming from a real environment (that uses friendly_name) using MQTT Explorer?

 

By default the FRIENDLY_NAME is the IEEE-address
But it’s changeable in the zigbee config(easiest way is at the zigbee/mqtt front end), ultimately you can make it anything you want.

There would be a topic for each device
Zigbee2mqtt/device_1
Zigbee2mqtt/device_2
And so on etc

- how to upload screenshot I don’t see an option for uploading image, maybe I’m just blind…

 

Just use copy/paste.

 

try this

 

Im going to read the zigbee2mqtt code and figure out if the IEEE addresses still work in MQTT even if the device has been given a friendly name, because that's currently working for me.

 

That's brilliant, @Benla. In-progress proposed code has been updated to handle friendly names, including those that are structured "Kitchen/Main lights".

This is so much fun doing this without an actual Coordinator. It's landed in Melbourne, but ... Easter. My aim is to be code evolved before it arrives, given the extra Easter coding free time. Much. Doco. Reading...

Thanks for your help.

 

resolveEntity in zigbee.ts suggests that the IEEE address will always work, probably for sets and gets. However, I doubt the IEEE address mqtt object will get updated if the light is updated from the zigbee side.

Probably safest to use friendly_name as the canonical name for the device, rather than the IEEE address.

 

Edit. Strike this: ... Even for status updates? Sounds unlikely given @Benla's screen shot, but probably works to set device status.
I read your message properly...

I've got the code subscribing to the friendly name/IEEE device status topics for two-way, and for lifting sensor values from the broker. (Edit: And for getting device availability if used...)