Run like Hell: ZigBee

Showing posts with label ZigBee. Show all posts

Nov 6, 2021

Fritz!Box monitoring with grafana, influx, collectd and fritzcollectd

A nice way to monitor your Fritz!Box is this here:

How can you achieve this:

https://fetzerch.github.io/2014/08/23/fritzcollectd/

and

https://github.com/fetzerch/fritzcollectd

Here a list of the software packages you have to install

apt install -y collectd python3-pip libxml2 libxml2-dev libxslt1-dev influxdb nodejs git make g++ gcc npm net-tools certbot mosquitto mosquitto-clients grafana-server

for grafana-server and influxdb you have to add new repositories, because they are still not included in ubuntu.

To tell collectd, that it shoud write to influxdb, you have to uncomment the following in collectd.conf:

<Plugin network>
Server "localhost" "25826"
</Plugin>

and in influxdb.conf:

[[collectd]]
enabled = true
bind-address = "localhost:25826"
database = "collectd"
retention-policy = ""
typesdb = "/usr/share/collectd/types.db"
parse-multivalue-plugin = "split"

and of course inside collectd.conf you have to add the fritzcollectd config from the github link above.

But with starting collectd you might get the error:

dlopen("/usr/lib/collectd/python.so") failed: /usr/lib/collectd/python.so: undefined symbol: PyFloat_Type

This can be solved with adding into /etc/default/collectd:

LD_PRELOAD=/usr/lib/python3.8/config-3.8-aarch64-linux-gnu/libpython3.8.so

Zigbee: Setup zigbee2mqtt with usbstick conbee II & influxdb on a raspberry pi

Just a short walkthrough of all steps which are necessary:

1.) insert the usbstick and check if this special device is there: /dev/ttyACM0

if this device is not showing up, it might be, that your kernel does not support usbserial. In my case i had to downgrade from ubuntu server 21.10 to 21.04.

2.) follow these steps: https://www.zigbee2mqtt.io/guide/installation/01_linux.html#installing

apt-get install -y nodejs git make g++ gcc npm
git clone https://github.com/Koenkk/zigbee2mqtt.git /opt/zigbee2mqtt
cd /opt/zigbee2mqtt
npm ci

if you get

prebuild-install WARN install EACCES: permission denied, access '/root/.npm/_cacache'

then you should not use root for running this command.

cd /opt/zigbee2mqtt
chown -R ubuntu node_modules
rm node_modules/*
npm ci

3.) install mqtt

apt install mosquitto mosquitto-clients

4.) add to /etc/mosquitto/mosqitto.conf the line

listener 1883 127.0.0.1

and restart mosquitto (systemctl restart mosquitto)

5.) then start the zigbee2mqtt:

cd /opt/zigbee2mqtt
npm start

if you get

Zigbee2MQTT:error 2021-11-06 09:05:23: Error: Error while opening serialport 'Error: Error: No such device or address, cannot open /dev/ttyACM0'

then you did not really check step 1.): please check that /dev/ttyACM0 is missing - if yes: for me the kernel module (to list: lsmod) usbserial was missing. It seems, that ubuntu missed that on 21.10 - so i reinstalled 21.04....

if you get

zigbee2MQTT:error 2021-11-06 14:54:11: MQTT failed to connect: connect ECONNREFUSED 127.0.0.1:1883

then you did not get mosquitto running. Check with systemctl status mosquitto and follow step 3 and 4.

6.) configure telegraf, so that the data from mosquitto is transferred to influxdb. So you have to add to telegraf.conf:

[[inputs.mqtt_consumer]]
   servers = ["tcp://127.0.0.1:1883"]
   topics = [
     "zigbee2mqtt/sensor/#",
   ]
   data_format = "json"

[[outputs.influxdb]]
   urls = ["unix:///var/run/influxdb/influxdb.sock"]
   username = "admin"
   password = "XXXXX"

7.) add this user to influxdb:

influx -ssl -unsafeSsl (only influx if you have not enabled SSL)
create user admin with password 'XXXXXXX' with all privileges

8.) if you have joined a device this the zigbee2mqtt, then you have to give a friendy name inside /opt/zigbee2mqtt/data/configuration.yaml

friendly_name: 'sensor/t1'

Dec 16, 2020

zigbee: moving data from mqtt to influxdb - transforming strings to integers

After some first steps with zigbee devices and storing the data in an influxdb, i noticed that string values are suboptimal for building graphs.

Moving the data from mqtt to influxdb was done with telegraf:

https://www.influxdata.com/time-series-platform/telegraf/

And i was wondering, how i can change string to integers, but this i very easy:

[[processors.enum]]
    order = 2
    [[processors.enum.mapping]]
      field = "state"
      [processors.enum.mapping.value_mappings]
        "ON" = 1
        "OFF" = 0
    [[processors.enum.mapping]]
      field = "contact"
      [processors.enum.mapping.value_mappings]
        "true" = 2
        "false" = 1
    [[processors.enum.mapping]]
      field = "tamper"
      [processors.enum.mapping.value_mappings]
        "true" = 1
        "false" = 0
    [[processors.enum.mapping]]
      field = "water_leak"
      [processors.enum.mapping.value_mappings]
        "true" = 1
        "false" = 0

Next problem: if the column "water_leak" was already added inside your influxdb, you can not add numbers - so you have to drop the table and loose your data...

(This is not the full truth: you can export the data via a select to a file and insert the data afterwards - with the appropriate numbers...)

Dec 5, 2020

Securing InfluxDB

In my monitoring setup i am heavily using InfluxDB. Starting with one linux server with grafana which loads the data from its local influxdb, i wanted to setup a second linux server.

My options:

new telegraf, new influxdb, new grafana
but then i have two url (because of two grafanas and i can not copy graphs from one dashboard to the other)
new telegraf, new influxdb, but grafana from first server
grafana has to get the data over the network
new telegraf, influxdb & grafana from first server
what is happening if telegraf can not reach influxdb, because of network problem? what if the first server is down?
completely remote monitoring
what is happening if telegraf can not reach the other server? what if the first server is down?

As you can see, option 2 is the favorite here.

But therefore InfluxDB has to be secured: SSL + user/password.

So let's start with creating some certificates:

openssl req -new -x509 -nodes -out server-cert.pem -days 3650 -keyout server-key.pem

So that you get:

zigbee:/etc/influxdb# ls -lrt *pem
-rw-r--r-- 1 influxdb root 1704 Nov 7 09:48 key.pem
-rw-r--r-- 1 influxdb root 1411 Nov 7 09:48 cert.pem

Then add this in /etc/influxdb/influxdb.conf

https-enabled = true
https-certificate = "/etc/influxdb/cert.pem"
https-private-key = "/etc/influxdb/key.pem"

But still a user is missing, so we have to create users (via bash):

influx -ssl -unsafeSsl
create user admin with password 'XXXXXXX' with all privileges

After that you can test this with

root@zigbee:# influx -ssl -unsafeSsl
Connected to https://localhost:8086 version 1.6.4
InfluxDB shell version: 1.6.4
> show databases
ERR: unable to parse authentication credentials
Warning: It is possible this error is due to not setting a database.
Please set a database with the command "use <database>".
> auth
username: admin
password:
> show databases
name: databases
name
----
_internal

Nov 20, 2020

ZigBee@Linux: Getting Data from ZigBee Devices via MQTT to InfluxDB and Grafana

Getting sensors with zigbee integrated with my linux raspberry pi, i did some monitoring tasks on my raspberry pi.

Monitoring my raspberry pi:
There is a very nice tutorial:
https://medium.com/@andreea.sonda31/monitor-raspberry-pi-resources-and-parameters-with-grafana-board-part-1-ab0567303e8
Or even better: Just use this from grafana:
https://grafana.com/grafana/dashboards/10578

add deb https://packages.grafana.com/oss/deb stable main to a file in /etc/apt/sources.list.d/
apt install grafana telegraf influxdb
configure telegraf for your influxdb
import the json from the grafana.com-link above

Monitoring my Fritz.Box with Grafana:
https://grafana.com/grafana/dashboards/713
and follow the given tutorial https://fetzerch.github.io/2014/08/23/fritzcollectd/

After these steps i have the following infrastructures running:

zigbee2mqtt --> MQTT -->FHEM
Fritz.box --> collectd --> InfluxDB --> Grafana
raspberry --> telegraf --> InfluxDB --> Grafana

For 2 and 3 it is very easy to create graphics and the presentation looks little bit prettier than 1 (imho).

AND there is only one frontend to configure. So what about the following chain for my zigbee sensors:

zigbee2mqtt --> MQTT -->telegraf --> InfluxDB --> Grafana

Looks like some more steps, but the telegraf --> InfluxDB --> Grafana chain is already there for monitoring my raspberry pi.

So i only had to add the following on /etc/telegraf/telegraf.conf:

[[inputs.mqtt_consumer]]
   servers = ["tcp://127.0.0.1:1883"]
   topics = [
     "zigbee2mqtt/0x00158d000542239e",
     "zigbee2mqtt/0x00158d00044a6378",
     "zigbee2mqtt/0x00158d0003f0faad",
     "zigbee2mqtt/0x00158d00044a72a2",
   data_format = "json"

And after that i was able to use the data in Grafana:

Nov 15, 2020

ZigBee@Linux: Securing zigbee2mqtt & MQTT@FHEM & FHEM

After my setup is running, just some words about securing the whole setup.

The web gui of FHEM was already setup with SSL/HTTPS but the MQTT server is listening for all ips.

The easiest way to get this secure is change the listener to localhost, so that no connections from outside can be made. Just change in /opt/fhem/fhem.cfg:

define MQTT2_FHEM_Server MQTT2_SERVER 1883 127.0.0.1

Just a checklist, if we secured everything:

FHEM

WebGUI: shown here
MQTT: s. above

zigbee2mqtt

add permit_join: false to configuration.yaml

Nov 14, 2020

ZigBee@Linux: Integration of zigbee2mqtt with FHEM (mqtt server) on ubuntu server

After the setup of FHEM and zigbee2mqtt the integration of both components has to be done.

What has to be done?

After reading the excellent documentation of FHEM it is very easy - FHEM can be configured, so that it is providing a mqtt server.

First you have to add the following line in /opt/zigbee2mqtt/data/configuration.yaml inside the "mqtt:" section:

client_id: 'zigbee_pi'

Then go to the command prompt of the FHEM webgui and enter the following:

define MQTT2_FHEM_Server MQTT2_SERVER 1883 global
defmod MQTT2_zigbee_pi MQTT2_DEVICE zigbee_pi
attr MQTT2_zigbee_pi IODev MQTT2_FHEM_Server
attr MQTT2_zigbee_pi bridgeRegexp zigbee2mqtt/([A-Za-z0-9]*)[/]?.*:.* "zigbee_$1"

After that you should see something like this:

(you can change the style of the page via "select style" on the left column)

Then you should save:

To create a graph just click on the file which is created for your zigbee device:

and then there should be something like:

Here you can click on "Create SVG plot" and on:

click on "write .gplot file" and your first graph is there... Repeat this and you can get:

Zigbee@Linux: Infrastructure - Setup

On my way to home automation with zigbee@linux my decision (as i wrote in this posting) was

Hardware

Raspberry Pi
usb zigbee stick (conbee II)

Ubuntu server

Software

FHEM (which is the acronym for Freundliche Hausautomation und Energie-Messung = Friendly home automation and energy metering)
This includes the server with MQTT infrastructure & webserver & gui based on perl
zigbee2mqtt
The server which does the communication with the usb zigbee stick and talking to the MQTT infrastructure based on nodejs

The installation of FHEM was quite easy (see here) and the installation of zigbee2mqtt just worked like described here.

Problem:
FHEM is per default installed without SSL/HTTPS and without user authentication
Problem:
The communication between both components has to be setup

Here the solution for problem 1:

cd /opt/fhem
chown fhem:dialout certs
cd certs/
openssl req -new -x509 -nodes -out server-cert.pem -days 3650 -keyout server-key.pem 
chown fhem:dialout *
apt  install libio-socket-ssl-perl

After that move the webgui (something like http://yourraspberry:8083) and submit the following commands on the prompt:

attr WEB sslVersion TLSv12:!SSLv3
attr WEB HTTPS 1

And then open your webfrontend with https://yourraspberry:8083.

To add a user:

@bash

echo -n fhem:MYPASSWD| base64

@Webfrontend:

attr WEB basicAuth BASE64String

The second problem will be solved on a future posting. Just wait...

Nov 8, 2020

Home automation with linux: How to use zigbee sensors on an ubuntu raspberry pi...

To the end of the year i wanted to start a new project: Home automation...

I decided to use a linux system (of course) on a raspberry pi (see the OS installation here) and the zigbee protocol.

The main problem: What packages are needed

to get a communication with zigbee components?
to get a website or app to get the data / visualize the data?
to set up a daemon/server which controls the devices?

Let's start with the third point: I will try FHEM.

The installation is described here:

https://debian.fhem.de/

wget -qO - http://debian.fhem.de/archive.key | apt-key add -
echo "deb http://debian.fhem.de/nightly/ /" >> /etc/apt/sources.list
apt update
apt upgrade
apt install fhem

After you follow the steps you can check, if FHEM is running with

root@zigbee:/home/ubuntu# netstat -ltnup | grep 8083
tcp 0 0 0.0.0.0:8083 0.0.0.0:* LISTEN 19446/perl

or just connect to your raspberry via browser: http://zigbee:8083