Feasibility of hosting Teddycloud

mrhanspeterwurst · October 24, 2024, 9:02am

Thanks for getting back this fast. I tried this but nginx fails to start because of certificate validation errors:

I did:

  112  openssl rsa -inform DER -in teddy-key.pem -out ttt-teddy-key.pem
  113  openssl rsa -inform DER -in ca-key.pem -out ttt-ca-key.pem
  114  cat teddy-cert.pem ttt-teddy-key.pem ca-root.pem ttt-ca-key.pem > ttt-fullchain.pem
  115  cp ttt-fullchain.pem /data/custom_ssl/npm-10/fullchain.pem
  116  cp ttt-teddy-key.pem /data/custom_ssl/npm-10/privkey.pem
  117  /usr/sbin/nginx -t

(/data/custom_ssl/npm-10 is the path where Nginx-Proxy-Manager expects the certs in my configuration)

this throws the following certificate validation error:

nginx: [emerg] cannot load certificate "/data/custom_ssl/npm-10/fullchain.pem": PEM_read_bio_X509() failed (SSL: error:068000DD:asn1 encoding routines::illegal padding error:0688010A:asn1 encoding routines::nested asn1 error:Field=serialNumber, Type=X509_CINF error:0688010A:asn1 encoding routines::nested asn1 error:Field=cert_info, Type=X509 error:0488000D:PEM routines::ASN1 lib)

Looks like the fullchain.pem is somehow malformed but I don’t know how. I googled around but haven’t found anything that matches this error.

I narrowed it down to ca-root.pem. As soon as ca-root.pem is part of/used as fullchain.pem. the error is thrown. openssl x509 -in ttt-fullchain.pem -text -noout works/looks fine though:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1729265846 (0x671280b6)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = DE, ST = NW, L = Duesseldorf, O = Team RevvoX, CN = TeddyCloud CA Root Cert.
        Validity
            Not Before: Nov  3 15:23:19 2015 GMT
            Not After : Jun 24 15:23:19 2040 GMT
        Subject: C = DE, ST = NW, L = Duesseldorf, O = Team RevvoX, CN = TeddyCloud Server
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:a4:1b:48:2b:e2:7a:62:58:cc:c1:97:8e:40:98:
                    72:67:04:03:b8:80:21:b8:e2:9f:61:1f:26:e7:3b:
                    a9:12:2a:2a:c6:b9:37:5f:0f:90:0e:cc:33:9f:24:
                    ba:99:2f:4e:5e:a5:ca:62:28:b5:4a:75:37:ce:aa:
                    3a:e5:71:ce:03:bd:7f:4e:24:26:02:dd:28:9b:56:
                    a9:46:21:f7:1e:88:15:ac:ef:71:26:cb:8f:6f:cd:
                    72:68:03:77:b4:f7:da:1f:07:d1:ef:74:d6:21:1c:
                    55:88:2b:ec:f2:da:f1:f3:ba:bb:8c:8c:81:e7:a8:
                    02:1a:a3:86:d8:ab:d2:63:44:10:bd:d9:50:f4:36:
                    fa:f6:54:19:87:5e:07:e9:26:c4:db:d6:7f:1f:f0:
                    6c:51:ad:09:b7:4b:3b:29:2e:79:1a:f0:d7:57:a6:
                    d9:1d:65:7f:15:7b:c5:32:52:81:48:b0:b3:ec:18:
                    6e:5d:f2:ab:06:bf:7c:38:c5:b2:2a:85:f3:48:24:
                    65:b9:af:16:c2:0c:8f:32:32:d4:8b:42:18:25:20:
                    39:77:97:72:1a:fb:10:4c:af:78:d0:69:88:55:e9:
                    89:fa:85:60:73:1a:f7:24:b5:83:23:e0:34:bb:31:
                    c2:dc:31:55:69:6c:47:15:36:5a:58:54:f8:b4:82:
                    ca:db
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                3B:F9:12:BD:C6:72:E5:B7:A5:08:97:E6:C1:12:56:90:16:05:A9:FC
            X509v3 Authority Key Identifier:
                C1:8C:E8:30:AE:C7:E5:3C:61:B0:1A:65:6E:02:09:EB:69:DE:C5:7C
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        6a:43:11:7e:a5:3d:28:d3:29:29:58:96:23:68:9e:c9:f4:ef:
        85:76:bc:ce:af:73:d0:e6:d1:5d:99:ca:7e:f3:b3:db:a7:f9:
        4f:80:dd:eb:40:86:85:5c:a7:2e:8b:7b:8f:bc:1d:a6:3d:fd:
        11:b5:1e:0d:2a:94:71:1b:ea:ec:b6:d7:11:16:1b:53:c4:7b:
        8f:57:24:31:26:c9:7d:23:6d:b9:39:4d:d3:c3:3d:75:36:ba:
        f8:a2:75:cc:c9:50:74:28:75:df:57:bc:ea:82:c3:9f:b4:39:
        c6:99:04:27:4a:ed:44:f2:69:c2:72:70:e5:48:a2:91:ab:0d:
        0f:78:b6:7f:90:06:8f:a3:97:3a:40:e2:80:a3:0b:35:78:4b:
        7d:06:5e:e4:f5:b0:28:96:f7:c0:40:61:d9:15:4d:5e:53:c4:
        7e:b1:5d:66:59:05:17:76:71:43:e0:b0:f6:ec:7d:39:f1:09:
        9b:2a:07:a2:28:ed:80:d7:fb:fe:f9:93:0c:ed:47:48:40:19:
        1e:2f:8b:a6:7f:0d:6b:d9:c6:87:17:61:a6:ae:24:d1:fc:e9:
        1a:c6:29:6a:6c:33:6c:61:0e:01:7c:ac:e5:38:23:14:05:9d:
        ff:ed:aa:73:20:07:e3:67:7b:a0:ad:a4:02:3d:01:15:f5:96:
        02:ac:8e:2f:9d:b2:55:73:bb:f8:02:d6:17:ef:16:f5:89:8c:
        6a:a9:49:56:b9:ce:d4:e7:ca:46:e5:bf:62:d3:54:8d:a1:3b:
        b1:0e:bc:5d:f2:f6:11:27:da:23:6b:46:76:94:0a:30:eb:af:
        bd:ff:6a:5c:73:44:3e:88:bf:c0:de:36:f6:f7:f4:15:8e:13:
        ee:55:d9:0f:9c:37:fe:39:9c:26:aa:b6:e5:5b:d7:7f:a0:f0:
        3b:21:a6:6d:9b:a7:06:d7:77:2c:64:88:09:26:a9:e9:5b:7c:
        4c:4b:0b:da:c2:24:4a:42:f5:ec:95:ca:2f:68:a8:6b:5f:2b:
        ba:b5:87:0f:fe:50:30:14:95:4f:17:52:6d:e7:33:d5:a7:d5:
        53:2e:fa:48:1b:a9:1c:ad:36:a8:18:78:f3:13:07:a6:ca:3a:
        40:91:dd:e4:5e:4e:4e:22:87:46:9f:18:4a:b8:6a:63:78:45:
        30:cf:94:36:74:5b:70:1d:fa:39:b7:90:ad:9f:2c:8b:96:f2:
        b0:f8:f3:6b:3c:57:56:36:d0:be:a0:4d:7e:75:92:0b:35:5e:
        00:9d:a6:95:3b:78:d4:60:86:5f:e0:0b:57:21:51:55:59:72:
        f9:27:04:16:f9:b3:d9:87:25:29:72:8f:1a:16:ef:c6:30:c0:
        67:dc:73:06:ec:93:ff:75

Do you have any idea on what could be wrong/differing from you setup here?

Gremlin · December 3, 2024, 11:39am

I am currently experiencing the same error. Have you found a solution for this?

ks61625 · December 8, 2024, 8:14pm

Am I correct, by using this config, anybody would have acces to TC backend, as long as SNI is not sent? Sounds “dangerous” to me… I think there should be added mTLS verification…? Or do I miss something, that this is not needed here?

adn77 · December 9, 2024, 10:56am

Yes, you’re correct. The backend (Boxine Emulation) is unprotected. The TCP connection is passed transparently to the backend if no SNI is present. There’s no way to inspect TCP traffic beyond the handshake (which sadly doesn’t contain an SNI in the case of the TonieBox).
mTLS in HAProxy is possible but can’t be done transparently like the Nginx hack (which is not really transparent and hardly secure).

If there was exploitable code in Teddycloud, this might be dangerous (and this is why the team doesn’t recommend public hosting).

Otherwise if somebody knows the Boxine protocol, they could read your media data from the server.

Only do this, if you’re willing to take this risk!

ks61625 · December 9, 2024, 6:01pm

Thanks for coming back on this.

What you mean with “hardly secure”? My understanding is, checking client cert with mTLS is most protection you can get here. As access would be limited to only clients providing with the correct cert. How could that be unsecure? What do I miss…?

adn77 · December 9, 2024, 6:49pm

Sorry, I was referring to the way Nginx is implementing this. Nginx is acting as a man in the middle in this scenario. There might be all kinds of issues due to that (timing, replay).

But then this is not a banking application

mrprohl · December 11, 2024, 10:03am

Hey, unfortunately I have the same problem. Does anyone have an idea?

ks61625 · December 13, 2024, 11:40am

Had no issue with the “fullchain” in nginx by exactly following what @cfelder explained here.

mrprohl · December 13, 2024, 6:36pm

I tried the last 3 hours to fix the problem, but no success. My error is:

fullchain.pem": PEM_read_bio_X509() failed (SSL: error:068000DD:asn1 encoding routines::illegal padding error:0688010A:asn1 encoding routines::nested asn1 error:Field=serialNumber, Type=X509_CINF error:0688010A:asn1 encoding routines::nested asn1 error:Field=cert_info, Type=X509 error:0488000D:PEM routines::ASN1 lib)

Does anyone have an idea?

My nginx:

nginx version: nginx/1.22.1
built with OpenSSL 3.0.8 7 Feb 2023 (running with OpenSSL 3.0.15 3 Sep 2024)
TLS SNI support enabled

My nginx settings:

server {
	## TeddyCloud subdomain
	listen                                    443 ssl;
    server_name								  DOMAIN;

	ssl_protocols SSLv2 SSLv3 TLSv1 TLSv1.1 TLSv1.2;
	ssl_ciphers ALL:!aNULL;

	# Server's certificate and private key
	ssl_certificate /etc/nginx/certs/teddycloud/server/fullchain.pem;
	ssl_certificate_key /etc/nginx/certs/teddycloud/server/teddy-key.pem;

	# Require client certificate
	ssl_client_certificate /etc/nginx/certs/teddycloud/client/ca.pem;
	ssl_verify_client optional_no_ca;

	location / {
		proxy_pass https://192.168.1.211;
		proxy_ssl_certificate /etc/nginx/certs/teddycloud/client/client.pem;
		proxy_ssl_certificate_key /etc/nginx/certs/teddycloud/client/private.pem;
		proxy_ssl_conf_command Options UnsafeLegacyRenegotiation;
	}
}

My commands:

server dir:
openssl rsa -inform DER -in teddy-key.pem -out ttt-teddy-key.pem
openssl rsa -inform DER -in ca-key.pem -out ttt-ca-key.pem
cat teddy-cert.pem ttt-teddy-key.pem ca-root.pem ttt-ca-key.pem > fullchain.pem

client dir:
openssl x509 -inform DER -in ca.der -out ca.pem
openssl x509 -inform DER -in client.der -out client.pem
openssl rsa -inform DER -in private.der -out private.pem

mrprohl · December 13, 2024, 9:23pm

Hey! I have found the problem. Teddycloud version >= 0.6.1 has a new CA keylength of 4096 bits. With <= 0.6.0 the SSL proxy is running fine with no PEM_read_bio_X509() error.

My workaround (edit):
Replace the files ca.der, ca-key.pem, ca-root.pem, teddy-cert.pem and teddy-key.pem in the “server”-dir on teddycloud_certs from version 0.6.0 (install a temp teddycloud) and patch/flash the toniebox again with and in your prod 0.6.2 version. Create a new fullchain.pem and now it works. It’s important that the teddybox ssl proxy config file from nginx is the first config in load order.

mrhanspeterwurst · December 14, 2024, 10:38am

Here is the corresponding commit: increase keylength from 2048 to 4096 · toniebox-reverse-engineering/teddycloud@9f48229 · GitHub

So you may change this var and run contrib/gencerts.sh by hand to get the old 2048 long certs without installing the older teddycloud version. You will still need to re-flash your box though.

mrprohl · December 14, 2024, 10:46am

Thanks, but the next problem, i think, is the other changes in cert.c in src:
(marked with *):

    const char *cacert = settings_get_string("core.server_cert.file.ca");
    const char *cacert_key = settings_get_string("core.server_cert.file.ca_key");
    *uint8_t serial[14];
    *size_t serial_length = 14;

    error_t error_ca = load_cert("internal.server.ca", "core.server_cert.file.ca", "core.server_cert.data.ca", 0);
    error_t error_ca_key = load_cert("internal.server.ca_key", "core.server_cert.file.ca_key", "core.server_cert.data.ca_key", 0);

    if (error_ca != NO_ERROR || error_ca_key != NO_ERROR)
    {

        /* create a proper ASN.1 compatible serial with no leading zeroes */
        cert_generate_serial(serial, &serial_length);
        *serial[0] = 0x00;

...

vs

error_t cert_generate_default()
{
    const char *cacert = settings_get_string("core.server_cert.file.ca");
    const char *cacert_key = settings_get_string("core.server_cert.file.ca_key");
    uint8_t serial[9];
    size_t serial_length;

    /* create a proper ASN.1 compatible serial with no leading zeroes */
    cert_generate_serial(serial, &serial_length);

...

yoxcu · December 15, 2024, 10:49am

Thanks for your haproxy config.
It can be cumbersome to list all domain names.
One can do a “catch all” by checking domain names for a dot “.” in their name (so all of them).

use_backend tcp2http if { req.ssl_sni -m sub . }

Gremlin · December 15, 2024, 12:53pm

ITS WORKING JEAHH

Thank you so much! I can finally sleep peacefully again haha

mrprohl · December 15, 2024, 1:00pm

No problem. For all, here is my final certicates cmd and nginx config. nginx above version 1.19.4 required.

Convert certificates:

server dir:
openssl rsa -inform DER -in teddy-key.pem -out ttt-teddy-key.pem
openssl rsa -inform DER -in ca-key.pem -out ttt-ca-key.pem
cat teddy-cert.pem ttt-teddy-key.pem ca-root.pem ttt-ca-key.pem > fullchain.pem

client dir:
openssl x509 -inform DER -in ca.der -out ca.pem
openssl x509 -inform DER -in client.der -out client.pem
openssl rsa -inform DER -in private.der -out private.pem

nginx http-config (must be the first load “server”-config with port 443):

## SSL SHA1 fingerprints with "grep -v ^- FILENAME  | base64 -d | sha1sum"
map $ssl_client_fingerprint $reject {
	default 1;
	"your_fingerprint" 0; # client/ca.pem
	"your_fingerprint" 0; # client/client.pem
}

server {
	## TeddyCloud subdomain
	listen                                    443 ssl;
    server_name								  YOUR_DOMAIN;
	
	## Reject connection if ssl fingerprint from remote is unknown
	if ($reject) {
		return 301                            https://www.google.de;
	}
	
	## Edit ssl protocols and ciphers
	ssl_protocols SSLv2 SSLv3 TLSv1 TLSv1.1 TLSv1.2;
	ssl_ciphers ALL:!aNULL;

	## Server's certificate and private key
	ssl_certificate /etc/nginx/certs/teddycloud/server/fullchain.pem;
	ssl_certificate_key /etc/nginx/certs/teddycloud/server/teddy-key.pem;

	## Require client certificate
	ssl_client_certificate /etc/nginx/certs/teddycloud/client/ca.pem;
	ssl_verify_client optional_no_ca;

	## Root location of subdomain
	location / {
		proxy_pass https://192.168.1.45:443;
		proxy_ssl_certificate /etc/nginx/certs/teddycloud/client/client.pem;
		proxy_ssl_certificate_key /etc/nginx/certs/teddycloud/client/private.pem;
		proxy_ssl_conf_command Options UnsafeLegacyRenegotiation;
	}
}

Belaith · January 4, 2025, 3:54pm

Hey, i use traefik for all my containers and didn’t want to use nginx just for this one so i adapted the configuration for traefik.
I already had a fully configured and working traefik instance.

Docker-Copose Labels:

labels:
       traefik.enable: true
       traefik.http.routers.teddycloudweb.entrypoints: httpsPort
       traefik.http.routers.teddycloudweb.rule: "Host(`teddycloudweb.MYDOMAIN .de`)"
       traefik.http.routers.teddycloudweb.service: teddycloudweb
       traefik.http.routers.teddycloudweb.middlewares: auth
       traefik.http.middlewares.auth.basicauth.usersFile: /etc/traefik/.htpasswd
       traefik.http.services.teddycloudweb.loadbalancer.server.port: 80
       traefik.tcp.routers.teddycloudapi.entrypoints: httpsPort
       traefik.tcp.routers.teddycloudapi.rule: "HostSNI(`*`)"
       traefik.tcp.routers.teddycloudapi.priority: 1
       traefik.tcp.routers.teddycloudapi.service: teddycloudapi
       traefik.tcp.routers.teddycloudapi.tls.passthrough: true
       traefik.tcp.services.teddycloudapi.loadbalancer.server.port: 443

The first router is for the webfrontend, if you don’t want to expose that just delete those lines.
It has a basic auth configured but could use any other auth that you already have configured.
Just don’t expose it without auth.
The second router catches all traffic that doesn’t match any other rule (of my other services) and forwards it to the teddycloud backend.

traefik.yml (only relevant bits):

 entryPoints:
   httpPort:
     address: ":80"
     http:
       redirections:
         entryPoint:
           to: httpsPort
           scheme: https
           permanent: false
   httpsPort:
     address: ":443"
     http:
       tls:
         certResolver: leresolver
         domains:
           - main: "*.MYDOMAIN .de"
             sans:
               - "MYDOMAIN .de"
       redirections:
 
 certificatesResolvers:
   leresolver:
     acme:
       email: admin@MYDOMAIN .de
       storage: etc/traefik/acme/acme.json
       caserver: https://acme-v02.api.letsencrypt.org/directory
       dnsChallenge:
         provider: cloudflare
         resolvers:
           - "1.1.1.1:53"
           - "1.0.0.1:53"
 
 # Docker configuration backend
 providers:
   file:
     filename: /etc/traefik/fileconf.yml
     watch: true

Part off my traefik configuration. The certificate resolver is only needed if you want to expose the webfrontend and want to do that with a real certificate.

fileconf.yml (only relevant bits):

 tls:
   options:
     default:
       minVersion: VersionTLS12
   stores:
     default:
       defaultCertificate:
         certFile: /etc/traefik/teddycloud/cert.crt
         keyFile: /etc/traefik/teddycloud/cert.key

What certificates to use for the “HostSNI(*)”-router:
cert.crt is same as fullchain.pem
and cert.key is ttt-teddy-key.pem

ks61625 · January 10, 2025, 9:13pm

Just for the records, a member of telegram group documented some smooth and working setup in his blog:

ks61625 · January 13, 2025, 10:18pm

Is this fully working with your TeddyCloud instance? Me, having cc3200 Toniebox, I am not able with this config, to see the Box beeing online inside TC under „Tonieboxes“. As well as connection seems not stable. Tonie/TAF download works somehow, but interrupts and resume multiple time. Radio-streams are instable as well.

Do you have cc3200 or ESP Box? Do you see „online“ status and which Tonie is currently placed under „Tonieboxes“ in TeddyCloud?

Cheers

mrprohl · January 16, 2025, 9:52pm

Hey, i use the ESP32 Box.

I think you need additional configs for CC:

ssl_protocols TLSv1.2;
ssl_ciphers 'HIGH:!aNULL:!MD5@SECLEVEL=0';
ssl_prefer_server_ciphers off;

ks61625 · January 20, 2025, 5:01pm

Thanks for feedback. Unfortunately this seems not to be the trick…I’ll keep on trying…
…any hints are very welcome