Started maintaining this file as: http://linuxmafia.com/~rick/jitsi.txt ...and largely following https://jitsi.github.io/handbook/docs/devops-guide/devops-guide-quickstart as my guide. EFFORT 1: Subsequently nuked and re-created. Skip to EFFORT 3 unless interested in what was tried and discarded. Picked EC2 region us-west-2 (Oregon). Picked AMI: debian-10-amd64-20200610-293 - ami-01f29bf534f24ebd2 Debian 10 (20200610-293) Instance type: m5a.xlarge vCPU: 4 RAM 16GB EBS storage EBS-Optimized Available: yes Network performance: up to 10 gigabit IPv6 Support: yes Network: vpc-65f6841d (default) Auto-assign Public IP: enable Security group 'jitsi' created with this design: 80/tcp for Web UI HTTP to redirect 443/tcp for Web UI HTTPS 4443/tcp for RTP media over TCP (transport for WebRTC A/V) 10000/udp for RTP media over UDP optional: 20000-20050/udp for jigasi (SIP access) Type Protocol Port Range Source Description SSH TCP 22 0.0.0.0/0 ssh SSH TCP 22 ::/0 ssh HTTP TCP 80 0.0.0.0/0 http HTTP TCP 80 ::/0 http HTTPS TCP 443 0.0.0.0/0 https HTTPS TCP 443 ::/0 https Custom TCP Rule TCP 4443 0.0.0.0/0 RTP media over TCP (transport for WebRTC A/V) Custom TCP Rule TCP 4443 ::/0 RTP media over TCP (transport for WebRTC A/V) Custom UDP Rule UDP 10000 0.0.0.0/0 RTP media over UDP Custom UDP Rule UDP 10000 ::/0 RTP media over UDP Custom UDP Rule UDP 20000 - 20050 0.0.0.0/0 Jigasi (SIP access) Custom UDP Rule UDP 20000 - 20050 ::/0 Jigasi (SIP access) Assign Public IP: Yes Storage: 8GB (default) Created new keypair 'Jitsi'. Downloaded keypair file 'Jitsi.pem.txt' to my laptop. (Other CNZ admin users will need a copy of this keypair file!) Running instance ID: i-0976fddcb2a9a938c Public IP: 44.230.198.10 chmod 600 Jitsi.pem.txt ssh -i "Jitsi.pem,txt" admin@ec2-44-230-198-10.us-west-2.compute.amazonaws.com sudo su - Finally, I'm into a generic Debian 10 'buster' host. Sole public service is sshd. Created DNS entry jitsi.linuxmafia.com for testing: $ dig -t a jitsi.linuxmafia.com +short 44.230.198.10 $ # apt-get update [...] # apt-get install gnupg2 apt-transport-https # hostnamectl set-hostname jitsi.linuxmafia.com # In /etc/hosts, change line 127.0.0.1 localhost to 127.0.0.1 localhost jitsi.linuxmafia.com Test: # echo $(hostname) jitsi.linuxmafia.com # ping "$(hostname)" PING localhost (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.023 ms 64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.028 ms 64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.046 ms ^C # Add Jitsi's repository to package sources: # curl https://download.jitsi.org/jitsi-key.gpg.key | sudo sh -c 'gpg --dearmor > /usr/share/keyrings/jitsi-keyring.gpg' # echo 'deb [signed-by=/usr/share/keyrings/jitsi-keyring.gpg] https://download.jitsi.org stable/' | sudo tee /etc/apt/sources.list.d/jitsi-stable.list > /dev/null # apt-get update # In the long term, we'll be using Let's Encrypt, but I'll defer that work until we have the real DNS name. For now, we'll generate a self-signed cert in /etc/ssl/private . I'll follow my own instructiosn on http://linuxmafia.com/faq/Security/ssl-cert-self-signed.html. Fetch to /tmp/ several text files to generate entropy from. # openssl genrsa -rand ssl-cert-self-signed.html:HTMLTerroristsHandbook.html:apache-lecture.html -out jitsi.key 2048 Generating RSA private key, 2048 bit long modulus (2 primes) .....................................+++++ .................................................................................................................+++++ e is 65537 (0x010001) # openssl req -new -sha256 -key jitsi.key -out jitsi.csr You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:CA Locality Name (eg, city) []:Menlo Park Organization Name (eg, company) [Internet Widgits Pty Ltd]:Linuxmafia.com Organizational Unit Name (eg, section) []: Common Name (e.g. server FQDN or YOUR name) []:jitsi.linuxmafia.com Email Address []:hostmaster@linuxmafia.com Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []: An optional company name []: # openssl x509 -req -sha256 -days 730 -in jitsi.csr -signkey jitsi.key -out jitsi.crt # ls -l jitsi.* -rw-r--r-- 1 root root 1334 Jul 2 01:51 jitsi.crt -rw-r--r-- 1 root root 1062 Jul 2 01:50 jitsi.csr -rw------- 1 root root 1679 Jul 2 01:46 jitsi.key # mv jitsi.csr jitsi.key /etc/ssl/private/ # mv jitsi.crt /etc/ssl/certs/jitsi.pem A self-signed SSL cert is OK for _most_ testing, but the Jitsi app for mobile (smartphones) refuses to connect to such a cert. (For the curious: The jitsi.csr file goes unused after cert-signing, and could be erased.) Need either nginx or Apache httpd, before installing jitsi-meet # apt-get install nginx-full nginx-doc nginx-common [...] # In /etc/nginx/snippets/ created jitsi-cert.conf as follows: ssl_certificate /etc/ssl/certs/jitsi.pem; ssl_certificate_key /etc/ssl/private/jitsi.key; #OMITTED (see: https://www.digitalocean.com/community/tutorials/how-to-create-a-self-signed-ssl-certificate-for-nginx-on-debian-8) #Define a strong cipher suite and some other desirable SSL parameters #for good security. # #New file /etc/nginx/snippets/ssl-params.conf # (didn't do this bit) Reference the cert & key to enable ssl: # cd /etc/nginx/sites-available/ # cp default default-ORIGINAL Change /etc/nginx/sites-available to say (in part): server { listen 80 default_server; listen [::]:80 default_server; server_name jitsi.linuxmafia.com; return 302 https://$server_name$request_uri; } server { # SSL configuration # listen 443 ssl default_server; listen [::]:443 ssl default_server; include snippets/jitsi-cert.conf; [...] Restart nginx. Verify that http://jitsi.linuxmafia.com is reachable. Now for the big install. Whole lot o' stuff to go in: # apt-get install jitsi-meet Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: ca-certificates-java coturn dnsutils java-common jicofo jitsi-meet-prosody jitsi-meet-turnserver jitsi-meet-web jitsi-meet-web-config jitsi-videobridge2 libasound2 libasound2-data libavahi-client3 libavahi-common-data libavahi-common3 libcups2 libevent-2.1-6 libevent-core-2.1-6 libevent-extra-2.1-6 libevent-openssl-2.1-6 libevent-pthreads-2.1-6 libhiredis0.14 libirs161 liblcms2-2 libmariadb3 libnspr4 libnss3 libpcsclite1 libpq5 libxext6 libxi6 libxrender1 libxtst6 lua-bitop lua-event lua-expat lua-filesystem lua-sec lua-socket lua5.2 mariadb-common mysql-common openjdk-11-jre-headless prosody sqlite3 ssl-cert telnet x11-common Suggested packages: sip-router rblcheck default-jre libasound2-plugins alsa-utils cups-common liblcms2-utils pcscd libnss-mdns fonts-dejavu-extra fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei | fonts-wqy-zenhei fonts-indic lua-dbi-mysql lua-dbi-postgresql lua-dbi-sqlite3 lua-zlib sqlite3-doc openssl-blacklist The following NEW packages will be installed: ca-certificates-java coturn dnsutils java-common jicofo jitsi-meet jitsi-meet-prosody jitsi-meet-turnserver jitsi-meet-web jitsi-meet-web-config jitsi-videobridge2 libasound2 libasound2-data libavahi-client3 libavahi-common-data libavahi-common3 libcups2 libevent-2.1-6 libevent-core-2.1-6 libevent-extra-2.1-6 libevent-openssl-2.1-6 libevent-pthreads-2.1-6 libhiredis0.14 libirs161 liblcms2-2 libmariadb3 libnspr4 libnss3 libpcsclite1 libpq5 libxext6 libxi6 libxrender1 libxtst6 lua-bitop lua-event lua-expat lua-filesystem lua-sec lua-socket lua5.2 mariadb-common mysql-common openjdk-11-jre-headless prosody sqlite3 ssl-cert telnet x11-common 0 upgraded, 49 newly installed, 0 to remove and 0 not upgraded. Need to get 135 MB of archives. After this operation, 322 MB of additional disk space will be used. Do you want to continue? [Y/n] [...] Puts me through installation of package jitsi-videobridge. Prompts for FQDN. Prompts for whether to use my own cert (yes). Prompts for full path to SSL key (/etc/ssl/private/jitsi.key). Prompts for full path to SSL cert (/etc/ssl/certs/jitsi.pem). Completes. Verified that Jitsi Meet now comes up for a functional A/V room when I test using Chromium Web browser to test room https://jitsi.linuxmafia.com/fnord . IMPORTANT: This test instance initially has ZERO access control. Anyone on the Internet can currently use the server and create/use any arbitrary Jitsi room on it. Adjust limits on system process/file handling. In /etc/systemd/system.conf set: DefaultLimitNPROC=65000 DefaultTasksMax=65000 # systemctl daemon-reload # systemctl restart jitsi-videobridge2 To adjust Jitsi Videobridge for NAT operation: Edit /etc/jitsi/videobridge/sip-communicator.properties to add: org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=172.31.63.37 org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=44.230.198.10 Also, comment out this line: org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443 # systemctl restart jitsi-videobridge2 # IMPORTANT: AWS assigns a new IP address upon instance restart. (Therefore, DNS also needs updating, at that point.) Tackling the authentication problem w/oath2 support (for GLUU): Fetched https://github.com/jitsi/jitsi-auth-portal code It's in Go (golang), so install the golang metapackage (which installs many dependencies along with essential golang support): # apt-get install golang Also install git # apt-get install git Setup instructions say: The Jitsi Authentication Portal is an OAuth2 frontend that allows authentication with a number of third party providers and generates short-lived, signed, JWT (jot) tokens to assert the users identity to Jitsi Meet. The package contains a number of handlers which can be used to build your own compatible login service. There is also an example service in the `cmd/jap` directory which provides a simple frontend. To get started, install the `jap` command and run it: ```go go get github.com/jitsi/jap go install github.com/jitsi/jap/cmd/jap jap -help ``` 'go get github.com/jitsi/jap' creates subtree 'go' Moving on to the next command results in a missing golang component: root@jitsi:~# go install github.com/jitsi/jap/cmd/jap go/src/github.com/jitsi/jap/cmd/jap/main.go:30:2: cannot find package "golang.org/x/text/language" in any of: /usr/lib/go-1.11/src/golang.org/x/text/language (from $GOROOT) root@jitsi:~# Fixing: # go get -v -u golang.org/x/text/language [...] # root@jitsi:~# cd go/bin/ ./jap -help The jap command launches an OAuth2 server that generates a JSON Web Signature (JWS) to prove the users identity to other Jitsi services. Environment To use the supported providers, a few environment variables must be set: GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET: Needed to support login with Google. The private key used to sign JWTs can also be loaded directly from an environment variable: JAP_PRIVATE_KEY Signals On POSIX based systems templates can be reloaded in a running process by sending the process a SIGHUP. For more information on POSIX signals, see the signal(7) man page. Usage of ./jap: -http string The address to listen on. (default ":http-alt") -key string An RSA private key in PEM format to use for signing tokens. Defaults to $JAP_PRIVATE_KEY_PATH. -maxconns int The maximum number of connections to service at once or 0 for unlimited. -origin string A domain that the /login endpoint will send a postMessage too (eg. https://meet.jit.si). -public string A directory containing static files to serve. (default "public/") -rpc string The RPC call to make to rcpaddr. This should be a function that takes a string (the token) and replies with a boolean. It should be compatible with Go's net/rpc package. (default "Permissions.Check") -rpcaddr string An address that can be used to make RPC calls to verify permissions for a user. -rpccodec string The type of RPC call to make (either "gob" for Go gobs or "json" for JSON-RPC). (default "gob") -rpcretries int The number of times to retry making RPC calls. (default 3) -templates string A directory containing templates to render. (default "templates/") root@jitsi:~/go/bin# /root/go/src/golang.org/x/text/language (from $GOPATH) Unresolved: Does Jitsi Authentication Portal support GLUU, and if so how? If not, it's possible that this alternative might: Jitsi Authentication Provider https://godoc.org/github.com/jitsi/jap Found in the source tree for Jisti Authentication Provider: # OAuth2 for Go [![Build Status](https://travis-ci.org/golang/oauth2.svg?branch=master)](https://travis-ci.org/golang/oauth2) [![GoDoc](https://godoc.org/golang.org/x/oauth2?status.svg)](https://godoc.org/golang.org/x/oauth2) oauth2 package contains a client implementation for OAuth 2.0 spec. ## Installation ~~~ go get golang.org/x/oauth2 ~~~ Or you can manually git clone the repository to `$(go env GOPATH)/src/golang.org/x/oauth2`. See godoc for further documentation and examples. * [godoc.org/golang.org/x/oauth2](https://godoc.org/golang.org/x/oauth2) * [godoc.org/golang.org/x/oauth2/google](https://godoc.org/golang.org/x/oauth2/google) ## Policy for new packages We no longer accept new provider-specific packages in this repo if all they do is add a single endpoint variable. If you just want to add a single endpoint, add it to the [godoc.org/golang.org/x/oauth2/endpoints](https://godoc.org/golang.org/x/oauth2/endpoints) package. ## Report Issues / Send Patches This repository uses Gerrit for code changes. To learn how to submit changes to this repository, see https://golang.org/doc/contribute.html. The main issue tracker for the oauth2 repository is located at https://github.com/golang/oauth2/issues. https://godoc.org/golang.org/x/oauth2/endpoints clarifies that package endpoints provides constants for using OAuth2 to access various services look like this: var GitHub = oauth2.Endpoint{ AuthURL: "https://github.com/login/oauth/authorize", TokenURL: "https://github.com/login/oauth/access_token", } Various examples are shown, and are implemented in ~/go/src/golang.org/x/oauth2/$THING/$THING.go files. Deferring OATH work. Instead, enabling hashed passwords. Docs: https://jitsi.github.io/handbook/docs/devops-guide/secure-domain In /etc/prosody/conf.avail/jitsi.linuxmafia.com.cfg.lua , in VirtualHost "jitsi.linuxmafia.com" stanza, comment out Authentication = "anonymous" Add authentication = "internal_hashed" Add new block: VirtualHost "guest.jitsi-meet.example.com" authentication = "anonymous" c2s_require_encryption = false In /etc/jitsi/meet/jitsi.linuxmafia.com-config.js Add line anonymousdomain: 'guest.jitsi-meet.example.com', In /etc/jitsi/jicofo/sip-communicator.properties , add additional line: org.jitsi.jicofo.auth.URL=XMPP:jitsi-meet.example.com Add usernames/passwords: prosodyctl register rickmoen jitsi.linuxmafia.com $PASS prosodyctl register test jitsi.linuxmafia.com $PASS (Real passwords in place of $PASS.) Updated /etc/jitsi/videobridge/sip-communicator.properties for new public IP. /etc/init.d/jitsi-videobridge2 restart /etc/init.d/jicofo restart /etc/init.d/prosody restart Update DNS for jitsi.linuxmafia.com for new public IP address. Upon trying to join conference 'test', I get told that I need to wait for the moderator to start the conference, unless I'm the moderator. I indicate I'm the moderator, provide username & password, submit, and get 'Oops! Something went wrong and we couldn't connect to the conference: connection.GET_SESSION_ID_ERROR' Hmm, that's a problem. Rolling back changes to: /etc/jitsi/jicofo/sip-communicator.properties /etc/jitsi/meet/jitsi.linuxmafia.com-config.js /etc/prosody/conf.avail/jitsi.linuxmafia.com.cfg.lua Re-verified that the no-authentication setup still works. Trying again, looking at https://dev.to/noandrea/self-hosted-jitsi-server-with-authentication-ie7 : Editing as indicated on that page: /etc/jitsi/meet/jitsi.linuxmafia.com-config.js /etc/prosody/conf.avail/jitsi.linuxmafia.com.cfg.lua /etc/jitsi/jicofo/sip-communicator.properties prosodyctl register rickmoen jitsi.linuxmafia.com $PASS prosodyctl register test jitsi.linuxmafia.com $PASS (Real passwords in place of $PASS.) /etc/init.d/jicofo restart /etc/init.d/jitsi-videobridge2 restart /etc/init.d/prosody restart This regime fails in a different way: No user authentication gets attempted, and the anonymous user can immediately create a room. Rolling back changes to: /etc/jitsi/meet/jitsi.linuxmafia.com-config.js /etc/prosody/conf.avail/jitsi.linuxmafia.com.cfg.lua /etc/jitsi/jicofo/sip-communicator.properties /etc/init.d/jicofo restart /etc/init.d/jitsi-videobridge2 restart /etc/init.d/prosody restart Re-verified that the no-authentication setup still works. Switched from self-signed cert to Let's Encrypt cert. First, backed up /etc/nginx to /root/nginx-2020-07-11.tar.gz . Then: apt-get install certbot python-certbot-nginx certbot --authenticator standalone --installer nginx -d jitsi.linuxmafia.com --pre-hook "service nginx stop" --post-hook "service nginx start" Follow instructions given. EFFORT 2. My friend Michael Paoli (another sysadmin) and I were worried enough about possible cumulative errors that we terminated the EC2 instance to start over with a clean slate. Picked EC2 region us-west-2 (Oregon). Picked AMI: debian-10-amd64-20200610-293 - ami-01f29bf534f24ebd2 Debian 10 (20200610-293) Instance type: m5a.xlarge vCPU: 4 RAM 16GB EBS storage EBS-Optimized Available: yes Network performance: up to 10 gigabit IPv6 Support: yes Network: vpc-65f6841d (default) Auto-assign Public IP: enable Security group 'jitsi' created with this design: 80/tcp for Web UI HTTP to redirect 443/tcp for Web UI HTTPS 4443/tcp for RTP media over TCP (transport for WebRTC A/V) 10000/udp for RTP media over UDP optional: 20000-20050/udp for jigasi (SIP access) Type Protocol Port Range Source Description SSH TCP 22 0.0.0.0/0 ssh SSH TCP 22 ::/0 ssh HTTP TCP 80 0.0.0.0/0 http HTTP TCP 80 ::/0 http HTTPS TCP 443 0.0.0.0/0 https HTTPS TCP 443 ::/0 https Custom TCP Rule TCP 4443 0.0.0.0/0 RTP media over TCP (transport for WebRTC A/V) Custom TCP Rule TCP 4443 ::/0 RTP media over TCP (transport for WebRTC A/V) Custom UDP Rule UDP 10000 0.0.0.0/0 RTP media over UDP Custom UDP Rule UDP 10000 ::/0 RTP media over UDP Custom UDP Rule UDP 20000 - 20050 0.0.0.0/0 Jigasi (SIP access) Custom UDP Rule UDP 20000 - 20050 ::/0 Jigasi (SIP access) Assign Public IP: Yes Storage: 8GB (default) new keypair 'Jitsi'. Downloaded keypair file 'Jitsi.pem.txt' to my laptop. (Other CNZ admin users will need a copy of this keypair file!) Running instance ID: i-00b62cfa5fcc33a02 Public IP address: 44.234.64.170 $ ssh -i "Jitsi.pem,txt" admin@44.234.64.170 $ sudo su - Created _new_ DNS entry 'meet.linuxmafia.com', this time with TTL set to 300 seconds (5 minutes). # dig -t a jitsi.linuxmafia.com +short 44.234.64.170 # # apt-get update [...] # apt-get install gnupg2 apt-transport-https nvi nginx-full nginx-doc # nginx-common [...] # hostnamectl set-hostname meet.linuxmafia.com # In /etc/hosts, change line 127.0.0.1 localhost to 127.0.0.1 localhost meet.linuxmafia.com # echo $(hostname) meet.linuxmafia.com # ping "$(hostname)" PING localhost (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.015 ms 64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.021 ms 64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.016 ms ^C --- localhost ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 27ms rtt min/avg/max/mdev = 0.015/0.017/0.021/0.004 ms # # curl https://download.jitsi.org/jitsi-key.gpg.key | sudo sh -c 'gpg --dearmor > /usr/share/keyrings/jitsi-keyring.gpg' # echo 'deb [signed-by=/usr/share/keyrings/jitsi-keyring.gpg] https://download.jitsi.org stable/' | sudo tee /etc/apt/sources.list.d/jitsi-stable.list > /dev/null # apt-get update # This time, we are _not_ manually hacking a self-signed SSL cert. Instead, we'll let the Jitsi installer routine do one, and convert to Let's Encrypt later. Update /etc/systemd/system.conf to have: DefaultLimitNOFILE=65000 DefaultLimitNPROC=65000 DefaultTasksMax=65000 Implement those changes: # systemctl daemon-reload # systemctl restart jitsi-videobridge2 Check with: # systemctl status jitsi-videobridge2 | grep Tasks Tasks: 42 (limit: 65000) # cat /proc/`cat /var/run/jitsi-videobridge/jitsi-videobridge.pid`/limits | grep -e processes -e open Max processes 65000 65000 processes Max open files 65000 65000 files # Finally the big install. # apt-get install jitsi-meet [...] Puts me through installation of package jitsi-videobridge2. Prompts for FQDN. (Answer meet.linuxmafia.com). Prompts for e-mail address. (Answer hostmaster@linuxmafia.com). Prompts for whether to use my own cert. Answer _no_. This results in Jitsi generating a self-signed cert with the option of retrofitting Let's Encrypt later, which docs say can be done by running script /usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh . As Amazon EC2 machines are behind NAT, need to update /etc/jitsi/videobridge/sip-communicator.properties to add two new lines: org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=172.31.51.199 org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=44.234.64.170 And comment out (using #) this line: org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443 NOTE: The public IP address line must be updated if EC2 assigns the instance a new public IP, e.g., because we stopped and then restarted the instance. Among other things, the 'apt-get install jitsi-meet' instructions create /etc/nginx/site-available/meet.linuxmafia.com.conf with hooks for Jitsi and for the self-signed cert. Complete steps for doing moderator auth with anoymous guest are here: https://jochen.kirstaetter.name/authentication-jitsi-meet/ Did those, and they were successful. Assigned Elastic IP 100.20.107.25 . Updated /etc/jitsi/videobridge/sip-communicator.properties and service jitsi-videobridge2 restart Requested FQDN jitsi.conzealand.nz from Z!, pointing to that IP. Reminder: Need to do Let's Encrypt & re-do service configs for the new FQDN when we have that. Edited /etc/jitsi/meet/meet.linuxmafia.com-config.js to set: startWithAudioMuted: true, startWithVideoMuted: true, prejoinPageEnabled: true, EFFORT 3: CNZ has decided it wants no-auth Jitsi Meet. To avoid burning bridges, creating a new EC2 instance. Picked EC2 region us-west-2 (Oregon). Picked AMI: debian-10-amd64-20200610-293 - ami-01f29bf534f24ebd2 Debian 10 (20200610-293) VPC: vpc-65f6841d Availability zone: us-west-2d Subnet: subnet-6b237240 Instance type: m5a.xlarge vCPU: 4 RAM 16GB EBS storage EBS-Optimized Available: yes Network performance: up to 10 gigabit IPv6 Support: yes Network: vpc-65f6841d (default) Auto-assign Public IP: enable (Will rebind to the elastic IP, later.) Security group 'jitsi' reused. Reused keypair file 'Jitsi.pem.txt'. Running instance ID: i-0365da97a2c92a3b0 Public IP (temporary): 44.234.115.157 ssh -i "Downloads/Jitsi.pem.txt" admin@44.234.115.157 $ sudo su - # apt-get update [...] # hostnamectl set-hostname jitsi.conzealand.nz # echo $(hostname) jitsi.conzealand.nz # ping jitsi.conzealand.nz PING localhost (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.020 ms 64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.011 ms ^C --- localhost ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 25ms rtt min/avg/max/mdev = 0.011/0.015/0.020/0.006 ms # apt-get install gnupg2 apt-transport-https nvi apt-transport-https nvi nginx-full nginx-doc nginx-common [...] # curl https://download.jitsi.org/jitsi-key.gpg.key | sudo sh -c 'gpg --dearmor > /usr/share/keyrings/jitsi-keyring.gpg' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3071 100 3071 0 0 56870 0 --:--:-- --:--:-- --:--:-- 56870 # echo 'deb [signed-by=/usr/share/keyrings/jitsi-keyring.gpg] https://download.jitsi.org stable/' | sudo tee /etc/apt/sources.list.d/jitsi-stable.list > /dev/null # apt-get update [...] # Update /etc/systemd/system.conf to have: DefaultLimitNOFILE=65000 DefaultLimitNPROC=65000 DefaultTasksMax=65000 Implement those changes: # systemctl daemon-reload # apt-get install jitsi-meet [...] This time, specify hostname as jitsi.conzealand.nz during config of package jitsi-videobridge2. Installation includes reminder output: "You can now switch to a Let’s Encrypt certificate. To do so, execute: /usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh" Check system limits: # systemctl status jitsi-videobridge2 | grep Tasks Tasks: 39 (limit: 65000) # cat /proc/`cat /var/run/jitsi-videobridge/jitsi-videobridge.pid`/limits | grep -e processes -e open Max processes 65000 65000 processes Max open files 65000 65000 files # Edit /etc/jitsi/videobridge/sip-communicator.properties to add two new lines: org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=172.31.51.199 org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=44.234.115.157 And comment out line org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443 Edited /etc/jitsi/meet/jitsi.conzealand.nz-config.js to set: startWithAudioMuted: true, startWithVideoMuted: true, # /etc/init.d/jitsi-videobridge2 restart [ ok ] Restarting jitsi-videobridge2 (via systemctl): jitsi-videobridge2.service. # /etc/init.d/jicofo restart [ ok ] Restarting jicofo (via systemctl): jicofo.service. # /etc/init.d/prosody restart [ ok ] Restarting prosody (via systemctl): prosody.service. # /etc/init.d/nginx restart [ ok ] Restarting nginx (via systemctl): nginx.service. # System ought to be fully working, but we need to switch from temporary IP 44.234.115.157 to Elastic IP 100.20.107.25 Then update /etc/jitsi/videobridge/sip-communicator.properties and do service jitsi-videobridge2 restart. EIP de-associated with instance i-00b62cfa5fcc33a02 . EIP re-associated with instance i-0365da97a2c92a3b0 . (Old instance is now given public IP 34.222.179.101.) Updated /etc/jitsi/videobridge/sip-communicator.properties to new public IP. # service jitsi-videobridge2 restart # Confirmed working (with self-signed cert.) # /usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh Verified that new server is working by going to https://jitsi.conzealand.nz/test . Add Zabbix monitoring. Problem: Debian 10 (buster) 'stable' provides only zabbix-agent v. 4.0.4. Need zabbix-agent 5.x, provided by backports. Create /etc/apt/sources.list.d/buster-backports.list containing deb http://deb.debian.org/debian buster-backports main # apt-get update # apt-get -t buster-backports install zabbix-agent # dpkg -l | grep zabbix ii zabbix-agent 1:5.0.1+dfsg-1~bpo10+1 amd64 network monitoring solution - agent # # systemctl stop zabbix-agent.service # systemctl disable zabbix-agent.service ## Temporarily ensure that it doesn't start until wanted. In /etc/zabbix create psk.conf containing 6a697473692e636f6e7a65616c616e642e6e7a32737461747573 Edit /etc/zabbix/zabbix_agentd.conf to have Server=traffic.conzealand.nz ServerActive=traffic.conzealand.nz Hostname=jitsi.conzealand.nz HostMetadata=Linux TLSConnect=psk TLSAccept=psk TLSPSKIdentity=jitsi-status TLSPSKFile=/etc/zabbix/psk.conf Edit /etc/nginx/sites-available to add in the port 80 section: location = /basic_status { stub_status; allow 127.0.0.1; allow 164.90.149.90; deny all; } Add in the port 4444 section: location = /basic_status { stub_status; allow 127.0.0.1; allow 164.90.149.90; deny all; } systemctl restart nginx.service systemctl start zabbix-agent.service systemctl enable zabbix-agent.service Narrative/history of the effort and its context. (I wrote this shortly after the experience, hence everything was still fresh in memory, at the request of local science fiction convention BayCon, in case they wanted to use the technology. Below is my e-mail text to a BayCon volunteer:) ------ As you will see, after consultation with other tech folks on my CoNZealand team, we used: Instance type: m5a.xlarge vCPU: 4 RAM 16GB EBS storage EBS-Optimized Available: yes Network performance: up to 10 gigabit IPv6 Support: yes In case you're not familiar with the above, this is jargon specific to Amazon Web Services (AWS) Elastic Compute Cloud (EC2) -- Amazon's main cloud computing offering. A Jitsi Meet server can run on practically any Unix machine with a public IP address anywhere. CNZ just happened to already use the EC2 cloud service for all of its infrastructure, so I spun up Debian 10 using their EC2 customer login credential. It could have been hosted on any cloud or non-cloud Unix host, anywhere on the planet, that has Internet connectivity. The above was my and my team's SWAG (scientific wild-assed guess) about what a first-pass at sufficient virtual hardware and bandwidth would be. That turned out to be _more_ than sufficient hardware resources and bandwidth for a virtual Worldcon's Dealer's Room needs. Vastly more. As I mentioned, in the event of our needing more, I would have slipped in additional EC2 hosts in the same cluster, running the Jitsi Videobridge2 component, to scale up capacity as required. We deployed in EC2 region us-west-2 (Oregon) because that was a tad cheaper than some of the other regions, and because some other CNZ infrastructure was already there. To clarify, the _main_ reason CNZ was interested in Jitsi Meet was to be a vehicle for interactive chat between dealers in the virtual Dealer's Hall and customers. We also made sure that _if_ the dreadful happened and Zoom's services collapsed during con, CNZ could if necessary cut over to Jitsi Meet as a fallback. The latter, doomsday-usage mode was not needed, and so it was used only for simultaneous dealer video presence for all dealers and for any and all customers wishing to enter their virtual stores. I honestly have absolutely no idea how much CNZ spent for the EC2 bill related to jitsi.conzealand.nz . I'm sure they could look that up. What surprised me just a little was that the above single instance of Jitsi Meet not only was never heavily loaded but seemed to never break a sweat. As you'll see in the write-up, we made the virtual EC2 host report to a Zabbix server with lots and lots of real-time and historical metrics. I followed the numbers and graphs during con, but never grabbed copies. Again, BayCon could request details from CNZ. My role was just to make sure it worked and be prepared to go into emergency debugging mode if necessary to fix problems. Happily, it was highly reliable. I should explain a few things in my (referenced) write-up: 1. Early on, I had been tasked to try to make Jitsi Meet seemlessly integrate into CNZ's grand plan for single sign-on, which was supposed to use a technology called oauth2 (an IETF standard for authentication/authorisation,https://oauth.net/2/), implemented by a company called GLUU. I did a great deal of research about how to shim in alternative authentication systems into Jitsi Meet, and ended up at a point where we needed company-specific implementation details from GLUU -- but somehow we never go that information. (Nobody's fault. Mostly, we just ran out of time.) So, full integration into SSO (single sign-on) was not possible, and got dropped from the project scope. If we'd had more time, that might have been a solvable problem, but as you might imagine, everything had to be built and tested super-fast. 2. We went ultimately with the default Jitsi Meet setup where nobody requires a password to enter a room, for simplicity's sake and because we were nervous about debugging integration with WordPress, which was where people entered video-chat rooms for Dealer's Hall. _However_, with only, IIRC, about 1-2 days to go before show time, I did manage to robustly solve the problem of reconfiguring Jitsi Meet so that _only_ a person with a pre-arranged username/password pair had the power to create and open a room. We _could_ have deployed that, but my team (wisely, IMO) decided to go with simpler (one more thing that we didn't have to worry about going wrong during con). 3. In the event of the system getting bogged down and starting to have bad performance / high load, the standard answer is to deploy and add into the cluster additional machines (virtual EC machines in this case) running the Jitsi Videobridge2 component -- adding more capacity. I don't think I specifically documented how to do that, but I remember that it looked pretty easy, and even pretty easy to script, so that the cluster can be scaled up on-the-fly. As it happened, the system was never even a tiny bit in need of more video capacity. It was almost comically not challenged by the con's demands. 4. I asked CNZ management whether they wanted to pony up for an SSL cert for the site, e.g., whether they had an existing deal with a Certificate Authority they wanted to use, or whether they preferred I be a cheap sombitch and use Let's Encrypt. There was a small amount of time lost on the question, but eventually the answer (as I expected) was "Be a cheap sombitch, Rick". Fortunately, I'd already by then thoroughly tested Debian's Let's Encrypt integration scripts, which -- thank the gods -- are bulletproof and abstract away the complexities. Things were actually so last-minute that I wasn't able to get the DNS name "jitsi.conzealand.nz" created for me until... um... late in the process of deploying and testing. So, for most of my write-up, you'll see that I used a temporary hostname of something like meet.linuxmafia.com, since I could make an addition to _my_ domain's DNS without borthering anyone. Production used jitsi.conzealand.nz, of course. I got practiced enough at following my instructions that, if memory serves, I was able to start from scratch with creating a generic Debian 10 image and have Jitsi Meet fully functional in about 20 minutes, even though manual baby-sitting was required (as described). It actually would not have been difficult to fully script, which would have taken time to deployment down to just a couple of minutes, but we didn't need that. 5. The Debian 10 Zabbix package was a big problem, as I detail, so I list painstakingly what we did to solve that. Fortunately, I was working with a Zabbix expert, who pretty much guided me through fixing it. 6. You'll see that, in the instructions, I go out of my way to change a couple of Jitsi Meet server defaults, to: startWithAudioMuted: true, startWithVideoMuted: true, This was part of our great caution to prevent the Jitsi Meet instance getting overwhelmed with too much user traffic: The more-important of those two lines is the second one, which causes users entering a conference to arrive with their video cameras (beaming out their smiling faces) initially switched off. It was probably a wise precaution, because we hadn't really had time, pre-con, to do load testing, and were afraid of overwhelming the EC2 host with lots of video streams whizzing between the instance and each connected user in a conference room. As mentioned, as it turned out, load and performance were at no point ever a problem. If you wish to try my instructions, by far the best thing would be to just put, say, somebody's machine with 8GB of RAM (or even 4GB, but bear in mind that this _is_ reakin' Java, so it's a RAM pig by my crufty Unix old-timer standards) up on a public IP address, install a server profile of Debian, and then just follow my recipe and play with it. I did my best to make it a complete do-this-and-you'll-have-it-running Betty Crocker-worthy recipe. And, honestly, I've been intending to do something very like that in order to write a less-frantic and more novice-friendly article about the experience. I just haven't had the time to do so. ------