First report after the school started again in Germany.

1. ffmuc: FFMEET what happened so far
1500 concurrent users and beyond

2. awlnx
● Senior Network Engineer @AS51324
● Twitter @awlnx
Krombel
● IT Consultant
● Twitter @kr0mbel
2
Who are we?

3. 3
Holidays are over … school starts - 20.04.2020
• All time record before was around 600 Users

4. 4
21.04.2020 - Another record! Sadly also another bug
● 20 min to find the issue, isolate and fix it!
● Unfortunately there was an other bug hiding after the crash
● We run at reduced capacity for a few hours but as we have so many servers
that was not an issue.
● Crash of prosody crashed some important DBs :/ ⇒ Loadbalancing failed

5. 5
New records every day!
• Today (29.04.2020) we hit our all time max of 1503 concurrent users. Servers
had no issues!

6. 6
We are running on the internet aren’t we?
• From time to time there are issues in the internet beyond our control (route
leaks, connectivity issues between providers, maintenance)
• We have many servers and providers so we can easily mitigate connectivity
issues

7. 7
Some figures?
• ~ 4000GB a day / ~ 28.000GB a week of videotraffic
• ~ 10.000.000 HTTPS Requests a day
• 180 CPU Cores / 240GB of RAM on 20 servers
• ~ 100 tickets solved (many schools, charity projects, individuals etc.)
• ~ 200 Euro / day
• 2 Admins and at least 15 people in the background for translations, press,
wiki etc

8. 8
What’s next?
• Hopefully more happy users!
• Establishing a long term roadmap for the service
• Gathering money and resources to secure the funding of the project even
more
• Every penny helps! paypal.me/ffmucspenden

9. 9
Thank you very much!
A special thanks to all who helped
writing/translating the FAQs.
Also thank you to all the users and of
course our supporters: