Bugs related to Cloud VPS infrastructure and not a specific Cloud VPS project (see VPS-Projects for that).
Issues which are related to Toolforge should go in Toolforge instead.
Request new projects by filing a task in Cloud-VPS (Project-requests)
Request increased quota for existing projects by filing a task in Cloud-VPS (Quota-requests)
From a quick chat with @taavi :
Let me try to shed some light:
Well, that mystery aside, the actual issue is resolved with
root@apt1002:~# reprepro ls helm helm | 2.17.0-1 | buster-wikimedia | amd64, source helm | 3.17.2-1 | bookworm-wikimedia | amd64 helm | 3.18.6-1 | bookworm-wikimedia | amd64 helm | 3.18.4-1 | trixie-wikimedia | amd64
Change #1191327 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] wmcs: have additional IPs survive reboots
Change #1191326 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] interface: new define for additional IPs
Oh, also: if you want to reimage either of them: everything currently needs to be Bookworm, and the debian installer sometimes gets caught up during the partman phase. I've had good luck connecting to the console, escaping out of the partition phase in the installer and trying again, it seems to always take the second time.
@fgiunchedi, 1050 and 1051 should already be fully puppetized with Ceph 18.x packages and mostly ready to go. The next step to get them in service is the 'wmcs.ceph.osd.bootstrap_and_add' cookbook.
For now I'm leaving codfw1 with osd_memory_target_autotune=true and eqiad1 with osd_memory_target_autotune=false and osd_memory_target=6442450944 -- I'm not convinced there's any change here which would be worth the extra complexity but I am interested in seeing what happens with autotune in the long rune.
So... it doesn't look like ceph will make use of more RAM even if we offer it up. If it did, changing the cluster-wide setting from 6 to 8 would run the risk of OOMing most of the cluster.
Here's another example: Cloudcephosd1016 is using around 29G or RAM according to grafana.
I can confirm the switch is already set to accept tagged traffic for the storage vlan on the ports connecting to both these two hosts (think it was set that way from before).
Change #1191086 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] Cloudcephosd1050: Configure ceph with a single nic
1050 and 1051 won't be pooled immediately, they're being reserved for T405478
root@cloudcephmon1004:~# ceph versions
{
"mon": {
"ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)": 3
},
"mgr": {
"ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)": 3
},
"osd": {
"ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)": 276
},
"rgw": {
"ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)": 3
},
"overall": {
"ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)": 285
}
}Change #1191027 merged by Andrew Bogott:
[operations/puppet@production] Update cloudceph specs to expect reef
Change #1191027 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] Update cloudceph specs to expect reef
Change #1191005 abandoned by Majavah:
[operations/puppet@production] P:wmcs::nfsclient: Allow granular control of mounted volumes
Filippo convinced me that this is probably not that useful.
Change #1191005 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] P:wmcs::nfsclient: Allow granular control of mounted volumes
Ok if we have systemd-networkd everywhere then we should be installing/removing drop-in files for networkd to pick up instead.
This has not happened recently.
Seems fixed?
My understanding is that the reason for this is the puppetization expecting ifupdown (with /etc/network/interfaces) but the Debian cloud images switched to netplan/systemd-networkd some releases ago.
I can confirm this is still the case, Profile::Wmcs::Nfs::Standalone does a manual Exec[ip addr add]
Server completed Reimage by andrew
I am now upgrading the cluster to Bookworm + Reef (18.x) and that seems to bypass this issue.