TechnicalOperations Status Report

Project Operations from 2018-10-01 to 2019-01-01

Help

Network Operations 205888 deploy PFW policy commit 99eb6f026 Screep Done None
Network Operations 189552 Rack/cable/configure ulsfo MX204 In-Scope Done None
Network Operations 210788 faulty VC link on asw2-c-eqiad Screep Done None
Network Operations 206872 cannot add cloudvirt1023 eth1 to cloud-instances2-b-eqiad vlan Screep Done None
Network Operations 211079 IPv6 ~20ms higher ping than IPv4 to gerrit Screep Done None
Network Operations 133387 Enabling IGMP snooping on QFX switches breaks IPv6 (HTCP purges flood across codfw) In-Scope Done None
Network Operations 210612 Remove neodymium/sarin from router ACLs Screep Done None
Network Operations 206704 Enable access from icinga1001 to mgmt interfaces Screep Done None
Network Operations 122406 Consider renumbering Labs to separate address spaces In-Scope Done None
Network Operations 208091 Fix missing PDU's for row C eqiad in netbox Screep Done None
Network Operations 210447 codfw row A recable and add QFX Elaborated Done None
Network Operations 208244 ntp broken in new region Screep Done None
Network Operations 211405 pc2007.codfw.wmnet network blip? Screep Done None
Network Operations 209588 asw2-a-eqiad FPC2 reboot Screep Done None
Network Operations 208726 Access to network devices for Riccardo (volans) Screep Done None
Network Operations 183585 Rack/cable/configure asw2-b-eqiad switch stack In-Scope Done None
Network Operations 207035 relabel switch interfaces formerly saiph.frack.codfw.wmnet to frpig2001.frack.codfw.wmnet Screep Done None
Network Operations 205985 Renumber office-DC interconnect link Screep Done None
Network Operations 174596 dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour In-Scope Done None
Network Operations 210467 codfw row D recable and add QFX Elaborated Done None
Network Operations 207175 add icinga1001 to send_nsca and pfw rules in FRACK Screep Done None
Network Operations 208272 codfw row C recable and add QFX Elaborated Done None
Network Operations 210683 lvs1006 down Screep Done None
Network Operations 209424 Permit routing from eqiad1-r instances to labnet1001 Screep Done None
Network Operations 205829 IPv6 ping to eqiad on ripe-atlas-eqiad IPv6 noisy alert Screep Done None
Network Operations 207428 cr2-esams - BGP WARNING - AS15426/IPv4 Screep Done None
Network Operations 210456 codfw row B recable and add QFX Elaborated Done None
Network Operations 208630 Display remote port name in LLDP output Screep Done None
Network Operations 206778 Configure v6 OOB for ulsfo Screep Done None
Network Operations 211699 Remove static routes for NS v6 IPs Screep Done None
Network Operations 196489 upgrade all codfw switch stacks to include additional 10G switch per row In-Scope Done None
Network Operations 170144 Evaluate NetBox as a Racktables replacement & IPAM In-Scope Done None
Network Operations 206972 asw2-a-eqiad FPC7 faulty PEM0 Screep Done None
Network Operations 206431 Qualys scans causing problematic pfw logspam Screep Done None
Network Operations 206637 icinga reports frbast2001.frack.eqiad.wmnet as host down Screep Done None
Network Operations 86541 setup wifi in codfw In-Scope Open None
Network Operations 187929 Cloud IPv6 subnets In-Scope Open None
Network Operations 136671 Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR In-Scope Open None
Network Operations 184067 Complete router migration from cr1-esams to cr3-esams In-Scope Open None
Network Operations 186021 reconfigure esams switch port for new bastion In-Scope Open None
Network Operations 212273 Spike of multicast traffic Screep Open None
Network Operations 174637 Setup esams atlas anchor In-Scope Open None
Network Operations 204281 Stop prioritizing peering over transit In-Scope Open None
Network Operations 196557 switch port configuration for frmon2001 In-Scope Open None
Network Operations 209145 Investigate network issues in codfw that caused 503 errors Screep Open None
Network Operations 187960 Rack/cable/configure asw2-a-eqiad switch stack In-Scope Open None
Network Operations 208788 Decommission asw-b-eqiad Screep Open None
Network Operations 191667 Juniper HA audit In-Scope Open None
Network Operations 172459 eqiad row D switch upgrade In-Scope Open None
Network Operations 200277 OSPF metrics In-Scope Open None
Network Operations 150264 Icinga check for VRRP In-Scope Open None
Network Operations 189689 Connection timeout from 195.77.175.64/29 to text-lb.esams.wikimedia.org In-Scope Open None
Network Operations 196432 Configure interface damping on primary links In-Scope Open None
Network Operations 205937 Interface errors on cr4-ulsfo:et-0/0/1 Screep Open None
Network Operations 212011 migrate netinsights from rhenium to sulfur Elaborated Open None
Network Operations 167841 Cleanup confed BGP peerings and policies In-Scope Open None
Network Operations 207321 Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster Screep Open 3.0
Network Operations 185151 replace msw1-esams In-Scope Open None
Network Operations 196487 upgrade row d to have 3 10G switches In-Scope Open None
Network Operations 201444 Refresh switch ports descriptions for recently renamed cloud servers In-Scope Open None
Network Operations 106056 set up a looking glass for WMF ASes In-Scope Open None
Network Operations 207753 esams/knams: advertise 185.15.58.0/23 instead of 185.15.56.0/22 Screep Open None
Network Operations 211730 Replace accepted-prefix-limit with prefix-limit Screep Open None
Network Operations 201039 IGMP snooping breaks IPv6 ND on Junos 14.1X53-D46 In-Scope Open None
Network Operations 167691 High amount of unexpected ICMP dest unreachable toward esams cache clusters In-Scope Open None
Network Operations 83992 Juniper monitoring In-Scope Open None
Network Operations 174616 set up cr3-esams In-Scope Open None
Network Operations 171032 Investigate lvs IP pages during codfw row C switch upgrade In-Scope Open None
Network Operations 211254 Free up 185.15.59.0/24 Screep Open None
Network Operations 193496 Allocate public v4 IPs for Neutron setup in eqiad In-Scope Open None
Network Operations 189522 Detect IP address collisions In-Scope Open None
Network Operations 210566 Netbox should use CN rather than UID for LDAP login username Screep Open None
Network Operations 207663 Renumber cloud-instance-transport1-b-eqiad to public IPs Screep Open None
Network Operations 209989 Bird multihop BFD Screep Open None
Network Operations 196946 switch port configuration for lvs200[7-10] In-Scope Open None
Network Operations 190364 eqiad 10G ports needs In-Scope Open None
Network Operations 212348 Move servers off asw2-a5-eqiad Elaborated Open None
Network Operations 207668 Increase network capacity (2018-19 Q2 Goal) Screep Open None
Network Operations 167306 ospf link-protection In-Scope Open None
Network Operations 211728 Outbound BGP graceful shutdown Screep Open None
Network Operations 180179 Evaluate the possibility to add Juniper images to Openstack In-Scope Open None
Network Operations 211930 Add eqsin routing special cases to jnt Screep Open None
Network Operations 167842 Find a new PIM RP IP In-Scope Open None
Network Operations 186550 Anycast recdns In-Scope Open None
Network Operations 82038 create a test for multicast relay In-Scope Open None
Network Operations 208734 Decommission asw-c-eqiad Screep Open None
Network Operations 185337 rack spare switches in c1-eqiad In-Scope Open None
Network Operations 201139 Intermittent connectivity issues in eqiad's row C In-Scope Open None
Network Operations 98006 Anycast (Auth)DNS In-Scope Open None
Network Operations 190090 Offload pings to dedicated server In-Scope Open None
Network Operations 163674 Frequent RST returned by appservers to LVS hosts In-Scope Open None
Network Operations 124843 Peer with SFMIX at ULSFO in 200 Paul In-Scope Open None
Traffic 128358 Uploading 1.2GB ogv results in 503 In-Scope Cut None
Traffic 206394 cp1076 hardware failure Screep Done None
Traffic 170606 Add Accept header to webrequest logs In-Scope Done 3.0
Traffic 178173 Renew unified certificates 2017 In-Scope Done None
Traffic 208390 Allow Let's Encrypt issue wildcard certificates Screep Done None
Traffic 204355 Allow traffic team to manage the traffic blog on phame In-Scope Done None
Traffic 209856 Deploy a certcentral managed TLS certificate for librenms Screep Done None
Traffic 164327 replace ulsfo aging servers In-Scope Done None
Traffic 97051 adding new languages to DNS langs.tmpl doesn't work until zone template is edited as well In-Scope Done None
Traffic 207476 Create production LE accounts Screep Done None
Traffic 207718 Errors trying to fetch RDF from Wikidata Screep Done None
Traffic 210295 ATS path normalization Screep Done None
Traffic 206688 SOA serial numbers returned by authoritative nameservers differ Screep Done None
Traffic 211970 kartotherian TLS support Screep Done None
Traffic 209805 Wikipedia sends WebP thumbnails when Opera claims to support it but lies Screep Done None
Traffic 207050 Migrate most standard public TLS certificates to CertCentral issuance Screep Done None
Traffic 206950 Icinga: check_confd_vcl_reload unknown when file is missing Screep Done None
Traffic 17357 Redirect dk.wiktionary and dk.wikibooks to da.wiktionary and da.wikibooks respectively. Screep Done None
Traffic 207364 Github: add verified domain Screep Done None
Traffic 207069 Mobile DNS entry for vewikimedia is missing Screep Done None
Traffic 131894 Collect Backend-Timing in Prometheus In-Scope Done None
Traffic 206308 Create VMs for certcentral hosts Screep Done None
Traffic 208588 Add ex cp-misc_codfw to text and upload Screep Done None
Traffic 128188 Make CI run Varnish VCL tests In-Scope Done None
Traffic 210890 Loading full versions of larger images from Commons stucks / repeatedly gets interrupted after a few MBs Screep Done None
Traffic 207315 Investigate 200-300ms increase in responseStart.p75 Screep Done None
Traffic 179050 setup bast4002/WMF7218 In-Scope Done None
Traffic 208752 webrequest data loss 2018-11-05 on upload partition Screep Done 5.0
Traffic 208574 varnish systemd service reaching TaskMax Screep Done None
Traffic 182028 DNS repo: add CI checks for obvious configuration errors In-Scope Done None
Traffic 205970 lvs2009/lvs2010 with no RAID configured Elaborated Done None
Traffic 207583 Add punjabi.wikimedia.org to DNS and Apache Screep Done None
Traffic 209703 trafficserver debian-glue builds failing on integration-slave-jessie-1001: No space left on device Screep Done None
Traffic 161148 AuthDNS CM/CI refactor In-Scope Done None
Traffic 206804 Renew GlobalSign Unified in 2018 Screep Done None
Traffic 206923 Remove *.cz domains from WMF's infrastructure Screep Done None
Traffic 206461 Provide a Let's Encrypt ACME v2 staging environment account Screep Done None
Traffic 207140 Add maint-announce@ to Equinix's recipient list for eqsin incidents Screep Done None
Traffic 206861 Power incident in eqsin Screep Done None
Traffic 208583 Reimage eeden to test role Screep Done None
Traffic 212215 Update Subject Alternative Name field in TLS certificates for swift Screep Done None
Traffic 32206 Non-existing wikis should redirect to Incubator Screep Done None
Traffic 207138 Document eqsin power connections in Netbox Screep Done None
Traffic 211860 Fix CAA iodef tags Screep Done None
Traffic 209021 ATS backend-side request-mangling Screep Done None
Traffic 163541 cache hosts should auto-repool iff OCSP files are sane In-Scope Done None
Traffic 207763 Traffic project in labs cannot talk HTTP with deployment-prep any longer Screep Done None
Traffic 137747 Parametrization of VCL is inconsistent In-Scope Open None
Traffic 207048 ATS production-ready as a backend cache layer Screep Open None
Traffic 184942 Deprecate python varnish cachestats In-Scope Open None
Traffic 81305 Make PyBal respect advertised BGP capabilities In-Scope Open None
Traffic 185239 Puppet hosts with signed certificate present on agent but not master In-Scope Open None
Traffic 118181 Planning for phasing out non-Forward-Secret TLS ciphers In-Scope Open None
Traffic 167400 Disable serving unpatrolled new files to Wikipedia Zero users In-Scope Open None
Traffic 91820 Create HTTP verb and sticky cookie DC routing in VCL In-Scope Open None
Traffic 102178 Fix RESTBase support for wikitech.wikimedia.org In-Scope Open None
Traffic 211697 clean up deprecated TLS certificates from the puppet repo Screep Open None
Traffic 204997 certcentral: delay deployment of renewed certs to wait out skewed client clocks In-Scope Open None
Traffic 117826 TEST: redirect small portion of unauthenticated desktop users to mobile web In-Scope Open None
Traffic 134447 letsencrypt puppetization: upgrade for scalability In-Scope Open None
Traffic 159412 Convert all of our site.pp/roles to the role/profile paradigm In-Scope Open None
Traffic 161517 Allow anonymous users to change interface language on Commons with ULS In-Scope Open None
Traffic 208584 Decommission old eqiad caches Screep Open None
Traffic 127573 wikiknihy.cz - transfer to Wikimedia Czech Republic? In-Scope Open None
Traffic 147209 etcd cluster has Raft Internal errors sporadically In-Scope Open None
Traffic 123854 Set up action API latency / error rate metrics & alerts In-Scope Open None
Traffic 204993 Update certspotter In-Scope Open None
Traffic 148134 OCSP Stapling for Intermediates In-Scope Open None
Traffic 164259 Add VSL error counters to Varnishkafka stats In-Scope Open None
Traffic 150673 Thumb API: Varnish / CDN questions In-Scope Open None
Traffic 127482 Enable VCL source-DC switching via confd In-Scope Open None
Traffic 144508 Point wikipedia.in to 205.147.101.160 instead of URL forward In-Scope Open None
Traffic 178815 decom cp40(09|1[078]) In-Scope Open None
Traffic 119372 Pybal IdleConnectionMonitor with TCP KeepAlive shows random fails if more than 100 servers are involved. In-Scope Open None
Traffic 120121 Improve Varnish XFF processing for trusted proxies In-Scope Open None
Traffic 126281 [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") In-Scope Open None
Traffic 203396 certcentral: challenge checking on *all* pooled backend hosts In-Scope Open None
Traffic 198620 Consider using vmod_var instead of temporary headers in VCL In-Scope Open None
Traffic 125170 Internal DNS resolver responds with NXDOMAIN for localhost AAAA In-Scope Open None
Traffic 109776 Tilerator should purge Varnish cache In-Scope Open None
Traffic 138093 Investigate query parameter normalization for MW/services In-Scope Open None
Traffic 120631 Security: Is it safe to enable Zero spoofing In-Scope Open None
Traffic 164768 Explicitly limit varnishd transient storage In-Scope Open None
Traffic 202046 cp3032 PS Redundancy Lost In-Scope Open None
Traffic 207340 Determine cause of upload.wikimedia.org requests routed to text-lb (404 Not Found) Screep Open None
Traffic 102367 Migrate tools.wmflabs.org to https only (and set HSTS) In-Scope Open None
Traffic 192082 lvs2006 Embedded Flash/SD-CARD iLO errors In-Scope Open None
Traffic 158599 Samsung Internet's desktop mode getting redirected to mobile site In-Scope Open None
Traffic 45250 Redo /beacon/impression system (formerly Special:RecordImpression) to remove extra round trips on all FR impressions (title was: S:RI should pyroperish) In-Scope Open None
Traffic 146619 DNS domains registered to WMF no longer redirecting In-Scope Open None
Traffic 158604 Investigate usefulness of SameSite cookies for logged-in accounts In-Scope Open None
Traffic 204987 Consider adding Must-Staple header to enforce revocation checking In-Scope Open None
Traffic 209337 lvs2006 crashed into (what it seems) an unrecoverable state Screep Open None
Traffic 208263 Refactor public-facing DYNA scheme for primary project hostnames in our DNS Screep Open None
Traffic 138546 Backend naming in VCL needs to use fqdn+port In-Scope Open None
Traffic 179025 LVS hosts should have static-mapped IPv6 on all virtual interfaces In-Scope Open None
Traffic 190607 cp3048 hardware issues In-Scope Open None
Traffic 200673 varnish-http-requests false positives when a DC is depooled In-Scope Open None
Traffic 99531 [Task] move wikiba.se webhosting to wikimedia cluster In-Scope Open None
Traffic 210484 Only serve debug HTTP headers when x-wikimedia-debug is present Screep Open None
Traffic 105657 Expires header for load.php should be relative to request time instead of cache time In-Scope Open None
Traffic 180655 Phabricator and Gerrit: Improve the way that maintenance downtime is communicated to users. In-Scope Open None
Traffic 192437 Pybal support of configuration from the kubernetes API In-Scope Open None
Traffic 117435 Spike: CentralNotice: Verify that our Special:HideBanners cookie storm works as efficiently as possible In-Scope Open 2.0
Traffic 206339 Separate Traffic layer caches for PHP7/HHVM Screep Open None
Traffic 96499 dbtree loads third party resources (from jquery.com and google.com) In-Scope Open None
Traffic 152622 Wikipedia.cz and other domains owned by WMCZ have invalid certificate In-Scope Open None
Traffic 190993 Upgrade pybal-test instances to stretch In-Scope Open None
Traffic 134807 Replace test hostnames in datecenter-specific subdomains with dashed names In-Scope Open None
Traffic 153468 Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs In-Scope Open None
Traffic 146332 Create short link for outreachdashboard.wmflabs.org In-Scope Open None
Traffic 147967 The WMF-Last-Access Set-Cookie header should follow RFC 2965 syntax rather than the pre-RFC Netscape format In-Scope Open None
Traffic 211131 DNS recursors TCP retransmits Screep Open None
Traffic 101002 Use Upgrade Insecure Requests on Wikimedia wikis In-Scope Open None
Traffic 132629 Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense In-Scope Open None
Traffic 23027 Requests with utf-8 in the URL return a outdated page revision In-Scope Open None
Traffic 210134 wikidata.org lacks SPF record Elaborated Open None
Traffic 36670 Check all wikis for inclusions of http resources on https In-Scope Open None
Traffic 191183 Enable avatars in gerrit In-Scope Open None
Traffic 193521 Consider adding expect-CT: header to enforce certificate transparency In-Scope Open None
Traffic 212310 varnishreqstats sends truncated statsd traffic Screep Open None
Traffic 174342 Missing IP addresses for Maroc Telecom In-Scope Open None
Traffic 165560 Artificial spike in offset of unique devices from November to February 6th on wikidata In-Scope Open None
Traffic 179197 Investigate what caused the the unattended varnish upgrade in Beta Cluster In-Scope Open None
Traffic 211661 Automatically clean up unused thumbnails in Swift Screep Open None
Traffic 138591 Backport iproute2 4.x from debian testing -> our jessie In-Scope Open None
Traffic 172124 PyBal Feature: progressive depooling strategy for monitored failures In-Scope Open None
Traffic 66214 Define an official thumb API In-Scope Open None
Traffic 177742 Investigate Chrony as a replacement for ISC ntpd In-Scope Open None
Traffic 88861 wikipedia.lol In-Scope Open None
Traffic 170567 Support TLSv1.3 In-Scope Open None
Traffic 204931 Re-evaluate use of EV certificates for payments.wm.o? In-Scope Open None
Traffic 89838 Move proxy IP lists to META for Varnish XFF decoding In-Scope Open None
Traffic 104681 HTTPS Plans (tracking / high-level info) In-Scope Open None
Traffic 201666 cp3040: kernel crash in ipsec code shortly after reboot In-Scope Open None
Traffic 167513 Redirect lzh.wikipedia to zh-classical.wikipedia In-Scope Open None
Traffic 137979 Support brotli compression In-Scope Open None
Traffic 177927 Refactor kafka_config.rb and and kafka_cluster_name.rb in puppet to avoid explicit hiera calls In-Scope Open None
Traffic 128374 Sort out analytics service dependency issues for cp* cache hosts In-Scope Open None
Traffic 148976 Strongswan Icinga check: do not report issues about depooled hosts In-Scope Open None
Traffic 198286 Decommission acamar and achernar In-Scope Open None
Traffic 205988 Simplify comment misc-frontend.inc.vcl.erb Screep Open None
Traffic 206951 Puppet doesn't restart ferm on failure Screep Open None
Traffic 131930 Set SPF (... -all) for toolserver.org In-Scope Open None
Traffic 128409 Detect tools.wmflabs.org tools which are HTTP-only In-Scope Open None
Traffic 192280 sda failure in hydrogen.wikimedia.org In-Scope Open None
Traffic 143562 High number of failed inbound TFO connections in esams Mon-Fri In-Scope Open None
Traffic 188561 SSL cert for links.email.wikimedia.org In-Scope Open None
Traffic 171850 Backport ipvsadm In-Scope Open None
Traffic 149873 CentralNotice: Review and update Varnish caching for Special:BannerLoader In-Scope Open 2.0
Traffic 209785 INMARSAT geolocates to the UK, leading to requests going to esams Screep Open None
Traffic 184715 pybal's "can-depool" logic only takes downServers into account In-Scope Open None
Traffic 210411 Applayer services without TLS Screep Open None
Traffic 79730 Add pybal check to ensure service IP is bound In-Scope Open None
Traffic 159411 Uniform cluster nomenclature across puppet In-Scope Open None
Traffic 117618 Add restrictive CSP to upload.wikimedia.org In-Scope Open None
Traffic 133821 Content purges are unreliable In-Scope Open None
Traffic 204013 Horizon Designate dashboard not allowing creation of NS records In-Scope Open None
Traffic 202564 https://sv.wikipedia.beta.wmflabs.org/ has invalid certificate In-Scope Open None
Traffic 202479 Investigate source of 404 Not Found responses from load.php In-Scope Open None
Traffic 128559 store.wikimedia.org HTTPS issues In-Scope Open None
Traffic 194031 Setup a new PKI software as an alternative to the puppet CA for managing services certificates In-Scope Open None
Traffic 106517 upload.wikimedia.org returns HTTP status code 503 for truncated urls, not 404 In-Scope Open None
Traffic 181368 Log source port for anonymous users and expose it for sysops/checkusers In-Scope Open None
Traffic 191393 Puppet: tlsproxy localssl default_server make a Notify at each run In-Scope Open None
Traffic 133548 Create a secure redirect service for large count of non-canonical / junk domains In-Scope Open None
Traffic 194724 Deprecate `base::service_unit` in puppet In-Scope Open None
Traffic 133001 Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames In-Scope Open 0.0
Traffic 193445 Update Media dashboard in Grafana to use Prometheus metrics In-Scope Open None
Traffic 205344 Inconsistent lists of labs-ns* nameservers In-Scope Open None
Traffic 119038 Image cache issue when 'over-writing' an image on commons In-Scope Open None
Traffic 191017 Unwanted service startups and their triggers In-Scope Open None
Traffic 185350 Vet reliability of the response_size field for data analysis purposes In-Scope Open None
Traffic 184293 rack/setup/install lvs101[3-6] In-Scope Open None
Traffic 202966 Make cp1099 the new pinkunicorn In-Scope Open None
Traffic 78963 Support ESI for ResourceLoader In-Scope Open None
Traffic 180712 VCL: handling of uncacheable responses in wikimedia-common In-Scope Open None
Traffic 82849 lvs servers report 'Memory allocation problem' on bootup In-Scope Open None
Traffic 109325 Outbound HTTPS for varnish backend instances In-Scope Open None
Traffic 108580 HTTPS for internal service traffic In-Scope Open None
Traffic 188087 Some etcd connections not established at startup In-Scope Open None
Traffic 208282 Increase EventLogging limit from 2K to 5K Screep Open None
Traffic 154801 Investigate varnishd child crashes when multiple nodes get depooled/pooled concurrently In-Scope Open None
Traffic 171498 Implement machine-local forwarding DNS caches In-Scope Open None
Traffic 50133 ForeignAPIRepo wrongly returns non-protocol-relative URLs for original "thumbs" In-Scope Open None
Traffic 167972 Respect host header in RESTBase, and redirect /rest_v1 to /rest_v1/ In-Scope Open None
Traffic 119366 Disable caching on the main page for anonymous users In-Scope Open None
Traffic 175636 prometheus -> grafana stats for per-numa-node meminfo In-Scope Open None
Traffic 141266 letsencrypt puppetization: add parallel rsa+ecdsa cert support In-Scope Open None
Traffic 204056 Move wikimedia.ee under WM-EE In-Scope Open None
Traffic 114104 pybal doesn't fully manage LVS table leaving stale services (on IP change) In-Scope Open None
Traffic 205378 Enable ESNI support on Wikimedia servers In-Scope Open None
Traffic 155314 Varnish does not cache Action API responses when logged in In-Scope Open None
Traffic 201409 Harmonise the identification of requests across our stack In-Scope Open None
Traffic 133895 Varnish configuration for mobile domains should be coherent with Apache configuration In-Scope Open None
Traffic 209707 tagged_interface sometimes exceeds IFNAMSIZ Screep Open None
Traffic 56783 Respect X-Forwarded-For only from trustworthy sources In-Scope Open None
Traffic 192559 Establish timeline and methodology for upcoming deprecation of non-forward-secret ciphers and TLSv1.0 In-Scope Open None
Traffic 202627 cp3036 PS Redundancy Lost In-Scope Open None
Traffic 91372 $wgMFAnonymousEditing = true is sometimes not respected: cache? In-Scope Open None
Traffic 189290 Tune systemd journal rate limiting for PyBal In-Scope Open None
Traffic 192206 Remove wildcard vhost for *.wikimedia.org In-Scope Open None
Traffic 147648 Unexplained increase in thumbnail 500s In-Scope Open None
Traffic 133178 RESTBase support for www.wikimedia.org missing In-Scope Open None
Traffic 183554 Unified certs bloat reduction? In-Scope Open None
Traffic 163141 dbtree: make wasat a working backend and become active-active In-Scope Open None
Traffic 199677 cp3033 unreacheable since 2018-07-15 11:47:31 In-Scope Open None
Traffic 174960 Varnish does not vary elasticsearch query by request body In-Scope Open None
Traffic 120509 Cache education dashboard pages In-Scope Open None
Traffic 104442 Investigate better DNS cache/lookup solutions In-Scope Open None
Traffic 83467 LVS testing needs to include internal services testing In-Scope Open None
Traffic 162818 icinga alerts on nodejs services when a recdns server is depooled In-Scope Open None
Traffic 212312 prometheus-based graph significantly slower than statsd equivalent Elaborated Open None
Traffic 102848 Split GeoIP into a new component In-Scope Open None
Traffic 136703 Add LVS public endpoint checks that bypass caches In-Scope Open None
Traffic 203423 certcentral: Provide script for certificate revocation In-Scope Open None
Traffic 120486 add a https-only option to dynamicproxy In-Scope Open None
Traffic 196560 rack/setup/install LVS200[7-10] In-Scope Open None
Traffic 202040 Decommission radon In-Scope Open None
Traffic 129839 restrict upload cache access for private wikis In-Scope Open None
Traffic 184534 Cached page previews not shown when refreshed In-Scope Open None
Traffic 134323 confctl: give regexen more freedom In-Scope Open None
Traffic 150479 Prometheus varnish metric churn due to VCL reloads In-Scope Open None
Traffic 207008 Create redirect to integration.wikimedia.org/zuul Screep Open None
Traffic 190992 prometheus: slow dashboards due to suboptimal query_range performance In-Scope Open None
Traffic 78421 m.{project}.org portal/redirect consistency In-Scope Open None
Traffic 137252 Redirect phabricator.mediawiki.org to phabricator.wikimedia.org In-Scope Open None
Traffic 196066 Add prometheus metrics for varnishkafka instances running on caching hosts In-Scope Open None
Traffic 122867 Evaluate the feasibility of cache invalidation for the action API In-Scope Open None
Traffic 199247 Decommission baham In-Scope Open None
Traffic 176388 pybal: race condition in alerts instrumentation In-Scope Open None
Traffic 175319 cp1066 unexplained 503 spikes In-Scope Open None
Traffic 152882 Many misc wikis lack mobile domains In-Scope Open None
Traffic 74186 Varnish: Mobile site redirect interferes with OAuth authorization process In-Scope Open None
Traffic 118468 point wikilovesmonuments.org ns to wmf In-Scope Open None
Traffic 208586 Decommission lvs1007-1012 Screep Open None
Traffic 179027 Puppetize LVS interface IP sets per-DC for easy use in ferm rules In-Scope Open None
Traffic 165765 Refactor pybal/LVS config for shared failover In-Scope Open None
Traffic 173966 Like nan.wikipedia.org, redirect other nan.*.org to the proper zh-min-nan.*.org domains In-Scope Open None
Traffic 180434 Uncacheable content handling: hfp vs hfm In-Scope Open None
Traffic 174932 Recurrent 'mailbox lag' critical alerts and 500s In-Scope Open None
Traffic 165764 Fully-redundant LVS clusters using Pybal per-service MED feature In-Scope Open None
Traffic 144187 Better handling for one-hit-wonder objects In-Scope Open None
Traffic 140365 Lower geodns TTLs from 600 (10min) to 300 (5min) In-Scope Open None
Traffic 141480 mixed-content issues on planet.wikimedia.org In-Scope Open None
Traffic 54253 Protocol-relative URLs are poorly supported or unsupported by a number of HTTP clients In-Scope Open None
Traffic 101525 Set up LVS for current AuthDNS In-Scope Open None
Traffic 164456 Migrate to nginx-light In-Scope Open None
Traffic 190244 en-wp.org certificate error In-Scope Open None
Traffic 198152 Size of headers processed by varnish? In-Scope Open None
Traffic 164460 Use DNS discovery record for deployment CNAME In-Scope Open None
Traffic 109331 Deleted files sometimes remain visible to non-privileged users if permanently linked In-Scope Open None
Traffic 130904 Host rewrite for /static/ not applied to purges In-Scope Open None
Traffic 175203 Implement stateless TCP balancing in our LVS servers In-Scope Open None
Traffic 170605 Unable to render file from upload.wikimedia.org "Error 349 ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION" In-Scope Open None
Traffic 180921 Referrer policy for browsers which only support the old spec In-Scope Open None
Traffic 204994 Integrate certspotter with certcentral to avoid certspotter notifying us on legitimate certs generated by our certcentral boxes In-Scope Open None
Traffic 208585 Decommission esams cache_misc hosts Screep Open None
Traffic 149847 RFC: Use content hash based image / thumb URLs In-Scope Open None
Traffic 150022 thumb.php should not set CC:no-cache on renderer 404 responses? In-Scope Open None
Traffic 181315 Varnish HTTP response from app servers taking 160s (only 0.031s inside Apache) In-Scope Open None
Traffic 146832 Clarify caching to enable direct Wikidata Query Service access by <mapframe/link> In-Scope Open None
Traffic 180257 Puppet / LVS: confusion in service vs IP name In-Scope Open None
Traffic 134324 confctl select needs a -y flag? In-Scope Open None
Traffic 204992 Puppetise OCSP stapling for all one-off HTTPS servers In-Scope Open None
Traffic 196248 TLS certificates renewal process In-Scope Open None
Traffic 152091 Block hotlinking In-Scope Open None
Traffic 169765 pybal should automatically reconnect to etcd In-Scope Open None
Traffic 113817 Connect Hadoop records of the same request coming via different channels In-Scope Open None
Traffic 127387 Split slash decoding from general percent normalization in Varnish VCL In-Scope Open None
Traffic 179026 LVS IPv6 IPs should all be recorded in DNS In-Scope Open None
Traffic 86915 nan and minnan subdomain redirects are a mess In-Scope Open None
Traffic 107236 Switch port 80 to nginx on primary clusters In-Scope Open None
Traffic 147162 upload.wikimedia.org returns HTTP 501 instead of 416 for non-satisfiable byte ranges In-Scope Open None
Traffic 94125 Central login notice appears on unencrypted API format=*fm pages, where reloading does not affect login status In-Scope Open None
Traffic 176875 Allow access to wdqs.svc.eqiad.wmnet on port 8888 In-Scope Open None
Traffic 156462 Framework to transfer files over the LAN In-Scope Open None
Traffic 188804 Investigate and fix odd uri_host values In-Scope Open None
Traffic 99216 Please set up a CNAME for videoserver.wikimedia.org to Video Editing Server In-Scope Open None
Traffic 136944 Set up LVS connection sync In-Scope Open None
Traffic 111588 RFC: API-driven web front-end In-Scope Open None
Traffic 186732 Decide on Cache-Control headers for map tiles In-Scope Open None
Traffic 180269 Wikimedia's recent upgrade to nginx v. 1.13.6 breaks older Android HTTP libraries In-Scope Open None
Traffic 161256 multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert In-Scope Open None
Traffic 206876 certcentral: check for SCTs, with optional disable per-account Screep Open None
Traffic 171470 Monitor DNS delegations In-Scope Open None
Traffic 208242 Investigate using RFC 7838 Alternate Services to better optimize edge connections Screep Open None
Traffic 172103 IPVS issues with UDP services, pybal depooling strategy In-Scope Open None
Traffic 98165 Figure out an etcd deploy strategy that includes multi DC failure scenarios. In-Scope Open None
Traffic 167906 Make API usage limits easier to understand, implement, and more adaptive to varying request costs / concurrency limiting In-Scope Open None
Traffic 161360 404 loading images from Virgin Media In-Scope Open None
Traffic 210167 Disable WMF-Last-Access cookies for wmfusercontent.org Screep Open None
Traffic 129682 Look into solutions for replaying traffic to testing environment(s) In-Scope Open None
Traffic 190843 Spammy events coming our way for sites such us https://ru.wikipedia.kim In-Scope Open None
Traffic 96844 Update TLS/HTTP documentation on wikitech In-Scope Open None
Traffic 200806 cp3031: Power required by the system exceeds the power supplied by the Power Supply Units In-Scope Open None
Traffic 192368 Unconditional return(deliver) in vcl_hit In-Scope Open None
Traffic 194814 Reduce amount of headers sent from web responses In-Scope Open None
Traffic 112316 Configure varnish to use "Unconfigured domain" page for 404 Not Served (instead of generic error) In-Scope Open None
Traffic 174432 Unclear LVS bandwidth graph in "load balancers" dashboard In-Scope Open None
Traffic 209515 Renew Digicert Unified in 2019 Screep Open None
Traffic 203191 prometheus-varnish-exporter@frontend.service: Unit entered failed state - invalid character 'C' In-Scope Open None
Traffic 125226 [feature request] Redirect root API path to docs page In-Scope Open None
Traffic 177961 Upgrade LVS servers to stretch In-Scope Open None
DBA 206313 Degraded RAID on db1072 Screep Done None
DBA 211537 Degraded RAID on db1063 Screep Done None
DBA 204928 Issues with mgmt interface on es2001 host In-Scope Done None
DBA 209858 Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) Screep Done None
DBA 205872 db2058: Disk #1 predictive failure Screep Done None
DBA 206191 rack/setup/install db2096 (x1 codfw expansion host) Screep Done None
DBA 212277 Upgrade db2057 firmware Screep Done None
DBA 210049 Degraded RAID on db2044 Screep Done None
DBA 208150 db1117 went away Screep Done None
DBA 206245 db1064 has disk smart error Screep Done None
DBA 208526 Database timeout error + significant lag when modifying a Partial Block with 10 items Screep Done None
DBA 207212 Degraded RAID on db2051 Screep Done None
DBA 208462 Error Unknown column ipb_sitewide in field list on query Screep Done None
DBA 202051 db2042 (m3) master RAID battery failed In-Scope Done None
DBA 206500 Degraded RAID on db1067 Screep Done None
DBA 205865 Investigate decrease in wikidata dispatch times due to eqiad -> codfw DC switch Screep Done None
DBA 206740 parsercache used disk space increase Screep Done None
DBA 209754 db1078 (s3 candidate master) crashed Screep Done None
DBA 126252 Populate the wikishared db on all dbstores In-Scope Done None
DBA 212185 Degraded RAID on db1072 Screep Done None
DBA 207259 rack/setup/install pc2007-pc2010 Screep Done None
DBA 206841 Evaluate the consequences of the parsercache being empty post-switchover Screep Done None
DBA 206254 Degraded RAID on db1073 Screep Done None
DBA 165677 Create a backend check for pybal to monitor the MySQL protocol being up In-Scope Open None
DBA 141968 Display lag on grafana (prometheus) and dbtree from pt-heartbeat instead (or in addition) of Seconds_Behind_Master In-Scope Open None
DBA 177779 Generate instance list of database hosts to be monitored automatically from exported resources In-Scope Open None
DBA 143896 MySQL metrics monitoring In-Scope Open None
DBA 133523 [RFC] improve parsercache replication and sharding handling In-Scope Open None
DBA 152427 Create a check/calendar alert for MariaDB TLS certs In-Scope Open None
DBA 162070 Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases In-Scope Open None
DBA 200297 Introduce a new namespace for collaborative judgements about wiki entities In-Scope Open None
DBA 100501 mysql user and group should be a system user/group In-Scope Open None
DBA 211613 rack/setup/install db11[26-38].eqiad.wmnet Screep Open None
DBA 176532 Gerrit is failing to connect to db on gerrit2001 thus preventing systemd from working In-Scope Open None
DBA 161754 eqiad: (2) hardware access request for labsdb1004 & 5 refresh In-Scope Open None
DBA 212625 Prepare and check storage layer for hywwiki Screep Open None
DBA 161755 eqiad: (2) hardware access request for labsdb1006 & 7 refresh In-Scope Open None
DBA 210992 Increase parsercache keys TTL from 22 days back to 30 days Screep Open None
DBA 127570 Rename be_x_oldwiki database to be_taraskwiki In-Scope Open None
DBA 208323 Predictive failures on disk S.M.A.R.T. status Screep Open None
DBA 134809 Apache <=> mariadb SSL/TLS for cross-datacenter writes In-Scope Open None
DBA 196378 Investigate solutions for MySQL connection pooling In-Scope Open None
DBA 112473 Better mysql monitoring for number of connections and processlist strange patterns In-Scope Open None
DBA 207258 rack/setup/install pc1007-pc1010 Screep Open None
DBA 141547 Setup automatic failover for misc database servers In-Scope Open None
DBA 210969 Decommission parsercache hosts: pc1004.eqiad.wmnet pc1005.eqiad.wmnet pc1006.eqiad.wmnet Screep Open None
DBA 160731 Decom db1048 (BBU Faulty - slave lagging) In-Scope Open None
DBA 196055 Remove table `math` from the database In-Scope Open None
DBA 164834 In some database hosts, performance schema loses digest statistics In-Scope Open None
DBA 119626 Eliminate SPOF at the main database infrastructure In-Scope Open None
DBA 204026 DBPerformance warning "Query returned 22186 rows: query: SELECT * FROM `translate_metadata`" on Meta-Wiki In-Scope Open None
DBA 166108 x1 master db1031: Faulty BBU In-Scope Open None
DBA 145072 Create a script to regenerate prometheus mysqld exporter listing that works with puppetdb In-Scope Open None
DBA 107610 Setup separate logical External Store for Flow in production In-Scope Open None
DBA 157702 Followup for TLS MariaDB server roll-out In-Scope Open None
DBA 175672 Make apache/maintenance hosts TLS connections to mariadb work In-Scope Open None
DBA 193224 Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished In-Scope Open None
DBA 185084 Allow use of EtcdConfig to configure slave databases In-Scope Open None
DBA 119154 Move echo tables from local wiki databases onto extension1 cluster for mediawikiwiki, metawiki, and officewiki In-Scope Open None
DBA 208383 Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] Screep Open None
DBA 197531 Data model for dbconfig In-Scope Open None
DBA 209815 Upgrade firmware on db1078 Screep Open None
DBA 109179 Migrate MySQLs to use ROW-based replication In-Scope Open None
DBA 104699 Firewall configurations for database hosts In-Scope Open None
DBA 83609 script & docs to rename wiki databases In-Scope Open None
DBA 112282 Multiple pages with no revisions In-Scope Open None
DBA 197126 Create tool to handle the state of database configuration in MediaWiki in etcd In-Scope Open None
Software Development 205868 Expand Netbox usage - Q2 2018-19 Goal Screep Done None
Software Development 209136 python3-etcd needs python3-dnspython Screep Done None
Software Development 148494 Add shell scripts CI validations In-Scope Open None
Software Development 203944 Create a spicerack cookbook for restoring an etcd cluster from backups In-Scope Open None
Software Development 201317 wmf-auto-reimage: 'execution expired' on first puppet run In-Scope Open None
Software Development 205899 Develop and deploy at least three Netbox reports to assist with data correctness and consistency Screep Open None
Software Development 157001 Puppet compiler: abort on git rebase conflict In-Scope Open None
Software Development 207845 debdeploy: show help message if invoked with no arguments Screep Open None
Software Development 198850 debmonitor: Race condition between package updated triggered by apt hook and daily cron run In-Scope Open None
Software Development 211750 Introduce Python code formatters usage Screep Open None
Software Development 177385 Upgrade Cumin masters to stretch In-Scope Open None
Software Development 203943 Convert automation scripts to spicerack cookbooks In-Scope Open None
Software Development 167504 New tool to track package updates/status for hosts and images (debmonitor) In-Scope Open None
Software Development 206448 Decommission script race condition Screep Open None
Software Development 157002 Puppet compiler: re-add the concurrency option NUM_THREADS In-Scope Open None
Software Development 144169 Flake8 for python files without extension in puppet repo In-Scope Open None
Software Development 204789 wmf-auto-reimage tries to remove from Debmonitor even with --new In-Scope Open None
Software Development 150560 More verbose messages from service-checker-swagger In-Scope Open None
Software Development 164587 cumin could use randomization/splay options In-Scope Open None
Software Development 152950 E901 SyntaxError: invalid syntax is wrongly raised on using python's abc by jenkins python CI linter In-Scope Open None
Software Development 155705 confctl: log to SAL even if the selection doesn't match any host In-Scope Open None
Software Development 205867 Expand Spicerack library and SRE Cookbooks - Q2 2018-19 Goal Screep Open None
Software Development 157133 Consider adding a --skip-conftool option to puppet-merge In-Scope Open None
Software Development 203964 Create a spicerack cookbook to empty a ganeti node from VMs In-Scope Open None
Software Development 212395 cergen CI fails to run on Debian Stretch because cryptography dependency cannot be built against newer openssl version Screep Open None
Software Development 201669 wmf-auto-reimage should retry on ipmi failures In-Scope Open None
Software Development 205900 Cumin: add backend for Netbox Screep Open None
Software Development 184435 Puppet tox: properly lint both Py2 and Py3 files In-Scope Open None
Software Development 203963 Convert makevm to spicerack cookbook In-Scope Open None
Software Development 203948 Covert deploy_apache_change.sh to a spicerack cookbook In-Scope Open None
Software Development 199911 Systemd session creation fails under I/O load In-Scope Open None
Software Development 154776 Puppet compiler: order resources for easy comparison between hosts In-Scope Open None
Software Development 159045 Update Puppet repo code that uses deprecated maniphest.update/.createtask/.query Conduit API In-Scope Open None
Hardware Requests 206017 Hardware for session storage service Screep Done None
Hardware Requests 181264 Refresh or replace oxygen In-Scope Done None
Hardware Requests 199673 eqiad | (14 + 6) hadoop hardware refresh and expansion In-Scope Done None
Hardware Requests 204589 eqiad: (1) misc single cpu server allocation for performance browser testing In-Scope Open None
Other Operations 199234 Find a better way to notify tool maintainers of schema and API changes In-Scope Cut None
Other Operations 211235 Degraded RAID on cloudvirtan1001 Screep Done None
Other Operations 184236 Puppet broken on deployment-ms-be0[34] with evaluation error in swift module In-Scope Done None
Other Operations 208432 Requesting access to Jupyter notebook / analytics-privatedata-users for jgleeson Screep Done None
Other Operations 207283 New list request for 1lib1ref Screep Done None
Other Operations 208422 Migrate operations/puppet CI job from Jessie to Stretch Screep Done None
Other Operations 195750 Mailman issues a "403 Forbidden" error when subscribing to a list In-Scope Done None
Other Operations 207643 Sanitizing input and increase throttling rate for wdqs errors to prevent spamming logstash Screep Done None
Other Operations 207009 Onboarding Cas Rusnov Screep Done None
Other Operations 209620 rack/setup/install dbstore100[3-5].eqiad.wmnet Screep Done None
Other Operations 164123 tools-k8s-master-01 has two floating IPs In-Scope Done None
Other Operations 211014 Requesting access to deployment for Christoph Jauera (WMDE-Fisch) Screep Done None
Other Operations 209408 Degraded RAID on cloudelastic1003 Screep Done None
Other Operations 209298 access to analytics-privatedata-users for @toddleroux, @Afandian, & @RyanSteinberg Screep Done None
Other Operations 211148 QIDs work locally but not in production with new translation-server Screep Done None
Other Operations 211353 Switch PHP-FPM on phab1002 Screep Done None
Other Operations 206105 Optimize networking configuration for WDQS Screep Done None
Other Operations 208945 Relabel labvirt1017.eqiad.wmnet as cloudvirt1017.eqiad.wmnet Screep Done None
Other Operations 210290 make wdqs-updater heap size configurable from puppet Screep Done None
Other Operations 211114 Some regressions in production with Zotero translation-server in production Elaborated Done None
Other Operations 206315 Decommission bohrium Screep Done 3.0
Other Operations 206303 Add sudo rules for wdqs-updater in puppet Screep Done None
Other Operations 211115 Requesting access to `researchers` group for joewalsh Screep Done None
Other Operations 207983 BUG: Bad page map in process hhvm Screep Done None
Other Operations 208395 Cleanup wdqs puppet profile to include the new changes based on refactoring Screep Done None
Other Operations 207764 apply hostname label for weblog1001/WMF4750 Screep Done None
Other Operations 201343 rack/setup/install mwmaint1002.eqiad.wmnet In-Scope Done None
Other Operations 208652 Create oversight-ko mailing list Screep Done None
Other Operations 211127 The new translation-server returns access date with the full time stamp; we should strip this Screep Done None
Other Operations 211094 Port DirectorySize diamond collector to a Prometheus exporter Elaborated Done None
Other Operations 153416 docker-engine pulled into our repositories only keeps the latest version In-Scope Done None
Other Operations 206766 Update Debian package of Blubber (0.6.0-1) Screep Done None
Other Operations 207793 Remove "aude" from "wmde" LDAP group Screep Done None
Other Operations 209066 Decommission asw-c8-codfw Screep Done None
Other Operations 212402 Broken power supply on elastic2026 Screep Done None
Other Operations 209569 Make nodes.bin cache file writable by osmupdater after it is created by osmimporter Screep Done None
Other Operations 210878 Degraded RAID on restbase2014 Screep Done None
Other Operations 209691 Upgrade to OTRS version 5.0.32 Screep Done None
Other Operations 211416 Put restbase201[3-8] into conftool and LVS Screep Done None
Other Operations 211184 Correctly collect logs from php-fpm pools Elaborated Done None
Other Operations 186748 [EPIC] New service request: chromium-render/deploy In-Scope Done 5.0
Other Operations 211654 puppet-provisioned dashboards not found in Grafana 5 Screep Done None
Other Operations 207930 Moving or deleting a translatable page on mediawiki.org triggers an error message Screep Done None
Other Operations 210846 Ship Grafana server logs to ELK Screep Done None
Other Operations 210036 upgrade people.wikimedia.org to stretch (replace rutherfordium with people1001) Screep Done None
Other Operations 208824 rename tegmen to icinga2001 and reinstall it with stretch Screep Done None
Other Operations 210027 access request for Jeena Huneidi (deployment, conint-admins, contint-docker) Screep Done None
Other Operations 207470 Additional ssh key for Antoine "hashar" Musso Screep Done None
Other Operations 210380 Icinga downtime script should fail on the passive hosts Screep Done None
Other Operations 208838 Degraded RAID on db2049 Screep Done None
Other Operations 212624 wtp1028 unresponsive Screep Done None
Other Operations 206579 Releases Jenkins Icinga check failing after restricting access Screep Done None
Other Operations 205882 Add Mathew.onipe(onimisionipe) to procurement group Screep Done None
Other Operations 209622 Update label and switch to rename labvirt1015 to cloudvirt1015 Screep Done None
Other Operations 82614 Neon sdb is failing Screep Done None
Other Operations 209395 rack/setup/install new ms-be servers ms-be204[4-9] ,ms-be2050 Screep Done None
Other Operations 18215 Incubator mailinglist. Screep Done None
Other Operations 211708 Blubberoid - Create Helm Chart Screep Done None
Other Operations 207036 relabel server saiph.frack.codfw.wmnet to frpig2001.frack.codfw.wmnet Screep Done None
Other Operations 206341 Evaluate scalability and performance of PHP7 compared to HHVM Screep Done None
Other Operations 207775 newer version of nagios-nrpe-plugin nrpe (check_nrpe) with fixed logging issue on stretch icinga Screep Done None
Other Operations 208706 Degraded RAID on analytics1039 Screep Done None
Other Operations 206187 reconfigure Icinga alert for elasticsearch_shard_size to reduce false positive alerts Screep Done None
Other Operations 210455 Ship prometheus logs to ELK Screep Done None
Other Operations 205898 Netbox: explore NAPALM integration Screep Done None
Other Operations 211719 docker-registry.wikimedia.org caches images missing instead of revalidating Screep Done None
Other Operations 211382 Requesting access to Proton for pmiazga, bearND, Mholloway, MSantos, Tgr Screep Done None
Other Operations 207788 Remove "daniel" from "wmde" LDAP group and add him to "wmf" Screep Done None
Other Operations 209757 Notifications disablement via puppet not working on icinga Screep Done None
Other Operations 209570 Disable proxy for beta cluster in maps Screep Done None
Other Operations 199198 Some swift filesystems reporting negative disk usage In-Scope Done None
Other Operations 206450 Reorganize our redis rdb1/rdb2 clusters Screep Done None
Other Operations 209074 Devices with wmf* names and status active Screep Done None
Other Operations 207918 Refactor current code base to support multiple elasticsearch instances/multiple elasticsearch clusters Elaborated Done None
Other Operations 208622 Import recommendations into production database Screep Done None
Other Operations 209693 Redirect from zh-yue.wiktionary.org is not working properly Screep Done None
Other Operations 206338 Allow directing users to PHP7 based on a cookie Screep Done None
Other Operations 212525 Administrator password recovery for wmfaliens@lists.wikimedia.org Screep Done None
Other Operations 158429 Switch to predictable network interface names? In-Scope Done None
Other Operations 209064 Changeprop: Error during deduplication Screep Done None
Other Operations 206166 puppet compiler set to eqiad as primary dc while prod is codfw Screep Done None
Other Operations 207377 Reboot WMCS servers for L1TF Screep Done None
Other Operations 205850 Procure and provision Logging pipeline hardware in multiple datacenters Screep Done None
Other Operations 211596 mtail seems broken on syslog::centralserver installations Screep Done None
Other Operations 209300 Review and make librdkafka-0.11.6 installable from stretch-wikimedia Screep Done None
Other Operations 209176 Add Mukunda to releasers-mediawiki Screep Done None
Other Operations 206909 Degraded RAID on heze-array1 Screep Done None
Other Operations 209456 Gerrit is down "502 Proxy Error" Screep Done None
Other Operations 178535 decommission lvs400[1-4].ulsfo.wmnet In-Scope Done None
Other Operations 209073 Missing rack face/position for 2 eqiad devices Screep Done None
Other Operations 136562 Audit/fix hosts with no RAID configured In-Scope Done None
Other Operations 211859 cronspam from elasticsearch-curator on stretch Elaborated Done None
Other Operations 181559 Investigate redis-cluster or other techniques for making Redis not a single point of failure. In-Scope Done None
Other Operations 206915 Degraded RAID on aqs1006 Screep Done None
Other Operations 209134 pwstore access for cdanis Screep Done None
Other Operations 206566 Create a mailing list for Wikipedia & Education User Group Screep Done None
Other Operations 206633 Setup rsyslog to be able to produce logs to Kafka Screep Done None
Other Operations 208375 mcrouter prometheus exporter stops working when mcrouter restarts Elaborated Done None
Other Operations 207843 increase restart interval of wdqs updater Screep Done None
Other Operations 164955 Degraded RAID on heze In-Scope Done None
Other Operations 210742 Add ahalfaker to ORES-related icinga contacts Screep Done None
Other Operations 189065 Outbound mail from Greenhouse is broken In-Scope Done None
Other Operations 181632 Celery manager implodes horribly if Redis goes down In-Scope Done None
Other Operations 209802 Cannot vote on votewiki Screep Done None
Other Operations 208533 Access for imarlier to wdqs servers Screep Done None
Other Operations 210299 Wikimedia-GIN Screep Done None
Other Operations 210880 Degraded RAID on restbase2017 Screep Done None
Other Operations 207724 Investigate reducing number of servers in the elasticsearch cluster Screep Done None
Other Operations 207817 WDQS Updater ran into issue and stopped working Screep Done None
Other Operations 210067 During scap sync-file error on one endpoint (mw1265) Screep Done None
Other Operations 208267 Requesting access to netbox for bd808 Screep Done None
Other Operations 207091 Parsoid no longer active-active Screep Done None
Other Operations 181546 Let the ORES application set log severity, not uWSGI In-Scope Done None
Other Operations 210877 Degraded RAID on restbase2013 Screep Done None
Other Operations 208141 Degraded RAID on db2048 Screep Done None
Other Operations 212403 Non-redundant power supply on ms-be2048 Screep Done None
Other Operations 208394 Separation of concerns for WDQS puppet module Screep Done None
Other Operations 201364 rack/setup/install sulfur.wikimedia.org In-Scope Done None
Other Operations 208066 Push check latency and check execution time to Prometheus Screep Done None
Other Operations 197689 Figure out service "externalization" with WMF (i.e. whether is possible, and what it takes to get service X running at WMF infrastructure, not being part of mediawiki) Screep Done None
Other Operations 207192 rack/setup/install an-worker10[78-95].eqiad.wmnet Screep Done None
Other Operations 208491 Requesting access to deployment for tarrow Screep Done None
Other Operations 207319 labvirt1018 -> cloudvirt1018: update physical label, network port description, netbox Screep Done None
Other Operations 205694 I get a "403 Forbidden" error when subscribing to a list In-Scope Done None
Other Operations 208717 Requesting access to deployment for WMDE-leszek Screep Done None
Other Operations 140442 reinstall rdb100[56] with RAID In-Scope Done None
Other Operations 209573 Gather metrics from php-fpm Screep Done None
Other Operations 209184 Upgrade to OTRS version 5.0.31 Screep Done None
Other Operations 210458 Ship PuppetDB logs to ELK Screep Done None
Other Operations 206318 wdqs1009 - cannot create /var/log/wdqs/wdqs_autodeployment.log Screep Done None
Other Operations 208729 Onboarding Chris Danis (CDanis) Screep Done None
Other Operations 210624 Create email alias for CPT Leads Screep Done None
Other Operations 208049 unrack/decom cr1-eqord Screep Done None
Other Operations 208108 httpd class and php7.0 - conflict with mpm_event module Screep Done None
Other Operations 206597 Refactor 'use_git_deploy' in wdqs puppet module to cater for scap3 and autodeployment modes Screep Done None
Other Operations 206612 Requesting access to servers for Release Engineering tasks for Lars Wirzenius Screep Done None
Other Operations 206544 Google Search Console access request Screep Done None
Other Operations 208393 Fix Type constraints in wdqs (init.pp) Screep Done None
Other Operations 209077 Rename asw-c4-codfw and asw2-c4-codfw Screep Done None
Other Operations 206478 Frdev1001 server and mysql access Screep Done None
Other Operations 208376 Upgrade memkeys to its latest upstream Elaborated Done None
Other Operations 206121 Cleanup WDQS logging configuration Screep Done None
Other Operations 208715 Onboard Fabián Sellés Rosa to SRE Screep Done None
Other Operations 209566 Update SQL location script for osm-initial-import Screep Done None
Other Operations 207530 Deleting pages on the English Wikipedia is very slow Screep Done None
Other Operations 211974 eqiad: 1 VM request for doc.wikimedia.org Screep Done None
Other Operations 210674 Point keyholder github mirror to gerrit Elaborated Done None
Other Operations 206423 The usual Lag pattern for wdqs2003 seems to be taking another turn Screep Done None
Other Operations 207194 rack/setup/install cloudvirtan100[1-5].eqiad.wmnet Screep Done None
Other Operations 207644 Degraded RAID on analytics1029 Screep Done None
Other Operations 209758 parsoid-rt repeated failures on ruthenium (parsoid::testing) Screep Done None
Other Operations 209427 Relabel labvirt1016.eqiad.wmnet as cloudvirt1016.eqiad.wmnet Screep Done None
Other Operations 191842 Deployment git server can't supply ORES hosts in parallel In-Scope Done None
Other Operations 207966 Why doesn't icinga notify the team-fr-tech-ops contact for services in WARNING state? Screep Done None
Other Operations 205919 TEC3:O3:O3.1:Q2 Goal - Move Blubberoid, ZoteroV2, and Graphoid through the production CD Pipeline Elaborated Done None
Other Operations 160644 Eventstreams graphite disk usage In-Scope Done None
Other Operations 210863 Reconfigure hardware and reimage restbase201[3-8].codfw.wmnet Screep Done None
Other Operations 207040 Graphite1001 disk usage at 96% Screep Done None
Other Operations 210882 Degraded RAID on restbase2016 Screep Done None
Other Operations 127008 hotmail users are reporting mailman issues Screep Done None
Other Operations 210881 Degraded RAID on restbase2018 Screep Done None
Other Operations 210948 Remove unused i18n messages for math extension Elaborated Done None
Other Operations 212669 icinga doesn't log ampersand in notes_url links Screep Done None
Other Operations 200960 Logstash packet loss In-Scope Done None
Other Operations 210265 Setup elasticsearch on new codfw servers Screep Done None
Other Operations 208201 Refactor puppet WDQS module Screep Done None
Other Operations 206261 Routing RFC1918 private IP addresses to/from WMCS floating IPs Elaborated Done None
Other Operations 207278 Move dumpsdata1001 Screep Done None
Other Operations 150356 Wikidata Query Service is overly verbose toward logstash In-Scope Done None
Other Operations 207239 Make aklapper a co-admin of the listadmins@ mailing list Screep Done None
Other Operations 207951 Requesting access to deployment, operational logs, and analytics cluster for jlinehan Screep Done None
Other Operations 184655 logstash group1 dashboard incorrectly shows testwikidatawiki In-Scope Done None
Other Operations 205896 Netbox: upgrade to the latest version (>= 2.4) Screep Done None
Other Operations 209628 Add option maxmemory-policy: 'volatile-lru' on Redis class for debian stretch Screep Done None
Other Operations 208392 refactor wdqs::updater Screep Done None
Other Operations 205873 Investigate Kafka main cluster usage for logging pipeline Screep Done None
Other Operations 209257 Refactor wdqs::gui - Separate cron tasks from the module Screep Done None
Other Operations 208549 HHVM CPU usage when deploying MediaWiki Screep Done None
Other Operations 210451 Kafka eqiad.mediawiki.page-delete topic is empty Screep Done None
Other Operations 211511 Fetching ORES API from en.wikipedia.org blocked in debug mode Screep Done None
Other Operations 207816 Create URL for Mexico Awareness Campaign Screep Done None
Other Operations 207834 Cleanup Wikidata Query Service logging configuration Screep Done None
Other Operations 209526 Add javamelody to gerrit Screep Done None
Other Operations 82983 neon has bad memory (or disk?) Screep Done None
Other Operations 211065 rack/setup/install codfw logstash elasticsearch storage servers Elaborated Done None
Other Operations 212007 Add eprodromou@wikimedia.org to cpt-leads@wikimedia.org Screep Done None
Other Operations 211170 Time on new servers different from time on puppetmaster1001 Screep Done None
Other Operations 208986 WDQS tests can no longer edit test.wikidata.org Screep Done None
Other Operations 208096 Degraded RAID on ms-be2021 Screep Done None
Other Operations 210667 Can exfat be used in WMF production? Screep Done None
Other Operations 117673 labs precise and jessie instance not accessible after provisioning In-Scope Done None
Other Operations 209081 Requesting creation of librarycard-dev mailing list Screep Done None
Other Operations 209615 rack/setup/install restbase201[3-8].codfw.wmnet Screep Done None
Other Operations 41785 Create a Cloud VPS SMTP smarthost In-Scope Done None
Other Operations 207833 Add Lars Wirzenius to releng LDAP groups Screep Done None
Other Operations 210720 Logrotate should restart services when more people are around Screep Done None
Other Operations 181630 Send celery and wsgi service logs to logstash In-Scope Done None
Other Operations 208433 Package and install php 7.2 in place of php 7.0 Screep Done None
Other Operations 210940 Cron <root@maps2001> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) Screep Done None
Other Operations 207900 Enable csp-report-only mode everywhere Screep Done None
Other Operations 136312 Encrypt syslog traffic In-Scope Done None
Other Operations 176153 Create affcom-staff email account In-Scope Done None
Other Operations 206454 Setup Kafka cluster, producers and consumers for logging pipeline Screep Done None
Other Operations 211217 Spin up 3 logstash/kibana frontend VMs in codfw Screep Done None
Other Operations 206224 WMCS: Fewer transitory middle-of-the-night puppet alerts Screep Done None
Other Operations 211715 Interface errors on cr1-codfw:xe-5/3/1 Screep Done None
Other Operations 207101 delete t206636-3 VM and revert quota bumps for project wikidata-query Screep Done None
Other Operations 205840 Server Access for Isaac Johnson Screep Done None
Other Operations 211095 Grant fdans permissions to deploy AQS in prod, and accessing the aqs hosts Screep Done None
Other Operations 210879 Degraded RAID on restbase2015 Screep Done None
Other Operations 205981 Add Mathew.onipe(onimisionipe) to deployment group Screep Done None
Other Operations 207792 Remove "jk" from "wmde" ldap group Screep Done None
Other Operations 207919 Write cookbooks to support spicerack's elasticsearch multi cluster/instance Elaborated Done None
Other Operations 211020 Create maps-root group and add Matt(onimisionipe) to maps-roots Screep Done None
Other Operations 207328 es2017 and es2019 have an idrac ethernet interface in Linux Screep Done None
Other Operations 207656 WDQS logging should be rate limited Screep Done None
Other Operations 207260 Jenkins mail delivery failure to betacluster-alerts@list.wikimedia.org Screep Done None
Other Operations 212305 restbase1011 fails to boot, ASSERT error lines Screep Done None
Other Operations 211945 Add LDAP to aezell for read/write access of Grafana Screep Done None
Other Operations 208750 Requesting access to graphite hosts for addshore Screep Done None
Other Operations 206345 Degraded RAID on db1064 Screep Done None
Other Operations 209829 Degraded RAID on labcontrol1001 Screep Done None
Other Operations 207852 Requesting access to deployment and analytics-privatedata-users for sbassett Screep Done None
Other Operations 196886 Replace wtp1043's sda In-Scope Done None
Other Operations 207629 scb2001: Power supply failure Screep Done None
Other Operations 207947 Switch wdqs1003 with one of the internal wdqs cluster Screep Done None
Other Operations 212266 Request to create mailing list for Wikimedians of Chicago User Group Screep Done None
Other Operations 207090 Requesting deployment access to servers for Performance Team task for perf-roots Screep Done None
Other Operations 179192 Check analytics1037 power supply status In-Scope Done None
Other Operations 209391 change my email address in the techcom alias Screep Done None
Other Operations 150771 Secondary production Jenkins for CI In-Scope Done None
Other Operations 208240 Ensure jenkins on puppet.git checks for yaml syntax errors Screep Done None
Other Operations 205814 Switch the main etcd cluster in eqiad to use conf1004-1006 Screep Done None
Other Operations 168967 Upload shiny-server .deb to our Stretch apt repository In-Scope Done None
Other Operations 207184 dumps mounts don't show up on eqiad1-r VMs Screep Done None
Other Operations 211641 frack / passive icinga checks: Errors connecting to icinga2001.wikimedia.org Screep Done None
Other Operations 211883 Move oxygen to weblog1001 Screep Done None
Other Operations 209860 Ship peopleweb apache2 error logs to ELK Screep Done None
Other Operations 208476 Requesting shh access to mwmaint1001 for kaldari Screep Done None
Other Operations 209568 The icinga web interface can't read the icinga log file Screep Done None
Other Operations 209112 kubestage1001.mgmt down or flapping Screep Done None
Other Operations 210843 Reshape RESTBase Cassandra cluster for server refresh Screep Done None
Other Operations 210469 Update Debian Package for Scap to 3.8.10-1 Screep Done None
Other Operations 209726 Need to shut down a list, mediation-en-l Screep Done None
Other Operations 208722 Add Michael Grosse to 'wmde' LDAP group Screep Done None
Other Operations 181621 What is causing ORES celery workers to suddenly require more CPU? In-Scope Done None
Other Operations 206089 Transfer mailman ownership of Wikimania lists Screep Done None
Other Operations 206314 Modify scap::target to define sudo rules for multiple services Screep Done None
Other Operations 170640 reports.frdev.wm.o -- still in use? In-Scope Done None
Other Operations 210416 Upgrade grafana to 5.x Screep Done None
Other Operations 208518 Requesting access to deployment for Lucas Werkmeister Screep Done None
Other Operations 132216 Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm In-Scope Done None
Other Operations 210701 ORES 500s since 2018-11-29 6:25 Screep Done None
Other Operations 183454 Deprovision Diamond collectors no longer in use In-Scope Done None
Other Operations 207330 Requesting access to production servers (mwlog*, mmaint* ?) for kharlan Screep Done None
Other Operations 210022 Allow access to Data Lake/Hive for Niharika Screep Done None
Other Operations 197242 Transition citoid to use Zotero's translation-server-v2 In-Scope Done None
Other Operations 196336 Icinga passive checks go awal and downtime stops working Screep Done None
Other Operations 206217 Allow Analytics team members to restart Turnilo and Superset Screep Done None
Other Operations 156398 Decommission or repair old asw-c2-eqiad In-Scope Open None
Other Operations 122127 Translation of namespaces for Gilaki In-Scope Open None
Other Operations 191491 Adjust bandwidth/connection limits, memory settings on labstore1006,7 as appropriate In-Scope Open None
Other Operations 175885 Toolforge's static webserver broken by Puppet changes and stale nginx packages In-Scope Open None
Other Operations 199890 missed pages from kafka outage on July 11 2018 In-Scope Open None
Other Operations 206870 decommission betelgeuse Screep Open None
Other Operations 196698 rack/setup/install auth1002 In-Scope Open None
Other Operations 210464 Tracking down gary@ and redirecting it to trustandsafety@ Screep Open None
Other Operations 206963 Perform a statsd and Graphite switch Screep Open None
Other Operations 203520 decommission thulium.frack.eqiad.wmnet In-Scope Open None
Other Operations 181971 Disable hiera autolookups In-Scope Open None
Other Operations 115757 document debian packaging guidelines In-Scope Open None
Other Operations 112774 solve mtp panel issue for row uplinks In-Scope Open None
Other Operations 162037 Use SSL certificates with discovery entry for elasticsearch In-Scope Open None
Other Operations 94215 decommission cp3001 & cp3002 In-Scope Open None
Other Operations 204479 Heating alerts and broken RAM on kafka1014 In-Scope Open None
Other Operations 126574 puppet should try to mount all mountable swift filesystems In-Scope Open None
Other Operations 180105 Set up a statsv-like endpoint for Prometheus In-Scope Open None
Other Operations 36947 Incorrect text positioning in SVG rasterization (scale/transform; font-size; kerning) In-Scope Open 0.0
Other Operations 199670 Integrate Stretch 9.5 point release In-Scope Open None
Other Operations 207721 Broken memory on thumbor1004 Screep Open None
Other Operations 211121 Usual git mechanism for aborting commit does not work on the private puppet repo Screep Open None
Other Operations 160412 Add lock_wait_timeout to maintain_views and maintain-meta_p In-Scope Open None
Other Operations 212397 conftool is failing flake8 Screep Open None
Other Operations 207760 setup/install weblog1001/WMF4750 as oxygen replacement Elaborated Open None
Other Operations 160101 Upgrade php5-json .deb to at least 1.3.8 In-Scope Open None
Other Operations 190693 Extend dpkg Icinga check to also check for inconsistent apt state In-Scope Open None
Other Operations 109089 EPIC: Cultivating the Elasticsearch garden (operational lessons from 1.7.1 upgrade) In-Scope Open None
Other Operations 144539 Remove /srv/deployment/wdqs/wdqs/rules.log symlink In-Scope Open None
Other Operations 164993 archiva artifact links point to 127.0.0.1 In-Scope Open None
Other Operations 189729 Build .deb package of python3-typing for jessie In-Scope Open None
Other Operations 207543 Move labmon (Graphite, StatsD) into a Cloud VPS Screep Open None
Other Operations 133674 HHVM is leaking memory on the API appservers In-Scope Open None
Other Operations 134811 Consider REST with SSL (HyperSwitch/Cassandra) for session storage In-Scope Open None
Other Operations 202535 decom rigel.frack.codfw.wmnet In-Scope Open None
Other Operations 138821 extend existing graphite whisper files retention to five years In-Scope Open None
Other Operations 102575 document graphite failover/backfill procedures In-Scope Open None
Other Operations 212102 Add `supervised` option to redis configuration Screep Open None
Other Operations 164238 move icinga contacts file to public repo In-Scope Open None
Other Operations 204047 investigate tilerator crash on maps eqiad In-Scope Open None
Other Operations 114337 Assign 3 more servers to video scaler duty In-Scope Open None
Other Operations 110169 Monitor redis memory/disk usage In-Scope Open None
Other Operations 170474 Decommisson and store old row D network gear. In-Scope Open None
Other Operations 143556 Setting up grafana should also setup Anonymous read-only access for the default org In-Scope Open None
Other Operations 138496 bring swift eqiad to one zone per row In-Scope Open None
Other Operations 175362 Split MXes into inbound and outbound In-Scope Open None
Other Operations 209292 Create a wmf production ready nginx image Screep Open None
Other Operations 190086 Decommission old server wmf4077 In-Scope Open None
Other Operations 118812 Investigate mysterious_sysctl settings and figure out what to do with them In-Scope Open None
Other Operations 209861 labvirt1007 predicted raid failure Screep Open None
Other Operations 211070 decommission of restbase200[1-6] (lease return in December 2018) Screep Open None
Other Operations 196697 rack/setup/add to spares tracking 2 single cpu misc class systems In-Scope Open None
Other Operations 135113 Rationalize our jobqueues redis topology In-Scope Open None
Other Operations 104352 Make scap able to depool/repool servers via the conftool API In-Scope Open None
Other Operations 190766 add ci test for admin module indentation In-Scope Open None
Other Operations 132632 puppetize turning off reserved space for cassandra /srv In-Scope Open None
Other Operations 188317 Detect high server load earlier – prometheus alert? In-Scope Open None
Other Operations 203786 Mcrouter periodically reports soft TKOs for mc1022 (was mc1035) leading to MW Memcached exceptions In-Scope Open None
Other Operations 153279 labnet/ labtestnet2001 - disk space - nova-api.log needs rotation In-Scope Open None
Other Operations 199406 rsyslog's in:imtcp thread stuck on old sockets In-Scope Open None
Other Operations 202255 Support for QLogic FastLinQ 41112 Dual Port 10Gb SFP+ Adapter In-Scope Open None
Other Operations 197173 Ship MX logs to ELK In-Scope Open None
Other Operations 104671 Rename 'restricted' group? In-Scope Open None
Other Operations 167689 Add RIPE atlas data to Prometheus In-Scope Open None
Other Operations 141756 audit / test / upgrade hp smartarray P840 firmware In-Scope Open None
Other Operations 205396 Evaluate/integrate rasdaemon as a replacement for mcelog In-Scope Open None
Other Operations 199780 labstore1003: more SMART failures In-Scope Open None
Other Operations 176335 logs sent to logstash are lost when the elasticsearch cirrus cluster is unavailable In-Scope Open None
Other Operations 169322 Research whether it makes sense to have OTRS installation in an HA setup In-Scope Open None
Other Operations 93531 secure.wikimedia.org entries still showing up in Google search results In-Scope Open None
Other Operations 133476 Proposal: Centralize OTRS login methodology In-Scope Open None
Other Operations 32716 Run our own Tor client for Tor block In-Scope Open None
Other Operations 210668 Elasticsearch health check for shards icinga check shows OK status when cluster health is yellow Screep Open None
Other Operations 93138 Procure hardware for Sentry In-Scope Open None
Other Operations 184462 Serve one production service via Kubernetes In-Scope Open None
Other Operations 109090 Investigate the need for master only (non data nodes) in our ES cluster In-Scope Open None
Other Operations 132324 Tracking and Reducing cron-spam to root@ In-Scope Open None
Other Operations 185195 tmpreaper doesn't play along with PrivateTmp systemd units In-Scope Open None
Other Operations 185814 /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 In-Scope Open None
Other Operations 187445 Decommission osm-db200[12] and osm-web200[1234] In-Scope Open None
Other Operations 179395 Cluster puppet variable and ganglia decommission In-Scope Open None
Other Operations 67270 Default license for operations/puppet In-Scope Open None
Other Operations 109606 Re-evaluate Limesurvey In-Scope Open None
Other Operations 135125 Install a second etcd cluster in codfw In-Scope Open None
Other Operations 178877 operations/software repo: flake8 check In-Scope Open None
Other Operations 46791 [[wikitech:Server_admin_log]] should not rely on freenode irc for logmsgbot entries In-Scope Open None
Other Operations 178575 Add require_package() variant with repository component to wmflib In-Scope Open None
Other Operations 121610 system users with UIDs > 500 In-Scope Open None
Other Operations 116767 limit the impact of heavy/large graphite queries In-Scope Open None
Other Operations 119401 Untangle labs/production roles from labs/instance roles In-Scope Open None
Other Operations 182702 Debian Jessie reimage/install ends up in kernel panic with 8.10 netboot image In-Scope Open None
Other Operations 198939 Decommission servermon In-Scope Open None
Other Operations 210721 Contact number of some WMDE staff should be avalible to SRE/RelEng Screep Open None
Other Operations 195392 Switch cronjobs on maintenance hosts to PHP7 In-Scope Open None
Other Operations 76203 Make ircecho run as its own user In-Scope Open None
Other Operations 170152 mc2023 / mc2025 fail to mount root partition within 90 seconds using Linux 4.9 In-Scope Open None
Other Operations 130590 Have dedicated master nodes for elasticsearch In-Scope Open None
Other Operations 159524 backup space is used unwisely In-Scope Open None
Other Operations 184065 Setup new access switches In-Scope Open None
Other Operations 198787 Revisit default settings for c-foreach-restart In-Scope Open None
Other Operations 196478 rack/setup/install backup1001 In-Scope Open None
Other Operations 207920 Test spicerack elasticsearch module Elaborated Open None
Other Operations 209149 Have linters/tests results show up as comments in files on gerrit Screep Open None
Other Operations 211368 update PDUs for eqsin (asset tag and other info) Screep Open None
Other Operations 202885 Migrate elasticsearch scripts to spicerack cookbooks In-Scope Open None
Other Operations 166368 Wipe of spare/replacement disks In-Scope Open None
Other Operations 187076 Deploy error: insufficient permission for adding an object to repository database .git/objects In-Scope Open None
Other Operations 207536 Move various support services for Cloud VPS currently in prod into their own instances Screep Open None
Other Operations 193072 TTS server deployment strategy In-Scope Open None
Other Operations 201342 rack/setup/install puppetmaster1003.eqiad.wmnet In-Scope Open None
Other Operations 144431 RESTBase k-r-v as Cassandra anti-pattern In-Scope Open None
Other Operations 134326 udpmxircecho should write stats of messages processed and we should alert when that drops to zero In-Scope Open None
Other Operations 88730 Nutcracker needs to automatically recover from MC failure - rebalancing issues In-Scope Open None
Other Operations 208576 Netbox: Usage guidelines for WMCS Screep Open None
Other Operations 136603 Update limit.sh to support systemd-based cgroup management In-Scope Open None
Other Operations 193473 Add HTTPS support to wdqs-internal service In-Scope Open None
Other Operations 150375 cronspam cleanup: Cron <www-data@terbium> /usr/local/bin/foreachwiki maintenance/cleanupUploadStash.php > /dev/null In-Scope Open None
Other Operations 212522 Update label and switch to rename labvirt1013 to cloudvirt1013 Screep Open None
Other Operations 198138 Disable agent forwarding to important hosts In-Scope Open None
Other Operations 97909 Upgrade jobrunners to redis 2.8 In-Scope Open None
Other Operations 199220 Cleanup cirrus keys in $wmfSwiftEqiadConfig In-Scope Open None
Other Operations 121240 Network isolation for production and semi-production services In-Scope Open None
Other Operations 193664 Knock down puppet 4 deprecation warnings In-Scope Open None
Other Operations 186069 Icinga: page in case all MediaWiki are throwing 5xx In-Scope Open None
Other Operations 118677 Nastaleeq font for Western Punjabi In-Scope Open None
Other Operations 187434 Include apache_exporter in puppet module apache In-Scope Open None
Other Operations 185644 Switch phabricator from using apache to nginx In-Scope Open None
Other Operations 123809 Module uwsgi doesn't allow passing multiple config params of same name In-Scope Open None
Other Operations 193160 Monitor the BIOS boot order and parameters In-Scope Open None
Other Operations 205712 wtp2020: correctable memory errors In-Scope Open None
Other Operations 164490 maintain-meta_p hangs on connecting to wikimedia.org.uk In-Scope Open None
Other Operations 176774 Reimage cobalt as stretch In-Scope Open None
Other Operations 209863 graph server temperature metrics Screep Open None
Other Operations 146090 High failure rate of account creation should trigger an alarm / page people In-Scope Open None
Other Operations 205862 Expand modern metrics infrastructure coverage (2018-19 Q2 goal) Screep Open None
Other Operations 177826 Upgrade ci ssh key to ecdsa In-Scope Open None
Other Operations 176666 Qualtrics cannot send email to wikimedia.org addresses In-Scope Open None
Other Operations 181803 Stop storing Mailman passwords in plain text In-Scope Open None
Other Operations 209189 Revisit and update python testing in puppet Screep Open None
Other Operations 95742 Decomission amssq31-62 (32 hosts) In-Scope Open None
Other Operations 45952 Incorrect "non-identical file already exists" error when undeleting file on Commons In-Scope Open None
Other Operations 160229 Back up of Commons files In-Scope Open None
Other Operations 207200 Revisit the logging work done on Q1 2017-2018 for the standard pod setup Elaborated Open None
Other Operations 161145 Fix the general problem of randomly-bad puppet agent cron timings within redundant clusters In-Scope Open None
Other Operations 191956 Document how to fix IPMI issues on Wikitech In-Scope Open None
Other Operations 211964 Make scap and opcache work consistently together Elaborated Open None
Other Operations 140594 svn.wikimedia.org redirects to Diffusion main page, hence hard to find e.g. "flexbisonparse" In-Scope Open None
Other Operations 101912 Network segmentation for WMF servers In-Scope Open None
Other Operations 127797 document all puppet classes / defined types!? In-Scope Open None
Other Operations 67394 [EPIC] Performance testing environment In-Scope Open None
Other Operations 184086 Add prometheus exporter to Gerrit In-Scope Open None
Other Operations 208191 Set hhvm.virtual_host[default][always_decode_post_data] = false Screep Open None
Other Operations 191199 Page allocation stalls on scb1001, scb1002 In-Scope Open None
Other Operations 135128 Turn on etcd TLS for intra-cluster communications In-Scope Open None
Other Operations 157306 Fix config file handling for /etc/hhvm/php.ini In-Scope Open None
Other Operations 197819 investigate caching of mailman listinfo pages In-Scope Open None
Other Operations 188453 Google Search Console access for Search Platform team In-Scope Open None
Other Operations 169318 Use multiple puppetdbs on puppet masters In-Scope Open None
Other Operations 170481 FY2017/18 Program 6 - Outcome 2 - Objective 2: Set up a continuous integration and deployment pipeline In-Scope Open None
Other Operations 151314 logrotate failing with $FILE.1.gz: File exists In-Scope Open None
Other Operations 141255 Separate host lookup from the sql shell script In-Scope Open None
Other Operations 119846 Redirect revisions from svn.wikimedia.org to https://phabricator.wikimedia.org/rSVN In-Scope Open None
Other Operations 123237 Provide production jessie image with node 4.2; use this for service-runner build command In-Scope Open None
Other Operations 156136 Increase swift replication factor for accounts In-Scope Open None
Other Operations 207294 Create icinga checks for certcentral Screep Open None
Other Operations 195981 require_package should mark packages as manually installed In-Scope Open None
Other Operations 120856 Remove all out of warranty unused cp10xx's from A2 In-Scope Open None
Other Operations 210223 Post hold because of "invalid headers" in wikimediacz-l Screep Open None
Other Operations 155761 DNS repo: add Jenkins job to ensure there are no duplicates In-Scope Open None
Other Operations 136094 Race condition in setting net.netfilter.nf_conntrack_tcp_timeout_time_wait In-Scope Open None
Other Operations 203402 in Commons, some PDFs are failing to render thumbnails. In-Scope Open None
Other Operations 202504 Evaluate VMWare's Harbour as a docker registry In-Scope Open None
Other Operations 203485 Revisit Grafana/Icinga notification strategy In-Scope Open None
Other Operations 175710 Add profiling for Varnish and VCL In-Scope Open None
Other Operations 211488 Audit and sync INI settings as needed between HHVM and PHP 7 Elaborated Open None
Other Operations 202764 Wikidata produces a lot of failed requests for recentchanges API In-Scope Open None
Other Operations 207754 Access requests process: People sometimes specify hostnames instead of admin groups in access requests Screep Open None
Other Operations 170995 Setup a mirror for R language dependencies (CRAN) In-Scope Open 10.0
Other Operations 148647 refresh swift hardware in codfw/eqiad In-Scope Open None
Other Operations 206639 Switch to unix socket connections for osmupdater / osmimporter for postgresql on maps Screep Open None
Other Operations 187473 Decommission old and unused/spare servers in eqiad In-Scope Open None
Other Operations 165511 Change automatic shortlink in blog theme In-Scope Open None
Other Operations 207706 LibreNMS upgrade to 1.44 Screep Open None
Other Operations 174172 unused grafana-dashboard indices on elasticsearch / logstash In-Scope Open None
Other Operations 179501 Use external dsh group to list pooled ORES nodes In-Scope Open None
Other Operations 164819 reprepro: Support for buildinfo files / dbgsym packages In-Scope Open None
Other Operations 166233 Update redis puppet class to support stretch In-Scope Open None
Other Operations 182819 custom fact interface_primary breaks under newer versions of facter In-Scope Open None
Other Operations 162612 codfw/eqiad hosts occasionally spend > 3 minutes starting networking.service with linux 4.9 In-Scope Open None
Other Operations 164042 Racktables: clearly show when hosts are decommissioned In-Scope Open None
Other Operations 152632 Explore hosting the multimedia commons use case In-Scope Open None
Other Operations 138799 Create a simple puppet role for setting up a singlenode kubernetes install In-Scope Open None
Other Operations 185815 The Rack Puppet master server is deprecated and will be removed in a future release. Please use Puppet Server instead. In-Scope Open None
Other Operations 193573 Consider allowing mailing lists to be indexed by archive.org In-Scope Open None
Other Operations 177196 Port non-deprecated Diamond collectors to Prometheus In-Scope Open None
Other Operations 76306 Set warning thresholds for average cluster utilization In-Scope Open None
Other Operations 210143 Canaries canaries canaries Screep Open None
Other Operations 211850 install2002 94% disk usage on "/" Screep Open None
Other Operations 98831 Honor DNT header for access logs & varnish logs In-Scope Open None
Other Operations 211721 Establish an SLA for session storage Screep Open None
Other Operations 208875 Update prometheus-node-exporter NTP metrics Screep Open None
Other Operations 210582 New node request: oresrdb[12]003 Screep Open None
Other Operations 186153 outdated DjVu file page thumbnail in cache In-Scope Open None
Other Operations 210717 Find an alternative to HHVM curl connection pooling for PHP 7 Elaborated Open None
Other Operations 124413 confctl should provide tags information after writing data In-Scope Open None
Other Operations 203861 decom radium In-Scope Open None
Other Operations 95053 ircecho should accept input via unix sockets In-Scope Open None
Other Operations 158022 make apt.wikimedia.org HA In-Scope Open None
Other Operations 200832 remove mathoid from scb In-Scope Open None
Other Operations 101141 UDP rcvbuferrors and inerrors on graphite hosts In-Scope Open None
Other Operations 142991 Enable "upload by url" feature at zhwiki In-Scope Open None
Other Operations 210269 Build helm charts for ORES Elaborated Open None
Other Operations 188913 "Obama" page on Beta Cluster often responds with 500 or 503 In-Scope Open None
Other Operations 205240 MCE errors on mw2181 / temperature warnings In-Scope Open None
Other Operations 211547 Cleanup the puppetmaster module so that we stop breaking expectations (and the puppet compiler) Screep Open None
Other Operations 204363 Modify elasticsearch_shard_size_check plugin to display only indices and shard size In-Scope Open None
Other Operations 210927 Update label and switch to rename labvirt1014 to cloudvirt1014 Screep Open None
Other Operations 204830 Temporarily redirect sgs.wikipedia.org to bat-smg.wikipedia.org until bat-smg->sgs move can be done In-Scope Open None
Other Operations 204801 Exec error "Possibly missing executable file: svn diff" from Special:Code In-Scope Open None
Other Operations 179353 Scap: Standardize git version In-Scope Open None
Other Operations 212231 Remove Diamond from production Screep Open None
Other Operations 199236 Handle SMART for multiple shelves and controllers In-Scope Open None
Other Operations 182759 Add Prometheus exporter to Jenkins instances In-Scope Open None
Other Operations 182274 Create custom per-job metric reporters capability In-Scope Open None
Other Operations 87220 Minimize differences between beta and production (Tracking) In-Scope Open None
Other Operations 212621 jalexander should be removed from security@ as his emails are bouncing Screep Open None
Other Operations 171191 Should puppet auto-restart slapd? In-Scope Open None
Other Operations 210076 set up a test node with new version, Redis as cache, a new Swift container and export metrics over graphana Screep Open None
Other Operations 205852 Onboard at least 10 new non-sensitive log producers to the logging pipeline Screep Open None
Other Operations 198648 Authentication for grafana In-Scope Open None
Other Operations 195553 Puppet class systemd needs to throw a more useful error In-Scope Open None
Other Operations 162122 Swiftrepl was stuck in an infinite loop since days In-Scope Open None
Other Operations 161834 Undo special tools-home and tools-project share definitions for NFS In-Scope Open None
Other Operations 206965 Degraded RAID on dbstore1002 Screep Open None
Other Operations 177914 Switch labstore servers to default SSH configuration In-Scope Open None
Other Operations 207666 Redefine privileges and access for perf-roots group Screep Open None
Other Operations 156232 confctl SubjectAltNameWarning after python-urllib3 upgrade In-Scope Open None
Other Operations 187658 Setup cron for foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php on Beta In-Scope Open None
Other Operations 202329 SRE query: Is it possible to measure how many e-mails are sent to "black hole" e-mail addresses? In-Scope Open None
Other Operations 199876 Migrate pool counters to stretch In-Scope Open None
Other Operations 150486 Deploy federation for Prometheus In-Scope Open None
Other Operations 134271 Replace ircd-ratbox with something newer/maintained In-Scope Open None
Other Operations 208284 Increase "check_legal_html" coverage to group0 wikis Screep Open None
Other Operations 161598 Monitor HHVM bytecode cache depletion on mediawiki app servers In-Scope Open None
Other Operations 88997 Improve graphite failover In-Scope Open None
Other Operations 207707 contint1001 store docker images on separate partition or disk Screep Open None
Other Operations 209590 HTTP/2 requests fail with too-long URLs Screep Open None
Other Operations 189741 Build .deb package of python3-aiokafka In-Scope Open None
Other Operations 210268 Build blubber file for ORES Screep Open None
Other Operations 170628 Jessie rsvg/cairo can't render specific SVG file on Commons In-Scope Open None
Other Operations 171482 Programmatic generation of grafana dashboards In-Scope Open None
Other Operations 167035 stretch acct monthly cron will spam when /var/log/wtmp.1 doesn't exist In-Scope Open None
Other Operations 165136 Ferm rules for labstore NFS hosts In-Scope Open None
Other Operations 142827 Enforce reference to Phabricator task for all commits to modules/admin/data/data.yaml In-Scope Open None
Other Operations 212189 New Service Request: Wikidata Termbox SSR Screep Open None
Other Operations 150396 Phabricator leaving old files in /tmp In-Scope Open None
Other Operations 211125 Move service-runner to new logging infrastructure Screep Open None
Other Operations 211270 Cron <www-data@mwmaint1002> /usr/local/bin/foreachwiki maintenance/cleanupUploadStash.php > /dev/null Screep Open None
Other Operations 128715 Add other Tools administrators to the Icinga notification group In-Scope Open None
Other Operations 207312 skylake CPU numa clustering settting discussion Screep Open None
Other Operations 191625 Create RC feed for login.wikimedia In-Scope Open None
Other Operations 210704 Migrate node-based services in production to node10 Screep Open None
Other Operations 152100 should we make privatewiki list available to puppet without maintaining two lists? In-Scope Open None
Other Operations 163823 During labservices1001 failover fqdn changed from foo.project.eqiad.wmflabs to foo.eqiad.wmflabs In-Scope Open None
Other Operations 149885 Investigate Swift as a storage backend for maps tiles In-Scope Open None
Other Operations 209489 Apply interface::rps to all the mc hosts Elaborated Open None
Other Operations 128615 Get rid of Tool Labs home page check from shinken In-Scope Open None
Other Operations 194855 Degraded RAID on cloudvirt1020 In-Scope Open None
Other Operations 158915 Setup reply emails for gerrit In-Scope Open None
Other Operations 211956 Add chi, psi and omega selector to the elasticsearch dashboards in grafana Screep Open None
Other Operations 212519 Massive spambot registrations at dinwiki Screep Open None
Other Operations 132325 Weak digest algorithm (SHA1) used to sign InRelease on apt.wikimedia.org In-Scope Open None
Other Operations 161920 logrotate for ruthenium In-Scope Open None
Other Operations 210450 rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4] Screep Open None
Other Operations 151049 Run systematic availability tests In-Scope Open None
Other Operations 210597 Broken elasticsearch-prometheus-exporter service on logstash nodes after reboot Screep Open None
Other Operations 160071 Add slabinfo prometheus exporter In-Scope Open None
Other Operations 124179 Improve access to and control over incident and metrics monitoring infrastructure In-Scope Open None
Other Operations 192610 prometheus on bast3002 misbehaving In-Scope Open None
Other Operations 149057 Designate seems very slow to delete records? In-Scope Open None
Other Operations 195847 Clean up artifacts from LaTeX based math rendering In-Scope Open None
Other Operations 107108 Flow notification links on mobile point to desktop In-Scope Open None
Other Operations 125442 es2009 degraded RAID In-Scope Open None
Other Operations 197862 Increase the CPU count for proton[12]00[12] In-Scope Open None
Other Operations 197171 Graph outbound mail volume on per-service or hostgroup level In-Scope Open None
Other Operations 163507 Intermittent DB connectivity problem on phabricator, needs investigation In-Scope Open None
Other Operations 148693 Deploy IDS rendering engine to production In-Scope Open None
Other Operations 163362 audit all codfw pdu tower draws In-Scope Open None
Other Operations 194997 Track more detailed disk usage on maps servers In-Scope Open None
Other Operations 162955 rebuild tools-grid-master as a large instance In-Scope Open None
Other Operations 137939 Increase frequency of OSM replication In-Scope Open None
Other Operations 111540 Clean up labs graphite datapoints In-Scope Open None
Other Operations 210806 Decreased internationalisation of automatic citations as a result of switch to new translation-server Elaborated Open None
Other Operations 116742 Track amount of package updates on systems In-Scope Open None
Other Operations 161864 404 error while accessing some images files e.g. djvu and jpg In-Scope Open None
Other Operations 209361 Phase out Nodepool from production Screep Open None
Other Operations 210486 Audit "misc" cluster hosts Screep Open None
Other Operations 170298 sshd stretch puppet support In-Scope Open None
Other Operations 153940 Logrotate fails for: "$FILE No such file or directory" In-Scope Open None
Other Operations 92471 enable authenticated access to Cassandra JMX In-Scope Open None
Other Operations 183236 After reimage Puppet order: sudo command failed In-Scope Open None
Other Operations 205855 Investigate approaches to ingest sensitive log producers Screep Open None
Other Operations 192547 Improve remote IPMI monitoring In-Scope Open None
Other Operations 159750 E-mail for people in different OIT LDAP object unit In-Scope Open None
Other Operations 212129 Use a multi-dc aware store for ObjectCache's MainStash if needed. Screep Open None
Other Operations 165323 Add Prometheus machine metric to track core dumps In-Scope Open None
Other Operations 196034 Define scap::sources in a way that is shared between prod and beta In-Scope Open None
Other Operations 193733 Move dispatching of wikidata to a dedicated node In-Scope Open None
Other Operations 210109 Spec out migrating ORES to kubernetes Screep Open None
Other Operations 206484 Manage Hue via systemd unit Screep Open None
Other Operations 182331 [Epic] Deploy ORES in kubernetes cluster In-Scope Open None
Other Operations 209460 CloudVPS: our ideal future model Screep Open None
Other Operations 194558 Enable CAPTCHA on mailman instances In-Scope Open None
Other Operations 120585 Make l10nupdate user a system user In-Scope Open None
Other Operations 131832 Unable to restore file that has a very large file size In-Scope Open None
Other Operations 129180 Preserve SSH host key when re-imaging hosts In-Scope Open None
Other Operations 184186 Fix unknown variables warning that occur with puppet 4.x In-Scope Open None
Other Operations 161904 decommission backup4001 In-Scope Open None
Other Operations 133913 Completely port l10nupdate to scap In-Scope Open None
Other Operations 211998 Decommission brokenasw-c2-eqiad Screep Open None
Other Operations 137229 Tune thread for osm2pgsql / postgres max connections for Maps In-Scope Open None
Other Operations 133093 Investigate idle appservers in codfw In-Scope Open None
Other Operations 200209 Decom graphite2001 In-Scope Open None
Other Operations 178663 Switch CI Docker Storage Driver to its own partition and to use devicemapper In-Scope Open None
Other Operations 140942 Tracking: Monitoring and alerts for "business" metrics In-Scope Open None
Other Operations 114446 move human users out of UID range for system accounts In-Scope Open None
Other Operations 211023 Decommission elastic2001-2024 Screep Open None
Other Operations 211404 More restrictive DMARC policy for the wikimedia.org domain Screep Open None
Other Operations 139971 access_new_install role vs. Labs vs. the future In-Scope Open None
Other Operations 179562 Create jenkins job for creating deployment artifacts for `docker-pkg-deploy` In-Scope Open None
Other Operations 146355 Replace etcd internal auth mechanism with a frontend proxy In-Scope Open None
Other Operations 40860 security@mediawiki.org : Create a public key and publish it on the public key servers In-Scope Open None
Other Operations 153068 Consider mounting labs NFS labstore1003.eqiad.wmnet:/scratch for server-side uploads In-Scope Open None
Other Operations 210993 Deprecate Diamond collectors in Cloud VPS Elaborated Open None
Other Operations 209271 improve docker registry architecture Elaborated Open None
Other Operations 46016 SVG fails to render properly due to several issues In-Scope Open None
Other Operations 211124 Move mediawiki to new logging infrastructure Screep Open None
Other Operations 149287 Heating alerts for mw servers in eqiad In-Scope Open None
Other Operations 147366 Setup automated topk wide row reporting In-Scope Open None
Other Operations 187257 puppetdb4: systemd config review In-Scope Open None
Other Operations 209088 Design pipeline image versioning scheme Screep Open None
Other Operations 186918 prometheus: ganglia-gen and outdated Ganglia:cluster resource name In-Scope Open None
Other Operations 155929 Create /community-beacon alternative entry point In-Scope Open None
Other Operations 169518 Decommission esams ms-fe / ms-be In-Scope Open None
Other Operations 147204 Update confd package In-Scope Open None
Other Operations 196019 setup/install phab1002(WMF4727) In-Scope Open None
Other Operations 204110 Add favicon to icinga and tendril In-Scope Open None
Other Operations 106664 Set up role accounts and feedback loops (FBL) with all providers In-Scope Open None
Other Operations 207533 Move labs-recursors in WMCS Screep Open None
Other Operations 114849 Log lines on flourine overflow at 8092 bytes. In-Scope Open None
Other Operations 203434 Decom mw2213 In-Scope Open None
Other Operations 200312 Remove data from Hadoop's HDFS as part of the user offboard workflow In-Scope Open None
Other Operations 17000 Special:Import error: "Import failed: Could not open import file" In-Scope Open None
Other Operations 193272 Prometheus vs. CPU usage vs. hyperthreading In-Scope Open None
Other Operations 119679 Rewrite http://download.wikimedia.org/mediawiki/ -> https://releases.wikimedia.org/mediawiki in less than 3 redirects In-Scope Open None
Other Operations 134875 udpmxircecho spam/not working if unable to connect to irc server In-Scope Open None
Other Operations 208259 OSError: [Errno 1] Operation not permitted when running git fat pull Screep Open None
Other Operations 212219 wmf-auto-restart fails on certain legacy services Screep Open None
Other Operations 184064 Prepare racks OE14, OE15 and OE16 with new infrastructure In-Scope Open None
Other Operations 196665 rack/setup/install bast2002.wikimedia.org In-Scope Open None
Other Operations 154665 Look into behaviour of /etc/exim4/update-exim4.conf.conf related to updates In-Scope Open None
Other Operations 163354 Find a way to verify mediawiki-config IPs ahead of datacenter switchovers In-Scope Open None
Other Operations 192532 Figure out a way to enable volunteers to use the puppet compiler In-Scope Open None
Other Operations 140270 Determine a core set or a checklist of permissions for deployment purpose In-Scope Open None
Other Operations 125085 Split the API MediaWiki appserver pool into two external/internal pools In-Scope Open None
Other Operations 180023 [DRAFT][RfC] Deployment of python applications in production In-Scope Open None
Other Operations 204106 Log slow queries on postgresql / maps In-Scope Open None
Other Operations 129188 mw2212 unresponsive In-Scope Open None
Other Operations 135122 Reduce etcd technical debt In-Scope Open None
Other Operations 179856 Improve documentation for mirrors.wikimedia.org In-Scope Open None
Other Operations 175876 document all scs connections In-Scope Open None
Other Operations 97524 ocg alarm ocg_job_status_queue 'flapping' In-Scope Open None
Other Operations 157038 Make it possible to run the mediawiki testsuite against a staging repo of apt.wikimedia.org In-Scope Open None
Other Operations 200103 requesting additional production ssh key for jmorgan In-Scope Open None
Other Operations 187456 Decommission labstore100[123] and their disk shelves In-Scope Open None
Other Operations 156937 Provide cross-dc redundancy (active-active or active-passive) to all important misc services In-Scope Open None
Other Operations 204135 Warn when CirrusSearch is not configured to use local DC for an extended time In-Scope Open None
Other Operations 189629 rename role::xenon In-Scope Open None
Other Operations 192457 Reallocate former image scalers In-Scope Open None
Other Operations 211027 puppet (systemd::service) attempts to start manually masked units Screep Open None
Other Operations 82350 update exim::listserve::private::mailing_lists value in puppet In-Scope Open None
Other Operations 179463 Create a single application to provision and manage developer (LDAP) accounts In-Scope Open None
Other Operations 156475 Investigate spike in 500s during asw-c2-eqiad replacement In-Scope Open None
Other Operations 104774 Publishing translations for central notice banners fails In-Scope Open 2.0
Other Operations 87790 decom amslvs1-4 (dc work) In-Scope Open None
Other Operations 124185 Evaluate alternative web interfaces to icinga 1 core In-Scope Open None
Other Operations 144933 Cleanup debconf handling in mailman puppet setup In-Scope Open None
Other Operations 200023 2018 data center switchover: Move all the things back to eqiad In-Scope Open None
Other Operations 209738 decom einsteinium Screep Open None
Other Operations 171048 Eventbus does not handle gracefully changes in DNS recursors In-Scope Open None
Other Operations 156544 Create backups of Wikimedia content in diverse geographic places In-Scope Open None
Other Operations 208844 Apply -R 200 to all the memcached mw object cache instances running in eqiad/codfw Elaborated Open None
Other Operations 209425 Reclaim rdb2001, rdb2002 Screep Open None
Other Operations 187984 Update OTRS to the latest stable version (6.x.x) In-Scope Open None
Other Operations 210267 The continuous release pipeline should support more than one service per repo Screep Open None
Other Operations 197470 find a way to systematically update the deployment server name across all repos In-Scope Open None
Other Operations 149643 Review Icinga alarms with disabled notifications In-Scope Open None
Other Operations 211272 Cron <root@mw2182> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) Screep Open None
Other Operations 126083 overhaul labstore setup [tracking] In-Scope Open None
Other Operations 130709 authoritative copy of 'root' files for upload.wikimedia.org is only in swift In-Scope Open None
Other Operations 163996 Icinga check for ipv6 host reachability In-Scope Open None
Other Operations 120377 labmon1001 graphite instance archiver keeps archiving the same instances In-Scope Open None
Other Operations 156140 Lots of hosts with hyperthreading disabled In-Scope Open None
Other Operations 113792 Change LDAP cn to something more useful (was Rename "Dzahn" to "Daniel Zahn" in Gerrit) In-Scope Open None
Other Operations 161096 confctl no longer logs a non-changing state change In-Scope Open None
Other Operations 191191 pdfrender logs to /var/log/syslog as well as to /srv/log/pdfrender In-Scope Open None
Other Operations 150300 icinga notification if elevated writing to badpass.log In-Scope Open None
Other Operations 191018 Provide an option menu when booting via PXE In-Scope Open None
Other Operations 125408 Regularly & Automatically backup WMDE metrics stored in graphite Screep Open None
Other Operations 210264 Investigate memory usage of ORES in kubernetes Elaborated Open None
Other Operations 201247 Sporadic puppet failures Screep Open None
Other Operations 151304 tmpreaper possible race condition In-Scope Open None
Other Operations 141038 implement icinga paging for non-ops teams In-Scope Open None
Other Operations 198041 graphite2001 crashed In-Scope Open None
Other Operations 202705 Degraded RAID on sodium In-Scope Open None
Other Operations 184066 Procure and install new PDUs In-Scope Open None
Other Operations 140075 investigate swift used space spikes since June 2016 In-Scope Open None
Other Operations 181967 Update puppet code to conform to puppet 4.x and later standards In-Scope Open None
Other Operations 171122 librenms: consider using Distributed Poller with multiple netmon servers In-Scope Open None
Other Operations 111838 Some files had disappeared from Commons after renaming In-Scope Open None
Other Operations 209357 Return graphite100[13] to spares pool (or decom) Screep Open None
Other Operations 201779 Have a check to prevent non-existent accounts from being added to LDAP groups In-Scope Open None
Other Operations 202033 Feedback Appreciated: Use of HTTP Without TLS In-Scope Open None
Other Operations 197873 how to structure wiki pages for Icinga reaction play books In-Scope Open None
Other Operations 64987 librsvg misinterpret quoted font family names that contain whitespaces In-Scope Open None
Other Operations 205974 logrotate cronspam on ms-be1040 Screep Open None
Other Operations 154627 Production error message (when servers are down) points users to donate link which is likely to produce the same error message In-Scope Open None
Other Operations 174959 swift-recon-cron on ms-be203[34]: [Errno 17] File exists: '/var/lock/swift-recon-object-cron' In-Scope Open None
Other Operations 207804 Upgrade calico in production to version 2.4+ Screep Open None
Other Operations 116627 Include 5xx numbers in fluorine fatalmonitor In-Scope Open None
Other Operations 210034 build grafana package for stretch Screep Open None
Other Operations 210592 Fix prometheus elasticsearch exporter to show all the metrics Screep Open None
Other Operations 196474 Externalize tile storage for maps In-Scope Open None
Other Operations 94329 secure Cassandra/RESTBase cluster In-Scope Open None
Other Operations 111653 Encrypt all the things In-Scope Open None
Other Operations 135595 mod_deflate + mod_uwsgi causing mangled apache responses In-Scope Open None
Other Operations 153246 Puppet failures with "Attempt to assign to a reserved variable name: 'trusted'" In-Scope Open None
Other Operations 149804 Review of ferm services without srange In-Scope Open None
Other Operations 83729 Fix monitoring of poolcounter service In-Scope Open None
Other Operations 203260 Outdated TLS config for MXes In-Scope Open None
Other Operations 209812 Review Elastic/maps Grafana dashboards Screep Open None
Other Operations 193155 IPMI Audit 2018-04 In-Scope Open None
Other Operations 163286 Tegmen: process spawn loop + failed icinga + failing puppet In-Scope Open None
Other Operations 169564 MD RAID: remove mdadm daily check In-Scope Open None
Other Operations 191627 Remove Cassandra 2.2.6 packages from jessie-wikimedia/thirdparty apt repo In-Scope Open None
Other Operations 178445 flapping monitoring for recommendation_api on scb In-Scope Open None
Other Operations 209389 rack/setup/install sessionstore200[123].codfw.wmnet Screep Open None
Other Operations 184634 Netbox: postgres cannot be restarted w/ current config In-Scope Open None
Other Operations 141959 Moving network::external to hiera broke much of labs In-Scope Open None
Other Operations 140316 Add granularity limiter (g=) to wikimedia.org DKIM record(s) In-Scope Open None
Other Operations 155209 Increase $wgHTTPImportTimeout to a higher value on WMF wikis In-Scope Open None
Other Operations 204032 Support meta tag refresh redirects in citoid to support elsevier's linking hub In-Scope Open None
Other Operations 182832 Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state In-Scope Open None
Other Operations 140813 Protect sensitive user-related information with a UserData / auth / session service In-Scope Open None
Other Operations 110171 Alert when ES indexes are freezed for more than 30 minutes In-Scope Open None
Other Operations 211668 mw1272 crashed: Bad page map in process hhvm Screep Open None
Other Operations 205849 Begin the implementation of Q1's Logging Infrastructure design (2018-19 Q2 Goal) Screep Open None
Other Operations 193628 tungsten disk 1 and 8 SMART failure In-Scope Open None
Other Operations 166038 Sync internal nutcracker package with Debian package In-Scope Open None
Other Operations 134551 Create functional cluster checks for all services (and have them page!) In-Scope Open None
Other Operations 206185 connect atlas-ulsfo to scs-ulsfo Screep Open None
Other Operations 205911 Track and install additional npm packages for all service container images Screep Open None
Other Operations 209507 issue pulling 1 layer of docker-registry.wikimedia.org/releng/composer-php71:latest Screep Open None
Other Operations 197003 Dismantle most of the old jobqueue infrastructure In-Scope Open None
Other Operations 163033 Create grafana dashboard for video scaler job runners In-Scope Open None
Other Operations 178592 decommission/replace bast4001.wikimedia.org In-Scope Open None
Other Operations 209616 rack/setup/install cloudvirt10[25-30].eqiad.wmnet Screep Open None
Other Operations 91404 Setup backups of elasticsearch indices In-Scope Open None
Other Operations 147923 Extract metrics from logs In-Scope Open None
Other Operations 174269 Two cases of local-multiwrite storage backend failure In-Scope Open None
Other Operations 163698 Add flood protection to the ircecho bot (icinga-wm) In-Scope Open None
Other Operations 118829 Automate the provisioning and management of MediaWiki clusters In-Scope Open None
Other Operations 198215 systemd-logind fails with result 'timeout' in db2093 and dns4001 In-Scope Open None
Other Operations 208813 Scap deployers should have the ability to depool and restart HHVM Screep Open None
Other Operations 197084 Report problems found in server's IPMI SEL In-Scope Open None
Other Operations 199427 Separate dev Change-Prop from production Kafka cluster In-Scope Open None
Other Operations 148017 lvs2002 repeated usb connect/disconnect message In-Scope Open None
Other Operations 150917 Remove deprecated features from book creator UI In-Scope Open None
Other Operations 184714 Puppet fail to properly refresh Icinga In-Scope Open None
Other Operations 209182 netbox won't allow me to upload photos of the rack Screep Open None
Other Operations 151048 Icinga monitoring for Yubikey components In-Scope Open None
Other Operations 200678 wtp2011 memory correctable errors In-Scope Open None
Other Operations 142002 Clean up puppet & configs for ORES In-Scope Open None
Other Operations 117508 Make ops-l a list for humans again (no cheating) In-Scope Open None
Other Operations 105780 Create a doc explaining the SLA between services and the monitoring tool In-Scope Open None
Other Operations 125411 Diamond load averages do not contain scaled versions In-Scope Open None
Other Operations 148986 Firewall sets not being loaded post-reboot due to a @resolve race on jessie In-Scope Open None
Other Operations 198901 Migrate production services to kubernetes using the pipeline In-Scope Open None
Other Operations 116951 Reprepro should bail if it can't read and sign using the root keys In-Scope Open None
Other Operations 185215 Puppet compiler failure to lookup some keys In-Scope Open None
Other Operations 202574 convert cloud VPS projects from apache to httpd module In-Scope Open None
Other Operations 194966 disk usage increase on maps servers In-Scope Open None
Other Operations 103886 Translation cache exhaustion caused by changes to PHP code in file scope In-Scope Open None
Other Operations 178690 Better organization for SRE grafana dashboards In-Scope Open None
Other Operations 94457 Install nodejs, nginx and other dependencies on francium In-Scope Open None
Other Operations 187101 Setup some alert mechanism when some 'critical' cron jobs fail In-Scope Open None
Other Operations 203959 SRE quarterly goal: allow MediaWiki requests to be served by PHP7 alongside HHVM In-Scope Open None
Other Operations 179696 Homepage for https://docker-registry.wikimedia.org In-Scope Open None
Other Operations 207041 Adapt Kafka dashboards to use metrics from prometheus-node-exporter Elaborated Open None
Other Operations 175738 Long term storage for frack prometheus data In-Scope Open None
Other Operations 187673 Build and deploy hhvm-luasandbox 3.0.1 to Wikimedia wikis In-Scope Open None
Other Operations 211747 Graphite generates a lot of 502 in Grafana Screep Open None
Other Operations 156570 Investigate issues with wikitech-static.wikimedia.org In-Scope Open None
Other Operations 188098 Add Prometheus collector for Tor In-Scope Open None
Other Operations 174916 electron/pdfrender hangs In-Scope Open None
Other Operations 200706 rack/setup/install centrallog1001.eqiad.wmnet In-Scope Open None
Other Operations 111595 Do not apply spam headers on email assessed NOT to be spam In-Scope Open None
Other Operations 150466 publish kartotherian / tilerator metrics by cluster In-Scope Open None
Other Operations 123560 investigate rsync between dcs with encryption In-Scope Open None
Other Operations 195364 Remove pear/mail packages from WMF MW app servers In-Scope Open None
Other Operations 210833 Filter potentially harmful PostScript commands in Commons upload/thumbor Screep Open None
Other Operations 152445 Move prometheus entry point off port 80 In-Scope Open None
Other Operations 119274 Check incoming requests to secure.wm.o In-Scope Open None
Other Operations 131326 smokeping config puppetization issue? In-Scope Open None
Other Operations 170817 Upgrade Thumbor servers to Stretch In-Scope Open None
Other Operations 211684 Toolforge: Port sge.py stats to Prometheus Screep Open None
Other Operations 129963 Update memcached package and configuration options In-Scope Open None
Other Operations 123276 URL parameters do not work with pages that have "?" in their names In-Scope Open None
Other Operations 168403 Aggregate prometheus functions yielding different results in grafana vs. prometheus console In-Scope Open None
Other Operations 203239 Create Debian packages for Node.js 10 upgrade In-Scope Open None
Other Operations 211250 Create a mediawiki::cronjob define Screep Open None
Other Operations 160941 Improve SSH access information in onboarding documentation In-Scope Open None
Other Operations 209642 Remove labnodepool1001.eqiad.wmnet Screep Open None
Other Operations 203092 Create Graphoid .pipeline files In-Scope Open None
Other Operations 119719 Enforce a minimum refresh period for grafana dashboards hitting graphite In-Scope Open None
Other Operations 125015 Requests to (hard) redirect pages return their target's contents but are counted as pageviews to the redirect page In-Scope Open None
Other Operations 204907 Scap is checking canary servers in dormant instead of active-dc In-Scope Open None
Other Operations 149589 Puppet tab in Horizon unusably slow In-Scope Open None
Other Operations 170740 PuppetDB misbehaving on 2017-07-15 In-Scope Open None
Other Operations 167091 Elasticsearch errors about BulkShardRequest In-Scope Open None
Other Operations 163288 Decide on /var/lib vs /home as locations of homedir for l10nupdate In-Scope Open None
Other Operations 94951 Enable the usage of `hhvm -m debug --debug-host ::1` from mw1017 so developers can step through code (think gdb) in production to see what is going wrong. In-Scope Open None
Other Operations 196507 Degraded RAID on cloudvirt1019 In-Scope Open None
Other Operations 138685 notebook1001 shown as DOWN in icinga, due to firewall rules In-Scope Open None
Other Operations 195421 update physical labels from naos.codfw.wmnet to deploy2001.codfw.wmnet In-Scope Open None
Other Operations 163667 Fix UIDs for deployment server users In-Scope Open None
Other Operations 203272 cp3038, cp3039 - power supply redundancy failure In-Scope Open None
Other Operations 209921 ms-be2047 spontaneous reboots Screep Open None
Other Operations 171619 [Epic] ORES should use a git large file plugin for storing serialized binaries In-Scope Open None
Other Operations 198784 Degraded RAID on cp3048 In-Scope Open None
Other Operations 78135 Provide a pxe-bootable rescue image In-Scope Open None
Other Operations 209881 Introduce state to Scap Screep Open None
Other Operations 161296 Upgrade mysqld_exporter in production In-Scope Open None
Other Operations 159242 Segmentation fault creating thumbnail In-Scope Open None
Other Operations 150872 Replace OCG in collection extension with Electron In-Scope Open None
Other Operations 108985 Monitor MediaWiki sessions In-Scope Open None
Other Operations 187987 Serve >= 50% of production Prometheus systems with Prometheus v2 In-Scope Open None
Other Operations 165631 move gerrit.wm.org SSH service to private/behind LVS like phab-vcs In-Scope Open None
Other Operations 191357 decom silver/WMF3434 In-Scope Open None
Other Operations 135991 Automated service restarts for common low-level system services In-Scope Open None
Other Operations 116750 2FA for SSH access to the production cluster In-Scope Open None
Other Operations 184063 Remove all decommissioned hardware In-Scope Open None
Other Operations 151317 stat user crontab on stat hosts for old file removal In-Scope Open None
Other Operations 211982 Find links to grafana.wikimedia.org and change them to use the new URL format Screep Open None
Other Operations 141704 Unable to delete certain files due to "inconsistent state within the internal storage backends" In-Scope Open None
Other Operations 212010 Degraded RAID on sodium Screep Open None
Other Operations 179568 all mailing lists should have descriptions Screep Open None
Other Operations 193766 Ship host syslogs to ELK In-Scope Open None
Other Operations 187651 Setting packages on 'hold' breaks puppet runs In-Scope Open None
Other Operations 157972 Puppet fails only once when restarting ferm is not successful In-Scope Open None
Other Operations 194249 kafka1023 correctable memory errors In-Scope Open None
Other Operations 179181 Puppet4: hiera() can only be called using the 4.x function API. In-Scope Open None
Other Operations 198209 Graphite returning 500 @ nagf and graphite url In-Scope Open None
Other Operations 133164 Document eqiad/codfw transition plan for OCG In-Scope Open None
Other Operations 190716 Deploying FileExporter and FileImporter In-Scope Open None
Other Operations 163339 codfw pdu phase inbalances: audit and correct In-Scope Open None
Other Operations 211231 ms-be raid setup / evaluation (currently using swraid on top of hwraid) Screep Open None
Other Operations 210850 WMCS-related dashboards using Diamond metrics Screep Open None
Other Operations 210108 icinga1001 mysterious reboots Screep Open None
Other Operations 200690 Wrong umask when deploying from screen In-Scope Open None
Other Operations 200210 Decom graphite2002 In-Scope Open None
Other Operations 203169 Logstash hardware expansion In-Scope Open None
Other Operations 194171 rdb2002 correctable memory errors In-Scope Open None
Other Operations 200984 Stop introducing new code expanded from erb templates In-Scope Open None
Other Operations 212429 Allow the deployment of users to a host without their ssh key via the admin module Screep Open None
Other Operations 161835 Convert labstore cluster configuration to hiera and profiles In-Scope Open None
Other Operations 156955 Standardizing our partman recipes In-Scope Open None
Other Operations 141128 determine/process/document bios firmware tracking/updating policies In-Scope Open None
Other Operations 211902 create IRC channel for the Service Operations SRE subteam Screep Open None
Other Operations 199479 Add alerts for Logstash rates in production In-Scope Open None
Other Operations 212212 eqiad: 1-2 VM requests for docker-registry-beta.wikimedia.org Screep Open None
Other Operations 207693 Evaluate (and potentially implement) upgrade of docker-engine to docker-ce 17+ for production (kubernetes) Screep Open None
Other Operations 186734 Clean up redundant ORES celery_workers defaults In-Scope Open None
Other Operations 201366 rack/setup/install scandium.eqiad.wmnet (parsoid test box) In-Scope Open None
Other Operations 205856 Retire udp2log: onboard its producers and consumers to the logging pipeline Screep Open None
Other Operations 194174 wtp2013 memory correctable errors In-Scope Open None
Other Operations 160158 Make disabled accounts visible in the corp mirror LDAP replica In-Scope Open None
Other Operations 161004 Remove disabled users from internal mailing lists In-Scope Open None
Other Operations 212504 Remove OOMScoreAdjust from nrpe unit file? Screep Open None
Other Operations 167376 Decommission cp300[3456] In-Scope Open None
Other Operations 207739 Access requests process: Consideration of 'indirect' sudo rules via e.g. keyholder Screep Open None
Other Operations 151046 Fully puppetise yubikey-val In-Scope Open None
Other Operations 131966 Default gateway unreachable on baham.wikimedia.org after reboot In-Scope Open None
Other Operations 184924 Utilize the deployment pipeline (stretch) In-Scope Open None
Other Operations 149543 Setup PAWS internal experimentally on notebook* nodes In-Scope Open None
Other Operations 199356 Contrabass MIDI instrument is unusable In-Scope Open None
Other Operations 182228 run-no-puppet leave puppet disabled on kill/crash In-Scope Open None
Other Operations 212424 restbase cassandra driver excessive logging when cassandra hosts are down Screep Open None
Other Operations 174475 update firmware on scs consoles In-Scope Open None
Other Operations 134237 Graphoid returns a 400 on MW API time-out In-Scope Open None
Other Operations 55457 setup a DB backed parser cache In-Scope Open None
Other Operations 209260 Integrate Stretch 9.6 point update Screep Open None
Other Operations 191362 decom promethium/WMF3571 In-Scope Open None
Other Operations 209139 Broken memory on mw1239 Screep Open None
Other Operations 179099 puppetmaster hostcert and hostprivkey point to nonexistent files In-Scope Open None
Other Operations 95054 Move ircecho config file to be YAML In-Scope Open None
Other Operations 166081 rack/setup/install conf1004-conf1006 In-Scope Open None
Other Operations 163393 Determine appropriate proxy_read_timeout setting for Tools Proxy In-Scope Open None
Other Operations 165618 Audit / document reasons for not enabling HT? In-Scope Open None
Other Operations 211139 Convert Gerrit to use H2 as the database after 2.16 upgrade Screep Open None
Other Operations 208783 Migrate tests from nose to pytest Screep Open None
Other Operations 148843 GPU upgrade for stats machine In-Scope Open None
Other Operations 185306 ms-be2023 unresponsive while rebuilding one disk In-Scope Open None
Other Operations 168767 Monitor PostgreSQL connection slots In-Scope Open None
Other Operations 82937 re-create script for manual paging In-Scope Open None
Other Operations 140141 Install mscorefonts on scaling servers for SVG rendering In-Scope Open None
Other Operations 190568 Reimage both phab1001 and phab2001 to stretch In-Scope Open None
Other Operations 203625 mwdebug1001 and mwdebug1002 are reliably the last two hosts to finish scap-cdb-rebuild In-Scope Open None
Other Operations 97368 Investigate more efficient memcached solution for CacheAwarePropertyInfoStore In-Scope Open None
Other Operations 207965 eqiad: Re-connect cage cameras Screep Open None
Other Operations 171188 Move the main WMCS puppetmaster into the Labs realm In-Scope Open None
Other Operations 133656 Have a paging check for Nova API accessible In-Scope Open None
Other Operations 158757 Puppet certificate missing subjectAltName In-Scope Open None
Other Operations 208231 Issues with purgeUnusedProjects.php cron job on mwmaint1002 (Fri Oct 26) Screep Open None
Other Operations 206626 Decommission conf100[1-3] Screep Open None
Other Operations 148061 Feasibility of hosting podcast setup on Wikimedia servers In-Scope Open None
Other Operations 142984 Review lists of config/sysctl recommendations by "kernel self-protection project" In-Scope Open None
Other Operations 151047 Integrate Yubikey into data.yaml In-Scope Open None
Other Operations 186288 replace all Ubuntu (trusty) hosts in production with Debian In-Scope Open None
Other Operations 127054 pinentry-gtk2 pulls in a lot of unneeded Gnome/GTK libs In-Scope Open None
Other Operations 185024 Readd complete URL parsing fix from 3.18.7 release In-Scope Open None
Other Operations 148048 Store Wikimedia unified account name (SUL) in LDAP directory In-Scope Open None
Other Operations 126158 [RFC] Alert about *when* partitions will run out of space, not a percentage/absolute number In-Scope Open None
Other Operations 211512 "sql" command fails with "sh: 1: mysql: not found" on mwdebug1002 Screep Open None
Other Operations 204450 Why doesn't profile::mediawiki::nutcracker create /var/run/nutcracker/ ? In-Scope Open None
Other Operations 210723 Address recurrent service check time out for "HP RAID" on swift backend hosts Screep Open None
Other Operations 210991 Deprecate Diamond collectors in Tool Labs / Tool Forge Elaborated Open None
Other Operations 122917 Provide a good download service of dumps from Wikimedia In-Scope Open None
Other Operations 100777 expose hosts in maintenance state so we can prevent scap from running on them In-Scope Open None
Other Operations 208257 stop using mod_php anywhere Screep Open None
Other Operations 153816 apache::static_site is not working In-Scope Open None
Other Operations 137176 catch-all apache vhost on the cluster should return 404 for non-existing sites In-Scope Open None
Other Operations 172584 Securing external binaries run by MediaWiki In-Scope Open None
Other Operations 167292 Collate jessie-wikimedia/backports into jessie-wikimedia/main In-Scope Open None
Other Operations 152782 Kibana functionality missing after upgrade: histograms In-Scope Open None
Other Operations 170456 FY2017/18 Program 6 - Outcome 2 - Objective 3: Integrated, container-based development environment In-Scope Open None
Other Operations 170480 FY2017/18 Program 6 - Outcome 2: Developers are able to develop and test their applications through a unified pipeline towards production deployment. In-Scope Open None
Other Operations 85451 scale graphite deployment (tracking) In-Scope Open None
Other Operations 186073 Rack/setup frmon1001 In-Scope Open None
Other Operations 128590 Cassandra uses default ip address for outbound packets while bootstrapping In-Scope Open None
Other Operations 212681 /dev/log symlink to /run/systemd/journal/dev-log disappeared on kubernetes1001 Screep Open None
Other Operations 126989 MediaWiki logging & encryption In-Scope Open None
Other Operations 209181 Decommission rdb1001, rdb1002, rdb1003, rdb1004, rdb1007, rdb1008 Screep Open None
Other Operations 185055 Stack overflow when Redis is down In-Scope Open None
Other Operations 197172 Improve outbound mail service alerting In-Scope Open None
Other Operations 199676 Community Relations support for the 2018 data center switchover In-Scope Open None
Other Operations 190111 VirtualHost for mod_status breaks debugging Apache/MediaWiki from localhost In-Scope Open None
Other Operations 116747 Meta task "Revamp user authentication" In-Scope Open None
Other Operations 205567 PHP Warning "Unable to delete stat cache" from file uploads In-Scope Open None
Other Operations 119718 Make it easier to ban misbehaving dashboards from graphite In-Scope Open None
Other Operations 199008 sql enwik gives a poor error message when db doesn't exist In-Scope Open None
Other Operations 120532 Use user-specific passwords for accessing EventLogging database In-Scope Open None
Other Operations 182597 Use EtcdConfig in production to allow automation of a datacenter switch In-Scope Open None
Other Operations 204267 Flood of WDQS requests from wbqc In-Scope Open None
Other Operations 205618 rsync puppet module doesn't delete removed config In-Scope Open None
Other Operations 205619 Upload to Commons fails with a common ADSL connection in Taiwan Screep Open None
Other Operations 196476 rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech In-Scope Open None
Other Operations 170108 Operations Q1 goal: Streamlined Service Delivery In-Scope Open None
Other Operations 209810 Ingest swift access logs for thumbnail/original analysis Screep Open None
Other Operations 140879 Request timeout while loading Wikidata:Database_reports/Constraint_violations/P570&curid=15087958&diff=358447430&oldid=358294930 In-Scope Open None
Other Operations 184564 Plan Puppet 5 upgrade In-Scope Open None
Other Operations 167245 prometheus-node-exporter - invalid group: ‘prometheus:prometheus' In-Scope Open None
Other Operations 190318 remove puppet_major_version and puppetdb_major_version variables. clean up puppet master/db hieradata In-Scope Open None
Other Operations 198790 Relabel hooft to bast3002 In-Scope Open None
Other Operations 163336 kube-proxy pulls in docker and starts service even when it isnt needed In-Scope Open None
Other Operations 179230 Puppet wmf-style-guide: array of classes not detected properly In-Scope Open None
Other Operations 159536 Puppet constantly trying to stop the already stopped puppetmaster process on Trusty In-Scope Open None
Other Operations 210008 upgrade krypton (webserver_misc_apps) to stretch Screep Open None
Other Operations 133179 Redis monitoring needs to be improved In-Scope Open None
Other Operations 167422 Monitoring: add link to graph for Icinga timeseries alarms In-Scope Open None
Other Operations 197086 Report problems found by mcelog In-Scope Open None
Other Operations 130617 Collect metrics on CirrusSearch usage of PoolCounter In-Scope Open None
Other Operations 194012 labsdb1004 and labsdb1005 some hard disks not healthy In-Scope Open None
Other Operations 118746 Goal: Strengthen Incident monitoring infrastructure In-Scope Open None
Other Operations 154619 Export ipsec counters as Prometheus metrics In-Scope Open None
Other Operations 197624 Improve visibility of incoming operations tasks In-Scope Open None
Other Operations 212418 Memory error on restbase1016 Screep Open None
Other Operations 207178 logstash HTTP Basic Auth prompt says "WMF Labs" Screep Open None
Other Operations 199251 furud: disconnect and power down all disk shelves In-Scope Open None
Other Operations 122825 Service Ownership and Maintenance In-Scope Open None
Other Operations 116580 monitor postgresql replication status In-Scope Open None
Other Operations 210071 debianize docker-registry 2.7.0-rc0 and upload in stretch-wikimedia Screep Open None
Other Operations 206636 Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service Screep Open None
Other Operations 158562 Manage apt sources via puppet? In-Scope Open None
Other Operations 184061 SRE 2017-18 Q3 goal Cleanup esams and refresh servers and infrastructure (tracking) In-Scope Open None
Other Operations 198622 migrate maps servers to stretch with the current style In-Scope Open None
Other Operations 178839 New upstream jvm-tools In-Scope Open None
Other Operations 170453 FY2017/18 Program 6: Streamlined Service delivery In-Scope Open None
Other Operations 86552 Monitor and alarm on SMART attributes In-Scope Open None
Other Operations 183146 Monitor resource usage on a per-cgroup basis In-Scope Open None
Other Operations 187474 Decommission old and unused/spare servers in codfw In-Scope Open None
Other Operations 196994 Open Phab tasks on SMART failure In-Scope Open None
Other Operations 89808 wikitech instances list is blank In-Scope Open None
Other Operations 176437 puppet ca_server confusion In-Scope Open None
Other Operations 185298 xfs_db blocked / timeout on ms-be2023 In-Scope Open None
Other Operations 160529 Sender email spoofing In-Scope Open None
Other Operations 205870 Provision >= 50% of statsd/Graphite-only metrics in Prometheus Screep Open None
Other Operations 187994 netfilter software at WMF: iptables vs nftables In-Scope Open None
Other Operations 151009 Provide authenticated access to Prometheus native web interface In-Scope Open None
Other Operations 124101 Specific revisions of multiple files missing from Swift - 404 Not Found returned In-Scope Open None
Other Operations 206152 Set up request profiling for PHP 7 Elaborated Open None
Other Operations 143552 Make elasticsearch configuration more robust to loss of network connectivity In-Scope Open None
Other Operations 199228 Define an SLO for Wikidata Query Service public endpoint and communicate it In-Scope Open None
Other Operations 173721 Track down the source of periodic increases in requests to swift eqiad In-Scope Open None
Other Operations 167412 host-vmem.erb is doing operations that make no sense In-Scope Open None
Other Operations 106937 Monitor [[Special:ListFiles]] for non 200 HTTP statuses in thumbnails In-Scope Open None
Other Operations 130554 Official support for upgrade from existing Mailman 2.1 lists to Mailman 3 In-Scope Open None
Other Operations 125976 Run mediawiki::maintenance scripts in Beta Cluster In-Scope Open None
Other Operations 199431 Consider the possibility of separating ChangeProp and JobQueue on Kafka level In-Scope Open None
Other Operations 194036 mw1230 sdb "Raw_Read_Error_Rate" SMART In-Scope Open None
Other Operations 158434 Phabricator: Make sure phabricator works properly including our puppet roles on jessie In-Scope Open None
Other Operations 176370 Migrate to PHP 7 in WMF production In-Scope Open None
Other Operations 207285 wmf-style adds 'has no call to hiera' violations for parameters already containing hiera calls Screep Open None
Other Operations 208934 mcrouter does not remove a memcached shard from consistent hashing when timeouts happen Elaborated Open None
Other Operations 101585 document redis upgrade/restart procedures In-Scope Open None
Other Operations 183381 Deploy JADE extension to production In-Scope Open None
Other Operations 177197 Export Prometheus-compatible JVM metrics from JVMs in production In-Scope Open None
Other Operations 201016 Include ADD operation in memcached stats and grafana dashboard In-Scope Open None
Other Operations 209029 cloudelastic1004: SMART/disk error Screep Open None
Other Operations 209101 ulsfo: install new PDUs in racks / phase out APC loaner PDU use Screep Open None
Other Operations 212674 Update Wikimedia logo on Mailman web pages from colored version to black and white version Screep Open None
Other Operations 186625 apply hostname labels to bast1002/WMF4749 In-Scope Open None
Other Operations 128821 reclaim and return all cisco servers In-Scope Open None
Other Operations 193408 SPF record for canonical domains In-Scope Open None
Other Operations 150875 Confirm attribution needs In-Scope Open None
Other Operations 204024 Store WikibaseQualityConstraint check data in persistent storage instead of in the cache Screep Open None
Other Operations 184461 Discourse migration from wmflabs to production In-Scope Open None
Other Operations 165885 Create a cron to clean clientbucket every day or hour In-Scope Open None
Other Operations 205851 Migrate >=90% of existing Logstash traffic to the logging pipeline Screep Open None
Other Operations 212640 logstash stuck on its persistent queue Screep Open None
Other Operations 114801 operations-apache-config-lint replacement doesn't check syntax In-Scope Open None
Other Operations 150822 Internal PKI for secure communication - Barcelona Ops offsite 2016 In-Scope Open None
Other Operations 180051 Reduce the number of fields declared in elasticsearch by logstash In-Scope Open None
Other Operations 182822 Generate a list of files that are supposed to exist but 404s In-Scope Open None
Other Operations 135385 investigate carbon-c-relay stalls/drops towards graphite2002 In-Scope Open None
Other Operations 175206 2017/18 Annual Plan Program 8: Multi-datacenter support In-Scope Open None
Other Operations 159480 Decommission bast3001 In-Scope Open None
Other Operations 205577 Puppet agent takes a long time to finish when adding IPv6 addresses In-Scope Open None
Other Operations 205507 Decommission analytics100[1,2] In-Scope Open None
Other Operations 211401 Wikipedia.org DMARC "rua" and "ruf" email addresses need verification Screep Open None
Other Operations 160060 Icinga check for sysctl settings In-Scope Open None
Other Operations 209618 rack/setup/install ms-be10[44-50].eqiad.wmnet Screep Open None
Other Operations 161528 incident 20170323-wikibase did not trigger Icinga paging In-Scope Open None
Other Operations 141897 Review new service 'pre-deployment to production' checklist In-Scope Open None
Other Operations 163673 Some swift disks wrongly mounted on 5 ms-be hosts In-Scope Open None
Other Operations 136311 Monitor the BMC's event log for hardware errors In-Scope Open None
Other Operations 211273 Cron <root@mwdebug2002> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) Screep Open None
Other Operations 142205 use granularity (g=) restrictions for wikimedia.org fundraising DKIM records In-Scope Open None
Other Operations 116063 Hardware Automation Workflow - Overall Tracking In-Scope Open None
Other Operations 132104 Consider moving policy.wikimedia.org away from WordPress.com In-Scope Open None
Other Operations 130593 investigate slapd memory leak In-Scope Open None
Other Operations 130883 decom cp3011-22 (12 machines) In-Scope Open None
Other Operations 202898 Decommission maps-test cluster In-Scope Open None
Other Operations 200022 2018 data center switchover: Move all the things over to codfw In-Scope Open None
Other Operations 154915 Get rid of "import realm.pp" in manifests/site.pp In-Scope Open None
Other Operations 206761 Create a jessie netboot image with the 4.9 Linux kernel Screep Open None
Other Operations 116805 DomainKeys Identified Mail (DKIM) for phabricator.wikimedia.org In-Scope Open None
Other Operations 184230 Disavow emails from wikipedia.com In-Scope Open None
Other Operations 150871 [EPIC] (Proposal) Replicate core OCG features and sunset OCG service In-Scope Open None
Other Operations 207292 Review prometheus_nodes params Screep Open None
Other Operations 159830 Sanity check global-multiwrite logs for ConfirmEdit usage In-Scope Open None
Other Operations 206336 SRE quarterly goal: Ability to serve a fraction of the production traffic from PHP7 Screep Open None
Other Operations 203883 Implement MTA-STS In-Scope Open None
Other Operations 202061 Implement an accurate and easy to understand status page for all wikis In-Scope Open None
Other Operations 167966 Look into feasibility of disabling sha-1 host keys on our ssh daemons In-Scope Open None
Other Operations 194186 rack/setup/install cloudelastic100[1-4].eqiad.wmnet systems In-Scope Open None
Other Operations 95801 Allow customizing the alert message from graphite In-Scope Open None
Other Operations 159661 Improve mwmaint servers (e.g. mwmain1001) userland to process server side uploads In-Scope Open None
Other Operations 199321 Return graphite200[12] to spares pool In-Scope Open None
Other Operations 191388 Puppet: tracking catalogs that changes at every run In-Scope Open None
Other Operations 168460 Update certificates on productions replicas of corp.wikimedia.org LDAP In-Scope Open None
Other Operations 188602 Decrease the amount of IRC spam in case of widespread puppet failures In-Scope Open None
Other Operations 207296 Rationalize default logrotate "rotated" file extensions Screep Open None
Other Operations 169287 etcd config depends on puppet certs, but puppet doesn't know In-Scope Open None
Other Operations 207837 wdqs updater should be better isolated from blazegraph and common workload should be shared between servers Screep Open None
Other Operations 187078 Re-consider ` >/dev/null 2>&1` as output of many cron'd MW maintenance scripts In-Scope Open None
Other Operations 113785 Make the Shinken IRC alert and icinga-wm bots use colors In-Scope Open None
Other Operations 175213 2017/18 Annual Plan Program 8: Multi-datacenter support, Q2 goals In-Scope Open None
Other Operations 180641 reinstall RT server with private IP and stretch In-Scope Open None
Other Operations 177371 Phase out DSA keys for SSH access (ssh-dss) In-Scope Open None
Other Operations 182249 Diagnose and fix 4.5k req/min ceiling for ores* requests In-Scope Open None
Other Operations 211414 Request for information about hosting services for WM-ES Screep Open None
Other Operations 209890 Memory consumption in Redis 3.2 vs Redis 2.8 Screep Open None
Other Operations 197564 cronspam for slow queries in PageAssessments In-Scope Open None
Other Operations 209393 rack/setup/install sessionstore100[123].eqiad.wmnet Screep Open None
Other Operations 148614 Icinga check for Tor In-Scope Open None
Other Operations 212556 frdb1001 RAID controller battery failure Screep Open None
Other Operations 126295 Spike: What do we have to package to run the Programs and Events dashboard on production? In-Scope Open None
Other Operations 211411 Citoid automated monitoring times out due to Zotero v2 Screep Open None
Other Operations 161003 Cross-check disabled accounts from corp LDAP against data.yaml In-Scope Open None
Other Operations 142815 Enhance account handling (meta bug) In-Scope Open None
Other Operations 179078 mpt raid controller not detected as fact on maps-test2* In-Scope Open None
Other Operations 128716 Make icinga-wm report Tools homepage check at #wikimedia-labs, too In-Scope Open None
Other Operations 169286 labstore1005 A PCIe link training failure error on boot In-Scope Open None
Other Operations 142821 Synchronise groups defined in data.yaml to LDAP In-Scope Open None
Other Operations 212697 uwsgi's logsocket_plugin.so causes segfaults during log rotation Screep Open None
Other Operations 158837 Consolidate performance website and related software In-Scope Open None
Other Operations 110240 [Discussion] Consider validating JSON schemas when running x-ample tests? In-Scope Open None
Other Operations 203003 Keyholder phab repo duplicate work In-Scope Open None
Other Operations 211580 blubber template for nodejs should allow defining configuration files to copy to the container Screep Open None
Other Operations 171157 Monitor internal CA expirations In-Scope Open None
Other Operations 162029 Migrate all jessie hosts to Linux 4.9 In-Scope Open None
Other Operations 86546 graphite-web logs are not rotated In-Scope Open None
Other Operations 203091 Move Graphoid to Kubernetes via the deployment pipeline In-Scope Open None
Other Operations 135318 Document how to handle 'inconsistent state within the internal storage backends' issues In-Scope Open None
Other Operations 124991 evaluate possibility for nscd use with useldap In-Scope Open None
Other Operations 166937 Broken /a/refinery-source/guard/run_all_guards.sh script on stat1002 In-Scope Open None
Other Operations 135124 Deploy etcddump (or another etcd dump & load tool) to production In-Scope Open None
Other Operations 204857 notebook1003 failed network mount on boot In-Scope Open None
Other Operations 150823 Puppet CA rollover In-Scope Open None
Other Operations 141520 "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) In-Scope Open None
Other Operations 133844 Improve Elasticsearch icinga alerting In-Scope Open None
Other Operations 185236 Password Vault for Security Team In-Scope Open None
Other Operations 129847 conftool-merge should report which node is setting attributes for In-Scope Open None
Other Operations 198256 RFC: Modern Event Platform - Choose Schema Tech In-Scope Open None
Other Operations 206131 add monitoring to alert on hosts without RAID Screep Open None
Other Operations 179022 Backport firejail 0.9.52 for use on Wikimedia appservers In-Scope Open None
Other Operations 185189 scap sudo violation on first puppet run In-Scope Open None
Other Operations 84163 Fix CirrusSearch monitoring In-Scope Open None
Other Operations 119660 Set up LVS for labs dns recursors In-Scope Open None
Other Operations 163068 More missing 'original' files on Commons In-Scope Open None
Other Operations 180944 Passenger spews Exception NoMethodError in Rack application object In-Scope Open None
Other Operations 84700 Setup management switch in OE12 In-Scope Open None
Other Operations 123918 'swift' user/group IDs should be consistent across the fleet In-Scope Open None
Other Operations 84845 improve cron spam visibility In-Scope Open None
Other Operations 146657 create notifications about user accounts that have not been used for a long time In-Scope Open None
Other Operations 184561 Modernize Puppet Configuration Management (2017-18 Q3 Goal) In-Scope Open None
Other Operations 145065 Decrease time required to fully restart the Cirrus elasticsearch clusters In-Scope Open None
Other Operations 211271 Cron <root@labweb1001> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) Screep Open None
Other Operations 193352 Update librsvg to ≥2.42.3 Elaborated Open None
Other Operations 204845 logstash-beta.wmflab throws multiple "Error: Could not locate that visualization" In-Scope Open None
Other Operations 141524 eventbus should send statsd in batches In-Scope Open None
Other Operations 205897 Netbox: fill network topology Screep Open None
Other Operations 189921 decom californium In-Scope Open None
Other Operations 180330 Add CI to all operations/* repositories and archive obsolete ones In-Scope Open None
Other Operations 191659 Configure a threshold for earlier notification of /srv/cassandra/instance-data In-Scope Open None
Other Operations 206649 Lessons learned: Communicating the server switch 2018 Elaborated Open None
Other Operations 158288 Unclean stop of jobrunner service via puppet In-Scope Open None
Other Operations 206327 Setup metrics monitoring for OpenLDAP/corp Elaborated Open None
Other Operations 118641 Implement proper AAA for lists.wikimedia.org (mailman) In-Scope Open None
Other Operations 210038 upgrade install servers to stretch Screep Open None
Other Operations 211403 Domains of most projects do not have DMARC policy Screep Open None
Other Operations 187991 Have swift metrics available in Prometheus In-Scope Open None
Other Operations 180853 Bring a discourse instance for technical questions to production In-Scope Open None
Other Operations 95052 Make ircecho much better In-Scope Open None
Other Operations 210289 Upgrade Ganeti clusters to 2.15.2-7+deb9u3 Screep Open None
Other Operations 210735 decommission labvirt101[01].eqiad.wmnet (Dec 2018 lease return) Screep Open None
Other Operations 159687 etcd switchover/enhancements In-Scope Open None
Other Operations 156143 High CPU usage from swift-proxy on frontend machines In-Scope Open None
Other Operations 203645 rspec-puppet fails with Could not find the daemon directory (tested [/etc/sv,/var/lib/service]) In-Scope Open None
Other Operations 151050 Proper documentation for Yubico 2FA for production use In-Scope Open None
Other Operations 201358 Increase job runners on video scalers to maximize load efficiency In-Scope Open None
Other Operations 209946 hhvm systemd service on deployment-prep reports: hhvm.service: Ignoring invalid environment assignment 'RUN_AS_GROUP=www-data Screep Open None
Other Operations 181200 Use "Charter" as preferred typeface on Electron In-Scope Open None
Other Operations 166291 Exim panics when spamd reaches maxchildren In-Scope Open None
Other Operations 98984 Check power supply balance settings on cp3030+ In-Scope Open None
Other Operations 172815 Improve stability and maintainability of our browser-based PDF render service In-Scope Open None
Other Operations 190455 Logstash no longer captures DB queries in debug mode In-Scope Open None
Other Operations 131748 Refresh the appservers puppet code/configs In-Scope Open None
Other Operations 111934 Nutcracker stats monitoring should only listen on localhost In-Scope Open None
Other Operations 166066 Integrate the puppet compiler in the puppet CI pipeline In-Scope Open None
Other Operations 151045 Extending Yubico 2FA for production use (meta bug) In-Scope Open None
Other Operations 195252 Update prometheus-varnish-exporter on debian to 1.4 In-Scope Open None
Other Operations 149421 Long running mediawiki web requests impacts service availability, specially databases In-Scope Open None
Other Operations 186311 wikitech-l is mangling my PGP/MIME emails, causing signature validation to fail In-Scope Open None
Other Operations 211881 graphoid: Code stewardship request Elaborated Open None
Other Operations 209886 Assess Thumbor upgrade options Elaborated Open None
Other Operations 188601 Gain visibility into httpd mod_proxy actions In-Scope Open None
Other Operations 84279 Admin module should allow group management of system users In-Scope Open None
Other Operations 206939 "Workers" data from prometheus for mw app servers alternates strangely Screep Open None
Other Operations 194669 Provide a mean to mass discard/reject subscription requests on Wikimedia mailing lists In-Scope Open None
Other Operations 178628 Improve puppet alerting In-Scope Open None
Other Operations 192370 Deploy mcrouter to production as a wancache backend In-Scope Open None
Other Operations 106346 setup an alertable threshold for Cassandra heap dumps In-Scope Open None
Other Operations 202982 Requests to MW 404 when on HTTPS In-Scope Open None
Other Operations 183565 Fix regex.yaml single-regex issue In-Scope Open None
Other Operations 212690 DBQueryTimeoutError on Wikidata's Special:Nuke Screep Open None
Other Operations 177195 Reduce technical debt in metrics monitoring In-Scope Open None
Other Operations 162123 Running swiftrepl is not puppetized In-Scope Open None
Other Operations 119612 Consider a serialization that supports random access for storage in the DB for Wikidata Screep Open None
Other Operations 146968 OTRS spam classification methods and systems In-Scope Open None
Other Operations 147040 Two recently uploaded files have disappeared (404) In-Scope Open None
Other Operations 187754 Figure out why HHVM isn't using error_document404 setting In-Scope Open None