TechnicalOperations Unpointed Tasks

Help.

82038 create a test for multicast relay Network Operations open 2017-01-02 2017-06-27
83119 Netflow Collector Project Network Operations open 2017-01-02 2017-06-27
83992 Juniper monitoring Network Operations open 2017-01-02 2017-06-27
86541 setup wifi in codfw Network Operations open 2017-01-02 2018-02-06
98006 Anycast (Auth)DNS Network Operations open 2017-01-02 2017-07-24
106056 set up a looking glass for WMF ASes Network Operations open 2017-01-02 2018-02-06
120425 dumps.wikimedia.org seems to have poor throughput towards some destinations Network Operations open 2017-01-02 2018-04-21
122210 Security audit for tftp on install1001 Network Operations open 2017-01-02 2018-03-22
122406 Consider renumbering Labs to separate address spaces Network Operations open 2017-01-02 2018-05-16
124843 Peer with SFMIX at ULSFO in 200 Paul Network Operations open 2017-01-02 2017-06-27
133387 Enabling IGMP snooping on QFX switches breaks IPv6 (HTCP purges flood across codfw) Network Operations open 2017-01-02 2017-08-03
136671 Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR Network Operations open 2017-01-02 2018-02-04
150264 Icinga check for VRRP Network Operations open 2017-01-02 2017-06-27
163674 Frequent RST returned by appservers to LVS hosts Network Operations open 2017-04-25 2017-11-09
167306 ospf link-protection Network Operations open 2017-06-08 2017-06-27
167691 High amount of unexpected ICMP dest unreachable toward esams cache clusters Network Operations open 2017-06-13 2017-11-09
167841 Cleanup confed BGP peerings and policies Network Operations open 2017-06-14 2017-06-27
167842 Find a new PIM RP IP Network Operations open 2017-06-14 2017-10-20
170144 Evaluate NetBox as a Racktables replacement & IPAM Network Operations open 2017-07-11 2018-03-20
171032 Investigate lvs IP pages during codfw row C switch upgrade Network Operations open 2017-07-20 2017-07-20
172459 eqiad row D switch upgrade Network Operations open 2017-08-04 2018-02-07
173698 Backfill librenms data in graphite with historical RRDs Network Operations open 2017-08-22 2017-08-21
174596 dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour Network Operations open 2017-08-31 2017-08-30
174616 set up cr3-esams Network Operations open 2017-08-31 2018-01-03
174637 Setup esams atlas anchor Network Operations open 2017-09-01 2018-01-03
180179 Evaluate the possibility to add Juniper images to Openstack Network Operations open 2017-11-10 2018-01-12
181036 Pull netflow data in realtime from Kafka via Tranquillity/Spark Network Operations open 2017-11-22 2018-04-19
183390 unrack/decom pfw1-eqiad and pfw2-eqiad Network Operations open 2017-12-21 2018-05-03
184067 Complete router migration from cr1-esams to cr3-esams Network Operations open 2018-01-04 2018-01-03
185151 replace msw1-esams Network Operations open 2018-01-18 2018-02-02
185171 replace mr1-eqiad Network Operations open 2018-01-19 2018-04-24
185337 rack spare switches in c1-eqiad Network Operations open 2018-01-20 2018-03-14
185926 review and fix scs config Network Operations open 2018-01-30 2018-01-29
186021 reconfigure esams switch port for new bastion Network Operations open 2018-01-31 2018-01-30
186550 Anycast recdns Network Operations open 2018-02-06 2018-02-13
187820 Rack/cable/configure mr1-eqiad Network Operations open 2018-02-21 2018-02-20
187929 Cloud IPv6 subnets Network Operations open 2018-02-22 2018-02-21
187960 Rack/cable/configure asw2-a-eqiad switch stack Network Operations open 2018-02-22 2018-04-24
187962 Rack/cable/configure asw2-c-eqiad switch stack Network Operations open 2018-02-22 2018-05-18
189519 Audit switch ports/descriptions/enable Network Operations open 2018-03-13 2018-04-24
189522 Detect IP address collisions Network Operations open 2018-03-13 2018-03-13
189552 Rack/cable/configure ulsfo MX204 Network Operations open 2018-03-14 2018-05-02
189689 Connection timeout from 195.77.175.64/29 to text-lb.esams.wikimedia.org Network Operations open 2018-03-15 2018-03-14
190090 Offload pings to dedicated server Network Operations open 2018-03-20 2018-04-03
190184 Netbox: setup backups Network Operations open 2018-03-21 2018-03-20
190364 eqiad 10G ports needs Network Operations open 2018-03-22 2018-03-26
190424 modify labs-hosts1-vlans for http load of installer kernel Network Operations open 2018-03-23 2018-03-22
191371 Enabling graceful-switchover causes core dumps on cr1-codfw Network Operations open 2018-04-05 2018-04-06
191667 Juniper HA audit Network Operations open 2018-04-07 2018-04-11
193496 Allocate public v4 IPs for Neutron setup in eqiad Network Operations open 2018-05-02 2018-05-16
194515 eqdfw: Patch GTT cross-connect Network Operations open 2018-05-12 2018-05-11
61115 Implement RPKI (Resource Public Key Infrastructure) Network Operations resolved 2017-01-02 2017-10-06
79755 Setup flow monitoring of *internal* network traffic Network Operations resolved 2017-01-02 2017-06-27
80253 Fix esams management network Network Operations resolved 2017-01-02 2015-09-08
80273 nagios monitor transit/peering links and alert on low/high traffic Network Operations resolved 2017-01-02 2017-05-09
80847 hook up and dns oob access Network Operations resolved 2017-01-02 2015-04-03
81345 Re-Label transcode1 and transcode2 Network Operations resolved 2017-01-02 2014-12-18
81440 mr1 asymetric routing Network Operations resolved 2017-01-02 2015-09-02
81989 Notify transits of new esams prefixes Network Operations resolved 2017-01-02 2016-06-17
82509 Replace asw-c5-eqiad or asw-c8-eqiad with EX4550 Network Operations resolved 2017-01-02 2015-02-11
83037 Filter outgoing BGP announcements on AS regex Network Operations resolved 2017-01-02 2017-06-27
83196 internal network packet loss alerting Network Operations resolved 2017-07-20 2017-07-19
83898 JSNMP flood of errors across multiple switches Network Operations resolved 2017-01-02 2017-05-23
84205 Add missing rack "locations" in Librenms Network Operations resolved 2017-01-02 2015-09-08
84502 interfaces that are down on mr1-eqiad Network Operations resolved 2017-01-02 2015-09-02
84775 Upgrade switch fabrics in cr2-codfw Network Operations resolved 2017-01-02 2015-08-04
86574 Connect Apple Airport to mr1-codfw Network Operations resolved 2017-01-02 2015-01-12
87600 labnet1001 should be able to send stats to tungsten Network Operations resolved 2017-01-02 2015-01-26
89227 Set up cr1-eqord & cr1-eqdfw Network Operations resolved 2017-01-02 2015-08-14
89294 Establish IPsec tunnel between codfw and eqiad pfw Network Operations resolved 2017-01-02 2015-07-09
89833 set up switch port for second ethernet interface for ms1001 Network Operations resolved 2017-01-02 2015-02-27
89999 cr1-eqiad Control Board error Network Operations resolved 2017-01-02 2016-07-19
90695 permit syslog from labs hosts subnets to lithium Network Operations resolved 2017-01-02 2015-06-08
92548 Network congestion between DTAG & eqiad Network Operations resolved 2017-01-02 2015-03-17
92616 cr1/cr2-codfw QSFP+ errors every second for qsfp-0/0/0 Network Operations resolved 2017-01-02 2015-11-10
92815 Codfw mediawiki appservers from any rows but row A can't communicate with the dhcp server Network Operations resolved 2017-01-02 2016-01-20
93932 determine networking for ganeti2001-2006 Network Operations resolved 2017-01-02 2015-03-26
95959 install/setup/deploy cobalt as replacement for gallium Network Operations resolved 2017-01-02 2016-06-08
96385 setup network switch ports / vlans for db2053-2070 Network Operations resolved 2017-01-02 2015-04-22
96569 pfw-eqiad JunOS upgrade Network Operations resolved 2017-01-02 2015-04-30
102411 Change the ip of labs-ns0 with our DNS registrar Network Operations resolved 2017-01-02 2015-06-17
104036 networking: adjust ACLs to allow analytics clusters to talk to new ganglia aggregator Network Operations resolved 2017-01-02 2015-07-01
104458 rack/setup new eqiad lvs machines Network Operations resolved 2017-01-02 2016-11-08
105984 Very slow downloads from Wikimedia sites in eqiad on Wikimania hotel network Network Operations resolved 2017-01-02 2015-07-27
107056 Puppetize a server with a role that sets up Cassandra on Analytics machines [13 pts] {slug} Network Operations resolved 2017-01-02 2016-03-16
107635 investigate ethernet errors: asw2-a5-eqiad port xe-0/0/36 Network Operations resolved 2017-01-02 2015-09-02
108715 smokeping loss of ping for codfw rows Network Operations resolved 2017-01-02 2015-08-11
110208 Requests from a specific network are blocked Network Operations resolved 2017-01-02 2015-09-22
110435 cr1-eqdfw PEM 0 failure Network Operations resolved 2017-01-02 2015-09-18
111274 Set up NTT transit @ eqdfw, eqord Network Operations resolved 2017-01-02 2015-09-18
112000 cr2-eqiad PEM 2 failure Network Operations resolved 2017-01-02 2015-09-24
112396 Wikimedia sites not reachable through CenturyLink ISP Network Operations resolved 2017-01-02 2015-09-13
112902 Various misc. sites (e.g. gerrit) load very slow or not at all Network Operations resolved 2017-01-02 2015-09-17
113640 Upgrade JunOS on cr1/cr2-codfw Network Operations resolved 2017-01-02 2016-01-21
113771 setup new equinix out of band mgmt access Network Operations resolved 2017-01-02 2015-10-27
115330 block labs IPs from sending data to prod ganglia Network Operations resolved 2017-01-02 2016-06-24
115491 Allocate labs subnet in dallas Network Operations resolved 2017-01-02 2016-04-28
115492 Allocate subnet for labs test cluster instances Network Operations resolved 2017-01-02 2016-04-28
115921 remove tele2(patchid 2953) from dmarc panel Network Operations resolved 2017-01-02 2015-10-20
115924 Return psw2-eqiad to spares Network Operations resolved 2017-01-02 2015-10-27
116766 attach mr1-codfw to new out of band mgmt link Network Operations resolved 2017-01-02 2015-10-27
116928 drain ULSFO of all traffic on 2015-11-02 before 0900 PST Network Operations resolved 2017-01-02 2015-11-30
118178 test new sfp-t Network Operations resolved 2017-01-02 2015-11-10
118256 Set up cr2-esams Network Operations resolved 2017-01-02 2016-05-30
118259 Figure out the source of QSFP+ errors with DAC + MX480 Network Operations resolved 2017-01-02 2015-11-25
118721 cr1-eqiad PEM 0 fan failed Network Operations resolved 2017-01-02 2015-11-20
119512 add new saltmaster (neodymium) to network exceptions for labvirt* etc hosts Network Operations resolved 2017-01-02 2015-11-30
119634 Fix edit permissions of the netops project Network Operations resolved 2017-01-02 2015-11-25
119759 Investigate why disabling an uplink port did not deprioritize VRRP on cr2-eqiad Network Operations resolved 2017-01-02 2017-01-09
120010 Add firewall exception to get to wdqs*:8888 from analytics cluster Network Operations resolved 2017-01-02 2017-02-08
122293 Orange S.A. searches a contact at WMF for tests Network Operations resolved 2017-01-02 2016-01-06
122885 turn-up/implement zayo wave (579171) for ulsfo-codfw Network Operations resolved 2017-01-02 2016-02-19
123732 Decommission nitrogen (IPv6 relay) Network Operations resolved 2017-01-02 2016-01-28
124804 All wikis under wikimedia.org (Meta, Commons, loginwiki, others) are using wikimedia.org VHost, including /wiki/ -> wikimediafoundation.org redirect Network Operations resolved 2017-01-02 2016-02-12
124955 stat1002 - redis can't connect to mira.codfw.wmnet Network Operations resolved 2017-01-02 2016-01-27
127295 mr1-ulsfo booted from backup JunOS Network Operations resolved 2017-01-02 2016-02-18
129136 Deploy libreNMS with scap3 Network Operations resolved 2017-01-02 2017-05-05
131218 Update the port description on the network switch for analytics 1017 and 1021 Network Operations resolved 2017-01-02 2016-03-30
131375 Make sure phab can talk to gearman and nodepool instances can talk to phabricator Network Operations resolved 2017-01-02 2016-06-21
133852 analytics hosts frequently tripping 'port utilization threshold' librenms alerts Network Operations resolved 2017-01-02 2017-05-23
134930 codfw-eqiad Zayo link is down (cr2-codfw:xe-5/0/1) Network Operations resolved 2017-01-02 2016-05-19
134932 cr2-codfw LUCHIP/trinity_pio error messages Network Operations resolved 2017-01-02 2016-06-01
136707 Upgrade cr1-codfw/cr2-codfw FPC 0 firmware Network Operations resolved 2017-01-02 2016-06-01
136717 Turn up new eqiad-esams wave (Level3) Network Operations resolved 2017-01-02 2016-08-14
136829 'German Wikipedia Broken Weblinks Bot' is ill-behaved and in danger of getting all of Labs blacklisted Network Operations resolved 2017-01-02 2018-01-22
138609 Network ACL rules to allow traffic from Analytics to Production for port 9160 Network Operations resolved 2017-01-02 2016-08-31
140257 Allocate contint1001 to releng and allocate to a vlan Network Operations resolved 2017-01-02 2016-11-16
140764 cr1/cr2-eqiad: install new SCBs and linecards Network Operations resolved 2017-01-02 2016-08-18
140765 Replace cr1/2-eqiad PSUs/fantrays with high-capacity ones Network Operations resolved 2017-01-02 2016-08-04
140770 Upgrade cr1/cr2-eqiad JunOS Network Operations resolved 2017-01-02 2016-08-18
140935 audit network ports in a4-codfw Network Operations resolved 2017-01-02 2016-08-09
140991 put pfw1- ge-2/0/11 in the 'fundraising' vlan for new host frqueue1001 Network Operations resolved 2017-01-02 2016-07-27
141085 deploy elasticsearch/plugins to relforge1001-1002 servers Network Operations resolved 2017-01-02 2016-08-03
141898 cr2-eqiad temperature alerts ("system warm") Network Operations resolved 2017-01-02 2016-08-11
142211 Enable access to relforge clusters from virtual machines running on labs Network Operations resolved 2017-01-02 2016-08-10
143248 configure port for frdb1001 Network Operations resolved 2017-01-02 2016-09-08
143335 Open hole in analytics vlan firewall to allow MirrorMaker to talk to main Kafka clusters Network Operations resolved 2017-01-02 2016-08-18
143800 codfw: rack/setup/deploy puppetmaster200[12]switch configuration Network Operations resolved 2017-01-02 2016-08-24
143913 Upgrade cr1-esams & cr2-knams to JunOS 13.3 Network Operations resolved 2017-01-02 2016-10-05
143914 Upgrade cr1-ulsfo & cr2-ulsfo to JunOS 13.3 Network Operations resolved 2017-01-02 2016-10-05
143915 Fix static IP fallbacks to Pybal LVS routes Network Operations resolved 2017-01-02 2016-10-26
144263 Connection problems (from NZ to ULSFO) Network Operations resolved 2017-01-02 2016-08-30
145270 Telkom/8ta (South Africa) users cannot connect to wikimedia sites Network Operations resolved 2017-01-02 2016-09-19
146391 eeden ethernet outage Network Operations resolved 2017-01-02 2017-08-29
146474 Add firewall exception to get to wdqs*.codfw.wmnet:8888 from analytics cluster Network Operations resolved 2017-01-02 2017-02-08
147061 Give elukey/ema access to network devices Network Operations resolved 2017-01-02 2016-10-05
147320 Decomission palladium Network Operations resolved 2017-01-02 2016-10-28
148506 Rack and setup new eqiad row D switch stack (EX4300/QFX5100) Network Operations resolved 2017-01-02 2018-02-21
149170 Move dbproxy1010 and dbproxy1011 to labs-support network, rename them to labsdbproxy1001 and labsdbproxy1002 Network Operations resolved 2017-01-02 2016-11-23
149196 Migrate links from cr1-eqiad/cr2-eqiad fpc 5 to fpc 3 Network Operations resolved 2017-01-02 2017-08-25
149224 Decommission psw1-eqiad Network Operations resolved 2017-01-02 2016-10-27
149226 cr1-eqiad:ae4 is disabled due to VRRP issue Network Operations resolved 2017-01-02 2016-11-08
149253 Firewall rules production/labs for prometheus-node-exporter Network Operations resolved 2017-01-02 2016-11-03
149829 Move labsdb1008 to production, rename it back to db1095, use it as a temporary sanitarium Network Operations resolved 2017-01-02 2016-11-15
150092 Provide read-only access to OpenStack APIs from WMF IP space Network Operations resolved 2017-01-02 2016-12-20
150380 kafka2003 switch port configuration Network Operations resolved 2017-01-02 2016-11-10
150400 Spare pool servers switch configuration Network Operations resolved 2017-01-02 2016-11-10
150769 restbase201[0-2] switch port configuration Network Operations resolved 2017-01-02 2016-11-15
151357 prometheus200[1-2] switch port configuration Network Operations resolved 2017-01-02 2016-11-22
152182 asw2-d-eqiad.mgmt.eqiad - JNX_ALARMS CRITICAL - 2 red alarms, Network Operations resolved 2017-01-02 2017-01-19
152215 Move novaobserver (and novaadmin) users out of ldap Network Operations resolved 2017-01-02 2016-12-07
152627 ms-fe200[5-8] switch port configuration Network Operations resolved 2017-01-02 2016-12-13
153094 wdqs2003 switch port configuration Network Operations resolved 2017-01-02 2016-12-14
153273 asw-a2-eqiad PEM 0 not powered Network Operations resolved 2017-01-02 2016-12-15
154507 Slight packet loss observed on the network starting Nov 2016 Network Operations resolved 2017-01-04 2017-04-10
154577 cr2-esams<->cr2-eqiad link flaps Network Operations resolved 2017-01-05 2017-01-18
154605 codfw: elastic2025-elastic2036/switch port configuration Network Operations resolved 2017-01-05 2017-01-04
154758 asw-a7-codfw is down Network Operations resolved 2017-01-07 2017-01-25
154952 cr2-esams<->cr2-eqiad link down Network Operations resolved 2017-01-10 2017-01-18
155487 Open temporary access from analytics vlan to new-labsdb one Network Operations resolved 2017-01-18 2017-01-26
155875 asw-c2-eqiad reboots & fdb_mac_entry_mc_set() issues Network Operations resolved 2017-01-22 2017-02-17
155999 DBA plan to mitigate asw-c2-eqiad reboots Network Operations resolved 2017-01-24 2017-02-09
156109 netops: switch all subnets to use install1002/2002 as DHCP Network Operations resolved 2017-01-25 2017-02-10
156212 codfw: mc2019-mc2036/switch port configuration Network Operations resolved 2017-01-26 2017-01-26
156225 Move db1054 to A3 Network Operations resolved 2017-01-26 2017-01-25
156381 pfws not on librenms Network Operations resolved 2017-01-27 2017-01-27
156957 asw-d-codfw public1-vlan addition review (blocks gerrit2001) Network Operations resolved 2017-02-02 2017-02-02
157435 Review ACLs for the Analytics VLAN Network Operations resolved 2017-02-08 2018-04-05
157593 Add firewall exception to get to wdqs[12]003.(codfw|eqiad).wmnet:8888 from analytics cluster Network Operations resolved 2017-02-09 2017-02-10
158601 git clone over EQIAD (wmflabs) CODFW timeout due to low bandwidth (~250 KiB/s) Network Operations resolved 2017-02-21 2017-03-13
158647 cr2-knams<->asw-esams GBLX fiber down Network Operations resolved 2017-02-22 2017-04-07
158714 codfw:ms-be2028-ms-be2039 switch port configuration Network Operations resolved 2017-02-22 2017-03-06
159336 deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001) Network Operations resolved 2017-03-02 2017-03-15
159464 asw-a1-codfw spontaneous reboot Network Operations resolved 2017-03-03 2017-04-03
159757 netmon1002 networking setup Network Operations resolved 2017-03-07 2017-05-02
160055 Audit and cleanup border-in ACL on core routers Network Operations resolved 2017-03-10 2017-04-04
160087 codfw: oresrdb2002 switch port configuration Network Operations resolved 2017-03-10 2017-03-09
162152 esams higher than usual temperature Network Operations resolved 2017-04-05 2017-04-24
162199 Faulty optics on asw-b-eqiad:xe-1/1/2 Network Operations resolved 2017-04-05 2017-04-10
162239 cr2-esams FPC 0 is dead Network Operations resolved 2017-04-06 2017-04-12
162601 knams equipment move Network Operations resolved 2017-04-11 2017-04-13
162944 db20[7-9][0-9] switch ports configuration Network Operations resolved 2017-04-14 2017-05-02
163002 Spread eqiad analytics Kafka nodes to multiple racks ans rows Network Operations resolved 2017-04-15 2017-04-28
163243 ripe-atlas-eqiad is down Network Operations resolved 2017-04-19 2017-05-25
163312 lvs2001: intermittent packet loss from Icinga checks Network Operations resolved 2017-04-20 2017-05-18
163323 Interface errors on asw-c-codfw:xe-7/0/46 Network Operations resolved 2017-04-20 2017-04-20
163324 switchover icinga.wikimedia.org from einsteinium to tegmen Network Operations resolved 2017-04-20 2017-08-03
163326 switchover oresrdb.svc.eqiad.wmnet from oresrdb1001 to oresrdb1002 and back (after T148506) Network Operations resolved 2017-04-20 2017-08-03
163447 Interface errors on pfw-codfw:xe-15/0/0 Network Operations resolved 2017-04-21 2017-04-24
163542 Interface errors on cr2-eqiad:xe-4/3/1 Network Operations resolved 2017-04-22 2017-07-12
164495 Replace old Prometheus VMs addresses with baremetal in firewall configuration Network Operations resolved 2017-05-05 2017-05-04
164554 pfw-eqiad.wikimedia.org - 3 interfaces down (fundraising hosts) Network Operations resolved 2017-05-06 2017-05-05
164594 codfw: ganeti2007-ganeti2008 switch port configuration Network Operations resolved 2017-05-06 2017-05-08
164911 LibreNMS improvements Network Operations resolved 2017-05-11 2017-06-09
164970 mr1-ulsfo crashed Network Operations resolved 2017-05-11 2017-05-18
164988 codfw: kubernetes200[1-4] switch port configuration Network Operations resolved 2017-05-11 2017-05-10
165006 Zayo Circuit ulsfo<->codfw down Network Operations resolved 2017-05-12 2017-05-11
165008 Interface errors on asw-c-eqiad:xe-8/0/38 Network Operations resolved 2017-05-12 2017-05-17
165288 Report of esams unreachable from Fastweb/Init7 Network Operations resolved 2017-05-16 2017-05-23
165584 Deploy pybal with BGP MED support (for primary/backup) in production Network Operations resolved 2017-05-18 2017-10-20
165614 LLDP on cache hosts Network Operations resolved 2017-05-18 2017-06-09
166156 ores200[1-9] switch port configuration Network Operations resolved 2017-05-24 2017-05-23
166171 rack/setup/wire/deploy msw2-c1-eqiad Network Operations resolved 2017-05-24 2017-10-30
166564 codfw: labtestvirt2002 swith port configuration Network Operations resolved 2017-05-31 2017-06-27
166663 Init7 routing loop Network Operations resolved 2017-06-01 2017-05-31
167261 Faulty link between cr2-codfw and cr1-eqdfw Network Operations resolved 2017-06-08 2017-06-15
167274 codfw row D switch upgrade Network Operations resolved 2017-06-08 2017-08-03
167281 Some BGP sessions to the SF Office down Network Operations resolved 2017-06-08 2017-06-29
167288 Rancid improvements Network Operations resolved 2017-06-08 2017-06-09
167321 codfw: labtestpuppetmaster2001 switch port configuration Network Operations resolved 2017-06-08 2017-07-11
167322 codfw:labtestnet2002 switch port configuration Network Operations resolved 2017-06-08 2017-06-12
167326 codfw: labtestneutron2002 switch port configuration Network Operations resolved 2017-06-08 2017-06-12
167840 Merge AS14907 with AS43821 Network Operations resolved 2017-06-14 2018-05-07
168462 codfw row A switch upgrade Network Operations resolved 2017-06-21 2017-08-03
168667 Temperature increase in esams Network Operations resolved 2017-06-23 2017-06-27
169345 codfw row B switch upgrade Network Operations resolved 2017-07-01 2018-03-23
169624 deploy diffscan2 Network Operations resolved 2017-07-05 2017-08-02
169637 zombie rstp configuration Network Operations resolved 2017-07-05 2017-07-05
169643 codfw: rack frack refresh equipment Network Operations resolved 2017-07-05 2017-10-19
169644 eqiad: rack frack refresh equipment Network Operations resolved 2017-07-05 2017-10-30
170131 Recurring varnish-be fetch failures in codfw Network Operations resolved 2017-07-11 2017-10-23
170138 Replace cr1/2-eqiad air filters Network Operations resolved 2017-07-11 2017-07-20
170369 Remove unsecure SSH algorithms on network devices Network Operations resolved 2017-07-13 2017-07-13
170380 codfw row C switch upgrade Network Operations resolved 2017-07-13 2017-08-03
170622 pfw-codfw still logging to indium Network Operations resolved 2017-07-14 2017-07-17
171167 Evaluate LibreNMS' Graphite backend Network Operations resolved 2017-07-21 2017-08-29
171714 "MySQL server has gone away" from librenms logs Network Operations resolved 2017-07-27 2017-08-02
171823 Grafana dashboards for librenms graphite data Network Operations resolved 2017-07-28 2017-09-20
171962 bonded/redundant network connections for fundraising hosts Network Operations resolved 2017-07-29 2018-02-09
171970 Move codfw frack to new infra Network Operations resolved 2017-07-29 2017-09-19
172262 Poor conectivity (Vodafone/THUS in UK) Network Operations resolved 2017-08-03 2017-08-15
172478 git-ssh.wikimedia.org and IPv6 are broken after switch to phab1001 Network Operations resolved 2017-08-05 2017-08-04
173489 pmacct should be upgraded to 1.6.2 on Stretch Network Operations resolved 2017-08-18 2017-11-21
174218 Move eqiad frack to new infra Network Operations resolved 2017-08-26 2017-10-05
174364 DDOS_PROTOCOL_VIOLATION_SET: Protocol Rejectv6:aggregate is violated Network Operations resolved 2017-08-29 2017-08-28
174366 Power alarm flap on asw-d-codfw:et-7/0/52 channel 3 Network Operations resolved 2017-08-29 2017-08-30
174397 Tracking task for network syslog messages Network Operations resolved 2017-08-29 2017-10-19
175737 two switches have same serial in racktables Network Operations resolved 2017-09-13 2017-09-14
176175 connect second ethernet interface for fundraising codfw hosts Network Operations resolved 2017-09-19 2017-11-06
176319 remove fundraising firewall rules related to ganglia Network Operations resolved 2017-09-21 2017-10-02
176337 esams: networking audit for support contract renewal Network Operations resolved 2017-09-21 2018-05-03
176338 eqiad: networking audit for support contract renewal Network Operations resolved 2017-09-21 2018-01-17
176350 connect new office link to asw-ulsfo Network Operations resolved 2017-09-21 2017-10-18
176427 unrack/decom pfw1-codfw and pfw2-codfw Network Operations resolved 2017-09-22 2018-03-15
176492 move frav1001's to the frack-fundraising VLAN so we can use it for database testing Network Operations resolved 2017-09-23 2017-09-22
176816 cr2-esams temperature warning Network Operations resolved 2017-09-27 2018-02-01
176972 reconfigure networking on frack-eqiad management interfaces Network Operations resolved 2017-09-29 2017-09-28
176975 connect second interface for each frack to opposite switch for each eqiad host Network Operations resolved 2017-09-29 2017-11-01
177332 vcp port down on fasw-c-codfw Network Operations resolved 2017-10-04 2017-10-05
177333 Interface error on fasw-c-eqiad:vcp-255/1/0 Network Operations resolved 2017-10-04 2017-10-05
177820 Allow syslog (-tls) from both wezen and lithium in labs Network Operations resolved 2017-10-11 2017-10-12
177821 Allow syslog-tls and syslog in analytics towards wezen/lithium Network Operations resolved 2017-10-11 2017-11-07
178608 Disable Jenkins autodiscovery system Network Operations resolved 2017-10-20 2017-12-21
179042 Setup eqsin RIPE Atlas anchor Network Operations resolved 2017-10-26 2018-03-13
180178 Request increased quota for traffic Cloud VPS project Network Operations resolved 2017-11-10 2018-01-12
181558 setup and deploy eqsin network infrastructure Network Operations resolved 2017-11-29 2018-03-08
184723 switch network port 2/0/3 (frdb1003) back to administration-vlan Network Operations resolved 2018-01-12 2018-01-16
185228 replace ulsfo access switches Network Operations resolved 2018-01-19 2018-02-08
185336 rack spare switches in c1-codfw Network Operations resolved 2018-01-20 2018-02-02
186172 switch port configuration for db2093 Network Operations resolved 2018-02-01 2018-02-21
187158 cp5010 - no link on primary ethernet port Network Operations resolved 2018-02-14 2018-03-06
187688 rhenium running out of disk space on / Network Operations resolved 2018-02-20 2018-02-20
187807 cr1-eqsin faulty interfaces Network Operations resolved 2018-02-21 2018-05-04
187816 codfw: mgmt switch replacement in D4 Network Operations resolved 2018-02-21 2018-02-26
187933 Labs to Cloud renaming for networking equipment Network Operations resolved 2018-02-22 2018-03-12
188045 wdqs1004 broken Network Operations resolved 2018-02-23 2018-03-13
188303 switch port configuration for wdq200[4-6] Network Operations resolved 2018-02-27 2018-02-26
188923 replace eqsin SFP-T/SFP+ Network Operations resolved 2018-03-06 2018-03-07
189060 return faulty MX104 to Juniper Network Operations resolved 2018-03-07 2018-03-08
189452 Interface errors on cr2-codfw: xe-5/3/1 Network Operations resolved 2018-03-13 2018-03-14
189588 Config discrepencies on network devices Network Operations resolved 2018-03-14 2018-04-04
190115 Switch port configuration for mw2259-mw2290 Network Operations resolved 2018-03-21 2018-03-20
190134 Can't login on netbox Network Operations resolved 2018-03-21 2018-03-20
190167 Add puppetmaster2001 to analytics-in4 Network Operations resolved 2018-03-21 2018-03-20
190317 Update BGP_sanitize_in filter Network Operations resolved 2018-03-22 2018-05-09
190322 Switch port configuration for ms-be204[0-3] Network Operations resolved 2018-03-22 2018-03-21
190323 Implement BGP graceful shutdown Network Operations resolved 2018-03-22 2018-05-04
191996 db1114 connection issues Network Operations resolved 2018-04-12 2018-04-19
192087 Rename lvs* LLDP port descriptions after upgrading to stretch Network Operations resolved 2018-04-13 2018-05-02
192104 Asymetric routing for cross-DC pfw syslog-tls Network Operations resolved 2018-05-11 2018-05-10
192114 ulsfo<->eqord BGP down Network Operations resolved 2018-04-13 2018-04-26
192125 asw1-eqsin vcp port flapping Network Operations resolved 2018-04-14 2018-05-08
193177 NAT for new fundraising bastion Network Operations resolved 2018-04-27 2018-04-26
193189 New PFW policy Network Operations resolved 2018-04-27 2018-04-26
193677 Interface errors on asw-d-codfw:xe-2/0/47 Network Operations resolved 2018-05-03 2018-05-08
193897 cr1-eqsin 4 onboard interfaces down Network Operations resolved 2018-05-05 2018-05-08
194858 asw-c8 rebooting Network Operations resolved 2018-05-17 2018-05-17
23027 Requests with utf-8 in the URL return a outdated page revision Traffic open 2017-01-02 2016-10-04
26413 Consider DNSSec Traffic open 2017-01-02 2017-10-23
36670 Check all wikis for inclusions of http resources on https Traffic open 2017-01-02 2016-10-04
45250 Redo /beacon/impression system (formerly Special:RecordImpression) to remove extra round trips on all FR impressions (title was: S:RI should pyroperish) Traffic open 2017-01-02 2017-03-27
50133 ForeignAPIRepo wrongly returns non-protocol-relative URLs for original "thumbs" Traffic open 2017-01-02 2017-06-27
50835 Separate Cache-Control header for proxy and client Traffic open 2017-01-02 2017-05-30
54253 Protocol-relative URLs are poorly supported or unsupported by a number of HTTP clients Traffic open 2017-01-02 2017-03-28
56783 Respect X-Forwarded-For only from trustworthy sources Traffic open 2017-01-02 2016-10-04
66214 Define an official thumb API Traffic open 2017-01-02 2018-04-25
74186 Varnish: Mobile site redirect interferes with OAuth authorization process Traffic open 2017-01-02 2017-11-04
75944 Monitor Varnish caches on beta cluster have two varnishd process running Traffic open 2017-01-02 2016-09-30
78421 m.{project}.org portal/redirect consistency Traffic open 2017-01-02 2017-08-08
78963 Support ESI for ResourceLoader Traffic open 2017-01-02 2016-12-05
79730 Add pybal check to ensure service IP is bound Traffic open 2017-03-31 2017-03-30
81305 Make PyBal respect advertised BGP capabilities Traffic open 2017-03-31 2017-03-30
81605 No IPv6 addresses on Wikimedia nameservers ns(0-2).wikimedia.org Traffic open 2017-01-02 2016-09-30
82849 lvs servers report 'Memory allocation problem' on bootup Traffic open 2017-02-11 2017-10-06
83467 LVS testing needs to include internal services testing Traffic open 2017-01-02 2016-09-30
86650 Add support for setting weight=0 when depooling Traffic open 2017-01-02 2017-07-31
86671 HTTPS RFC5077 session tickets encryption key rollovers Traffic open 2017-01-02 2017-10-23
86915 nan and minnan subdomain redirects are a mess Traffic open 2017-01-02 2017-08-24
88861 wikipedia.lol Traffic open 2017-01-02 2016-09-30
89838 Move proxy IP lists to META for Varnish XFF decoding Traffic open 2017-01-02 2017-02-06
91372 $wgMFAnonymousEditing = true is sometimes not respected: cache? Traffic open 2017-01-02 2016-05-04
91820 Create HTTP verb and sticky cookie DC routing in VCL Traffic open 2017-01-02 2018-05-20
92298 Investigate our mitigation strategy for HTTPS response length attacks Traffic open 2017-01-02 2016-11-15
94125 Central login notice appears on unencrypted API format=*fm pages, where reloading does not affect login status Traffic open 2017-01-02 2016-10-04
94697 implement better failure-scenario geoip mapping in gdnsd Traffic open 2017-01-02 2016-09-30
96499 dbtree loads third party resources (from jquery.com and google.com) Traffic open 2017-01-02 2016-10-04
96844 Update TLS/HTTP documentation on wikitech Traffic open 2017-01-02 2016-09-30
96852 Define 3-host infra cluster for traffic pops Traffic open 2017-01-02 2017-03-21
96853 Evaluate Apache Traffic Server Traffic open 2017-01-02 2017-06-21
97051 adding new languages to DNS langs.tmpl doesn't work until zone template is edited as well Traffic open 2017-01-02 2016-10-04
97333 Align prod + beta-cluster LVS usage Traffic open 2017-01-02 2016-09-30
98165 Figure out an etcd deploy strategy that includes multi DC failure scenarios. Traffic open 2017-01-02 2017-07-11
99216 Please set up a CNAME for videoserver.wikimedia.org to Video Editing Server Traffic open 2017-01-02 2016-09-30
99531 [Task] move wikiba.se webhosting to wikimedia misc-cluster Traffic open 2017-07-22 2018-05-07
101002 Use Upgrade Insecure Requests on Wikimedia wikis Traffic open 2017-01-02 2017-10-23
101048 Policy decisions for new (and current) DNS domains registered to the WMF Traffic open 2017-01-02 2018-03-26
101525 Set up LVS for current AuthDNS Traffic open 2017-01-02 2017-04-25
102178 Fix RESTBase support for wikitech.wikimedia.org Traffic open 2017-01-02 2017-07-11
102367 Migrate tools.wmflabs.org to https only (and set HSTS) Traffic open 2017-01-02 2018-05-14
102848 Split GeoIP into a new component Traffic open 2017-01-02 2017-11-13
104442 Investigate better DNS cache/lookup solutions Traffic open 2017-01-02 2017-07-26
104681 HTTPS Plans (tracking / high-level info) Traffic open 2017-01-02 2017-07-11
105657 Expires header for load.php should be relative to request time instead of cache time Traffic open 2017-01-02 2018-02-07
106517 upload.wikimedia.org returns HTTP status code 503 for truncated urls, not 404 Traffic open 2017-01-02 2017-11-30
107236 Switch port 80 to nginx on primary clusters Traffic open 2017-01-02 2016-09-30
108580 HTTPS for internal service traffic Traffic open 2017-01-02 2016-11-16
109321 Inbound TLS for tier-1 varnish backend caches Traffic open 2017-01-02 2016-09-30
109325 Outbound HTTPS for varnish backend instances Traffic open 2017-01-02 2016-09-30
109331 Deleted files sometimes remain visible to non-privileged users if permanently linked Traffic open 2017-01-02 2017-07-27
109776 Tilerator should purge Varnish cache Traffic open 2017-01-02 2017-03-08
111588 RFC: API-driven web front-end Traffic open 2017-01-02 2018-05-15
111899 Deprecate pybal SSH health checks Traffic open 2017-01-02 2016-09-30
112316 Configure varnish to use "Unconfigured domain" page for 404 Not Served (instead of generic error) Traffic open 2017-01-02 2016-09-30
113597 pybal-related issue on host start can break service IPs... Traffic open 2017-01-02 2017-02-10
113817 Connect Hadoop records of the same request coming via different channels Traffic open 2017-01-02 2016-10-04
114104 pybal doesn't fully manage LVS table leaving stale services (on IP change) Traffic open 2017-03-29 2017-03-30
114979 Run IPVS in a separate network namespace Traffic open 2017-01-02 2017-09-18
116132 missing H2 coalesce for upload.wm.o for images ref'd in projects' page outputs Traffic open 2017-01-02 2018-03-19
117826 TEST: redirect small portion of unauthenticated desktop users to mobile web Traffic open 2017-01-02 2017-10-02
118181 Planning for phasing out non-Forward-Secret TLS ciphers Traffic open 2017-01-02 2017-05-08
118468 point wikilovesmonuments.org ns to wmf Traffic open 2017-01-02 2016-09-30
119038 Image cache issue when 'over-writing' an image on commons Traffic open 2017-01-02 2016-04-28
119366 Disable caching on the main page for anonymous users Traffic open 2017-02-13 2018-03-29
119372 Pybal IdleConnectionMonitor with TCP KeepAlive shows random fails if more than 100 servers are involved. Traffic open 2017-01-02 2017-03-30
120121 Improve Varnish XFF processing for trusted proxies Traffic open 2017-01-02 2016-10-04
120486 add a https-only option to dynamicproxy Traffic open 2017-01-02 2017-12-29
120509 Cache education dashboard pages Traffic open 2017-01-02 2016-09-30
120631 Security: Is it safe to enable Zero spoofing Traffic open 2017-01-02 2017-07-20
122867 Evaluate the feasibility of cache invalidation for the action API Traffic open 2017-01-02 2017-07-19
123854 Set up action API latency / error rate metrics & alerts Traffic open 2017-01-02 2017-07-11
124418 Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan Traffic open 2017-01-02 2018-03-19
125170 Internal DNS resolver responds with NXDOMAIN for localhost AAAA Traffic open 2017-01-02 2016-09-30
125226 [feature request] Redirect root API path to docs page Traffic open 2017-06-15 2017-06-14
127387 Split slash decoding from general percent normalization in Varnish VCL Traffic open 2017-01-02 2017-11-14
127482 Enable VCL source-DC switching via confd Traffic open 2017-01-02 2016-09-30
127573 wikiknihy.cz - transfer to Wikimedia Czech Republic? Traffic open 2017-01-02 2016-10-04
128188 Make CI run Varnish VCL tests Traffic open 2017-01-02 2016-09-30
128358 Uploading 1.2GB ogv results in 503 Traffic open 2017-01-02 2016-12-13
128374 Sort out analytics service dependency issues for cp* cache hosts Traffic open 2017-01-02 2018-04-02
128409 Detect tools.wmflabs.org tools which are HTTP-only Traffic open 2017-01-02 2017-10-15
128559 store.wikimedia.org HTTPS issues Traffic open 2017-01-02 2018-04-26
129682 Look into solutions for replaying traffic to testing environment(s) Traffic open 2017-01-02 2017-07-12
129839 restrict upload cache access for private wikis Traffic open 2017-01-02 2016-09-30
130904 Host rewrite for /static/ not applied to purges Traffic open 2017-01-02 2016-09-30
131894 Collect Backend-Timing in Graphite (or Prometheus) Traffic open 2017-01-02 2018-04-19
131930 Set SPF (... -all) for toolserver.org Traffic open 2017-01-02 2017-06-14
132629 Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense Traffic open 2017-01-02 2017-07-21
133178 RESTBase support for www.wikimedia.org missing Traffic open 2017-05-31 2018-01-31
133410 Deploy TemplateStyles to WMF production Traffic open 2017-01-02 2018-05-03
133548 Create a secure redirect service for large count of non-canonical / junk domains Traffic open 2017-01-02 2018-05-18
133717 Letsencrypt all the prod things we can - planning Traffic open 2017-01-02 2017-01-18
133821 Content purges are unreliable Traffic open 2017-01-02 2018-01-15
133895 Varnish configuration for mobile domains should be coherent with Apache configuration Traffic open 2017-01-02 2016-09-30
134323 confctl: give regexen more freedom Traffic open 2017-01-02 2016-10-04
134324 confctl select needs a -y flag? Traffic open 2017-01-02 2016-10-04
134447 letsencrypt puppetization: upgrade for scalability Traffic open 2017-01-02 2016-09-30
134807 Replace test hostnames in datecenter-specific subdomains with dashed names Traffic open 2017-01-02 2016-10-03
134893 Unhandled pybal error causing services to be depooled in etcd but not in lvs Traffic open 2017-01-02 2017-01-26
136703 Add LVS public endpoint checks that bypass caches Traffic open 2017-01-02 2016-09-30
136944 Set up LVS connection sync Traffic open 2017-01-02 2016-09-30
137252 Redirect phabricator.mediawiki.org to phabricator.wikimedia.org Traffic open 2017-01-02 2016-10-07
137747 Parametrization of VCL is inconsistent Traffic open 2017-01-02 2016-09-30
137979 Support brotli compression Traffic open 2017-01-02 2017-10-10
138093 Investigate query parameter normalization for MW/services Traffic open 2017-01-02 2017-07-12
138546 Backend naming in VCL needs to use fqdn+port Traffic open 2017-01-02 2016-09-30
138591 Backport iproute2 4.x from debian testing -> our jessie Traffic open 2017-01-02 2017-07-27
140365 Lower geodns TTLs from 600 (10min) to 300 (5min) Traffic open 2017-01-02 2016-09-30
141266 letsencrypt puppetization: add parallel rsa+ecdsa cert support Traffic open 2017-01-02 2016-09-30
141480 mixed-content issues on planet.wikimedia.org Traffic open 2017-01-02 2016-10-04
143562 High number of failed inbound TFO connections in esams Mon-Fri Traffic open 2017-01-02 2016-10-04
144187 Better handling for one-hit-wonder objects Traffic open 2017-01-02 2017-11-06
144508 Point wikipedia.in to 205.147.101.160 instead of URL forward Traffic open 2017-01-02 2016-10-04
146332 Create short link for outreachdashboard.wmflabs.org Traffic open 2017-02-28 2017-02-28
146619 DNS domains registered to WMF no longer redirecting Traffic open 2017-01-02 2016-09-30
146832 Clarify caching to enable direct Wikidata Query Service access by <mapframe/link> Traffic open 2017-01-02 2016-12-15
147162 upload.wikimedia.org returns HTTP 501 instead of 416 for non-satisfiable byte ranges Traffic open 2018-03-30 2018-03-29
147202 Removing support for AES128-SHA TLS cipher Traffic open 2017-01-02 2016-10-03
147209 etcd cluster has Raft Internal errors sporadically Traffic open 2017-01-02 2016-10-04
147648 Unexplained increase in thumbnail 500s Traffic open 2017-01-02 2016-11-18
147967 The WMF-Last-Access Set-Cookie header should follow RFC 2965 syntax rather than the pre-RFC Netscape format Traffic open 2017-01-02 2016-10-17
148134 OCSP Stapling for Intermediates Traffic open 2017-01-02 2016-11-01
148422 cp3009: memory scrubbing error Traffic open 2017-01-02 2018-01-03
148595 Visting [[c:File:FEZ_trial_gameplay_HD.webm]] in IE11 shows errors in developer console about insecure data:image/png;base64 "URL" Traffic open 2017-01-02 2016-10-19
148976 Strongswan Icinga check: do not report issues about depooled hosts Traffic open 2017-01-02 2016-10-24
149847 RFC: Use content hash based image / thumb URLs Traffic open 2017-01-02 2018-04-12
150022 thumb.php should not set CC:no-cache on renderer 404 responses? Traffic open 2017-01-02 2018-05-17
150479 Prometheus varnish metric churn due to VCL reloads Traffic open 2017-01-02 2017-03-16
150673 Thumb API: Varnish / CDN questions Traffic open 2017-01-02 2017-08-16
152091 Block hotlinking Traffic open 2017-01-02 2016-12-05
152622 Wikipedia.cz and other domains owned by WMCZ have invalid certificate Traffic open 2017-01-02 2017-07-14
152882 Many misc wikis lack mobile domains Traffic open 2017-01-02 2017-07-25
153468 Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances Traffic open 2017-01-02 2018-03-31
153563 Consider switching to HTTPS for Wikidata query service links Traffic open 2017-01-02 2017-11-12
154015 announce breaking change: http > https for entities in rdf Traffic open 2017-01-02 2018-02-22
154017 compile number of http uses for http://www.wikidata.org/entity Traffic open 2017-01-02 2018-02-22
154702 Fix broken referer categorization for visits from Safari browsers Traffic open 2017-01-06 2017-09-01
154801 Investigate varnishd child crashes when multiple nodes get depooled/pooled concurrently Traffic open 2017-01-07 2017-02-07
155314 Varnish does not cache Action API responses when logged in Traffic open 2017-01-15 2017-07-19
156462 Framework to transfer files over the LAN Traffic open 2017-01-28 2018-05-17
157786 Unhandled error stopping pybal: 'RunCommandMonitoringProtocol' object has no attribute 'checkCall' Traffic open 2017-02-11 2017-02-10
158599 Samsung Internet's desktop mode getting redirected to mobile site Traffic open 2017-02-21 2017-07-19
158604 Investigate usefulness of SameSite cookies for logged-in accounts Traffic open 2017-10-25 2017-10-25
159411 Uniform cluster nomenclature across puppet Traffic open 2017-03-03 2017-03-26
159412 Convert all of our site.pp/roles to the role/profile paradigm Traffic open 2017-03-03 2017-12-01
161148 AuthDNS CM/CI refactor Traffic open 2017-03-23 2017-03-24
161256 multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert Traffic open 2017-03-24 2017-03-28
161360 404 loading images from Virgin Media Traffic open 2017-03-25 2017-03-28
161517 Allow anonymous users to change interface language on Commons with ULS Traffic open 2017-03-30 2017-07-25
162818 icinga alerts on nodejs services when a recdns server is depooled Traffic open 2017-04-13 2017-06-27
163141 dbtree: make wasat a working backend and become active-active Traffic open 2017-04-18 2017-12-11
163541 cache hosts should auto-repool iff OCSP files are sane Traffic open 2017-04-22 2017-04-21
164259 Add VSL error counters to Varnishkafka stats Traffic open 2017-05-03 2018-04-05
164327 replace ulsfo aging servers Traffic open 2017-05-03 2017-12-18
164456 Migrate to nginx-light Traffic open 2017-05-05 2017-05-09
164460 Use DNS discovery record for deployment CNAME Traffic open 2017-05-05 2017-05-09
164609 Merge cache_misc into cache_text functionally Traffic open 2017-05-06 2017-06-27
164768 Explicitly limit varnishd transient storage Traffic open 2017-05-09 2017-06-20
165252 cp1053 possible hardware issues Traffic open 2017-05-16 2017-10-23
165560 Artificial spike in offset of unique devices from November to February 6th on wikidata Traffic open 2017-05-18 2017-06-26
165764 Fully-redundant LVS clusters using Pybal per-service MED feature Traffic open 2017-05-20 2017-11-23
165765 Refactor pybal/LVS config for shared failover Traffic open 2017-05-20 2017-06-01
166782 wikimediafoundation.org's language selector is confusing to most visitors who don't have accounts there Traffic open 2017-06-02 2017-08-24
167239 Redirect status.wikipedia.org to status.wikimedia.org Traffic open 2017-06-07 2017-06-14
167400 Disable serving unpatrolled new files to Wikipedia Zero users Traffic open 2017-06-16 2018-02-06
167513 Redirect lzh.wikipedia to zh-classical.wikipedia Traffic open 2017-06-10 2017-06-12
167906 Make API usage limits easier to understand, implement, and more adaptive to varying request costs / concurrency limiting Traffic open 2017-06-15 2017-12-22
167972 Respect host header in RESTBase, and redirect /rest_v1 to /rest_v1/ Traffic open 2017-06-16 2017-06-20
168539 Unhandled pybal error: OpenSSL.SSL.Error - ssl handshake failure Traffic open 2017-06-22 2017-06-22
169765 pybal should automatically reconnect to etcd Traffic open 2017-07-06 2018-02-16
170567 Support TLSv1.3 Traffic open 2017-07-14 2017-10-23
170605 Unable to render file from upload.wikimedia.org "Error 349 ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION" Traffic open 2017-07-14 2017-08-09
171168 cp1050 apparently stuck while "Initializing firmware interfaces..." Traffic open 2017-07-21 2017-11-09
171470 Monitor DNS delegations Traffic open 2017-07-25 2017-07-24
171498 Implement machine-local forwarding DNS caches Traffic open 2017-07-25 2017-09-13
171850 Backport ipvsadm Traffic open 2017-07-28 2017-10-11
172103 IPVS issues with UDP services, pybal depooling strategy Traffic open 2017-08-01 2017-08-01
172123 Determine how to upload Zim files to Swift infrastructure Traffic open 2017-08-01 2018-04-06
172124 PyBal Feature: progressive depooling strategy for monitored failures Traffic open 2017-08-01 2017-09-06
172148 Determine URL paths for Zim files Traffic open 2017-08-01 2018-03-12
173966 Like nan.wikipedia.org, redirect other nan.*.org to the proper zh-min-nan.*.org domains Traffic open 2017-08-24 2017-08-23
174342 Missing IP addresses for Maroc Telecom Traffic open 2017-08-29 2017-11-16
174432 Unclear LVS bandwidth graph in "load balancers" dashboard Traffic open 2017-08-30 2017-08-30
174932 Recurrent 'mailbox lag' critical alerts and 500s Traffic open 2017-09-05 2018-03-29
174960 Varnish does not vary elasticsearch query by request body Traffic open 2017-09-05 2017-09-07
175203 Implement stateless TCP balancing in our LVS servers Traffic open 2017-09-07 2017-09-07
175319 cp1066 unexplained 503 spikes Traffic open 2017-09-09 2017-09-19
175636 prometheus -> grafana stats for per-numa-node meminfo Traffic open 2017-09-12 2017-10-02
176366 Decom cp4005-8,13-16 (8 nodes) Traffic open 2017-09-21 2018-05-03
176388 pybal: race condition in alerts instrumentation Traffic open 2017-09-22 2017-09-28
176875 Allow access to wdqs.svc.eqiad.wmnet on port 8888 Traffic open 2017-09-28 2018-03-05
177742 Investigate Chrony as a replacement for ISC ntpd Traffic open 2017-10-10 2018-03-23
177927 Refactor kafka_config.rb and and kafka_cluster_name.rb in puppet to avoid explicit hiera calls Traffic open 2017-10-12 2018-04-16
177961 Upgrade LVS servers to stretch Traffic open 2017-10-12 2018-03-12
178173 Renew unified certificates 2017 Traffic open 2017-10-14 2017-11-16
178535 decommission lvs400[1-4].ulsfo.wmnet Traffic open 2017-10-19 2017-11-27
178592 decommission/replace bast4001.wikimedia.org Traffic open 2017-10-20 2017-12-12
178815 decom cp40(09|1[078]) Traffic open 2017-10-24 2017-11-17
179025 LVS hosts should have static-mapped IPv6 on all virtual interfaces Traffic open 2017-10-26 2017-10-25
179026 LVS IPv6 IPs should all be recorded in DNS Traffic open 2017-10-26 2017-10-25
179027 Puppetize LVS interface IP sets per-DC for easy use in ferm rules Traffic open 2017-10-26 2017-11-20
179050 setup bast4002/WMF7218 Traffic open 2017-10-27 2017-11-09
179197 Investigate what caused the the unattended varnish upgrade in Beta Cluster Traffic open 2017-10-28 2017-11-07
179953 cp3043 disk failure Traffic open 2017-11-08 2018-01-03
180257 Puppet / LVS: confusion in service vs IP name Traffic open 2017-11-11 2017-11-14
180269 Wikimedia's recent upgrade to nginx v. 1.13.6 breaks older Android HTTP libraries Traffic open 2017-11-11 2017-11-14
180329 Add CI to all operations/software/varnish/* repositories and archive obsolete ones Traffic open 2017-11-14 2017-11-20
180407 Change "CP" cookie from subdomain to project level Traffic open 2017-11-14 2017-11-14
180434 Uncacheable content handling: hfp vs hfm Traffic open 2017-11-15 2017-11-16
180655 Phabricator and Gerrit: Improve the way that maintenance downtime is communicated to users. Traffic open 2017-11-17 2017-12-07
180712 VCL: handling of uncacheable responses in wikimedia-common Traffic open 2017-11-17 2018-03-23
180921 Referrer policy for browsers which only support the old spec Traffic open 2017-11-20 2018-05-20
181315 Varnish HTTP response from app servers taking 160s (only 0.031s inside Apache) Traffic open 2017-11-25 2018-05-05
181368 Log source port for anonymous users and expose it for sysops/checkusers Traffic open 2017-11-27 2017-11-27
181569 rack/setup scs-eqsin.mgmt.eqsin.wmnet Traffic open 2017-11-29 2018-02-13
183177 memory errors not showing in icinga Traffic open 2017-12-19 2018-05-14
183554 Unified certs bloat reduction? Traffic open 2017-12-23 2017-12-22
184293 rack/setup/install lvs101[3-6] Traffic open 2018-01-06 2018-04-30
184534 Cached page previews not shown when refreshed Traffic open 2018-01-10 2018-03-30
184942 Deprecate python varnish cachestats Traffic open 2018-01-16 2018-04-30
185239 Puppet hosts with signed certificate present on agent but not master Traffic open 2018-01-19 2018-01-22
185350 Vet reliability of the response_size field for data analysis purposes Traffic open 2018-01-20 2018-02-05
186732 Decide on Cache-Control headers for map tiles Traffic open 2018-02-08 2018-05-08
187157 cp5006 unresponsive Traffic open 2018-02-14 2018-04-13
188087 Some etcd connections not established at startup Traffic open 2018-02-24 2018-02-23
188561 SSL cert for links.email.wikimedia.org Traffic open 2018-03-23 2018-04-17
188776 Move Foundation Wiki to new URL when new Wikimedia Foundation website launches Traffic open 2018-03-03 2018-04-30
188804 Investigate and fix odd uri_host values Traffic open 2018-03-06 2018-03-07
188807 Update documentation for "https" field in X-Analytics Traffic open 2018-03-04 2018-03-12
189250 WP Zero workarounds for eqsin Traffic open 2018-03-09 2018-03-21
189290 Tune systemd journal rate limiting for PyBal Traffic open 2018-03-10 2018-04-16
189305 cp3034: Uncorrectable Memory Error Traffic open 2018-03-10 2018-03-19
190244 en-wp.org certificate error Traffic open 2018-03-22 2018-04-18
190607 cp3048 hardware issues Traffic open 2018-03-25 2018-03-26
190843 Spammy events coming our way for sites such us https://ru.wikipedia.kim Traffic open 2018-03-28 2018-03-29
190992 prometheus: slow dashboards due to suboptimal query_range performance Traffic open 2018-03-30 2018-03-29
190993 Upgrade pybal-test instances to stretch Traffic open 2018-03-30 2018-04-12
191017 Unwanted service startups and their triggers Traffic open 2018-03-30 2018-03-29
191393 Puppet: tlsproxy localssl default_server make a Notify at each run Traffic open 2018-04-05 2018-04-18
191940 Investigate 2018-04-10 global traffic drop Traffic open 2018-04-11 2018-04-16
192082 lvs2006 Embedded Flash/SD-CARD iLO errors Traffic open 2018-04-13 2018-04-12
192206 Remove wildcard vhost for *.wikimedia.org Traffic open 2018-04-15 2018-05-07
192280 sda failure in hydrogen.wikimedia.org Traffic open 2018-04-17 2018-04-17
192368 Unconditional return(deliver) in vcl_hit Traffic open 2018-04-18 2018-04-27
192437 Pybal support of configuration from the kubernetes API Traffic open 2018-04-19 2018-05-21
192555 Begin execution of non-forward-secret ciphers deprecation Traffic open 2018-04-20 2018-05-14
192559 Establish timeline and methodology for upcoming deprecation of non-forward-secret ciphers and TLSv1.0 Traffic open 2018-04-20 2018-04-19
193445 Update Media dashboard in Grafana to use Prometheus metrics Traffic open 2018-05-03 2018-05-07
193521 Consider adding expect-CT: header to enforce certificate transparency Traffic open 2018-05-02 2018-05-07
193865 Enable numa_networking on all caches Traffic open 2018-05-05 2018-05-07
194031 Setup a new PKI software as an alternative to the puppet CA for managing services certificates Traffic open 2018-05-08 2018-05-21
194380 Identify bots using AES128-SHA maintainers running on toolforge Traffic open 2018-05-11 2018-05-17
194724 Deprecate `base::service_unit` in puppet Traffic open 2018-05-16 2018-05-16
194757 cp1068 memory correctable errors Traffic open 2018-05-16 2018-05-16
194814 Remove unnecessary response headers Traffic open 2018-05-17 2018-05-17
194962 Create and deploy a centralized letsencrypt service Traffic open 2018-05-19 2018-05-20
194965 gdnsd plugin support for ACME DNS challenges Traffic open 2018-05-19 2018-05-22
195327 Normalise the Accept-Language header for REST API requests Traffic open 2018-05-23 2018-05-22
1147 Project Proposal: Label style projects for common operations tools Traffic resolved 2017-01-02 2016-05-04
2225 Secure login on Wikimedia wikis via SSL Traffic resolved 2017-01-02 2016-08-05
17357 Redirect dk.wiktionary and dk.wikibooks to da.wiktionary and da.wikibooks respectively. Traffic resolved 2017-08-24 2017-08-23
18822 Provide SSL/HTTPS interface to upload.wikimedia.org and use it for SSL-served pages Traffic resolved 2017-01-02 2016-08-05
20496 Non-secure logos on secure pages Traffic resolved 2017-01-02 2016-08-05
21429 secure links in email notification Traffic resolved 2017-01-02 2016-08-05
22643 Serve SSL/HTTPS sites out of same domain names as HTTP access: https://en.wikipedia.org/ Traffic resolved 2017-01-02 2016-08-05
25559 Secure Server 404 Error Message: Did you mean URL is wrong Traffic resolved 2017-01-02 2016-08-05
26332 OpenSSL on secure.wikimedia.org is possibly vulnerable to CVE-2009-3555 Traffic resolved 2017-01-02 2016-08-05
28971 Thumbnail generation for private wikis on https://secure.wikimedia.org/ is broken Traffic resolved 2017-01-02 2016-08-05
29448 meta:Special:BannerLoader is loaded using http protocol when using secure server (https) Traffic resolved 2017-01-02 2016-08-05
29622 Setup internal wikis as https only Traffic resolved 2017-01-02 2016-08-05
29909 SSL RSA keys should be at least 2048 bits Traffic resolved 2017-01-02 2016-08-05
29912 status.wikimedia.org should include the status of the secure gateway Traffic resolved 2017-01-02 2016-08-05
29968 JavaScript (geoip lookups) included via plain HTTP on HTTPS sites Traffic resolved 2017-01-02 2016-08-05
31014 Disable SSLv2 on secure.wikimedia.org Traffic resolved 2017-01-02 2016-08-05
31878 Resolution of protocol-independent wgServer for email messages should not be dependent on the performer's url protocol Traffic resolved 2017-01-02 2016-08-05
31898 User preference for enforcing HTTPS Traffic resolved 2017-01-02 2016-08-05
31977 Description url in imageinfo outputs protocol relative urls Traffic resolved 2017-01-02 2016-08-05
31981 Location: redirects on https sites always point to http sites Traffic resolved 2017-01-02 2016-08-05
32294 CSS @import should support relative urls on secure server Traffic resolved 2017-01-02 2016-08-05
32329 SSL termination gateway can't handle large-ish uploads Traffic resolved 2017-01-02 2016-08-05
32560 Bugzilla links to Wikipedia and to MW revisions should use secure server Traffic resolved 2017-01-02 2016-08-05
33323 Mandate only-SSL for accounts with access to private information Traffic resolved 2017-01-02 2016-08-05
33333 No SSL for MobileFrontend m. domain: https://en.m.wikipedia.org/ does not respond Traffic resolved 2017-01-02 2016-08-05
33334 Secure https://en.wikipedia.org/ redirects to insecure http://en.m.wikipedia.org/ on mobile Traffic resolved 2017-01-02 2016-08-05
33335 Rename wikis with multiple subdomains Traffic resolved 2017-01-02 2017-08-23
33369 Non-canonical HTTPS URLs quietly redirect to HTTP Traffic resolved 2017-01-02 2016-08-05
33437 https://www.mediawiki.org/xml/api/ shouldn't have links to http Traffic resolved 2017-01-02 2016-08-05
33539 Use protocol relative URLs on wgUploadNavigationUrl Traffic resolved 2017-01-02 2016-08-05
33563 https://secure.wikimedia.org should redirect to the new same-domain SSL sites Traffic resolved 2017-01-02 2016-08-05
34144 When user is logging out via HTTPS, insecure HTTP cookies keeping logged in state should be cleared as well Traffic resolved 2017-01-02 2016-05-31
34769 Notification emails should link to https, not http Traffic resolved 2017-01-02 2016-08-05
34796 status.wikimedia.org has no (valid) HTTPS Traffic resolved 2017-01-02 2016-06-02
35890 Support SPDY Traffic resolved 2017-01-02 2016-08-05
38456 Native "force https" option Traffic resolved 2017-01-02 2016-08-05
39790 shop.wikimedia.org should be HTTPS only Traffic resolved 2017-01-02 2016-03-02
40516 Enable HSTS on Wikimedia sites Traffic resolved 2017-01-02 2016-06-14
40763 Our *.wikimedia.org cert doesn't properly cover https://pa.us.wikimedia.org/ Traffic resolved 2017-08-24 2017-08-23
41380 Set $wgSecureLogin = true; on Wikimedia wikis Traffic resolved 2017-01-02 2016-08-05
45466 when serving Uzbek Wikipedia, make HTTPS canonical Traffic resolved 2017-01-02 2016-05-29
47765 Firefox search bar not using Secure search Traffic resolved 2017-01-02 2016-08-05
49832 Force all Wikimedia cluster traffic to be over SSL for all users (logged-in and anon) Traffic resolved 2017-01-02 2016-08-05
50402 rel=canonical of https pages should point to http Traffic resolved 2017-01-02 2016-08-05
51467 Server at http://upload.wikimedia.org isn't HTTP/1.1 compliant: doesn't allow absolute URI in request Traffic resolved 2017-01-02 2015-09-22
51698 When uploading from Flickr, unencrypted XHRs are sent from client to Flickr despite being on https:// Traffic resolved 2017-01-02 2016-08-05
53002 Point rel=canonical to HTTPS on every page Traffic resolved 2017-01-02 2016-08-05
54302 Varnish vagrant role Traffic resolved 2017-01-02 2016-05-04
54949 Use "https:" on keywords description Traffic resolved 2017-01-02 2016-08-05
55379 unchecking "Always use a secure connection when logged in" leaves forceHTTPS cookie set Traffic resolved 2017-01-02 2016-08-05
55526 Recent Changes IRC feed should link to the HTTPS diff Traffic resolved 2017-01-02 2016-08-05
63443 Enable gzip transfer encoding for png and ico files in /static Traffic resolved 2017-01-02 2016-05-04
63528 shop switches HTTPS -> HTTP when showing login prompt (on clicking checkout) Traffic resolved 2017-01-02 2016-04-04
63782 Add varnish logs to logstash Traffic resolved 2017-01-02 2017-11-07
70528 stream.wikimedia.org - redirect http(s) to docs Traffic resolved 2017-01-02 2017-07-19
70861 Performance review #2 of Hovercards (Popups extension) Traffic resolved 2017-01-02 2018-04-25
71319 Allow option to uncheck "Always use a secure connection when logged in" at login.wikimedia.org/wiki/Special:Preferences Traffic resolved 2018-03-03 2018-03-02
73267 Enable IPv6 on donate.wikimedia.org Traffic resolved 2017-01-02 2015-09-22
74072 Disable SSL 3.0 on Wikimedia sites to mitigate POODLE attack (CVE-2014-3566) Traffic resolved 2017-06-08 2017-06-07
74514 links.email.donate.wikimedia.org should offer HTTPS Traffic resolved 2017-01-02 2016-03-18
75644 Payment processor website uses RC4 for https encryption Traffic resolved 2017-01-02 2016-08-05
75943 Investigate Varnish 3.0.5plus~x-wm7 throwing on beta cluster "Could not mmap SILO" Traffic resolved 2017-01-02 2016-05-04
76562 Chrome warns about insecure certificate on gerrit.wikimedia.org Traffic resolved 2017-01-02 2016-08-05
81543 Enable IPSec between datacenters Traffic resolved 2017-01-02 2015-09-22
82096 update the multicast purging documentation Traffic resolved 2017-01-02 2017-12-06
82747 pybal health checks are ipv4 even for ipv6 vips Traffic resolved 2017-01-02 2017-07-18
84543 more robust certificate chain creation in puppet Traffic resolved 2017-01-02 2017-10-23
84980 Varnish: the lower the Age value, the slower the request Traffic resolved 2017-01-02 2016-05-31
86063 HTTPS phabricator project(s) Traffic resolved 2017-01-02 2016-07-18
86654 Switch to ECDSA hybrid certificates Traffic resolved 2017-01-02 2015-07-13
86664 HTTPS performance & UA adoption metrics Traffic resolved 2017-01-02 2015-08-27
86666 HTTPS performance tuning Traffic resolved 2017-01-02 2015-08-07
86718 Upgrade eqiad-misc varnish cluster from 2 to 4 systems. Traffic resolved 2017-01-02 2015-08-07
86993 Static image files from en.m.wikipedia.org are served with cache-suppressing headers Traffic resolved 2017-01-02 2016-01-13
87276 Set an explicit "Origin When Cross-Origin" referer policy via the meta referrer tag Traffic resolved 2017-01-02 2018-05-20
87465 create a project for tasks related to WMF domain names Traffic resolved 2017-01-02 2016-07-18
88366 Some traffic is not identified as Zero in Varnish Traffic resolved 2017-01-02 2016-05-31
88862 Enforce HTTPS for authenticated public connections Traffic resolved 2017-01-02 2016-07-12
88873 register .wiki gTLD domains Traffic resolved 2017-01-02 2017-11-16
89120 Upgrade eqiad LVS to 10G Traffic resolved 2017-01-02 2015-07-07
89688 Varnish GeoIP is broken for HTTPS+IPv6 traffic Traffic resolved 2017-01-02 2016-08-18
91347 EventLogging query strings are truncated to 1014 bytes by ?(varnishncsa? or udp packet size?) Traffic resolved 2017-01-02 2016-06-14
91612 acquire SSL certificate for w.wiki Traffic resolved 2017-01-02 2015-12-11
92435 Code changes in VCL to add a Last-Access cookie to track "top domain" uniques [13 pts] {bear} Traffic resolved 2017-01-02 2015-04-27
92438 Changing the URL for the Wikimedia Shop Traffic resolved 2017-06-08 2017-06-07
92602 Secure inter-datacenter web request log (Kafka) traffic Traffic resolved 2017-01-02 2016-01-13
92709 wikitech.wikimedia.org SSL certificate considered "outdated security" in Chrome Traffic resolved 2017-06-08 2017-06-07
93927 Make OCSP Stapling support more generic and robust Traffic resolved 2017-01-02 2016-11-15
94896 Optimize prod's resource domains for SPDY/HTTP2 Traffic resolved 2017-01-02 2017-03-27
95229 Set up an API base path for REST and action APIs Traffic resolved 2017-01-02 2015-08-05
95448 Move bits traffic to text/mobile clusters Traffic resolved 2017-01-02 2017-03-27
96375 Upgrade codfw,ulsfo,esams LVS to jessie Traffic resolved 2017-01-02 2015-09-24
96688 Proxying new services through RESTBase Traffic resolved 2017-01-02 2016-10-12
96847 Refactor varnish puppet config Traffic resolved 2017-01-02 2016-02-19
96848 Support HTTP/2 Traffic resolved 2017-01-02 2018-05-07
96850 Test then switch to openssl 1.0.2 + nginx 1.9.2 Traffic resolved 2017-01-02 2015-06-28
96854 Reboot caches for kernel 3.19.6 globally Traffic resolved 2017-01-02 2015-05-26
96873 reinstall/rename dysprosium as cp1099 (upload eqiad) Traffic resolved 2017-01-02 2015-05-27
97029 integrate (pybal|varnish)->varnish backend config/state with etcd or similar Traffic resolved 2017-01-02 2015-11-30
97206 RFC: Re-evaluate varnish-level request-restart behavior on 5xx Traffic resolved 2017-01-02 2018-03-08
97316 review/rebase/merge the final sslcert patch... Traffic resolved 2017-01-02 2015-06-02
97351 Fix annoying varnishncsa+initsystem issues on jessie Traffic resolved 2017-01-02 2016-05-31
97411 Build a non-trunk 3.19 kernel for jessie Traffic resolved 2017-01-02 2015-06-03
97506 Use OCSP Stapling on misc cluster Traffic resolved 2017-01-02 2015-05-20
97750 wmgUseBits = false still loads favicon from bits.wikimedia.org Traffic resolved 2017-01-02 2015-05-03
97970 Package a modern version of etcd for jessie, trusty Traffic resolved 2017-01-02 2015-08-07
97972 Figure out a security model for etcd Traffic resolved 2017-01-02 2016-01-13
97973 Create an etcd puppet module + find suitable servers for deployment Traffic resolved 2017-01-02 2015-06-11
97974 Create a confd puppet module Traffic resolved 2017-01-02 2015-06-16
97975 Integrate confd into the varnish configuration to generate the list of active backends Traffic resolved 2017-01-02 2015-07-01
97978 Create a tool to sync static configuration from a repository to the consistent k/v store Traffic resolved 2017-01-02 2015-06-16
98003 Upgrade prod DNS daemons to gdnsd 2.2.0 Traffic resolved 2017-01-02 2015-08-07
98203 Fix cpufrequtils issues on jessie Traffic resolved 2017-01-02 2015-05-26
98309 Mobile site broken Traffic resolved 2017-01-02 2015-05-06
98373 Replace bits URL in Firefox app, if possible Traffic resolved 2017-01-02 2016-08-03
98676 Create fishbowl wiki for Wikimedia User Group China Traffic resolved 2017-08-24 2017-08-23
98758 Upgrade beta-cluster caches to jessie Traffic resolved 2017-01-02 2016-01-11
98803 Varnish does not honor Cache-Control for Graphoid Traffic resolved 2017-01-02 2015-05-14
99096 Make Varnish cache for /static/$wmfbranch/ expire when resources change within branch lifetime Traffic resolved 2017-01-02 2017-11-02
99226 Switch Varnish's GeoIP code to libmaxminddb/GeoIP2 Traffic resolved 2017-01-02 2016-08-24
99343 Link to switch back to desktop in mobile view is forcing HTTP even if viewing over HTTPS Traffic resolved 2017-12-02 2018-03-07
99372 Graphoid access logs are missing from Hadoop Traffic resolved 2017-01-02 2016-08-11
99839 Sanitize varnish director-level retries Traffic resolved 2017-01-02 2015-05-21
100519 Phabricator needs to expose ssh Traffic resolved 2017-01-02 2016-08-18
100902 Get rid of geoiplookup service Traffic resolved 2017-01-02 2017-08-19
101060 wikipedia.es Traffic resolved 2017-01-02 2016-09-02
101204 setup/install/deploy cp2001-cp2026 Traffic resolved 2017-01-02 2015-06-19
101206 rack & on-site setup of cp2001-cp2026 Traffic resolved 2017-01-02 2015-06-19
101435 SPDY traffic isn't gzipped for Firefox Traffic resolved 2017-01-02 2015-06-08
101858 Create a confd template for pybal files that will work with our etcd schema. Traffic resolved 2017-01-02 2015-06-29
101892 Ensure {text-domain}/w/load.php requests do not bypass cache for session cookies Traffic resolved 2017-01-02 2015-06-28
102393 Make pybal accept 30[12] for ProxyFetch Traffic resolved 2017-01-02 2015-11-30
102394 Implement pybal pool state monitoring and alerting via icinga Traffic resolved 2017-01-02 2016-02-09
102457 Make Magnus tools on tools.wmflabs.org work in HTTPS Traffic resolved 2017-01-02 2017-01-29
102814 Decom old multiple-subdomain wikis in wikipedia.org Traffic resolved 2017-01-02 2015-09-14
102815 Decom www.$lang hostnames/redirects Traffic resolved 2017-01-02 2015-09-14
102824 Clean up DNS/redirects for TLS Traffic resolved 2017-01-02 2016-07-27
102826 Fix/decom multiple-subdomain wikis in wikimedia.org Traffic resolved 2017-01-02 2015-09-22
102827 Decide what to do with *.donate.wikimedia.org subdomain + TLS Traffic resolved 2017-01-02 2016-04-12
102991 Verify traffic to static resources from past branches does indeed drain Traffic resolved 2017-01-02 2016-01-13
103226 eqiad: investigate thermal issues with some cp10xx machines Traffic resolved 2017-01-02 2016-01-29
103660 Migrate Parsoid cache from Trusty to Jessie Traffic resolved 2017-01-02 2016-05-04
103773 check if services behind misc-web enforce http->https redirect or not Traffic resolved 2017-01-02 2015-06-28
103919 let all services on misc-web enforce http->https redirects Traffic resolved 2017-01-02 2016-04-13
103921 pybal DNS lookup issues causing outage risks Traffic resolved 2017-01-02 2017-07-04
104004 Can't download large datasets from datasets.wikimedia.org Traffic resolved 2017-01-02 2015-12-11
104076 Puppet failing on deployment-prep caches Traffic resolved 2017-01-02 2015-09-12
104143 openssl 1.0.2 packaging for jessie Traffic resolved 2017-01-02 2015-07-01
104211 Renew toolserver.org SSL certificate Traffic resolved 2017-06-08 2017-06-07
104225 enwiki Main_Page timeouts Traffic resolved 2017-01-02 2017-07-22
104244 Preload HSTS Traffic resolved 2017-01-02 2016-06-06
104274 Switch to explicit ciphersuite Traffic resolved 2017-01-02 2015-06-30
104281 Sort out DHE for Forward Secrecy w/ older clients Traffic resolved 2017-01-02 2015-07-07
104359 https://wikitech.wikimedia.org/beacon/statsv 404 Not Found Traffic resolved 2017-01-02 2016-05-04
104532 /static generates (and caches!) redirect loops on cache-miss Traffic resolved 2017-01-02 2016-05-04
104728 make blog links from wmfwiki front page use HTTPS links Traffic resolved 2017-01-02 2016-08-19
104936 move annual report from zirconium to bromine Traffic resolved 2017-01-02 2015-09-24
104937 move transparency report from zirconium to bromine Traffic resolved 2017-01-02 2015-07-13
104942 TLS and *.wap/*.mobile multi-level subdomains of wikipedia.org Traffic resolved 2017-01-02 2015-08-14
105076 Set up standard HTTPS Termination -> 2layer caching for maps service Traffic resolved 2017-01-02 2015-08-27
105455 Xbox 360 Internet Explorer unable to view Wikipedia Traffic resolved 2017-01-02 2016-09-16
105734 Varnish caches Grafana dashboard configuration too strongly Traffic resolved 2017-01-02 2015-09-22
105794 Insecure POST traffic Traffic resolved 2017-01-02 2016-09-23
105905 Switch blog to HTTPS-only Traffic resolved 2017-01-02 2017-04-06
105999 Redirect yue.wikipedia.org to zh-yue.wikipedia.org Traffic resolved 2017-01-02 2017-06-09
106311 pywikipedia.org is not responding; pywikibot.org is not registered Traffic resolved 2017-01-02 2016-12-16
106620 Figure out purging of static logos for updates Traffic resolved 2017-01-02 2016-05-04
106662 deployment-parsoidcache02 fails varnish VCL compilation Traffic resolved 2017-01-02 2016-05-04
107030 Restbase insecure POST requests to MW api.php Traffic resolved 2017-01-02 2015-08-07
107060 DNS request for wikimedia.org Traffic resolved 2017-01-02 2015-08-07
107086 Provide an API listing at /api/ Traffic resolved 2017-01-02 2016-02-29
107430 Decom bits.wikimedia.org hostname Traffic resolved 2017-01-02 2017-04-06
107575 download.wiki[mp]edia.org are using an invalid certificate Traffic resolved 2017-01-21 2017-01-20
107602 Set up a public interface to the wikidata query service Traffic resolved 2017-01-02 2015-09-22
107627 Quarry should be HTTPS-only Traffic resolved 2017-01-02 2016-05-03
107940 DNS request for wikimedia.org (let 3rd party send mail as wikimedia.org) Traffic resolved 2017-01-02 2015-08-27
108579 Getting ssl_error_inappropriate_fallback_alert very rarely Traffic resolved 2017-01-02 2016-05-31
108612 Fix Varnish TTLs across the board Traffic resolved 2017-01-02 2015-09-22
108649 Set up "w.wiki" domain for usage with UrlShortener Traffic resolved 2017-01-02 2017-11-16
108827 Investigate TCP Fast Open for tlsproxy Traffic resolved 2017-01-02 2016-08-15
109162 Set up proper edge Varnish caching for maps cluster Traffic resolved 2017-01-02 2016-04-26
109226 Sometimes apache error 503s redirect to /503.html and this redirect gets cached Traffic resolved 2017-01-02 2016-10-25
109286 Merge mobile cache into text cache Traffic resolved 2017-01-02 2016-02-08
109712 Invalid OCSP signing certificate in OCSP response : can't read Wikimedia websites on Firefox Traffic resolved 2017-01-02 2015-08-27
109737 ocsp updater: handle openssl "trylater" and similar more-gracefully Traffic resolved 2017-01-02 2016-10-03
109738 ocsp updater: validate the signature expiry lifetime Traffic resolved 2017-01-02 2016-10-03
109740 ocsp updater: re-enable automatic updates Traffic resolved 2017-01-02 2015-08-27
110065 Switch codfw caches to tier2, begin pushing some traffic through them to test Traffic resolved 2017-01-02 2015-09-22
110197 SSL certificate for policy.wikimedia.org Traffic resolved 2017-01-02 2016-07-28
110472 Decom parsoidcache cluster Traffic resolved 2017-01-02 2016-04-19
110475 Remove restbase from parsoidcache Traffic resolved 2017-01-02 2016-01-29
110476 Remove citoid from parsoidcache Traffic resolved 2017-01-02 2016-01-29
110477 Remove graphoid from parsoidcache Traffic resolved 2017-01-02 2016-01-28
110478 Remove cxserver from parsoidcache cluster Traffic resolved 2017-01-02 2016-01-29
110530 Fix ethernet startup race on HP LVS w/ jessie Traffic resolved 2017-01-02 2015-12-11
110899 figure out what's wrong with confd + varnish currently Traffic resolved 2017-01-02 2015-09-22
110954 Upgrade Pybal to 1.08 Traffic resolved 2017-01-02 2017-03-08
111045 zerofetcher in production is getting throttled for API logins Traffic resolved 2017-01-02 2015-11-30
111715 ru.wikinews.org Parsoid backend is down Traffic resolved 2017-01-02 2015-09-22
112195 Beta giving Error: 403, Insecure POST Forbidden Traffic resolved 2017-01-02 2015-09-22
112521 Track/notify cert expiries better Traffic resolved 2017-01-02 2017-10-23
112765 Phabricator needs to expose notification daemon (websocket) Traffic resolved 2017-01-02 2017-11-07
112836 allow maps cluster Varnish cache purging Traffic resolved 2017-01-02 2016-05-26
113007 [Research] Investigate 30% load.php reqs increase since 2015-07-30 Traffic resolved 2017-01-02 2016-08-19
113122 maps: support wikivoyages in incubator Traffic resolved 2017-01-02 2015-10-26
113151 pybal fails to detect dead servers under production lb IPs for port 80 Traffic resolved 2017-01-02 2016-02-09
113154 LVS HTTPS IPv6 on mobile-lb.eqiad alert occasionally flapping Traffic resolved 2017-01-02 2015-10-05
113184 cp1046 Varnish backend panic Traffic resolved 2017-01-02 2015-09-25
113223 Varnish not letting through RESTBase back-end service responses for rest.wm.org Traffic resolved 2017-01-02 2016-05-04
113319 Saving preferences or blocking (and probably various other things) give 403 error Traffic resolved 2017-01-02 2015-09-22
113591 Enable caching for the Mobile Content Service's RESTBase public endpoints Traffic resolved 2017-01-02 2016-05-04
113639 cp1046 is crashing and becoming unresponsive Traffic resolved 2017-01-02 2015-10-26
114059 ssl expiry tracking in icinga - we don't monitor that many domains Traffic resolved 2017-01-02 2016-03-23
114370 Flag in x-analytics in varnish any request that comes with no cookies whatsoever {bear} [5 pts] Traffic resolved 2017-01-02 2015-10-26
114659 Re-balance North American traffic in DNS for codfw Traffic resolved 2017-01-02 2016-09-14
114922 Transfer of domain names to WMF servers Traffic resolved 2017-01-02 2016-02-24
114995 [Bug] Varnish cache invalidation not working for m.wikidata.org Traffic resolved 2017-01-02 2016-05-04
115028 SSL cert needed for benefactorevents.wikimedia.org Traffic resolved 2017-01-02 2016-03-09
115191 ml.wikipedia.org not redirecting to mobile site while accessing from a mobile device; many "Error: Module not found" errors Traffic resolved 2017-01-02 2016-12-24
116752 Split HTCP multicast addresses Traffic resolved 2017-01-02 2015-11-30
116806 phabricator.wikimedia.org has no SPF record Traffic resolved 2017-01-02 2016-05-10
116898 On beta cluster varnish stats process points to production statsd Traffic resolved 2017-01-02 2016-08-29
117015 Investigate PyBal dead on lvs1007-12 Traffic resolved 2017-01-02 2015-11-30
117746 cp4007 crashed Traffic resolved 2017-01-02 2015-11-30
118078 Clean up nginx / nginx::ssl classes and usage Traffic resolved 2017-01-02 2016-05-31
118365 Increase request limits for GETs to /api/rest_v1/ Traffic resolved 2017-01-02 2017-06-14
118388 configure https for www.wikilovesmonuments.org Traffic resolved 2017-01-02 2016-09-08
118394 cache_upload should give an informative 404 rather than 403 on req.http.host != upload.wikimedia.org Traffic resolved 2017-01-02 2016-10-04
118428 mobile [[w:en:GrandCentral]] was not purged on edit Traffic resolved 2017-01-02 2016-05-04
118787 releases.wikimedia.org should be https only and have hsts set Traffic resolved 2017-01-02 2017-06-07
118790 Decom observium.wikimedia.org Traffic resolved 2017-01-02 2017-06-07
118830 Backport etcd 2.2 to jessie Traffic resolved 2017-01-02 2016-01-13
118831 Upgrade the production etcd cluster to 2.2 Traffic resolved 2017-01-02 2016-01-13
118833 Upgrade conftool to support credentials form a config file Traffic resolved 2017-01-02 2016-01-13
118834 Upgrade python-etcd to 0.4.2+ Traffic resolved 2017-01-02 2016-01-13
118892 Update CP cookie VCL once HTTP/2 support lands Traffic resolved 2017-01-02 2016-08-05
118948 Pybal 1.12 has issues with executing ipvsadm commands Traffic resolved 2017-01-02 2016-01-13
119144 EventLogging sees too few distinct client IPs {oryx} [8 pts] Traffic resolved 2017-01-02 2017-06-07
119394 Convert misc cluster to 2-layer Traffic resolved 2017-01-02 2015-11-30
119515 Upgrade LVS servers to a 4.3+ kernel Traffic resolved 2017-01-02 2016-02-11
119517 upgrade lvs1001-3 to jessie Traffic resolved 2017-01-02 2015-12-11
119576 Mark cookies from varnish as secure Traffic resolved 2017-01-02 2016-04-19
119917 Set up backend per-IP limits on varnish for WDQS Traffic resolved 2017-01-02 2017-02-22
120143 Investigate if www.m.wikipedia.org needs to stay around Traffic resolved 2017-01-02 2016-02-22
120151 Improve handling of mobile variants in Varnish Traffic resolved 2017-01-02 2016-06-15
120322 donate.wikimedia.org needs an MX record Traffic resolved 2017-01-02 2018-01-11
121135 Banners fail to show up occassionally on Russian Wikivoyage Traffic resolved 2017-01-02 2016-04-28
122319 ssl certificate replacement: tendril.wikimedia.org (expires 2016-02-15) Traffic resolved 2017-01-02 2016-05-03
122321 ssl certificate replacement: dumps.wikimedia.org (expires 2016-02-26) Traffic resolved 2017-01-02 2016-02-23
122453 Wikimedia servers won't serve images Traffic resolved 2017-01-02 2016-01-13
122455 Varnish leaks memory Traffic resolved 2017-01-02 2016-05-31
122462 Varnish Assert error in VGZ_Ibuf() Traffic resolved 2017-01-02 2016-05-31
122673 Varnish apparently unconditionally varies on session cookies Traffic resolved 2017-01-02 2016-01-13
122880 Evaluate and Test Limited Deployment of Varnish 4 Traffic resolved 2017-01-02 2016-08-01
122881 Install XKey vmod Traffic resolved 2017-01-02 2016-11-28
123431 Investigate if login.m.wikimedia.org needs to stay around Traffic resolved 2017-01-02 2016-03-28
124165 Fix mobile purging Traffic resolved 2017-01-02 2016-01-22
124166 Fix varnish handling of mobile hostname rewriting Traffic resolved 2017-01-02 2016-01-20
124277 Forward-port Varnish 3 patches to Varnish 4 Traffic resolved 2017-01-02 2016-01-29
124279 Forward-port maps VCL to Varnish 4 Traffic resolved 2017-01-02 2016-03-31
124281 Create separate packages for required vmods Traffic resolved 2017-01-02 2016-04-01
124356 Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) Traffic resolved 2017-01-02 2017-07-27
124482 Use Text IP for Mobile hostnames to gain SPDY/H2 coalesce between the two Traffic resolved 2017-01-02 2016-05-04
124711 Varnish failed to provision Traffic resolved 2017-01-02 2016-05-04
124835 Automate and/or better-document varnish ban procedure for operations staff, so it can be accomplished with more speed and confidence in outage conditions Traffic resolved 2017-01-02 2016-01-27
124954 Decrease max object TTL in varnishes Traffic resolved 2017-01-02 2017-10-23
125397 pybal etcd coroutine crashed Traffic resolved 2017-01-02 2016-02-09
125401 3 Varnish cache_upload servers crashed in a short time window Traffic resolved 2017-01-02 2016-05-31
125485 esams cache cluster re-arrangements, early 2016 Traffic resolved 2017-01-02 2016-03-24
125486 eqiad cache cluster re-arrangements Traffic resolved 2017-01-02 2016-03-24
125510 Traffic Infrastructure support for Mar 2016 codfw rollout Traffic resolved 2017-01-02 2016-03-23
125841 Purge pages cached with mobile editlinks Traffic resolved 2017-01-02 2016-04-06
125938 PHP fatal errors causing Varnish to return 503 - "Junk after gzip data" Traffic resolved 2017-01-02 2018-03-26
125979 Disable SPDY on cache_text for a week Traffic resolved 2017-01-02 2016-08-05
126015 Understand and improve streaming behaviour from Varnish Traffic resolved 2017-01-02 2016-05-09
126053 Send X-Carrier + X-Carrier-Meta headers on all responses Traffic resolved 2017-01-02 2016-02-29
126062 cp30[34]x hw/firmware/BMC issues Traffic resolved 2017-01-02 2016-05-31
126206 Upgrade to Varnish 4: things to remember Traffic resolved 2017-01-02 2017-04-18
126321 lvs2002 Embedded Flash/SD-CARD iLO errors Traffic resolved 2017-01-02 2017-04-07
126616 openssl-1.0.2f introduced minor bug with nginx Traffic resolved 2017-01-02 2016-02-29
127014 Empty result on a tree query Traffic resolved 2017-01-02 2017-07-14
127021 Stop default redirecting Samsung Smart TVs to mobile web Traffic resolved 2017-01-02 2016-05-04
127094 ulsfo possible downtime - PDU swaps in both cabinets Traffic resolved 2017-01-02 2016-02-18
127294 https://integration.wikimedia.org/ci/api/json is corrupted when required more than one time in a raw Traffic resolved 2017-01-02 2016-02-18
127370 Varnish not purging RESTBase URIs Traffic resolved 2017-01-02 2017-10-17
127436 Logging out of a wiki leaves an XXwikiSession= Cookie behind Traffic resolved 2017-01-02 2016-08-15
127481 Refactor VCL for 3-tier capabilities and source-DC switching Traffic resolved 2017-01-02 2016-03-04
127484 Refactor VCL for applayer datacenter-switching Traffic resolved 2017-01-02 2016-03-07
127492 Switch ulsfo to backend to codfw rather than eqiad Traffic resolved 2017-01-02 2016-03-08
127931 Number of 5xx (mainly 503) errors surge from text caches since February 17th Traffic resolved 2017-01-02 2016-02-24
128085 figure out if we can park wicipediacymraeg.org Traffic resolved 2017-01-02 2016-03-01
128182 Server certificate is classified as invalid on government computers Traffic resolved 2017-01-02 2017-10-23
128199 confctl: improve/upgrade --tags/--find Traffic resolved 2017-01-02 2016-04-28
128587 http://dumps.wikimedia.org should redirect to https:// Traffic resolved 2017-01-02 2016-04-08
128709 Segment Navigation Timing data by continent Traffic resolved 2017-01-02 2017-05-26
128788 Port varnishlog.py to new VSL API Traffic resolved 2017-01-02 2016-03-31
128813 cache_misc's misc_fetch_large_objects has issues Traffic resolved 2017-01-02 2016-04-24
128961 Images not showing up at Commons Traffic resolved 2017-01-02 2016-04-20
128968 Register nlwikipedia.org to prevent squatting Traffic resolved 2017-01-02 2016-11-30
129270 Fix puppet on deployment-cache* hosts in beta cluster Traffic resolved 2017-01-02 2016-03-25
129344 varnishkafka logrotate cronspam Traffic resolved 2017-01-02 2016-04-14
129424 Varnish support for shutting users out of a DC Traffic resolved 2017-01-02 2016-04-01
130414 delete links.email.donate.wikimedia.org (and all other email.donate.*?) from DNS Traffic resolved 2017-01-02 2016-04-22
130449 Do something better than an "Unauthorized" error page at https://upload.wikimedia.org/ Traffic resolved 2017-01-02 2016-03-23
130647 alias /apple-app-site-association and /.well-known/apple-app-site-association Traffic resolved 2017-01-02 2017-01-17
130790 Add Charles Roslof to dns-admin alias Traffic resolved 2017-01-02 2016-03-24
130937 Updating DNS records for Major Gifts subdomain (benefactors.wikimedia.org) Traffic resolved 2017-01-02 2017-06-26
130981 irc.wikimedia.org talks HTTP but not HTTPS Traffic resolved 2017-01-02 2016-03-31
131053 investigate radon crash Traffic resolved 2017-01-02 2016-04-11
131131 Canonical URL in Store points to HTTP address, should be HTTPS Traffic resolved 2017-01-02 2017-06-26
131163 Improve varnish stop for backend instances Traffic resolved 2017-01-02 2016-03-29
131353 Port remaining scripts depending on varnishlog.py to new VSL API Traffic resolved 2017-01-02 2016-10-21
131499 Upgrade all cache clusters to Varnish 4 Traffic resolved 2017-01-02 2016-11-24
131501 Convert misc cluster to Varnish 4 Traffic resolved 2017-01-02 2016-07-27
131502 Convert upload cluster to Varnish 4 Traffic resolved 2017-01-02 2016-11-02
131503 Convert text cluster to Varnish 4 Traffic resolved 2017-01-02 2016-11-24
131760 Add icinga monitoring for varnish statistics daemons Traffic resolved 2017-01-02 2016-04-13
131761 Solve large-object/stream/pass/chunked in upload cluster better Traffic resolved 2017-01-02 2016-08-19
131775 codfw: (1) phabricator host (backup node) Traffic resolved 2017-01-02 2016-06-14
131830 Varnish 4 panic log registered on cp1044 Traffic resolved 2017-01-02 2016-04-08
131880 codfw/eqiad/esams/ulsfo: (4) servers for maps caching cluster Traffic resolved 2017-01-02 2016-04-26
131908 Support TLS chacha20-poly1305 AEAD ciphers Traffic resolved 2017-01-02 2016-09-02
132337 MediaWiki-Vagrant varnish role fails to provision Traffic resolved 2017-01-02 2016-05-04
132346 cronspam from cpXXXX hosts related to varnishkafka non existent processes Traffic resolved 2017-01-02 2016-04-27
132374 Generate a list of junk CN cookies being sent by clients Traffic resolved 2017-01-02 2016-07-20
132430 varnishstatsd crashes with ValueError in vsl_callback without being restarted by systemd Traffic resolved 2017-01-02 2016-05-11
132450 enable https for (ubuntu|apt|mirrors).wikimedia.org Traffic resolved 2017-01-02 2016-04-26
132452 HSTS preload for wmfusercontent.org Traffic resolved 2017-01-02 2016-04-19
132457 Move wdqs to an LVS service Traffic resolved 2017-01-02 2016-10-12
132458 Move logstash.wikimedia.org (kibana) to an LVS service Traffic resolved 2017-01-02 2016-10-21
132459 HTTPS redirects for config-master.wikimedia.org Traffic resolved 2017-01-02 2016-04-13
132460 HTTPS redirects for git.wikimedia.org Traffic resolved 2017-01-02 2016-04-13
132461 HTTPS redirects for graphite.wikimedia.org Traffic resolved 2017-01-02 2016-04-13
132462 HTTPS redirects for parsoid-tests.wikimedia.org Traffic resolved 2017-01-02 2016-04-13
132463 HTTPS redirects for datasets.wikimedia.org Traffic resolved 2017-01-02 2016-04-25
132464 HTTPS redirects for transparency.wikimedia.org Traffic resolved 2017-01-02 2016-04-13
132465 HTTPS redirects for stats.wikimedia.org Traffic resolved 2017-01-02 2016-04-13
132474 varnishmedia: repeated calls to flush_stats() Traffic resolved 2017-01-02 2016-06-17
132521 Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination Traffic resolved 2017-01-02 2017-07-11
132543 enable HSTS on *.planet.wikimedia.org Traffic resolved 2017-01-02 2016-04-13
132685 Preload STS for wikimedia.org Traffic resolved 2017-01-02 2016-06-17
132812 Sort out letsencrypt puppetization for simple public hosts Traffic resolved 2017-01-02 2016-04-27
132835 cronspam from cpXXXX hosts due to update-ocsp-all and zero_fetch Traffic resolved 2017-01-02 2016-06-17
132969 Wikipedia app hits loads.php on bits.wikimedia.org Traffic resolved 2017-01-02 2016-08-18
133217 Fix apache-2.4 + DHE ciphersuites issue Traffic resolved 2017-01-02 2016-04-26
133360 Fix wikitech-static TLS config Traffic resolved 2017-01-02 2016-04-21
133485 Configure apache rules for UrlShortener Traffic resolved 2017-01-02 2016-06-23
133490 Wikidata Query Service REST endpoint returns truncated results Traffic resolved 2017-01-02 2016-05-09
133791 check_dns needs to be rewritten Traffic resolved 2017-01-02 2017-10-13
133866 Varnish seems to sometimes mangle uncompressed API results Traffic resolved 2017-01-02 2016-05-10
133915 lvs1012 eth1 NIC Link flapping Traffic resolved 2017-01-02 2016-04-28
133936 Beta cluster varnish sets overly broad domain on GeoIP cookie Traffic resolved 2017-01-02 2016-08-18
133988 Questions about map tile cache performance Traffic resolved 2017-01-02 2016-04-29
134346 varnish text on beta is unreachable / stuck Traffic resolved 2017-01-02 2016-06-30
134362 beta cluster varnish cache can't apt-get upgrade nginx-full: nginx: [emerg] unknown "spdy" variable Traffic resolved 2017-01-02 2016-06-17
134404 Varnish support for active:active backend services Traffic resolved 2017-01-02 2017-10-23
134542 Graph results are not being cached in Varnish Traffic resolved 2017-01-02 2016-06-17
134869 Secure connection failed when attempting to send POST request using HTTP/2 (if connection has been idle for a certain time) Traffic resolved 2017-01-02 2016-07-27
134870 Support websockets in cache_misc Traffic resolved 2017-01-02 2016-06-17
134871 Move stream.wikimedia.org (rcstream) behind cache_misc Traffic resolved 2017-01-02 2016-08-11
134989 WDQS empty response - transfer clsoed with 15042 bytes remaining to read Traffic resolved 2017-01-02 2016-07-27
135227 varnish.clients graphite metric spammed with jsessionid Traffic resolved 2017-01-02 2016-06-17
135384 Raise cache frontend memory sizes significantly Traffic resolved 2017-01-02 2016-08-25
135470 Lots of Title::purgeExpiredRestriction from API DELETE FROM `page_restrictions` WHERE (pr_expiry < '20160517063108') without batching/throttling potentially causing lag on s5-api Traffic resolved 2017-01-02 2016-05-31
135480 arbcom-nl.wikipedia.org doesn't have a functioning mobile website Traffic resolved 2017-01-02 2017-09-09
135515 graphite.wikimedia.org 503s on some css/js resources Traffic resolved 2017-01-02 2016-05-18
135696 Sort out vcl_deliver vs vcl_synth mess with v4 VCL Traffic resolved 2017-01-02 2016-06-17
135700 Varnishlog doesn't properly rotates logs, varnish.log is empty since forever (was: deployment-cache-upload04 (m1.medium) / is almost full) Traffic resolved 2017-01-02 2016-06-17
135762 A/B Testing solid framework Traffic resolved 2017-01-02 2017-10-23
136067 Ashburn servers almost unreachable from part of Europe Traffic resolved 2017-01-02 2016-05-24
136557 Create moon.wikimedia.org and redirect it to https://meta.wikimedia.org/wiki/Wikipedia_to_the_Moon Traffic resolved 2017-01-02 2016-06-17
136674 Help contact bot owners about the end of HTTP access to the API Traffic resolved 2017-01-02 2016-07-19
136737 Fix lvs1001-6 storage Traffic resolved 2017-01-02 2018-02-27
137106 californium and gallium ferm rules should disallow public access to port 80 Traffic resolved 2017-01-02 2016-06-17
137114 Scripts depending on varnishlog.py maxing out CPU usage on cache_misc Traffic resolved 2017-01-02 2016-06-17
137161 Fix nits in HTTPS/HSTS configs in externally-hosted fundraising domains Traffic resolved 2017-01-02 2017-07-11
137238 Tune WDQS caching headers Traffic resolved 2017-01-02 2016-08-24
137258 policy.wikimedia.org SSL vulnerability Traffic resolved 2017-01-02 2016-06-08
137609 Make upload.wikimedia.org cookieless Traffic resolved 2017-01-02 2017-03-07
137620 Send traffic to new maps200? servers Traffic resolved 2017-01-02 2016-06-17
137707 Antigng-bot improper non-api http requests Traffic resolved 2017-01-02 2016-07-14
137848 Remove referrer check from varnish for maps cluster Traffic resolved 2017-01-02 2016-06-17
138010 Investigate Android app API request latency regression Traffic resolved 2016-06-17 2016-06-22
138084 unix domain socket listening for varnish4 Traffic resolved 2017-01-02 2017-04-18
138682 https://ores.wikimedia.org redirects me to HTTP when I don't include a trailing slash Traffic resolved 2017-01-02 2016-08-12
140658 Make sure we're not relying on HTTP_PROXY headers Traffic resolved 2017-01-02 2017-04-18
141170 Strip query parameters from w.wiki domain Traffic resolved 2017-01-02 2016-08-03
141373 Age header reset to 0 after 24 hours on varnish frontends Traffic resolved 2017-01-02 2017-10-24
141454 piuparts fail with WARN: Broken symlinks: /etc/systemd/system... Traffic resolved 2017-01-02 2016-11-18
141687 Cached outdated revisions served to logged-out users Traffic resolved 2017-01-02 2016-08-11
141786 TLS stats regression related to Chrome/41 on Windows Traffic resolved 2017-01-02 2017-09-13
142076 Analyze Range requests on cache_upload frontend Traffic resolved 2017-01-02 2016-08-15
142131 Retire gerrit.wikimedia.org SSL cert Traffic resolved 2017-01-02 2016-08-08
142233 Varnish 4 stalls with two consecutive Range requests using HTTP persistent connections Traffic resolved 2017-01-02 2016-08-24
142410 Correct cache_status field on webrequest dataset Traffic resolved 2017-01-02 2016-08-11
142810 varnishd: Assert error in smp_oc_getobj(), storage/storage_persistent_silo.c line 417 Traffic resolved 2017-01-02 2016-10-03
142848 Stop using persistent storage in our backend varnish layers. Traffic resolved 2017-01-02 2016-10-03
142944 Performance and caching considerations for article placeholders accesses Traffic resolved 2017-01-02 2017-01-31
143078 Unhandled pybal ValueError: need more than 1 value to unpack Traffic resolved 2017-01-02 2017-04-03
143132 Expose the PDF rendering service via RESTBase Traffic resolved 2017-01-02 2016-12-05
143270 ULS GeoIP should not use meta.wm.o/geoiplookup Traffic resolved 2017-01-02 2016-08-26
143694 Preliminary Design document for A/B testing Traffic resolved 2017-01-02 2017-03-22
144158 varnishkafka frequently disconnects from kafka servers Traffic resolved 2017-01-02 2016-08-30
144194 Varnish-triggered CN campaign about browser security Traffic resolved 2017-01-02 2017-10-23
144293 adjust ssl certificate montioring to differentiate between standard and LE certificates. Traffic resolved 2017-01-02 2016-11-16
144523 OpenSSL 1.1 deployment for cache clusters Traffic resolved 2017-01-02 2016-11-10
144626 Strong cipher preference ordering for cache terminators Traffic resolved 2017-01-02 2017-10-23
145120 update *.wmflabs.org certificate (existing expires on 2016-09-16) Traffic resolved 2017-01-02 2016-09-12
145244 rcstream support defaults to stream.wikimedia.org:80 Traffic resolved 2017-01-02 2016-09-10
145661 varnish backends start returning 503s after ~6 days uptime Traffic resolved 2017-01-02 2018-03-29
145758 Maps - move traffic to eqiad instead of codfw Traffic resolved 2017-01-02 2016-10-24
146451 repeated 503 errors for 90 minutes now Traffic resolved 2017-01-02 2016-11-26
146638 Remove "GeoIP lookup" service from https://status.wikimedia.org Traffic resolved 2017-01-02 2016-09-27
147116 Upload cache in beta is broken Traffic resolved 2017-01-02 2016-10-04
147199 Removing support for DES-CBC3-SHA TLS cipher (drops IE8-on-XP support) Traffic resolved 2017-01-02 2018-01-26
147424 Port varnish metrics from ganglia to prometheus Traffic resolved 2017-01-02 2016-11-14
147426 Port gdnsd statistics from ganglia to prometheus Traffic resolved 2017-01-02 2017-10-13
147429 Port vhtcpd statistics from ganglia to prometheus Traffic resolved 2017-01-02 2016-12-06
147480 Upgrade conftool to 0.3.1 Traffic resolved 2017-01-02 2016-10-24
147569 Evaluate/Deploy TCP BBR when available (kernel 4.9+) Traffic resolved 2017-01-02 2017-05-30
147784 Mediawiki thumbnail requests for 0px should result in http 400 not 500 Traffic resolved 2017-01-02 2016-12-15
147814 Favicon broken on doc.wikimedia.org and integration.wikimedia.org (HTTP 500) Traffic resolved 2017-01-02 2016-10-11
147844 Standardize varnish applayer backend definitions Traffic resolved 2017-01-02 2017-10-23
147845 Move rcstream to an LVS service Traffic resolved 2017-01-02 2016-12-19
147847 Move pybal_config to an LVS service Traffic resolved 2017-01-02 2016-10-19
147848 Use hostnames (not IPs) in deployment-prep varnish app_directors Traffic resolved 2017-01-02 2016-10-12
148045 Windows 10 & MacOS Sierra Certificate errors due to GlobalSign Traffic resolved 2017-01-02 2017-06-12
148069 GlobalSign intermediate updates for one-offs Traffic resolved 2017-01-02 2016-10-14
148076 OCG failing with new GlobalSign intermediate workaround Traffic resolved 2017-01-02 2016-10-13
148131 Deploy redundant unified certs Traffic resolved 2017-01-02 2017-10-23
148490 Extend check_sslxnn to check OCSP Stapling Traffic resolved 2017-01-02 2016-10-31
148497 Some JPGs are being served as text Traffic resolved 2017-01-02 2017-01-27
148723 reimage cp1047 Traffic resolved 2017-01-02 2016-11-02
148780 mobile-safari has very few internally-referred pageviews Traffic resolved 2017-01-02 2018-03-06
148830 cache_upload: uncompressed images with Content-Encoding: gzip cause content decoding issues Traffic resolved 2017-01-02 2018-02-13
148891 cp1052 ethernet link down 2016-10-22 14:11 Traffic resolved 2017-01-02 2017-04-04
148893 nginx SSL_do_handshake spam filling disks Traffic resolved 2017-01-02 2016-10-31
148917 Thumbnails failing to render sporadically (ERR_CONNECTION_CLOSED or ERR_SSL_BAD_RECORD_MAC_ALERT) Traffic resolved 2017-01-02 2016-11-15
149217 cp1066.mgmt.eqiad.wmnet is unreachable Traffic resolved 2017-01-02 2017-04-27
149295 Restbase redirects with cors not working on Android 4 native browser Traffic resolved 2017-01-02 2016-11-10
149784 varnish-backend: weekly cron restart for all clusters Traffic resolved 2017-01-02 2016-11-24
149843 reimage cp4016 and cp1055 Traffic resolved 2017-01-02 2016-11-07
149865 503 errors for users connecting to esams Traffic resolved 2017-01-02 2016-11-11
149875 mgmt hosts that exist but don't resolve to an IP Traffic resolved 2017-01-02 2016-11-16
149881 varnish-be not restarting correctly because of disk space issues Traffic resolved 2017-01-02 2016-11-15
150168 Make our docker registry public Traffic resolved 2017-01-02 2016-11-08
150247 Varnish4 is unexpectedly retrying certain applayer failure cases Traffic resolved 2017-01-02 2017-11-06
150503 Failure to save recent changes Traffic resolved 2017-01-02 2016-11-21
150561 Extra RTT on TLS handshakes Traffic resolved 2017-01-02 2017-01-05
150660 Post Varnish 4 migration cleanup Traffic resolved 2017-01-02 2018-02-28
150879 Cannot connect to cp3039.mgmt.esams.wmnet:22 Traffic resolved 2017-01-02 2016-11-17
151078 Investigate major HTTP 500 spike since 2016-09-23 Traffic resolved 2017-01-02 2016-11-23
151273 lvs4002 power supply failure Traffic resolved 2017-01-02 2017-10-23
151444 Huge increase in cache_upload 404s due to buggy client-side code from graphiq.com Traffic resolved 2017-01-02 2017-04-20
151561 varnishapi.py AttributeError: VSM_Close Traffic resolved 2017-01-02 2016-11-28
151643 python-varnishapi daemons seeing "Log overrun" constantly Traffic resolved 2017-01-02 2017-07-24
151686 several 502 Bad Gateway Traffic resolved 2017-01-02 2016-11-30
152093 Ganglia varnishkafka python module crashing repeatedly Traffic resolved 2017-01-02 2016-12-06
152690 Refreshing mobile-sections does not purge mobile-sections-lead Traffic resolved 2017-01-02 2016-12-13
152870 OTRS-Wiki link to mobile website Traffic resolved 2017-01-02 2016-12-12
153042 Enable TLS termination on the MediaWiki clusters Traffic resolved 2017-01-02 2016-12-16
153336 Strip query string in varnish upload Traffic resolved 2017-01-02 2016-12-19
154558 Periodic 500s from piwik.wikimedia.org Traffic resolved 2017-01-05 2017-04-21
154704 Rate-limit browsers without referers Traffic resolved 2017-01-31 2017-06-29
154759 Pybal not happy with DNS delays Traffic resolved 2017-01-07 2017-07-18
154780 Cache and media (images) issues on all Wikimedia wikis - creates problems on upload, display and generation of thumbnails and files Traffic resolved 2017-01-07 2017-08-27
154826 Donate wiktionary.pl to the Foundation Traffic resolved 2017-01-07 2017-03-07
154864 Plot number of cached objects on a per-server per-DC basis Traffic resolved 2017-01-09 2017-01-24
154913 convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24) Traffic resolved 2017-01-10 2017-01-17
154917 convert lists.wikimedia.org certificate to LetsEncrypt (deadline:2017-03-02) Traffic resolved 2017-01-10 2017-03-01
154919 convert librenms.wikimedia.org from GS to LE cert (expires: 2017-02-11) Traffic resolved 2017-01-10 2017-01-18
154938 convert tendril to use Letsencrypt for SSL cert (deadline 2017-03-17) Traffic resolved 2017-01-10 2017-04-14
154939 convert ganglia to use Letsencrypt for SSL cert (deadline: 2017-02-07) Traffic resolved 2017-01-10 2017-01-10
154940 convert dumps to use Letsencrypt for SSL cert (deadline: 2017-04-26) Traffic resolved 2017-01-10 2017-01-19
154942 convert archiva to use Letsencrypt for SSL cert (deadline 2017-05-08) Traffic resolved 2017-01-10 2017-01-26
155504 Fix universal link support in iOS when the OS requests the site association file from m.wikipedia.org Traffic resolved 2017-01-18 2017-11-15
155806 Add CAA records to our domains Traffic resolved 2017-01-21 2017-10-23
155807 Monitor Certificate Transparency (CT) logs Traffic resolved 2017-01-21 2017-02-13
156026 Enable Service in Asia Cache DC Traffic resolved 2017-01-24 2018-05-15
156027 Configuration for Asia Cache DC hosts Traffic resolved 2017-01-24 2018-03-08
156028 Name Asia Cache DC site Traffic resolved 2017-01-24 2017-09-26
156029 Select location for Asia Cache DC Traffic resolved 2017-01-24 2017-04-11
156030 Select site vendor for Asia Cache Datacenter Traffic resolved 2017-01-24 2017-09-26
156031 Turn up network links for Asia Cache DC Traffic resolved 2017-01-24 2018-03-09
156032 Server hardware installation for Asia Cache DC Traffic resolved 2017-01-24 2018-03-08
156033 Server hardware purchasing for Asia Cache DC Traffic resolved 2017-01-24 2018-03-08
156256 Allocate address space for Singapore (APNIC) Traffic resolved 2017-01-26 2017-11-08
157353 prometheus-vhtcpd-stats cronspamming if vhtcpd is not running yet Traffic resolved 2017-02-07 2017-07-19
157398 Book collections communicate with pediapress using http: Traffic resolved 2017-02-07 2017-02-15
157430 varnish-be returning 503s upon VCL reload Traffic resolved 2017-02-08 2018-03-13
157794 diamond crashing on hosts using systemd-timesyncd Traffic resolved 2017-02-11 2017-03-13
158638 Using wikimedia.ee mail address as Google account Traffic resolved 2017-02-22 2017-04-07
158684 update SSL certificate for benefactorevents.wikimedia.org by 2017-03-02 Traffic resolved 2017-02-22 2017-02-23
159056 cp2017 froze and stopped serving traffic Traffic resolved 2017-02-26 2017-10-23
159137 certspotter: Error retrieving STH from log Traffic resolved 2017-02-28 2017-10-23
159346 convert mail servers from GS to LE certificates Traffic resolved 2017-03-02 2018-01-25
159424 Upgrade text and upload cache clusters to varnish 4.1.5 Traffic resolved 2017-03-03 2017-03-08
159429 Allow setting varnish connection timeouts in puppet Traffic resolved 2017-03-03 2017-10-26
159574 LDF endpoint ordering is not stable between servers when paging Traffic resolved 2017-03-09 2017-07-14
159870 baham (ns1) CPU-related issues Traffic resolved 2017-03-08 2017-04-18
159956 Prepare pywikibot for http -> https switch in entity uri Traffic resolved 2017-03-15 2017-04-30
159995 Separate subdomain for parsoid visual diff test service on ruthenium Traffic resolved 2017-03-09 2017-03-17
160109 What happened 2017-03-09 04:00 - 06:00 UTC Traffic resolved 2017-03-14 2017-04-03
160156 Add node_exporter ipvs ipv6 support Traffic resolved 2017-03-11 2017-05-24
160405 pybal stops logging Traffic resolved 2017-03-15 2017-03-15
160433 Upgrade twisted on load balancers to 16.2.0 Traffic resolved 2017-03-15 2017-03-30
160616 Enable HTTPS for swift clients Traffic resolved 2017-03-17 2017-07-05
161101 Performance impact evaluation of enabling nginx-lua and nginx-lua-prometheus on tlsproxy Traffic resolved 2017-03-23 2017-07-13
161343 mediawiki.org DNS TXT entry for Google Search Console site verification Traffic resolved 2017-03-25 2017-04-07
161819 Investigate 502 errors from nginx when backend returns 302 Traffic resolved 2017-03-31 2017-04-10
162035 Some PNG thumbnails and JPEG originals delivered as [text/html] content-type and hence not rendered in browser Traffic resolved 2017-04-04 2017-04-20
162073 Ops Onboarding for Arzhel Younsi Traffic resolved 2017-04-04 2017-05-04
162099 lvs2002 random shut down Traffic resolved 2017-04-04 2017-10-23
162111 Make WDQS active / active Traffic resolved 2017-04-05 2017-05-03
162129 Someone is parsing all enwiki pages using the action api at a rate of ~2M pages/hour Traffic resolved 2017-04-05 2017-04-17
162348 swift-object-server 1.13.1: Wrong Content-Type returned on 304 Not Modified responses Traffic resolved 2017-04-07 2017-04-25
162362 Make maps active / active Traffic resolved 2017-04-07 2017-09-22
162683 Network hardware purchasing for Asia Cache DC Traffic resolved 2017-04-12 2018-03-08
162684 Network hardware configuration for Asia Cache DC Traffic resolved 2017-04-12 2018-03-09
162976 dbtree broken (for some users?) Traffic resolved 2017-04-15 2017-04-17
163233 Implement Varnish-level rough ratelimiting Traffic resolved 2017-04-19 2017-09-01
163251 Communicate dropping IE8-on-XP support (a security change) to affected editors and other community members Traffic resolved 2017-04-19 2018-01-26
164376 [Discuss] Split ORES scores in datacenters based on wiki Traffic resolved 2017-05-04 2018-04-18
164579 Investigate nginx reload behavior Traffic resolved 2017-05-06 2017-07-24
164608 Merge cache_maps into cache_upload functionally Traffic resolved 2017-05-06 2018-01-15
164610 Unprovision cache_misc @ ulsfo Traffic resolved 2017-05-06 2017-06-27
165063 varnish frontend transient memory usage keeps growing Traffic resolved 2017-05-12 2017-05-15
165324 Can't upload large files with X-Wikimedia-Debug turned on Traffic resolved 2017-05-16 2017-05-29
165651 Collect Google IPs pinging the load balancers Traffic resolved 2017-05-19 2018-03-02
165736 Update Varnishkafka to support TLS encryption/authentication Traffic resolved 2017-05-20 2017-08-08
166120 phab.wmfusercontent.org "homepage" yields a 500 Traffic resolved 2017-05-25 2017-06-27
166179 singapore caching center: eqiad staging tracking task Traffic resolved 2017-05-24 2018-02-15
166229 Mediawiki replies with 500 on wrongly formatted CSP report Traffic resolved 2017-05-25 2017-06-06
166695 Substantive HTTP and mediawiki/database traffic coming from a single ip Traffic resolved 2017-06-01 2017-09-04
166758 cp3032 ethernet link down (bnx2x dump in the dmesg) Traffic resolved 2017-06-02 2017-10-03
166965 Degraded RAID on lvs3001 Traffic resolved 2017-06-05 2018-01-09
167060 en.wiki domain owned by us, but isn't hosted by us?? Traffic resolved 2017-06-06 2018-02-01
167492 Accessing zh-classical.wikipedia.org on a mobile device does not redirect to zh-classical.m.wikipedia.org Traffic resolved 2017-06-10 2017-06-14
167920 Impending load test Traffic resolved 2017-06-15 2017-06-19
168013 Remove disableImages handling from VCL Traffic resolved 2017-06-16 2017-07-04
168019 runUpdate.sh script in wikidata stand-alone has abruptly started incurring numerous 429 errors. Traffic resolved 2017-06-17 2017-06-20
168529 Upgrade to Varnish 5 Traffic resolved 2017-06-22 2018-02-28
168699 Verify that the codfw lvs is configured correctly for Phabricator Traffic resolved 2017-06-24 2017-12-21
168919 stream.wikimedia.org: remove legacy rcstream/socket.io HTTPS redirect hole punches Traffic resolved 2017-06-27 2017-07-11
169600 Enable diamond PowerDNSRecursor collector on dnsrecursors Traffic resolved 2017-07-05 2017-10-13
169893 pybal should reset the etcdindex it's looking at after losing a connection Traffic resolved 2017-07-07 2017-07-18
170140 revoke benefactorevents.wikimedia.org SSL certificate Traffic resolved 2017-07-11 2017-07-11
170192 remove eventdonations.wikimedia.org CNAME Traffic resolved 2017-07-11 2017-07-11
170193 revoke eventdonations.wikimedia.org SSL cert if there is one... Traffic resolved 2017-07-11 2017-07-11
170295 remove benefactorevents.wikimedia.org cname from DNS Traffic resolved 2017-07-12 2017-07-11
170598 Extending our HSTS value beyond ~1y Traffic resolved 2017-07-14 2017-09-13
170843 Determine where to host zim files for the Android app Traffic resolved 2017-07-18 2017-08-09
170847 Icinga check for pybal HTTP connections to etcd Traffic resolved 2017-07-18 2018-02-22
171028 Degraded RAID on cp1008 Traffic resolved 2017-07-20 2017-08-29
171145 cp3048 down, mgmt console not reachable Traffic resolved 2017-07-21 2017-11-14
171318 logster should not resolve statsd's IP every time it sends a metric Traffic resolved 2017-07-22 2017-07-26
171421 503 error for certain JPG thumbnail: "Backend fetch failed" Traffic resolved 2017-07-26 2017-07-28
171710 pybal: add prometheus metrics Traffic resolved 2017-07-27 2017-10-06
171966 setup/install cp402[34] Traffic resolved 2017-07-29 2017-10-23
171967 setup/install cp4022 Traffic resolved 2017-07-29 2017-10-23
172101 OCSP update failed for /etc/update-ocsp.d/globalsign-2016-ecdsa-unified.conf Traffic resolved 2017-08-01 2017-07-31
172116 Improve OCSP fetching and monitoring strategies Traffic resolved 2017-08-01 2017-10-23
172198 setup/install cp402[5-8].ulsfo.wmnet Traffic resolved 2017-08-02 2017-10-23
172417 Create wikimedia.org/resources redirect for Wikimedia Resource Center Traffic resolved 2017-08-04 2017-08-09
172418 Get translations for "IE8 on XP won't work" Traffic resolved 2017-08-04 2017-10-27
173028 pybal vs firewall failover - BGP session down Traffic resolved 2017-09-22 2017-09-28
173422 Investigate the increase in the number of requests to Swift after the Page Previews deploy Traffic resolved 2017-08-17 2017-08-23
173506 cp3036 crashed Traffic resolved 2017-08-18 2017-08-17
173787 setup CAA record for policy.wikimedia.org for namecheap (used by WP VIP GO) due by 2017-09-27 Traffic resolved 2017-08-22 2017-08-25
174891 cp4024 kernel errors Traffic resolved 2017-09-04 2017-11-15
175473 Multiple 503 Errors Traffic resolved 2017-09-11 2017-09-11
175585 cp4021 memory hardware issue - DIMM B1 Traffic resolved 2017-09-12 2017-09-25
175803 Text eqiad varnish 503 spikes Traffic resolved 2017-09-14 2018-03-07
175864 CI: operations-dns-lint broken due to missing Maxmind DB file Traffic resolved 2017-09-14 2017-09-26
176386 upload@ulsfo strange ethernet / power / switch issues, etc... Traffic resolved 2017-09-22 2017-10-19
176905 Evaluate requesting a rate limit change from Letsencrypt Traffic resolved 2017-09-28 2017-10-23
177199 Add Prometheus client support for varnish/statsd metrics daemons Traffic resolved 2017-10-03 2018-01-15
177228 Multiple systems in esams OE10 showing PSU failures Traffic resolved 2017-10-03 2018-01-09
177233 Upgrade cache_misc to Varnish 5 Traffic resolved 2017-10-03 2017-10-17
177815 Alerts on LVS services with one single realserver Traffic resolved 2017-10-11 2017-10-16
178011 cp4026 memory error Traffic resolved 2017-10-12 2017-10-17
178078 RESTBase logs disappeared from logstash Traffic resolved 2017-10-14 2017-10-19
178149 RunCommandMonitoringProtocol throws an exception if runcommand.arguments is not specified Traffic resolved 2017-10-14 2017-10-19
178151 Add UDP monitor for pybal Traffic resolved 2017-10-14 2018-04-10
178423 rack/setup/install cp40(29|3[012]).ulsfo.wmnet Traffic resolved 2017-10-18 2017-11-22
178436 rack/setup/install lvs400[567].ulsfo.wmnet Traffic resolved 2017-10-18 2017-11-22
178567 Server error (500) while trying to download files from Commons from PAWS Traffic resolved 2017-10-21 2018-01-30
178841 Beta cluster is down Traffic resolved 2017-10-24 2017-12-21
179156 503 spikes and resulting API slowness starting 18:45 October 26 Traffic resolved 2017-10-28 2018-03-02
179204 setup/deploy dns400[12]/wmf721[56] Traffic resolved 2017-10-28 2018-02-08
179494 restbase.svc.eqiad.wmnet directs requests to staging if the origin is staging too Traffic resolved 2017-11-02 2017-11-01
180041 Please create a phame blog for the Traffic team Traffic resolved 2017-11-09 2017-12-21
180069 Pybal should be able to advertise to multiple routers Traffic resolved 2017-11-09 2018-03-21
180256 authdns prometheus metrics are not available anymore Traffic resolved 2017-11-11 2017-11-15
180424 cp3048 crashed Traffic resolved 2017-11-15 2017-12-22
180433 Upgrade cache_upload to Varnish 5 Traffic resolved 2017-11-15 2018-02-15
180792 Remove 3DES patch from OpenSSL builds Traffic resolved 2017-11-18 2018-04-06
181554 rack/setup/install bast5001 Traffic resolved 2017-11-29 2018-03-08
181556 rack/setup/install dns500[12] Traffic resolved 2017-11-29 2018-03-08
181557 rack/setup/install cp50(0[1-9]|1[0-2]) Traffic resolved 2017-11-29 2018-03-08
182171 rack/setup/install lvs500[123] Traffic resolved 2017-12-07 2018-03-08
182825 PUT blocked by Varnish Traffic resolved 2017-12-14 2017-12-14
183176 cp4032 memory error Traffic resolved 2017-12-19 2017-12-22
183902 Swift invalid range requests causing 501s Traffic resolved 2018-01-03 2018-04-04
183916 Create subdomain for Research landing page Traffic resolved 2018-01-03 2018-02-06
183926 Limit http methods reported by varnishmtail Traffic resolved 2018-01-03 2018-01-09
184068 Procure and install LVS and miscellaneous servers Traffic resolved 2018-01-04 2018-05-03
184255 Lower varnish caching length on doc.wikimedia.org Traffic resolved 2018-01-06 2018-02-07
184448 Upgrade cache_text to Varnish 5 Traffic resolved 2018-01-09 2018-02-28
184715 pybal's "can-depool" logic only takes downServers into account Traffic resolved 2018-01-12 2018-01-17
184721 Alert instrumentation returning 500 errors Traffic resolved 2018-01-12 2018-01-17
184833 Inconsistent behavior when fetching redirected pages with Cache-Control header Traffic resolved 2018-01-14 2018-01-18
185282 Create subdomain for Design and Wikimedia User Interface Style Guide Traffic resolved 2018-01-20 2018-03-20
185718 Phabricator is slow on mobile ISP's and home ISP's Traffic resolved 2018-01-26 2018-01-25
185968 varnish 5.1.3 frontend child restarted Traffic resolved 2018-01-30 2018-04-04
186902 dns5002 mgmt console unreachable Traffic resolved 2018-02-10 2018-03-06
187014 Proxies information gone from Zero portal. Opera mini pageviews geolocating to wrong country Traffic resolved 2018-04-04 2018-04-18
187035 Ops Onboarding for Valentín Gutiérrez Traffic resolved 2018-02-13 2018-02-16
187090 Migrate dns caches to stretch Traffic resolved 2018-02-13 2018-04-17
187778 varnish: discard cold vcl Traffic resolved 2018-02-21 2018-03-26
188089 VCL discards crash varnish frontend child process Traffic resolved 2018-02-24 2018-02-27
188111 Cannot fetch Zero carriers/proxies JSON files from eqsin Traffic resolved 2018-02-24 2018-02-24
188362 Move "transparency.wikimedia.org/private" to "transparency-private.wikimedia.org" Traffic resolved 2018-02-28 2018-03-02
188545 Post Varnish 5 migration cleanup Traffic resolved 2018-03-01 2018-04-04
189168 How to purge misc-web varnishes for wikitech changes? Traffic resolved 2018-03-08 2018-03-19
189252 Define turn-up process and scope for eqsin service to regional countries Traffic resolved 2018-03-09 2018-05-15
190450 Figure out how Varnish errorpage was enabled for empty 404 text/html from mw/index.php?action=raw Traffic resolved 2018-03-23 2018-03-29
190527 pybal 1.15.2 dies with obscure errors without python-prometheus-client Traffic resolved 2018-03-24 2018-04-06
190540 cp[2006,2008,2010-2011,2017-2018,2022].codfw.wmnet: Uncorrectable Memory Error Traffic resolved 2018-03-24 2018-04-11
191028 Thumbor incorrectly normalizes .jpe and .jpeg into .jpg for Swift thumbnail storage Traffic resolved 2018-03-30 2018-04-09
191223 cp2006 memory replacement Traffic resolved 2018-04-03 2018-04-09
191224 cp2008 memory replacement Traffic resolved 2018-04-03 2018-04-11
191225 cp2010 memory replacement Traffic resolved 2018-04-03 2018-04-09
191226 cp2011 memory replacement Traffic resolved 2018-04-03 2018-04-11
191227 cp2017 memory replacement Traffic resolved 2018-04-03 2018-04-09
191228 cp2018 memory replacement Traffic resolved 2018-04-03 2018-04-11
191229 cp2022 memory replacement Traffic resolved 2018-04-03 2018-04-10
191897 Reimage LVS servers as stretch Traffic resolved 2018-04-11 2018-05-02
191905 eqsin hosts don't allow remote ipmi Traffic resolved 2018-04-11 2018-04-11
193376 Gather 24h data cluster wide of AES128-SHA usage Traffic resolved 2018-05-01 2018-05-14
193489 Refactor varnishospital and varnishslowlog Traffic resolved 2018-05-02 2018-05-03
2323 edits where the rev_user_text and ar_user_text fields contain underscores, initial lower-case letters or consecutive spaces, which could occur in the Phase I and II software, are inaccessible using Special:Contributions DBA open 2017-01-02 2017-12-08
3268 Database replication lag issues (tracking) DBA open 2017-01-02 2017-10-09
17441 Some tables lack unique or primary keys, may allow confusing duplicate data DBA open 2017-01-02 2018-04-03
29112 Select of revisions for stub history files does not explicitly order revisions DBA open 2017-01-02 2018-05-08
33863 Fix use of DB schema so RenameUser is trivial DBA open 2017-01-02 2017-09-12
39591 History of a page was lost after several move actions on zhwp DBA open 2017-01-02 2017-12-11
48098 Disable miser mode ($wgMiserMode) on small wikis (wikis in small.dblist) DBA open 2017-02-14 2017-03-27
50694 Show replication lags in Graphite DBA open 2017-01-02 2016-03-25
50930 Database replication problems - production and labs (tracking) DBA open 2017-01-02 2017-03-23
51190 Truncate SHA-1 indexes DBA open 2017-01-02 2017-04-26
51191 Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases DBA open 2017-01-02 2017-04-28
51195 Drop fj_new_sha1 field DBA open 2017-01-02 2017-12-08
51593 Rename two indexes in the Echo extension DBA open 2017-01-02 2017-04-26
54921 Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) DBA open 2017-01-02 2018-05-16
56401 Backups systems for userDBs hosted on replicas DBA open 2017-01-02 2018-01-29
57455 Provide dynamic report of differences between replica databases and production databases DBA open 2017-01-02 2016-11-15
59176 ApiQueryExtLinksUsage::run query has crazy limit DBA open 2017-01-02 2018-05-15
59186 blob_tracking indexes apparently unused DBA open 2017-01-02 2016-07-20
59617 Make watchlist table available as curated foo_p.watchlist_count on labsdb DBA open 2017-01-02 2018-05-04
67448 Dropping rc_cur_time on wmf databases DBA open 2017-01-02 2017-04-26
68025 [Story] Monitor size of some Wikidata database tables DBA open 2017-01-02 2016-05-11
68961 Logging needs an index to optimize searching by log_title DBA open 2017-02-16 2017-11-20
71088 Queries of commonswiki_p.filearchive for fa_sha1 are slow DBA open 2017-01-02 2017-10-26
71127 Discrepancies with logging table on different wikis DBA open 2017-01-02 2017-08-07
71182 Database upgrade MariaDB 10: Metadata access in INFORMATION_SCHEMA causes complete blocks DBA open 2017-01-02 2015-09-23
71222 list=logevents slow for users with last log action long time ago DBA open 2017-01-02 2016-05-23
72005 Apply enum changes to (img|oi|fa)_major_mime on production DBA open 2017-01-02 2018-03-05
79922 Set up backup strategy for es clusters DBA open 2017-01-02 2017-04-12
83609 script & docs to rename wiki databases DBA open 2017-01-02 2017-10-14
85266 Look into Maria 10 parallel-replication DBA open 2017-01-02 2017-03-02
85756 Make (redacted) log_search table available on ToolLabs DBA open 2017-02-09 2018-01-10
85757 Dropping user.user_options on wmf databases DBA open 2017-01-02 2017-04-26
86338 Dropping page.page_counter on wmf databases DBA open 2017-01-02 2017-04-26
86339 Dropping site_stats.ss_total_views on wmf databases DBA open 2017-01-02 2017-04-26
86342 Dropping page.page_no_title_convert on wmf databases DBA open 2017-01-02 2017-04-26
89737 Make several mediawiki table fields unsigned ints on wmf databases DBA open 2017-01-02 2018-03-01
89741 Expose ar_content_format and ar_content_model columns of archive table on Labs replicas DBA open 2017-03-27 2018-02-09
92739 Remove AFT tables from the analytics slaves DBA open 2017-01-02 2016-12-14
95799 wfWaitForSlaves in JobRunner can massively slow down run rate if just a single slave is lagged DBA open 2017-01-02 2018-05-01
97414 [Task] Use an own load section for Wikibase DB queries DBA open 2017-01-02 2016-02-05
100501 mysql user and group should be a system user/group DBA open 2017-01-02 2017-06-22
101502 Slow API list=recentchanges query (DBError: Lost connection to MySQL server during query) DBA open 2017-01-02 2016-12-20
101631 rev_len should be available also for deleted revisions in database replicas DBA open 2017-01-02 2017-10-26
104459 Automatize the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves DBA open 2017-01-02 2017-09-14
104699 Firewall configurations for database hosts DBA open 2017-01-02 2018-04-04
105395 UserMerge using SELECT img_name FROM image WHERE img_user = ... but img_user doesn't have an index DBA open 2017-01-02 2015-07-09
106151 write irc bot to report high replag of s{1,2,3}.labsdb on #wikimedia-labsdb DBA open 2017-01-02 2016-06-22
106854 Duplicate key errors (tracking) DBA open 2017-01-02 2018-03-26
107610 Setup separate logical External Store for Flow in production DBA open 2017-01-02 2018-04-06
107676 Undiscoverable log entries DBA open 2017-01-02 2015-11-05
108255 Enable MariaDB/MySQL's Strict Mode DBA open 2017-01-02 2018-04-27
108434 latin1 encoding in sql databases DBA open 2017-01-02 2015-08-14
108782 Special:Contributions: "contributions page filtered for namespace or RevisionDeleted edits" error DBA open 2017-01-02 2016-07-11
108968 prop=revisions with rvdir=newer gives internal_api_error_DBQueryError for page with many version DBA open 2017-01-02 2017-02-06
109179 Migrate MySQLs to use ROW-based replication DBA open 2017-01-02 2018-02-07
109964 Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections. DBA open 2017-01-02 2015-09-29
110115 Possible to run writes (e.g. UPDATE) on Beta Cluster replica DBA open 2017-01-02 2018-05-01
111929 Puppetize grants for mysql hosts that are the source of recovery (dbstore, passive misc) DBA open 2017-01-02 2017-04-12
112282 Multiple pages with no revisions DBA open 2017-01-02 2016-09-23
112473 Better mysql monitoring for number of connections and processlist strange patterns DBA open 2017-01-02 2017-08-04
113842 `pr_index` to be replicated to Labs public databases DBA open 2017-09-01 2018-04-26
114117 Drop externallinks.el_from_namespace on wmf databases DBA open 2017-01-02 2018-03-05
114476 Puppetize grants for mysql analytics servers DBA open 2017-01-02 2017-02-20
114902 Remove numeric entity IDs from database schema DBA open 2017-01-02 2017-11-17
114903 Migrate wb_terms to using prefixed entity IDs instead of numeric IDs DBA open 2017-01-02 2018-03-06
114904 Migrate wb_items_per_site to using prefixed entity IDs instead of numeric IDs DBA open 2017-01-02 2017-11-14
115081 Pages where page.page_latest = 0 on the English Wikipedia DBA open 2017-01-02 2015-12-05
115303 Expunge old AFTv5 log entries on WMF wikis DBA open 2017-01-02 2016-08-12
116793 Investigate slow query logging/digest for Beta Cluster DBA open 2017-01-02 2016-08-05
117272 Wikidata: Possibility to add comments with more than 255 bytes, potential db issue DBA open 2017-05-05 2017-05-04
119154 Move echo tables from local wiki databases onto extension1 cluster for mediawikiwiki, metawiki, and officewiki DBA open 2017-01-02 2017-02-09
119371 Enable MariaDB/MySQL strict mode on CI db hosts DBA open 2017-01-02 2018-05-07
119479 Create a script that'll allow easier swatting of queries that're overloading labsdb DBA open 2017-01-02 2015-11-24
119601 Certain tools users create multiple long running queries that take all memory and/or CPU from labsdb hosts, slowing it down and potentially crashing (tracking) DBA open 2017-01-02 2018-05-11
119626 Eliminate SPOF at the main database infrastructure DBA open 2017-01-02 2017-11-30
121094 Throttling linkwatcher tool user as it is consuming 100% CPU DBA open 2017-01-02 2017-04-19
121161 CentralAuthCreateLocalAccountJob failing on meta due to Echo deadlocks DBA open 2017-01-02 2017-12-11
121857 Implement a system to automatically deploy schema changes without needing DBA intervention DBA open 2017-01-02 2016-11-10
124214 Allow filtering based on tag on Special:NewFiles DBA open 2017-01-02 2017-06-27
126218 Replicate wikitech wikis to analytics-store.eqiad.wmnet DBA open 2017-01-02 2016-02-11
126251 Get HA db support for labs internal services DBA open 2017-01-02 2017-02-14
126252 Populate the wikishared db on all dbstores DBA open 2017-01-02 2016-02-09
127363 mysqlbinlog doesn't recognize the ssl-* options in [client] DBA open 2017-01-02 2017-01-06
127570 Rename be_x_oldwiki database to be_taraskwiki DBA open 2017-01-02 2017-10-14
127636 MariaDB missing logrotate for error and slow logs DBA open 2017-01-02 2017-12-11
127792 Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace DBA open 2017-01-02 2017-08-10
128623 Purge all Schema:Echo data after 90 days DBA open 2017-01-02 2017-12-11
128821 reclaim and return all cisco servers DBA open 2017-01-02 2018-05-03
129093 SHOW SLAVE STATUS as a health check should have a low timeout DBA open 2017-01-02 2016-03-08
129830 large s2 database traffic DBA open 2017-01-02 2016-03-18
131065 API: Contributions: Database query error DBA open 2017-01-02 2017-12-11
131266 Querying the logging table on labs is slow DBA open 2017-01-02 2016-12-04
132838 Certain wiki databases missing from replicas? DBA open 2017-01-02 2017-12-11
133185 Database error while saving an article (1205 Lock wait exceeded in Title::invalidateCache) DBA open 2017-01-02 2016-04-25
133333 Audit MySQL configurations DBA open 2017-01-02 2016-11-10
133523 [RFC] improve parsercache replication and sharding handling DBA open 2017-01-02 2018-02-12
134476 Decommission old coredb machines (<=db1050) DBA open 2017-01-02 2018-05-03
134809 Apache <=> mariadb SSL/TLS for cross-datacenter writes DBA open 2017-01-02 2018-01-18
135414 Entries in revision table where rev_len is NULL DBA open 2017-01-02 2017-12-11
135851 Preserve InnoDB table auto_increment on restart DBA open 2017-01-02 2017-05-08
136335 Allow self-serve database credential and permissions management for Tool Labs projects DBA open 2017-01-02 2018-02-14
137654 Tendril DB graphs say 24hr but at 12hr DBA open 2017-01-02 2016-06-14
138517 mysqldump is timing out preventing all tables from being included in the dump DBA open 2017-01-02 2017-04-20
138562 Improve regular production database backups handling DBA open 2017-01-02 2018-04-17
138915 OTRS database is "too large" DBA open 2017-01-02 2018-03-12
140302 plwiki API request is excessively slow when including a badrevid DBA open 2017-01-02 2017-12-11
140751 Create a logstash input filter to preprocess mysqld syslog messages DBA open 2017-01-02 2017-12-11
140832 Investigate moving labsdb (replicas) user credential management to 'Striker' (codename) DBA open 2017-01-02 2017-08-22
141095 Explore 'Analyze' statement as substitute for Explain DBA open 2017-01-02 2017-12-11
141255 Separate host lookup from the sql shell script DBA open 2017-01-02 2017-12-11
141547 Setup automatic failover for misc database servers DBA open 2017-01-02 2016-08-17
141765 Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) DBA open 2017-01-02 2017-05-08
141968 Display lag on grafana (prometheus) and dbtree from pt-heartbeat instead (or in addition) of Seconds_Behind_Master DBA open 2017-01-02 2016-08-25
142807 Migrate all users to new Wiki Replica cluster and decommission old hardware DBA open 2017-01-02 2018-03-26
143366 Removal of {{#coordinates:}} leaves DB entries behind DBA open 2017-07-13 2018-03-02
143870 Some dump hosts are accessing main traffic servers DBA open 2017-01-02 2018-04-25
143896 MySQL metrics monitoring DBA open 2017-01-02 2017-10-09
144010 Drop eu_touched in production DBA open 2017-01-02 2017-10-05
145072 Create a script to regenerate prometheus mysqld exporter listing that works with puppetdb DBA open 2017-01-02 2017-12-11
146055 Improve privilege separation for phabricator's config files and mysql credentials DBA open 2017-03-16 2018-04-23
147148 Wikipedia requires a patch to load its data from the dumps with mwdumper DBA open 2017-01-02 2017-05-08
149077 Certain ApiQueryRecentChanges::run api query is too slow, slowing down dewiki DBA open 2017-01-02 2017-12-11
150547 Add visitingwatchers to watchlist_count DBA open 2017-01-02 2017-12-11
150548 Reduce watchlist_count threshold DBA open 2017-01-02 2017-02-22
151583 Use tls for dump backup generation DBA open 2017-01-02 2017-12-11
151681 DispatchChanges: Avoid long-lasting connections to the master DB DBA open 2017-01-02 2017-12-11
151999 Create script to monitor db dumps for backups are successful DBA open 2017-01-02 2018-03-09
152427 Create a check/calendar alert for MariaDB TLS certs DBA open 2017-01-02 2017-02-21
153638 Drop echo tables from local wiki databases DBA open 2017-01-02 2017-07-12
156461 [META ticket] Automation for our DBs tracking task DBA open 2017-01-28 2017-01-27
156465 For switchovers: A way to check if slaves are up to date DBA open 2017-01-28 2017-01-27
157359 labsdb1006/1007 (postgresql) maintenance DBA open 2017-02-07 2018-02-17
157702 Followup for TLS MariaDB server roll-out DBA open 2017-02-10 2018-03-22
157762 Drop ptrl_comment in production DBA open 2017-10-02 2018-03-05
158166 Discuss labsdb visibility of rev_text_id and ar_comment DBA open 2017-02-16 2017-03-13
158724 Increase size of categorylinks.cl_collation column DBA open 2017-02-23 2018-03-20
159423 Meta ticket: Migrate multi-source database hosts to multi-instance DBA open 2017-03-03 2018-05-15
160731 Decom db1048 (BBU Faulty - slave lagging) DBA open 2017-03-18 2017-09-13
160983 Optimize SpecialAllPages::showChunk for large wikis DBA open 2017-03-22 2017-12-11
161066 Merge pagelinks, templatelinks and imagelinks tables DBA open 2017-05-20 2017-05-24
161754 eqiad: (2) hardware access request for labsdb1004 & 5 refresh DBA open 2017-03-30 2017-04-07
161755 eqiad: (2) hardware access request for labsdb1006 & 7 refresh DBA open 2017-03-30 2017-04-07
162070 Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases DBA open 2017-04-04 2018-02-07
162558 Use memcached (or something similar) to keep the latest chd_seen state, only flush to table every once in a while DBA open 2017-04-10 2017-05-05
162562 Choose a field length for entity id columns and stick to it DBA open 2017-04-10 2017-05-05
163339 pdu phase inbalances: ps1-a3-codfw, ps1-c6-codfw, & ps1-d6-codfw DBA open 2017-04-20 2017-05-08
163681 Use a random timestamp on Special:NewFiles on Commons doesn't loads the page (504 Gateway Timeout) DBA open 2017-08-02 2017-12-11
164167 change_tag table needs redesign DBA open 2017-05-01 2018-01-29
164382 Evaluate the need for FORCE INDEX (ls_field_val) [now IGNORE INDEX (ls_log_id)], delete the index hint if not needed anymore DBA open 2017-05-04 2017-12-11
164504 Tracking: Cleanup x1 database connection patterns DBA open 2017-05-09 2018-02-21
164810 Switch phabricator production to codfw DBA open 2017-05-17 2018-04-01
164834 In some database hosts, performance schema loses digest statistics DBA open 2017-05-10 2017-10-04
165358 Set up and package wmfmariadbpy helper scripts so they can easily be deployed to all database server and client hosts DBA open 2017-05-16 2018-02-07
165625 Evaluate future of wmf puppet module "mysql" DBA open 2017-05-18 2018-05-14
165674 Investigate slow servermon updating queries on db1016 DBA open 2017-05-19 2018-02-14
165677 Create a backend check for pybal to monitor the MySQL protocol being up DBA open 2017-05-19 2017-05-25
165756 Create summary templates on Wikitech wiki to stop writing the same things everywhere, everytime DBA open 2017-05-20 2017-09-12
166108 x1 master db1031: Faulty BBU DBA open 2017-05-24 2018-01-19
166798 Decide whether back-compat views for upcoming major schema changes will be provided in the Labs replicas DBA open 2017-06-02 2017-09-07
167973 Move wikitech and labstestwiki to s5 DBA open 2017-06-16 2018-03-13
168349 enwiki_p logging vs logging_userindex returning dramatically different results DBA open 2017-06-21 2017-09-12
170508 The "show ip" action should also provide a distinct list of user-agents for each IP DBA open 2017-07-20 2017-11-22
171071 Perform testing for TLS effect on connection rate DBA open 2017-07-20 2018-04-06
171570 Rename database labswiki to wikitech DBA open 2017-07-26 2017-10-14
171643 Clean up skin properties DBA open 2017-07-26 2018-02-01
171898 Deleting large watchlist takes > 4 seconds causing rollback due to write time limit DBA open 2017-07-29 2017-09-10
172489 Monitor read_only variable and/or uptime on database masters, make it page DBA open 2017-08-05 2017-10-09
172490 Monitor swap/memory usage on databases DBA open 2017-08-05 2017-08-07
172492 Improve database alerting (tracking) DBA open 2017-08-05 2017-10-09
172497 Change pt-heartbeat model to not use super-user, avoid SPOF and switch automatically to the real master without puppet dependency DBA open 2017-08-05 2017-08-07
173511 Implement technical details and process for "datasets_p" on wikireplica hosts DBA open 2017-08-23 2017-11-15
173917 Cannot add a previously used change tag to an abuse filter DBA open 2018-04-23 2018-05-01
173943 Display count of remaining content space errors DBA open 2018-02-01 2018-02-07
174044 Deploy MCR storage layer DBA open 2017-09-26 2018-05-15
174648 Create MW Schema Diff maintenance script DBA open 2017-09-01 2017-09-12
174802 Drop education program (ep_*) tables on wikis where it is not enabled DBA open 2017-09-02 2017-09-01
175672 Make apache/maintenance hosts TLS connections to mariadb work DBA open 2017-09-13 2018-05-08
176277 Provision a separate DB shard for wbc_entity_usage DBA open 2017-09-20 2017-09-25
176532 Gerrit is failing to connect to db on gerrit2001 thus preventing systemd from working DBA open 2017-09-24 2018-03-22
177778 Improve database application performance monitoring visibility DBA open 2017-10-10 2018-01-09
177779 Generate instance list of database hosts to be monitored automatically from exported resources DBA open 2017-10-10 2017-10-10
177782 Reduce false positives on database pages DBA open 2017-10-10 2017-11-16
178290 Check recentchanges table and query errors on wikis other than commonswiki and ruwiki DBA open 2017-10-17 2018-03-16
178877 operations/software repo: flake8 check DBA open 2017-10-25 2017-10-25
179010 Re-enable Wikidata Recent Changes integration on Commons DBA open 2017-10-26 2018-05-09
179604 Move AbuseFilter slow filters data from Logstash to per-filter profiling DBA open 2018-04-29 2018-04-28
180158 Investigate using macbre/index-digest over WMF MW sql query workload DBA open 2017-11-10 2017-11-10
180636 Make Dispenser's principle_links table accessible in new Wiki replica cluster DBA open 2017-11-16 2017-12-11
180979 List of Newsletters should have a column showing the number of issues DBA open 2017-12-21 2017-12-21
182563 Identify and delete non-useful files on database hosts taking space unnecesarily DBA open 2017-12-12 2017-12-11
183983 Re-institute query killer for the analytics WikiReplica DBA open 2018-01-03 2018-03-22
184054 Decommission db1029 and db1031 DBA open 2018-01-04 2018-05-07
184540 Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running DBA open 2018-01-10 2018-01-18
184615 Once MCR is deployed, drop the rev_text_id, rev_content_model, and rev_content_format fields to be dropped from revision DBA open 2018-01-11 2018-05-15
184805 Move some wikis to s5 DBA open 2018-01-13 2018-02-21
185084 Allow use of EtcdConfig to configure slave databases DBA open 2018-01-18 2018-02-07
185673 Add base36 functions to ToolForge database DBA open 2018-01-26 2018-02-07
186188 Failover DB masters in row D DBA open 2018-02-02 2018-04-25
186320 Decommission db1051-db1060 (DBA tracking) DBA open 2018-02-03 2018-05-21
187089 Fix WMF schemas to not break when comment store goes WRITE_NEW DBA open 2018-02-13 2018-04-17
187295 Apply AbuseFilter patch-fix-index DBA open 2018-02-15 2018-05-21
187521 Optimize recentchanges and wbc_entity_usage table across wikis DBA open 2018-02-17 2018-04-17
187579 Create ip_logging table to query for logged actions by IP ranges DBA open 2018-02-17 2018-02-16
187584 Investigation: Using log_search to query for logged actions against IPs in a given range DBA open 2018-02-17 2018-03-06
188299 Schema change for refactored actor storage DBA open 2018-02-27 2018-04-18
188992 Make wb_terms table fancy DBA open 2018-03-07 2018-05-09
189026 IndexPager::buildQueryInfo (contributions page unfiltered) query needs tuning DBA open 2018-03-07 2018-04-28
189107 DB meta task for next DC failover issues DBA open 2018-03-08 2018-03-20
190148 Change DEFAULT 0 for rev_text_id on production DBs DBA open 2018-03-21 2018-05-07
190704 Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy DBA open 2018-03-27 2018-05-18
191316 Schema change to make archive.ar_rev_id NOT NULL DBA open 2018-04-04 2018-04-24
191391 Apply schema changes to an isolated database and examine the results DBA open 2018-04-05 2018-05-22
191519 Schema change for rc_namespace_title_timestamp index DBA open 2018-04-06 2018-04-18
191581 Nullify ct_rc_id and ts_rc_id when recent changes entries are being deleted DBA open 2018-04-07 2018-04-06
192358 Setup database logical backups on eqiad DBA open 2018-04-18 2018-05-04
192875 Debmonitor: request for misc DB allocation DBA open 2018-04-25 2018-05-03
192926 Schema change to drop archive.ar_text and archive.ar_flags DBA open 2018-04-25 2018-05-15
192979 Productionize 8 eqiad hosts DBA open 2018-04-26 2018-05-16
193187 Prepare and check storage layer for idwikimedia DBA open 2018-04-27 2018-05-21
193224 Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished DBA open 2018-04-28 2018-04-27
193226 Test MySQL 8.0 with production data and evaluate its fit for WMF databases DBA open 2018-04-28 2018-04-27
193732 Decommission db1060 DBA open 2018-05-04 2018-05-15
193736 Decommission db1056 DBA open 2018-05-04 2018-05-08
194118 Decommission db1055 DBA open 2018-05-09 2018-05-15
194270 Drop 'tmp1' index from wb_terms table in production DBA open 2018-05-10 2018-05-22
194273 Clean up indexes of wb_terms table DBA open 2018-05-10 2018-05-22
194343 quentinv57-tools/tools/globalcontribs.php generates slow/complex SQL queries which impact server performance DBA open 2018-05-11 2018-05-18
194634 Decommission db1053 DBA open 2018-05-15 2018-05-16
194780 rack/setup/install db112[45].eqiad.wmnet (sanitarium expansion) DBA open 2018-05-16 2018-05-18
194781 rack/setup/install db209[45].codfw.wmnet (sanitarium expansion) DBA open 2018-05-16 2018-05-18
194870 Failover s2 primary master DBA open 2018-05-18 2018-05-21
195008 Prepare and check storage layer for pmswikisource DBA open 2018-05-19 2018-05-21
195228 db2064 crashed and totally broken - decommission it DBA open 2018-05-22 2018-05-22
12817 Filtering Special:NewPages by page size DBA resolved 2017-01-02 2016-11-10
26675 Revision 186704908 on en.wikipedia.org, Fatal exception: unknown "cluster16" DBA resolved 2017-01-02 2016-03-09
32824 Integrate Facebooks "Online Schema Change for MySQL" into MediaWiki DBA resolved 2017-05-24 2017-09-04
42867 Update change tag indexes DBA resolved 2017-01-02 2017-10-09
48047 Database error on es.wikipedia.org in function "Revision::insertOn": "Duplicate entry for key 'PRIMARY'" DBA resolved 2017-01-02 2017-11-06
52422 Replicate the Phabricator database to labsdb DBA resolved 2017-01-02 2015-10-22
54924 Drop PovWatch extension-related database tables from Wikimedia wikis DBA resolved 2017-01-02 2016-12-14
54927 Drop database table "hashs" from Wikimedia wikis DBA resolved 2017-01-02 2017-04-27
54929 Drop database table "namespaces" from Wikimedia wikis DBA resolved 2017-01-02 2015-06-09
54932 Drop *_old database tables from Wikimedia wikis DBA resolved 2017-01-02 2018-05-02
54934 Drop database table "optin_survey" from Wikimedia wikis DBA resolved 2017-01-02 2015-08-26
57385 Investigate dropping "edit_page_tracking" database table from Wikimedia wikis after archiving it DBA resolved 2017-01-02 2018-04-30
57676 Drop database table "email_capture" from Wikimedia wikis DBA resolved 2017-01-02 2016-10-10
59185 Archive and drop all article feedback related tables from all wikis DBA resolved 2017-01-02 2016-08-12
62539 [Task] Convert wb_terms term_row_id from INT to BIGINT on wikidatawiki DBA resolved 2017-01-02 2016-02-25
69223 Schema change for page content language DBA resolved 2017-01-02 2018-02-22
70942 Database upgrade MariaDB 10: Engine / Option mismatch on table `user_properties` DBA resolved 2017-01-02 2017-02-24
71463 Create a table in labs with replication lag data DBA resolved 2017-01-02 2016-04-22
72221 internal_api_error_DBQueryError on page edit; db1024 DBA resolved 2017-01-02 2015-12-07
72893 meta_p.wiki and meta_p.legacy tables not filled on all servers DBA resolved 2017-01-02 2015-06-09
73176 Discrepancy between enwiki_p.pagelinks on labs and production DBA resolved 2017-01-02 2015-06-26
73563 *_minor_mime are varbinary(32) on WMF sites, out of sync with varbinary(100) in MW core DBA resolved 2017-01-02 2017-04-20
74413 Different results with queries in labs versus production DBA resolved 2017-01-02 2016-06-29
74908 Replication behind or missing records DBA resolved 2017-01-02 2015-08-17
75047 Replicate flowdb from X1 to analytics-store DBA resolved 2017-01-02 2015-11-02
78276 allpages filterlanglinks DBQueryError DBA resolved 2017-01-02 2016-09-11
84178 investigate RAID BBU auto-learn on db hosts DBA resolved 2017-01-02 2015-09-02
85813 "sites" table missing from some wikis DBA resolved 2017-01-02 2015-11-16
86340 Dropping hitcounter table on wmf databases DBA resolved 2017-01-02 2017-01-19
86482 puppet stopped mysqld using orphan pid file from puppet agent DBA resolved 2017-01-02 2017-06-23
86958 install/deploy dbproxy1003 through dbproxy1011 DBA resolved 2017-01-02 2016-12-02
88718 Make sure tools-db is replicated somewhere DBA resolved 2017-01-02 2015-10-05
88770 sleeper database connection surges during outage DBA resolved 2017-01-02 2016-07-29
88986 [Task] Avoid inconsistencies in Wikidata editing due to slave lag DBA resolved 2017-01-02 2016-09-07
89689 Missing record in replica DBA resolved 2017-01-02 2015-09-26
89986 Update GeoData schema DBA resolved 2017-01-02 2015-10-23
90679 contacts.wikimedia.org drupal unpuppetized / retire contacts DBA resolved 2017-01-02 2015-06-01
91797 mariadb puppet module doesn't start mysql service in labs (possibly anywhere) DBA resolved 2017-01-02 2016-12-01
91859 Update diagram of core database tables to MediaWiki 1.24 DBA resolved 2017-01-02 2015-04-06
92693 Get Labs openstack service dbs on a proper db server DBA resolved 2017-01-02 2015-06-19
92841 Database replicas: replicate user.user_touched DBA resolved 2017-01-02 2015-11-29
93211 Grant access to the databases to codfw appserver networks DBA resolved 2017-01-02 2016-01-20
93708 Can't access x1-analytics-slave DBA resolved 2017-01-02 2015-03-26
94512 Bug: DB gather_list.gl_id autonumber is always odd DBA resolved 2017-01-02 2015-04-29
95421 Database error when trying to connect to NL wikipedia DBA resolved 2017-01-02 2015-04-08
95835 Better backup coverage for X1 database cluster DBA resolved 2017-01-02 2015-05-02
95871 Use External Store on Beta Cluster DBA resolved 2017-01-02 2016-08-12
96383 install/setup/deploy db2043-db2070 DBA resolved 2017-01-02 2015-08-28
96638 wikimania2016wiki not recognised by sql command DBA resolved 2017-01-02 2015-06-26
98110 Pagelinks table contains a row having pl_from = 0 DBA resolved 2017-01-02 2017-11-15
98682 labswiki DB is inaccessible from tin, terbium, etc. DBA resolved 2017-01-02 2015-08-20
98706 internal_api_error_DBQueryError: Database query error while (mass) deleting file over api DBA resolved 2017-01-02 2015-07-02
98816 move tendril to gerrit repo and puppetize cloning DBA resolved 2017-01-02 2015-07-14
98998 Database connection failure issues on s7 shard DBA resolved 2017-01-02 2015-05-15
99459 ips_site_page is too short to store some (full) page titles DBA resolved 2017-01-02 2016-08-11
99485 implement performance_schema for mysql monitoring DBA resolved 2017-01-02 2017-02-14
99520 On a maintenance window, upgrade db1063 to 14.04 and its MariaDB package to 10.0.16 DBA resolved 2017-01-02 2015-05-26
100441 Santitize recent wikis: wikimania 2016 and cn.wikimedia.org at labs dbs DBA resolved 2017-01-02 2015-06-26
100527 Document/fix scripts for echo x1 DB requirements DBA resolved 2017-01-02 2017-12-06
100846 Add Story points (from Sprint Extension) to the phabricator data dump DBA resolved 2017-01-02 2015-06-18
101009 Enabling automatic buffer pool dumping on start/stop (puppet) for all servers DBA resolved 2017-01-02 2015-07-03
101084 es[12]00[123] maintenance and upgrade DBA resolved 2017-01-02 2015-06-25
101182 Permission problem on parsercache db servers DBA resolved 2017-01-02 2015-06-16
101347 need 'spaces' database for phabricator DBA resolved 2017-01-02 2015-06-04
101516 Upgrade db1022, which has an older kernel DBA resolved 2017-01-02 2017-05-04
101561 labsdb1002.eqiad.wmnet seems to have stopped replicating DBA resolved 2017-01-02 2015-07-02
101567 Rebuild s6 and s7 on labsdb1002 DBA resolved 2017-01-02 2015-06-12
101735 db1058 (s5 master) degraded RAID DBA resolved 2017-01-02 2015-07-12
101750 Delete dbname = 'centralauth' from meta_p.wikis table in labs replicas DBA resolved 2017-01-02 2015-10-02
101937 setup logging for #wikimedia-databases DBA resolved 2017-01-02 2016-11-25
102484 Tokudb requires disabling huge pages; puppetize on labsdb* hosts DBA resolved 2017-01-02 2015-06-15
102647 Add Wikipedia Northern Luri and Wikipedia Goan Konkani to labs replicas DBA resolved 2017-01-02 2015-07-23
102915 Special:GlobalUsers varies between claiming a user is or isn't attached DBA resolved 2017-01-02 2016-12-15
103230 Disk issue on db1028 DBA resolved 2017-01-02 2015-09-03
103417 Change character set on etherpad- lite database to utf8mb4_bin DBA resolved 2017-01-02 2015-06-23
103435 BadMethodCallException in sql object cache DBA resolved 2017-01-02 2015-07-06
103611 Update Gather DB schema for flagging backend DBA resolved 2017-01-02 2015-07-06
103843 Faulty memory on es2004 (purchase one module) DBA resolved 2017-01-02 2015-07-23
103936 review eqiad database server quantities / warranties / service(s) DBA resolved 2017-01-02 2015-12-22
104310 Create a whitelist of tables to checksum on all wikis DBA resolved 2017-01-02 2015-09-29
104313 Revision Deletion page won't load for too many (10) revisions DBA resolved 2017-01-02 2016-10-01
104471 Tables created on the s7 master have not been replicated to dbstore2001 and dbstore2002. Replication issue? DBA resolved 2017-01-02 2015-07-23
104476 Provision a labsdb useraccount that can be used to run replica-addusers.pl DBA resolved 2017-01-02 2015-12-01
104573 codfw frontends cannot connect to mysql at db2029 DBA resolved 2017-01-02 2015-07-06
104670 recentchanges deadlocks for dewiki (db1058) DBA resolved 2017-01-02 2015-07-26
104743 Check wikilove_log table exists on all wikis and create it where it doesn't DBA resolved 2017-01-02 2015-07-09
104748 db1046 innodb signal 6 abort and restart DBA resolved 2017-01-02 2016-01-25
104922 [Bug] Deadlock in EntityUsageTable::addUsages DBA resolved 2017-01-02 2015-09-21
105135 Implement mariadb 10.0 masters DBA resolved 2017-01-02 2016-05-02
105469 Page deletion fails when MathSearch is enabled (due to a foreign key restriction) DBA resolved 2017-01-02 2015-07-11
105503 Tables corrupted or impossible to work with them DBA resolved 2017-01-02 2015-07-13
105698 Uncommitted DB writes in GeoData::getAllCoordinates() DBA resolved 2017-01-02 2016-11-07
105699 Uncommitted DB writes in GlobalUsage::getLinksFromPage() DBA resolved 2017-01-02 2016-11-07
105768 Remove db1002-db1007 from production DBA resolved 2017-01-02 2017-02-10
105843 new external storage cluster(s) DBA resolved 2017-01-02 2016-01-27
105879 db1022 duplicate key errors DBA resolved 2017-01-02 2015-12-29
105911 tools.tools-info credentials are not functioning DBA resolved 2017-01-02 2016-01-27
105964 HELP! Database is getting Slow: A test which took less than 10 min, now it takes 3 hours. I cannot do my experiments. DBA resolved 2017-01-02 2017-08-02
106312 m1-master switch from db1001 to db1016 DBA resolved 2017-01-02 2016-06-22
106470 Tool Labs enwiki_p replicated database missing rows DBA resolved 2017-01-02 2016-06-29
106647 mariadb multi-source replication glitch with site_identifiers DBA resolved 2017-01-02 2015-08-15
106682 SpecialWhatLinksHere::showIndirectLinks - Lost connection to MySQL server during query (10.64.16.28) DBA resolved 2017-01-02 2016-08-29
106798 Special:AbuseFilter on all large Wikipedias is returning errors DBA resolved 2017-01-02 2015-07-30
107029 Lost connection to MySQL server during query DBA resolved 2017-01-02 2015-08-02
107070 Measure capacity and utilization of labsdb*** boxes DBA resolved 2017-01-02 2016-04-01
107072 Spikes of job runner new connection errors to mysql "Error connecting to 10.64.32.24: Can't connect to MySQL server on '10.64.32.24' (4)" - mainly on db1035 DBA resolved 2017-01-02 2015-12-16
107282 Reduce memory commitment on database hosts with many objects, specially s3, dbstore/research and labs DBA resolved 2017-01-02 2017-02-14
107319 [Bug] EntityUsageTable::touchUsages slow query DBA resolved 2017-01-02 2015-09-08
107753 Puppet failure on all hosts with Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Mysql::Error: Out of range value for column 'id' at row 1: INSERT INTO `fact_values` (`updated_at`, `host_id`, `created_at`, `fact_name_id`, `value`) VALUES('date', etc) DBA resolved 2017-01-02 2016-05-01
107923 Watchlist shows pages still bold when "mark all pages as read" button clicked DBA resolved 2017-01-02 2015-09-04
108033 duplicate key error on db1056 DBA resolved 2017-01-02 2015-08-18
108316 huge (140GB) decrease on available disk space on db1042 DBA resolved 2017-01-02 2015-08-14
108648 Migrate OAuth data from mediawikiwiki to metawiki DBA resolved 2017-01-02 2016-01-06
108856 Set up bucketization of editCount fields {tick} DBA resolved 2017-01-02 2017-05-03
108857 Delete obsolete schemas {tick} [5 pts] DBA resolved 2017-01-02 2015-12-01
109069 TokuDB crashes frequently -consider upgrade it or search for alternative engines with similar features DBA resolved 2017-01-02 2016-08-25
109279 Phabricator creates MySQL connection spikes: Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections. DBA resolved 2017-01-02 2017-09-26
109282 db2034 crashed DBA resolved 2017-01-02 2015-11-05
109401 dbtree shows 0 lag for db1047 DBA resolved 2017-01-02 2016-05-30
109676 Add artificial primary key to flow_wiki_ref and flow_ext_ref DBA resolved 2017-01-02 2016-03-01
109777 decom ishmael DBA resolved 2017-01-02 2015-10-06
109943 Long running query from LinksUpdate::incrTableUpdate job causing general lag DBA resolved 2017-01-02 2016-05-13
109972 tools.citationhunt can't access databases DBA resolved 2017-01-02 2016-04-08
110504 defragment db1015, db1035 and db1027 DBA resolved 2017-01-02 2016-12-21
110693 MySQL database for Nodepool DBA resolved 2017-01-02 2015-09-11
110963 Create CentralNotice campaign mixin tables DBA resolved 2017-01-02 2015-10-21
111084 Do backfill and final schema change for "Segregate Reference objects by source wiki." DBA resolved 2017-01-02 2015-10-22
111200 Grant puppet script access to "phabricator_project" DB DBA resolved 2017-01-02 2015-09-03
111254 Run "patch-remove_unique_ref_indices.sql" and phase 1 of "Segregate Reference objects by source wiki." on officewiki DBA resolved 2017-01-02 2015-09-07
111266 Make LoadBalancer slave lag check and read-only mode more robust (for example, using pt-heartbeat) DBA resolved 2017-01-02 2016-08-08
111333 wikishared database has latin1 as the default encoding DBA resolved 2017-01-02 2015-09-10
111371 Potential templatelinks data integrity issue on Tool Labs' enwiki_p DBA resolved 2017-01-02 2015-09-07
111547 phabricator metrics script should use slave, not master DBA resolved 2017-01-02 2015-09-10
111654 Set up TLS for MariaDB replication DBA resolved 2017-01-02 2017-07-11
111656 Replica database tables may need to be ANALYZEd again DBA resolved 2017-01-02 2015-09-07
111766 Add icinga check for all MySQL/MariaDB hosts to check they have the right read_only value DBA resolved 2017-01-02 2016-10-01
111992 Physical location SPOF because of database server distribution on a single rack (D1) DBA resolved 2017-01-02 2016-06-16
112041 New database for designate pool manager DBA resolved 2017-01-02 2015-09-10
112079 Upgrade x1 cluster DBA resolved 2017-01-02 2016-06-17
112080 the bounce_records table in x1 wikishared has latin1 encoding DBA resolved 2017-01-02 2016-01-29
112103 fix m5 shard mysql issues with default encoding binary DBA resolved 2017-01-02 2016-12-16
112193 phabricator dump script should use slave db, not master DBA resolved 2017-01-02 2015-10-07
112478 Upgrade db1055 mysql version and configuration, and reduce its pool weight DBA resolved 2017-01-02 2015-10-08
112479 Check, test and tune pool-of-connections and max_connections configuration DBA resolved 2017-01-02 2015-10-29
112573 Drop phlegal_* databases from m3 DBA resolved 2017-01-02 2016-07-06
112732 Truncate table cx_suggestions, cx_lists DBA resolved 2017-01-02 2015-10-01
112835 Document stricter MySQL policy and settings for MediaWiki DBA resolved 2017-01-02 2015-11-05
113901 Odd plan for simple query on revisions + page DBA resolved 2017-01-02 2016-04-22
114068 Clean up search user preferences in user_properties table on Wikimedia wikis DBA resolved 2017-01-02 2015-09-29
114208 User skin preference for MonoBook changed to default (Vector) on small and medium wikis (about 95% of wikis) DBA resolved 2017-01-02 2017-12-21
114396 Grant 'show view' permissions on s1-analytics-slave/jmorgan to user jmorgan DBA resolved 2017-01-02 2015-10-02
114513 Slow running query on cawiki.labsdb; incomplete/missing table stats? DBA resolved 2017-01-02 2015-10-02
114519 Another slow running query on - lvwiki.labsdb this time. DBA resolved 2017-01-02 2015-10-02
114559 s51053 (tools.jackbot) is abusing resources on labsdbs, throttle his grants DBA resolved 2017-01-02 2017-06-20
114752 Puppetize pt-heartbeat on MariaDB10 masters and its corresponding checks on the several monitoring backends DBA resolved 2017-01-02 2016-08-03
115275 Replicate Echo tables to analytics-store DBA resolved 2017-01-02 2018-03-30
115517 Data missing from June 11/12 on s3.labsdb DBA resolved 2017-01-02 2016-12-21
115682 hewiki's categorylinks shown as not empty though it is; purging does not help DBA resolved 2017-01-02 2017-05-20
115711 Drop `user_daily_contribs` table from all production wikis DBA resolved 2017-01-02 2015-12-11
115825 action=query&list=pagepropnames really slow on a big wiki, got error with ppnlimit=500 function: /* ApiQueryPagePropNames::execute */ DBA resolved 2017-01-02 2017-04-27
115982 Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas DBA resolved 2017-01-02 2018-04-04
116425 Rename user creates lag on enwiki DBA resolved 2017-01-02 2016-08-04
116557 AFComputedVariable::compute query timeouts DBA resolved 2017-01-02 2017-04-28
116599 Eventlogging tables not replicating from master to slave DBA resolved 2017-01-02 2015-10-27
116903 Adapt wmf-mariadb101 package for stretch and adapt its service to systemd DBA resolved 2017-01-02 2017-06-22
117089 Deadlock found when trying to get lock; try restarting transaction DBA resolved 2017-01-02 2016-08-11
117398 number of database updates multiplied x3 since 29 October DBA resolved 2017-01-02 2016-08-08
117783 Add ref_id to flow_wiki_ref and flow_ext_ref (DBA phase 1) DBA resolved 2017-01-02 2015-11-05
117785 Deploy artificial primary key reference change to all production wikis simultaneously DBA resolved 2017-01-02 2015-11-22
117786 Run maintenance/FlowPopulateRefId.php on all production wikis DBA resolved 2017-01-02 2015-12-17
117787 Make ref_id of flow_wiki_ref and flow_ext_ref not null and PK (DBA phase 2) DBA resolved 2017-01-02 2015-12-22
117881 Several hosts on virt1002 are down DBA resolved 2017-01-02 2015-11-05
118079 Bug on MariaDB use_stat_tables DBA resolved 2017-01-02 2016-04-22
118159 dbtree fails to render correctly on a new server (mw1152) both with zend php and hhvm DBA resolved 2017-01-02 2015-11-10
118162 Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections DBA resolved 2017-01-02 2015-11-19
118764 Drop old devwikiinternal and rel13testwiki from s3 DBA resolved 2017-01-02 2018-05-07
118816 mysql privs: restrict access to racktables to krypton DBA resolved 2017-01-02 2016-01-29
118859 Drop backup tables for old schema changes (aka "x_pre_y" tables) DBA resolved 2017-01-02 2018-04-23
119056 External Storage on codfw (es2005-2010) is consuming 100-90GB of disk space per server and per month and it has 370GB available DBA resolved 2017-01-02 2016-03-10
119206 Requests to globally reset a user's skin preferences DBA resolved 2017-01-02 2016-01-26
119305 s2/s51072__dwl_p is stuck DBA resolved 2017-01-02 2015-11-23
119315 labsdb1002 and labsdb1003 crashed, affecting replication DBA resolved 2017-01-02 2016-05-22
119370 Send deployment-db1 deployment-db2 syslog to beta cluster logstash DBA resolved 2017-01-02 2016-07-19
119380 db1046 running out of disk space DBA resolved 2017-01-02 2016-02-08
119461 Evaluate security concerns of logging beta cluster db queries on tendril DBA resolved 2017-01-02 2015-11-26
119488 dbstore1002.mgmt.eqiad.wmnet: "No more sessions are available for this type of connection!" DBA resolved 2017-01-02 2016-07-21
119509 Cleanup ptwikibooks conversion DBA resolved 2017-01-02 2017-03-14
119525 Create "echo_unread_wikis" table on wikishared database for Echo cross-wiki notifications DBA resolved 2017-01-02 2016-05-18
119567 Run Flow External Store migration in dry-run mode on Beta DBA resolved 2017-01-02 2016-06-22
119577 Client loses connection to database replica and cannot connect to it any further DBA resolved 2017-01-02 2015-11-27
119604 Faebot is crashing labsdb1002 DBA resolved 2017-01-02 2016-01-08
119619 Decide storage backend for performance schema monitoring stats DBA resolved 2017-01-02 2017-02-14
119642 Create a Master-master topology between datacenters for easier failover (setup circular replication dallas -> eqiad for mysql databases) DBA resolved 2017-01-02 2016-03-16
119648 Fix getLagFromPtHeartbeat() to use the pt entry for the root master, not the immediate one DBA resolved 2017-01-02 2016-08-05
119695 tools.joanjoc is executing the same >1h query every 5 minutes DBA resolved 2017-01-02 2015-11-26
119704 HHVM default configuration kills queries after one minute, disable that DBA resolved 2017-01-02 2015-11-26
119742 Database error at uk:Сетакон page DBA resolved 2017-01-02 2015-12-04
119751 Rename Database to DBA, create blocked-by-schema-change DBA resolved 2017-01-02 2016-09-12
119752 Review schema changes for T119495 DBA resolved 2017-01-02 2016-09-12
119811 Drop localisation and localisation_file_hash tables, l10nwiki databases too DBA resolved 2017-01-02 2018-04-04
119841 labs db inconsistent data DBA resolved 2017-01-02 2015-12-11
120119 Query runs extremely slow on analytics-store DBA resolved 2017-01-02 2018-03-30
120122 Perform a rolling restart of all MySQL slaves (masters too for those services with low traffic) DBA resolved 2017-01-02 2016-04-22
120187 Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) DBA resolved 2017-01-02 2016-02-04
120513 Apply recentchanges special partitioning to a redundant slave at least on s1, s4, and s5 DBA resolved 2017-01-02 2017-03-29
120674 Drop tables MobileWebClickTracking_* from eventlogging db DBA resolved 2017-01-02 2015-12-15
120967 Make EL create tables using TokuDB by default [5] DBA resolved 2017-01-02 2015-12-14
121120 2 hour outage to update mysql on EL slaves {oryx} [3 pts] DBA resolved 2017-01-02 2015-12-15
121207 Implement slave_run_triggers_for_rbr at sanitarium for labs filtering DBA resolved 2017-01-02 2017-07-20
121306 Error: 2013 Lost connection to MySQL server during query on IndexPager::buildQueryInfo (LogPager) DBA resolved 2017-01-02 2017-11-15
121335 Create oauth_registered_consumer.oarc_owner_only column on metawiki DBA resolved 2017-01-02 2015-12-18
121509 Correct the schema to make revision id columns of cx_translations nullable DBA resolved 2017-01-02 2015-12-21
122048 db1024 (s2 master) will run out of disk space in ~4 months DBA resolved 2017-01-02 2016-02-25
122111 ref_target should be NOT NULL DBA resolved 2017-01-02 2016-02-25
122148 DBs provided on Tool Labs should have utf8 charset and collation by default DBA resolved 2017-01-02 2017-05-15
122338 Rename user database DBA resolved 2017-01-02 2015-12-24
122429 Batch updates create slave lag on s3 over WAN DBA resolved 2017-01-02 2016-02-05
122658 MySQL connections die in less than 30 seconds using tools-login tunnels DBA resolved 2017-01-02 2016-01-18
122998 setup/install/deploy db2033 DBA resolved 2017-01-02 2016-04-29
123108 Unthrottle faebot DBA resolved 2017-01-02 2016-01-19
123130 Labs: High Replication Lag (s6 on c1, s1 and s5 on c2) DBA resolved 2017-01-02 2016-01-18
123509 "db1047/eventlogging_sync processes" icinga alert is flaky since at least early January DBA resolved 2017-01-02 2017-04-12
123557 Database query error (internal_api_error_DBQueryError) while getting list=allrevisions DBA resolved 2017-01-02 2018-04-06
123634 EL replication having issues since at least January 11th DBA resolved 2017-01-02 2016-02-08
123731 Migrate labsdb1005/1006/1007 to jessie DBA resolved 2017-01-02 2017-03-31
123985 Inconsistencies between labs replicas DBA resolved 2017-01-02 2017-01-01
124002 Untangle wikitech/labtestwikitech and s7 DBs and networking and mysql grants DBA resolved 2017-01-02 2016-01-20
124205 Error: 2013 Lost connection to MySQL server during query on NewFilesPager DBA resolved 2017-01-02 2016-08-01
124406 s7 master is executing 54K transactions/s DBA resolved 2017-01-02 2016-12-15
124443 (Potential) ORES database schema improvements DBA resolved 2017-01-02 2016-09-16
124464 db1069 is running low on space DBA resolved 2017-01-02 2016-02-13
124649 "Lock wait timeout exceeded" errors regarding module_deps DBA resolved 2017-01-02 2016-07-27
124681 Architecture decision to solve the need larger serves (for better capacity and consolidation) vs. more, smaller servers (for high availability) DBA resolved 2017-01-02 2016-08-12
124697 Stress-test mediawiki application servers at codfw (specially to figure out db weights configuration) and basic buffer warming DBA resolved 2017-01-02 2016-04-01
124699 Change configuration to make codfw db masters as the masters of all datacenters DBA resolved 2017-01-02 2016-04-20
124704 puppetize mysql m5-master access on parsoid-test server DBA resolved 2017-01-02 2016-01-27
124705 New database for access from stat1003 (project_illustration) DBA resolved 2017-01-02 2016-01-25
124795 codfw is in read only according to mediawiki DBA resolved 2017-01-02 2016-03-10
124805 What would be the preferred way to run a series of read queries on DB replica, each on two *_p databases ? DBA resolved 2017-01-02 2017-10-06
125027 upgrade dbproxy1001/1002 to jessie DBA resolved 2017-01-02 2016-07-28
125028 reimage or decom db servers on precise DBA resolved 2017-01-02 2016-07-22
125215 Prepare db1018 and s2-slaves for s2 master failover DBA resolved 2017-01-02 2016-02-10
125386 Clarify how mysql dumps will be architectured during codfw failover DBA resolved 2017-01-02 2016-02-08
125435 Grant mysql client access to testreduce_vd and testreduce_0715 databases DBA resolved 2017-01-02 2016-02-08
125591 Set up additional filters for Echo tables DBA resolved 2017-01-02 2016-02-04
125927 labs copy of nlwiktionary db corrupted DBA resolved 2017-01-02 2016-02-05
126096 Replicate wikimania2017wiki to labs DBA resolved 2017-01-02 2016-11-02
126209 Reimage db2012 DBA resolved 2017-01-02 2016-02-11
126832 Recreate a wiki for Wikimedia Portugal DBA resolved 2017-01-02 2017-08-23
126942 labs -Database is offline DBA resolved 2017-01-02 2017-09-22
126946 disk failure on labsdb1002 DBA resolved 2017-01-02 2016-07-28
126996 Reimage db1021 database DBA resolved 2017-01-02 2016-02-25
127040 Beta cluster update database is broken because of Flow db patch db_patches/patch-ref_target_not_null.sql DBA resolved 2017-01-02 2016-02-17
127107 Copy user database to c1 DBA resolved 2017-01-02 2016-02-17
127164 Some databases cannot be backed up/replicated on toolsdb DBA resolved 2017-01-02 2017-06-23
127219 Drop wikilove_image_log table from Wikimedia wikis DBA resolved 2017-01-02 2017-06-26
127228 tools.ptwikis throttled for excessive usage of labs db replica resources DBA resolved 2017-01-02 2016-03-11
127266 Tool Labs queries die DBA resolved 2017-01-02 2016-12-21
127330 Migration from es2001-es2010 to es2011-es2019 DBA resolved 2017-01-02 2016-03-10
127385 Add pigz package to all wmf mariadbs DBA resolved 2017-01-02 2016-02-22
127638 Email spam from some MariaDB's logrotate DBA resolved 2017-01-02 2016-06-08
127769 es2010 controller issue DBA resolved 2017-01-02 2016-05-24
127811 MariaDB package improvements DBA resolved 2017-01-02 2016-11-17
128185 Prepare mysql account and options for prometheus DBA resolved 2017-01-02 2016-08-03
128353 Switchover to new s3 master DBA resolved 2017-01-02 2016-04-20
128355 s52490 created 338 simultanous conections to the same server executing the same query DBA resolved 2017-01-02 2016-04-04
128737 Move labs pdns database off of m5-master DBA resolved 2017-01-02 2016-06-17
129236 Improve documentation about database switchover DBA resolved 2017-01-02 2016-04-28
129350 es2014 revert data compression DBA resolved 2017-01-02 2017-03-24
130067 Add wl_id to watchlist tables on production dbs DBA resolved 2017-01-02 2017-06-12
130098 Create a new x1 slave in codfw (just in case) DBA resolved 2017-01-02 2016-03-23
130128 Fix dbstore2001 and dbstore2002 DBA resolved 2017-01-02 2017-03-10
130351 Configure and provision db1074-db1078 DBA resolved 2017-01-02 2016-03-24
130692 Add new indexes from eec016ece6d2b30addcdf3d3efcc2ba59b10e858 to production databases DBA resolved 2017-01-02 2016-06-17
130700 Create central OATHAuth table for CentralAuth wikis DBA resolved 2017-01-02 2016-12-15
130702 Several es20XX servers keep crashing (es2017, es2019, es2015, es2014) since 23 March DBA resolved 2017-01-02 2017-11-24
130819 Fix cluster name for databases that go into the misc group DBA resolved 2017-01-02 2016-04-04
131146 Analyze memory usage on db1011 DBA resolved 2017-01-02 2017-10-06
131236 Identical EventLogging queries give different results on db1047 and dbstore1002 DBA resolved 2017-01-02 2017-03-17
131630 Tgr unable to login on Horizon DBA resolved 2017-01-02 2017-12-09
131705 dbstore1001 low available space DBA resolved 2017-01-02 2016-06-30
131753 Semi-synchronous replication status in all MySQL production clusters DBA resolved 2017-01-02 2017-03-23
131897 High load on tools labsdb1005 DBA resolved 2017-01-02 2016-04-07
131933 BaGLAMa2 has very long running queries DBA resolved 2017-01-02 2016-04-07
131937 Throttling Cyberbot tool user as it is consuming most of the CPU DBA resolved 2017-01-02 2016-05-24
131956 Disabling general.confirmeduser from dbreports for using up too much db resources DBA resolved 2017-01-02 2017-11-17
132011 Reimage db2047 - check for hardware errors DBA resolved 2017-01-02 2017-09-24
132279 db1053 transient replica lag during dumping DBA resolved 2017-01-02 2016-08-02
132318 Job Queue growing and then running a lot of jobs at once on commonswiki DBA resolved 2017-01-02 2016-04-11
132416 Rampant differences in indexes on enwiki.revision across the DB cluster DBA resolved 2017-01-02 2018-04-13
132431 labsdb1001 and labsdb1003 short on available space DBA resolved 2017-01-02 2017-04-27
132515 db1070, db1071 and db1065 overheating problems DBA resolved 2017-01-02 2016-04-27
132539 Dumps for misc triggers unnecessary pages DBA resolved 2017-01-02 2016-12-14
132696 DBA-related homes on tin DBA resolved 2017-01-02 2017-06-23
132837 hitcounter and _counter tables are on the cluster but were deleted/unsused? DBA resolved 2017-01-02 2016-12-13
133086 Killed very long transaction that was blocking the replica on labsdb1001 DBA resolved 2017-01-02 2017-07-18
133122 Backfill data into db1065 and db1066 DBA resolved 2017-01-02 2016-04-20
133160 Jobqueue increase activity on refreshlinks for frwiktionary DBA resolved 2017-01-02 2016-04-22
133205 Switchback to eqiad DBA resolved 2017-01-02 2016-04-21
133251 archive/archive_userindex is not filled in eswiki_p DBA resolved 2017-01-02 2017-11-08
133265 Improve external storage MySQL configuration for connection spikes DBA resolved 2017-01-02 2016-04-21
133309 db1041 regular mysql stalls creates connection failures DBA resolved 2017-01-02 2016-04-22
133321 `s51187__xtools_tmp` database using 272G on labsdb1001 and 118G on labsdb1003 DBA resolved 2017-01-02 2017-07-10
133322 u3532__ (=marcmiquel) table using 64G on labsdb1001 and 108 GB on labsdb1003 DBA resolved 2017-01-02 2017-03-28
133323 u2815__old_p (dispenser) database using 20G on labsdb1001 (enwiki) DBA resolved 2017-01-02 2016-04-21
133325 s51127__dewiki_lists (merlbot) database using 13G on labsdb1001 (enwiki) DBA resolved 2017-01-02 2017-06-21
133326 p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) DBA resolved 2017-01-02 2017-07-21
133337 Automate database datacenter switchover steps DBA resolved 2017-01-02 2016-07-14
133338 Create salt database groups DBA resolved 2017-01-02 2016-05-10
133339 Improve lag detection mechanism's reliability and agility DBA resolved 2017-01-02 2016-05-18
133385 Implement GTID replication on MariaDB 10 servers DBA resolved 2017-01-02 2016-06-06
133386 Change dbstore1001 delayed slave to be a direct slave of the eqiad masters DBA resolved 2017-01-02 2016-12-09
133398 Install, configure and provision recently arrived db core machines DBA resolved 2017-01-02 2017-03-14
133584 Reduce stress caused by IABot on the DB DBA resolved 2017-01-02 2016-08-05
133588 Gerrit 285208 broke eventlogging_sync.sh DBA resolved 2017-01-02 2016-11-10
133705 Multiple concurrent long running queries from s51434 overloading labsdb1003/labsdb1009 DBA resolved 2017-01-02 2018-03-22
133715 Missing data on labs replica database DBA resolved 2017-01-02 2016-12-21
133780 Multiple Puppet class make MySQL load /etc/my.cnf twice DBA resolved 2017-01-02 2016-10-13
133801 SELECT /* CategoryMembershipChangeJob::run 127.0.0.1 */ GET_LOCK('CategoryMembershipUpdates:XXXX', 10) AS lockstatus DBA resolved 2017-01-02 2016-10-09
133906 install and puppetize memcached on dbmonitor hosts (tendril frontends) DBA resolved 2017-01-02 2018-05-09
134114 Investigate db1040 for overload DBA resolved 2017-01-02 2016-05-03
134203 enwiki_p replica on s1 is corrupted DBA resolved 2017-01-02 2017-01-25
134349 Upgrade db1069 DBA resolved 2017-01-02 2016-05-10
134392 Add shared edit rights between eranbot and plagiabot tools DBA resolved 2017-01-02 2017-10-28
134480 Improve replication lag detection for multi-dc environment DBA resolved 2017-01-02 2016-05-24
134481 Refactor $master in mariadb::core Puppet class DBA resolved 2017-01-02 2016-06-17
134555 db1033 (old s7 master) needs backup and reimage DBA resolved 2017-01-02 2016-05-20
134755 Decommission es2005-es2010 DBA resolved 2017-01-02 2016-09-23
134977 Throttle mysql backups on dbstore1001 in order to not saturate the node DBA resolved 2017-01-02 2017-01-03
135027 MASTER_POS_WAIT() alternative that works cross-DC DBA resolved 2017-01-02 2017-04-03
135029 adywiki and jamwiki are missing the associated *_p databases with appropriate views DBA resolved 2017-01-02 2016-11-25
135100 High replication lag to dewiki DBA resolved 2017-01-02 2016-05-26
135278 Import page restrictions to Cassandra restriction table DBA resolved 2017-01-02 2016-10-18
135405 Replicate CentralNotice tables to Labs DBA resolved 2017-01-02 2017-08-31
135471 Hundreds of concurrent SELECT MASTER_POS_WAIT('db1049-bin.002927', 354219766, 10) without debugging information (no user, not function) on s5 API node DBA resolved 2017-01-02 2016-05-26
135690 DBReplicationWaitError: Could not wait for slaves to catch up to 10.64.0.7 DBA resolved 2017-01-02 2016-07-20
135699 Schema changes for Echo moderation DBA resolved 2017-01-02 2016-06-06
135944 db1034 was killed 22-05-16 at 14:17:06 DBA resolved 2017-01-02 2016-06-06
135947 Missing grants on tools.labsdb DBA resolved 2017-01-02 2016-05-31
135971 labsdb1001 crashed yesterday at 21:48:07 DBA resolved 2017-01-02 2016-05-31
135973 Upgrade m1 db servers DBA resolved 2017-01-02 2016-06-27
136135 DB backup plan for ptwikibooks Flow cleanup DBA resolved 2017-01-02 2016-05-26
136136 db1056 BBU Failed DBA resolved 2017-01-02 2016-05-25
136214 Query from stat1003 brought down db1047 DBA resolved 2017-01-02 2017-07-03
136319 Labs users missing grants on replicas DBA resolved 2017-01-02 2016-07-02
136333 dbstore1002 crashed DBA resolved 2017-01-02 2016-06-06
136398 Empty sql-mode in my.cnf DBA resolved 2017-01-02 2016-06-17
136583 db2034 degraded RAID DBA resolved 2017-01-02 2016-07-14
136598 Wikidata master database connection issue DBA resolved 2017-01-02 2016-06-16
136618 Wrong page title in labs database replica enwiki page table DBA resolved 2017-01-02 2017-03-03
136789 High replag on db1069 DBA resolved 2017-01-02 2016-06-02
137191 Defragment db1070, db1082, db1087, db1092 DBA resolved 2017-01-02 2017-03-24
137567 Review ORES extension for data leaks DBA resolved 2017-01-02 2016-06-10
137691 Add shared edit rights between eranbot and copypatrol tools DBA resolved 2017-01-02 2016-06-15
137895 hidenondamaging is slow DBA resolved 2017-01-02 2016-08-12
138283 Some users use persistent connections that are idle, wasting memory and other resources that could be used for other users DBA resolved 2017-01-02 2017-04-11
138450 maintain-replicas.pl unmaintained, unmaintainable DBA resolved 2017-01-02 2016-10-18
138460 Upgrade m3 (phabricator) db servers DBA resolved 2017-01-02 2016-08-25
138516 etherpad database issues DBA resolved 2017-01-02 2016-07-07
138559 Correct m1 backups, that were being done supposing m1-master was still a slave DBA resolved 2017-01-02 2016-06-24
138778 Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 DBA resolved 2017-01-02 2016-10-01
138967 Labs database replica drift DBA resolved 2017-01-02 2017-10-18
139055 Test InnoDB compression DBA resolved 2017-01-02 2016-12-01
139056 API database error on continue with generator=categorymembers and prop=linkshere DBA resolved 2017-01-02 2017-04-12
139090 Deploy I2b042685 to all databases DBA resolved 2017-01-02 2016-11-08
139280 db1034 decommission DBA resolved 2017-01-02 2016-10-13
139283 pc2006 down DBA resolved 2017-01-02 2016-07-08
139342 DROP OAI-related tables DBA resolved 2017-01-02 2017-06-14
139346 All s4 (commons) slaves failed replication DBA resolved 2017-01-02 2016-07-08
139552 Create tables for PageAssessments in enwiki database DBA resolved 2017-01-02 2018-03-28
139618 Petscan is being used with excesive parallelism by a user on Wikidata DBA resolved 2017-01-02 2016-07-11
139714 BIOS upgrade on certain codfw machines DBA resolved 2017-01-02 2016-08-03
139755 Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades DBA resolved 2017-01-02 2016-07-08
140108 ApiQueryRecentChanges::run is spiking, nuking API servers DBA resolved 2017-01-02 2016-10-25
140337 dbstore1002 disk errors DBA resolved 2017-01-02 2016-07-22
140341 dbstore1002 lag on s7 DBA resolved 2017-01-02 2016-07-14
140452 Initial setup and provision of labsdb1009, labsdb1010 and labsdb1011 DBA resolved 2017-01-02 2016-12-14
140537 Query for Special:Contributions?contribs=newbie&dir=prev is slow on enwiki DBA resolved 2017-01-02 2016-07-19
140609 Add page_props.pp_value index to Wiki Replicas DBA resolved 2017-01-02 2018-05-06
140650 Special:Block request makes the database timeout on db1055 DBA resolved 2017-01-02 2016-10-20
140788 Labs databases rearchitecture (tracking) DBA resolved 2017-01-02 2018-03-26
140803 Remove oresc_rev index DBA resolved 2017-01-02 2017-07-03
140983 dbproxy1002 down DBA resolved 2017-01-02 2016-07-28
141090 Investigate increase in "readonly" saving errors in Content Translation DBA resolved 2017-01-02 2016-10-09
141096 Having lots of accounts with separate grants makes auditing difficult. DBA resolved 2017-01-02 2017-02-14
141097 Implement a frontend failover solution for labsdb replicas DBA resolved 2017-01-02 2016-12-21
141252 icinga hp raid check timeout on busy ms-be and db machines DBA resolved 2017-01-02 2017-08-16
141601 db2069 crashed due to RAID controller DBA resolved 2017-01-02 2016-07-29
141988 After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia DBA resolved 2017-01-02 2016-08-16
142223 Enable access to Wikipedia Tulu (tcywiki) on labs replicas DBA resolved 2017-01-02 2016-12-17
142231 The Optimistic Guy -> RainFall global rename completed without log entry DBA resolved 2017-01-02 2016-12-15
142273 dbstore2002 stopped providing mysql service despite the process being running DBA resolved 2017-01-02 2016-08-09
142305 Cannot drop table DBA resolved 2017-01-02 2016-08-06
142310 Replication seems to be halted for multiple databases DBA resolved 2017-01-02 2016-08-12
142545 Create production database and users for Striker DBA resolved 2017-01-02 2017-03-04
142725 Clean up revision UNIQUE indexes DBA resolved 2017-01-02 2017-06-06
143220 es2004 has a dead disk, but it is not under warranty DBA resolved 2017-01-02 2016-08-24
143862 s3 throughput tripled since 24 august DBA resolved 2017-01-02 2016-08-25
143932 MySQL chooses poor query plan for link counting query DBA resolved 2017-01-02 2016-11-17
143934 s2 replag currently 8 hours DBA resolved 2017-01-02 2016-08-26
144180 u13367 is running 2 inefficient 9-day-long queries, causing high cpu usage DBA resolved 2017-01-02 2016-08-29
144278 Queries on PageContentSaveComplete are starting to pileup DBA resolved 2017-01-02 2016-09-01
144382 Spiky write pattern on core db masters DBA resolved 2017-01-02 2018-03-09
144661 Add section for long-running tasks on the Deployment page (specially for database maintenance) DBA resolved 2017-01-02 2018-03-05
144723 Reimage & upgrade db1064 DBA resolved 2017-01-02 2016-09-07
145077 mysqld process hang in db1069 - S2 mysql instance DBA resolved 2017-01-02 2016-10-04
145079 Investigate slow transcludedin query DBA resolved 2017-01-02 2016-10-25
145356 Ensure ORES data violating constraints do not affect production DBA resolved 2017-01-02 2016-10-11
145366 Create and populate babel database table on Wikimedia wikis DBA resolved 2017-01-02 2017-08-01
145487 Investigate (and if possible, drop) _counters DBA resolved 2017-01-02 2016-10-17
145533 Investigate db1082 crash DBA resolved 2017-01-02 2017-10-18
145607 db1082 hardware check DBA resolved 2017-01-02 2016-09-15
145630 Cannot delete two pages with large histories even having the appropriate permissions to do so DBA resolved 2017-01-02 2016-09-14
145844 db2017 failed disk (degraded RAID) DBA resolved 2017-01-02 2016-10-05
146018 Investigate db1061 crash DBA resolved 2017-01-02 2017-04-05
146079 s5-master contention caused (?) by refreshlinksprioritized/addUsagesForPage jobs running for all wikis DBA resolved 2017-01-02 2016-09-20
146111 hidenondamaging=1 query is extremely slow on enwiki DBA resolved 2017-01-02 2016-10-11
146121 db1069: convert user_groups table to InnoDB across all the wikis DBA resolved 2017-01-02 2016-10-03
146261 Reimage dbstore2001 as jessie DBA resolved 2017-01-02 2017-02-14
146265 db1019: Decommission DBA resolved 2017-01-02 2017-02-16
146449 db1060: Degraded RAID DBA resolved 2017-01-02 2016-09-26
146529 5-10 times more rows are currently read from the s5 master than from all other masters together DBA resolved 2017-01-02 2016-09-28
146570 Give user_properties a primary key DBA resolved 2017-08-05 2017-08-29
146673 Contention on search phabricator database creating full phabricator outages DBA resolved 2017-01-02 2016-10-14
146718 [Discuss] Hosting the monthly article quality dataset on labsDB DBA resolved 2017-04-15 2017-08-17
147051 Provide at least 2 separate service endpoints: one for slow, long running queries; and another for quick, web requests DBA resolved 2017-01-02 2016-12-21
147052 Provision with data the new labsdb servers and provide replica service with at least 1 shard from a sanitized copy from production DBA resolved 2017-01-02 2016-12-21
147166 Apply change_tag and tag_summary primary key schema change to Wikimedia wikis DBA resolved 2017-01-02 2017-05-16
147172 db1055: degraded array DBA resolved 2017-01-02 2016-10-06
147296 img_metadata queries for PDF files saturates s4 slaves DBA resolved 2017-01-02 2016-12-02
147302 Prepare and check production and labs-side filtering for olowiki DBA resolved 2017-01-02 2016-11-02
147305 Unify commonswiki.revision DBA resolved 2017-01-02 2017-06-02
147396 db1065: Degraded RAID DBA resolved 2017-01-02 2016-10-10
147734 Review and deploy 309825 DBA resolved 2017-01-02 2016-10-21
147769 es2015 crashed with no os logs (kernel logs or other software ones) - it shuddenly went down DBA resolved 2017-01-02 2017-03-13
147774 Physically move db1053 to a different rack DBA resolved 2017-01-02 2016-10-13
147957 CirrusSearch SQL query for locating pages for reindex performs poorly DBA resolved 2017-01-02 2017-03-24
148078 Decommission db1015, db1035, db1044 and db1038 DBA resolved 2017-01-02 2017-11-30
148373 db2037: Disk in predictive failure DBA resolved 2017-01-02 2016-10-20
148434 es2014 raid controler temporary failure DBA resolved 2017-01-02 2016-10-18
148500 Implement proxysql both for labs and for later production usage DBA resolved 2017-01-02 2016-10-26
148560 Create maintain-views user for labsdb1001 and labsdb1003 DBA resolved 2017-01-02 2016-11-02
148561 Replicate ores_classification and ores_model tables in labs DBA resolved 2017-01-02 2017-02-22
148609 Review and deploy Linter extension to Wikimedia wikis DBA resolved 2017-04-15 2017-10-13
148633 Degraded RAID on db1046 DBA resolved 2017-01-02 2016-10-20
148790 apply events killing long running queries to db1070; any other production server DBA resolved 2017-01-02 2016-10-28
148866 DBA review of Linter extension DBA resolved 2017-01-02 2017-07-04
148955 Puppetize tendril web user creation DBA resolved 2017-01-02 2018-02-22
148967 Fix PK on S5 dewiki.revision DBA resolved 2017-01-02 2017-06-02
148988 Cognate DB review DBA resolved 2017-01-02 2017-06-07
149079 codfw: Fix S4 commonswiki.templatelinks partitions DBA resolved 2017-01-02 2017-04-21
149099 db2011 disk media errors DBA resolved 2017-01-02 2016-11-03
149186 Check wikitech has the right db grants DBA resolved 2017-01-02 2016-11-03
149357 dbtree broken DBA resolved 2017-01-02 2016-11-02
149377 Degraded RAID on db2052 DBA resolved 2017-01-02 2016-11-03
149418 Deploy gtid_domain_id flag in our mysql hosts DBA resolved 2017-01-02 2017-03-07
149422 Prepare for mariadb 10.1 DBA resolved 2017-01-02 2016-11-03
149457 install new disks into dbstore2001 DBA resolved 2017-01-02 2016-11-21
149509 Degraded RAID on db1050 DBA resolved 2017-01-02 2016-10-31
149526 es2019 crashed again DBA resolved 2017-01-02 2017-05-09
149553 db2034: investigate its crash and reimage DBA resolved 2017-01-02 2017-04-11
149624 Labsdbs for WMF tools and contributors: get more data, faster DBA resolved 2017-01-02 2017-05-12
149633 db1065 paged for NRPE timeout DBA resolved 2017-01-02 2016-11-03
149637 Implement functionality for RC page 'Experience level' filters DBA resolved 2017-01-02 2017-03-05
149728 Multiple hardware issues on db1073 DBA resolved 2017-01-02 2016-12-16
149793 Decommission db1042 DBA resolved 2017-01-02 2017-04-12
149819 Add primary keys to remaining Flow tables DBA resolved 2017-01-02 2017-01-25
149844 Decide between proxysql and haproxy for labsdbproxy service DBA resolved 2017-01-02 2016-11-14
149933 Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too DBA resolved 2017-01-02 2017-03-27
149936 Drop flow_subscription table DBA resolved 2017-01-02 2018-04-23
150017 Reimage dbstore2002 DBA resolved 2017-01-02 2016-11-04
150211 labsdb1009 boot issues (power supply and controller?) DBA resolved 2017-01-02 2016-11-22
150233 db2034 crashes meta ticket DBA resolved 2017-01-02 2017-01-16
150306 truncate l10n_cache table on WMF wikis DBA resolved 2017-01-02 2018-03-05
150334 db2042 reimage DBA resolved 2017-01-02 2016-11-16
150438 Meta ticket: Deploy InnoDB compression where possible DBA resolved 2017-01-02 2017-12-01
150474 Under high load, there is replication check pile-ups on coredbs, specially enwiki API servers DBA resolved 2017-01-02 2017-02-14
150511 db2035: RAID disk about to fail DBA resolved 2017-01-02 2016-11-22
150518 Import S5 to dbstore2001 and dbstore2002 + compression DBA resolved 2017-01-02 2016-11-18
150553 High replication activity filled up labsdb1004 with binlogs DBA resolved 2017-01-02 2017-02-14
150644 Wikidatawiki revision table needs unification DBA resolved 2017-01-02 2017-02-27
150747 Compress InnoDB on S5 (wikidatawiki, dewiki) DBA resolved 2017-01-02 2016-11-18
150802 Provision db1095 with at least 1 shard, sanitize and test slave-side triggers DBA resolved 2017-01-02 2016-12-07
150832 Replicate page_assessments and page_assessments_projects tables on Labs DBA resolved 2017-01-02 2016-12-07
150850 Decouple roles from mariadb.pp into their own file DBA resolved 2017-01-02 2017-12-14
150851 Moving backup and otrs role into their own .pp DBA resolved 2017-01-02 2016-12-01
150876 db2049 overheated and restarted DBA resolved 2017-01-02 2017-06-06
150949 Set barracuda InnoDB file format as the default configuration everywhere DBA resolved 2017-01-02 2018-05-08
150960 Initial data tests for db1095 (temporary db1069 - sanitarium replacement) DBA resolved 2017-01-02 2016-12-07
150965 phabricator close to saturate its database connections DBA resolved 2017-01-02 2016-11-17
150974 db2042 disk predictive failure DBA resolved 2017-01-02 2016-12-07
151014 Provisioning MySQL replica users fails on tool labs DBA resolved 2017-01-02 2017-01-20
151029 duplicate key problems DBA resolved 2017-01-02 2017-08-14
151039 Unknown cause is creating lag on db1048 under write load (but not on the other m3 slaves) DBA resolved 2017-01-02 2016-12-15
151076 labsdbadmin needs to be able to drop users and update passwords DBA resolved 2017-01-02 2016-11-18
151203 db2041: Disk RAID predictive failure DBA resolved 2017-01-02 2016-12-02
151228 Editing a recurring event overrides all past instances DBA resolved 2017-01-02 2016-12-09
151272 db1092 crash DBA resolved 2017-01-02 2016-11-28
151356 Wikibase\Repo\Store\Sql\SqlEntityIdPager::fetchIds query slow DBA resolved 2017-01-02 2016-11-28
151384 phabricator_conduit.conduit_methodcalllog failed replicating on dbstore1002, probably m3 needs a reload on that server DBA resolved 2017-01-02 2016-11-23
151465 Degraded RAID on db1053 DBA resolved 2017-01-02 2017-03-03
151552 Import S2,S6,S7,m3 and x1 to dbstore2001 and dbstore2002 DBA resolved 2017-01-02 2017-03-06
151607 Rebuild old timestamp format tables DBA resolved 2017-01-02 2017-09-12
151717 Usage tracking: record which statement group is used DBA resolved 2017-03-08 2018-03-02
151746 fstrim: Operation not supported on Labs DBs DBA resolved 2017-01-02 2017-04-12
151752 Prepare and check storage layer for the future private wiki arbcom-cs.wikipedia.org DBA resolved 2017-01-02 2016-12-11
151756 Prepare and check storage layer for new fi.wikivoyage.org DBA resolved 2017-01-02 2016-12-01
151763 Degraded RAID on db2068 DBA resolved 2017-01-02 2016-12-07
151993 Implement ChangeDispatchCoordinator based on RedisLockManager DBA resolved 2017-01-02 2017-04-09
151995 Rolling restart of external storage servers for TLS certificate update DBA resolved 2017-01-02 2016-12-01
152029 Rolling restart of parsercache servers for TLS certificate update DBA resolved 2017-01-02 2016-12-01
152043 Tool Labs: Add skin, language, and variant to user_properties_anon DBA resolved 2017-01-02 2017-04-12
152081 Moving eventlogging mariadb role into its own .pp DBA resolved 2017-01-02 2017-07-14
152188 Restart pending mysql hosts with old TLS cert DBA resolved 2017-01-02 2017-02-08
152194 Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 DBA resolved 2017-01-02 2016-12-21
152214 Degraded RAID on db1033 DBA resolved 2017-01-02 2017-01-04
152364 db1047 out of disk space, eventlogging_sync spam DBA resolved 2017-01-02 2016-12-07
152377 Create labsdb_accounts db on m5 to store state about labsdb accounts DBA resolved 2017-01-02 2016-12-05
152385 66 rows from external storage (dewiki) gave duplicate key errors on master failover DBA resolved 2017-01-02 2018-03-13
152411 Degraded RAID on db1048 DBA resolved 2017-01-02 2017-01-04
152585 rcshow=oresreview is slow DBA resolved 2017-01-02 2017-03-16
152761 db1028 increased lag after extensions/CentralAuth/maintenance/populateLocalAndGlobalIds.php DBA resolved 2017-01-02 2016-12-15
152766 dbstore1001 duplicate entry on s4 commonswiki.recentchanges DBA resolved 2017-01-02 2016-12-12
152894 high s1 replag to labs DBA resolved 2017-01-02 2016-12-11
153033 Drop MoodBar tables from all wikis DBA resolved 2017-01-02 2017-10-23
153058 LabsDB infrastructure pending work DBA resolved 2017-01-02 2018-03-26
153182 Perform schema change to add externallinks.el_index_60 to all wikis DBA resolved 2017-01-02 2018-04-18
153184 s3 database resource usage and contention increased 2-10x times DBA resolved 2017-01-02 2016-12-15
153194 Remove partitions from metawiki.pagelinks DBA resolved 2017-01-02 2016-12-15
153213 Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" DBA resolved 2017-01-02 2017-06-26
153300 Remove partitions from metawiki.pagelinks in s7 DBA resolved 2017-01-02 2017-04-27
153440 Create a full backup of all external storage records that would be easy to restore/setup a temporary delayed slave DBA resolved 2017-01-02 2018-04-06
153634 db1047 paged low on disk space DBA resolved 2017-01-02 2016-12-20
153680 Create a cronjob/check to run check_private_data data script and report back DBA resolved 2017-01-02 2017-04-27
153739 Defragment db1015 DBA resolved 2017-01-02 2017-01-26
153740 Degraded RAID on db2011 DBA resolved 2017-01-02 2017-01-30
153743 Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts DBA resolved 2017-01-02 2018-03-26
153768 Install and reimage dbstore1001 as jessie DBA resolved 2017-01-02 2017-03-13
153826 Defragment db1044 DBA resolved 2017-01-02 2017-01-18
154031 db2060 crashed (RAID controller) DBA resolved 2017-01-02 2017-01-24
154097 Remove partitions from enwiktionary.templatelinks in s2 DBA resolved 2017-01-02 2017-03-28
154189 db2035 reset DBA resolved 2017-01-02 2017-01-03
154327 Gerrit: Schedule downtime for T154205 (To do with data loss) DBA resolved 2017-01-02 2017-01-06
154355 page_lang column of the page table is not replicated to Labs DBA resolved 2017-01-02 2017-03-22
154465 Defragment db1038 DBA resolved 2017-01-04 2017-01-20
154485 run pt-table-checksum on s2 (WAS: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038) DBA resolved 2017-01-04 2018-01-17
154490 Delete prefstats tables DBA resolved 2017-01-04 2018-04-11
154497 Spatial database for tool-labs DBA resolved 2017-01-04 2017-01-12
154685 Sqoop doesn't run anymore - Seem related to a DB change (analytics store) DBA resolved 2017-01-06 2017-01-18
154929 db1026 (s5) needs some compression DBA resolved 2017-01-10 2017-01-19
155041 Prepare and check storage layer for wikimania2018wiki DBA resolved 2017-01-11 2017-08-07
155042 Increase quota for wikidata-dev project DBA resolved 2017-01-11 2017-01-24
155091 "Invalid DB key" errors on various special pages DBA resolved 2017-01-12 2017-10-13
155099 Database maintenance scheduled while eqiad datacenter is non primary (after the DC switchover) DBA resolved 2017-01-12 2017-05-03
155102 Review capacity on codfw DBA resolved 2017-01-12 2017-03-02
155190 Build an environment to test change dispatching using Redis-based locking DBA resolved 2017-01-13 2017-03-07
155399 db1045 disk space - compression needed DBA resolved 2017-01-17 2017-02-22
155500 Fatal exception of type "DBQueryError" on sorting ORES contributions DBA resolved 2017-01-18 2017-01-24
155605 Schema changes for expiring user groups DBA resolved 2017-01-19 2017-04-25
155769 Spurious completely empty `image` table row on commonswiki DBA resolved 2017-01-20 2017-04-26
155902 Reset password for database user p50380g50491 DBA resolved 2017-01-22 2017-01-25
156004 Move db1051 to row B3 DBA resolved 2017-01-24 2017-01-24
156005 Reimage db1065 and db1066 DBA resolved 2017-01-24 2017-01-25
156006 Move db1052 to row B3 DBA resolved 2017-01-24 2017-01-24
156008 Switchover s1 master db1057 -> db1052 DBA resolved 2017-01-24 2018-01-04
156126 Move db1073 to B3 DBA resolved 2017-01-25 2017-02-09
156161 db2060 not accessible DBA resolved 2017-01-25 2017-02-14
156166 db1072 change_tag schema and dataset is not consistent DBA resolved 2017-01-25 2017-01-31
156226 Reimage and clone db1072 DBA resolved 2017-01-26 2017-02-06
156285 Labs users reporting timouts when connecting to labsdb-web.eqiad.wmnet DBA resolved 2017-01-26 2017-01-25
156305 Update page_assessments_projects schema for subprojects in production DBA resolved 2017-01-26 2017-02-14
156373 During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time DBA resolved 2017-01-27 2017-02-01
156478 Change rack for servers in s1 in codfw DBA resolved 2017-01-28 2017-02-17
156560 lab's templatelinks contain possible garbage rows DBA resolved 2017-01-29 2017-01-28
156758 Drop m3 from dbstore servers DBA resolved 2017-02-01 2017-08-10
156844 Decommission old dbstore hosts (db1046, db1047) DBA resolved 2017-02-01 2017-12-11
156905 Phabricator master and slave crashed DBA resolved 2017-02-02 2017-02-14
157348 Have search recognise STL files as a new kind of media file ('type:3d' or whatever) DBA resolved 2017-05-19 2017-07-29
157358 labsdb1005 (mysql) maintenance for reimage DBA resolved 2017-02-07 2017-05-15
157714 Better mysql command prompt info for Beta DBA resolved 2017-02-11 2017-06-14
157931 s5: db1070 not using file per table DBA resolved 2017-02-14 2017-03-17
158188 db1082 MySQL crashed DBA resolved 2017-02-16 2017-10-18
158193 Reimage db1060 due to pysical disk corruption (was: Degraded RAID on db1060) DBA resolved 2017-02-16 2017-03-08
158194 Replace BBU for db1060 DBA resolved 2017-02-16 2017-02-22
158246 codfw: switch ports clean up DBA resolved 2017-02-16 2017-02-16
158446 fundraising database tuning DBA resolved 2017-02-18 2017-04-17
158761 Degraded RAID on db1049 DBA resolved 2017-02-23 2017-02-22
158764 m1 slaves all broke replication due to bacula.DelCandidates temporary table DBA resolved 2017-02-23 2017-03-02
158893 dbstore1001 troubleshoot IPMI issue DBA resolved 2017-02-24 2017-07-10
158969 Degraded RAID on db1070 DBA resolved 2017-02-25 2017-03-17
159023 Possible labs imagelinks drift DBA resolved 2017-02-26 2017-03-06
159266 db1047 BBU RAID issues (was: Investigate db1047 replication lag) DBA resolved 2017-03-01 2017-06-19
159410 Degraded RAID on db1056 DBA resolved 2017-03-03 2017-03-03
159414 Rampant differences in indexes and PK on s6 (frwiki, jawiki, ruwiki) for revision table DBA resolved 2017-03-03 2017-03-14
159493 Data integrity issue with enwiki_p user_groups on Wikimedia Tool Labs (missing rows) DBA resolved 2017-03-03 2017-03-05
159572 labsdb1004 MySQL crash DBA resolved 2017-03-04 2017-03-08
159665 Degraded RAID on db2044 DBA resolved 2017-03-07 2017-03-08
159666 Degraded RAID on db2048 DBA resolved 2017-03-07 2017-03-08
159707 Import x1 on dbstore2001 DBA resolved 2017-03-07 2017-03-13
159718 Evaluate how to best add a column for full entity ID to wb_terms without affecting wikidata.org users DBA resolved 2017-03-07 2017-05-05
159753 Concerns about ores_classification table size on enwiki DBA resolved 2017-05-17 2017-10-17
159826 Use redis-based lock manager in dispatch changes in production DBA resolved 2017-03-08 2017-05-08
159828 Use redis-based lock manager for dispatchChanges on test.wikidata.org DBA resolved 2017-03-08 2017-04-12
160062 Allow setting the watchlist table to read-only on a per-wiki basis DBA resolved 2017-03-10 2018-02-20
160242 es2015 crashed on 2017-03-11 DBA resolved 2017-03-12 2017-03-23
160390 Unify revision table on s7 DBA resolved 2017-03-15 2017-05-19
160392 Reset db1070 idrac DBA resolved 2017-03-15 2017-07-10
160407 Apply wikishared.cx_translations index change DBA resolved 2017-03-15 2017-03-15
160415 Review schema changes for T125071 - Add index to image table on all wikis DBA resolved 2017-03-16 2017-04-12
160435 db1057 does not react to powercycle/powerdown/powerup commands DBA resolved 2017-03-15 2017-04-04
160509 run pt-tablechecksum on s6 DBA resolved 2017-03-16 2018-01-18
160611 Make "linter" table available on Labs DBA resolved 2017-03-27 2017-05-09
160686 ug_expiry column of the user_groups table is not present on Labs DBA resolved 2017-03-17 2017-04-24
160691 Refreshing testreduce_vd database on ruthenium DBA resolved 2017-03-17 2017-03-17
160713 Replicate babel db table on Labs DBA resolved 2017-03-17 2017-07-07
160832 db1094 crash DBA resolved 2017-03-19 2017-03-22
160869 Prepare and check storage layer for kbp.wikipedia.org DBA resolved 2017-03-20 2017-07-04
160887 Wikibase\Repo\Store\Sql\SqlEntitiesWithoutTermFinder::getEntitiesWithoutTerm can take 19 hours to execute and it is run by the web requests user DBA resolved 2017-03-21 2017-06-25
160984 Reduce max execution time of interactive queries or a better detection and killing of bad query patterns DBA resolved 2017-03-22 2018-03-21
160991 Check OATHAuth tables are in each private or fishbowl wiki DBA resolved 2017-03-22 2017-03-21
161007 Decouple Mariadb semi-sync replication from $::mw_primary DBA resolved 2017-03-22 2017-05-03
161024 dbstore2001 creates bad query plans for wikidata.wb_changes DBA resolved 2017-03-22 2017-03-22
161088 Migrate some s4 hosts to file per table DBA resolved 2017-03-23 2017-11-02
161159 Cannot access the database: Can't connect to MySQL server on '10.192.48.41' (111) (10.192.48.41) DBA resolved 2017-03-23 2017-03-23
161232 Bump MediaWiki's minimum supported MySQL Version to 5.5.8 DBA resolved 2017-03-24 2017-11-04
161294 run pt-tablechecksum on s5/s8 DBA resolved 2017-03-25 2018-01-09
161524 Implements ContentHandler abstraction for ProofreadPage Index: pages DBA resolved 2017-03-28 2017-08-02
161600 Degraded RAID on db1067 DBA resolved 2017-03-29 2017-03-29
161631 [Task] Remove "all" option for Special:EntitiesWithout*" DBA resolved 2017-03-29 2017-04-17
161683 Remove partitioning from db2019 (codfw master) commonswiki.templatelinks DBA resolved 2017-03-30 2017-04-05
161712 codfw: (1) spare pool system for temp allocation as database failover DBA resolved 2017-03-30 2017-05-04
161781 s2 on db1069 stuck on a query for trwiki flaggedrevs_tracking table DBA resolved 2017-03-31 2017-04-18
162133 Replace some masters in eqiad while it is not active DBA resolved 2017-04-05 2017-04-26
162135 Decommission db1057 DBA resolved 2017-04-05 2017-05-03
162138 How many revision comments are exactly the same? Get some stats. DBA resolved 2017-04-05 2017-04-07
162159 codfw rack/setup 22 DB servers DBA resolved 2017-04-05 2018-04-29
162183 tendril cert expiry alerts on dbmonitor hosts DBA resolved 2017-04-05 2017-04-14
162212 dbstore1002 in bad shape DBA resolved 2017-04-05 2017-04-05
162233 eqiad rack/setup 11 new DB servers DBA resolved 2017-04-06 2018-04-29
162252 Create SQL database and Tables for Cognate extension to be used on Wiktionaries DBA resolved 2017-04-06 2017-04-13
162290 setup tempdb2001(WMF6407) DBA resolved 2017-04-06 2017-04-10
162519 s51362 has been rate limited to 2 concurrent connections for creating hundreds of 1400-second queries to labsdb1001 and labsdb1003 every 10 seconds DBA resolved 2017-04-09 2017-07-31
162539 Deploy schema change for adding term_full_entity_id column to wb_terms table DBA resolved 2017-04-27 2017-07-24
162556 Consider only updating wb_changes_dispatch after a successful run DBA resolved 2017-04-10 2017-06-29
162557 Use replica for reading the last dispatch position (chd_seen) DBA resolved 2017-04-10 2017-06-29
162593 Run pt-table-checksum on s4 (commonswiki) DBA resolved 2017-04-11 2017-12-28
162611 Unify revision table on s2 DBA resolved 2017-04-11 2017-05-25
162681 Network maintenance on row D (databases) DBA resolved 2017-04-12 2017-04-28
162699 Decomissions old s2 eqiad hosts (db1018, db1021, db1024, db1036) DBA resolved 2017-04-12 2017-12-05
162774 FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks DBA resolved 2017-04-13 2017-08-29
162789 Create less overhead on bacula jobs when dumping production databases DBA resolved 2017-04-13 2018-03-09
162807 Run pt-table-checksum on s1 (enwiki) DBA resolved 2017-04-13 2018-03-13
163023 S1 replag at 3 hours DBA resolved 2017-04-15 2017-04-17
163109 Reclone db1063 to become a slave in s5 DBA resolved 2017-04-18 2017-04-21
163110 Reclone db1068 to become master in s4 DBA resolved 2017-04-18 2017-04-20
163143 dbtree: don't return 200 on error pages DBA resolved 2017-04-18 2017-09-07
163183 db1069 s7 replication thread stuck on fawiki.flaggedrevs_tracking DBA resolved 2017-04-19 2017-04-18
163190 Checksum data on s7 DBA resolved 2017-04-19 2018-01-19
163351 codfw API slaves overloaded during the 2017-04-19 codfw switch DBA resolved 2017-04-20 2017-04-21
163413 Pool new server db2071 DBA resolved 2017-04-21 2017-04-21
163450 I99pema reset watchlist DBA resolved 2017-04-21 2017-04-20
163548 Drop the useless wb_terms keys "wb_terms_entity_type" and "wb_terms_type" on "wb_terms" table DBA resolved 2017-04-22 2017-05-11
163665 Reclone db1062 from db1041 (s7 master) DBA resolved 2017-04-25 2017-04-25
163778 Decommission db1022 (Was: db1022 broke while changing topology on s6- evaluate if to fix or directly decommission) DBA resolved 2017-04-26 2017-11-16
163895 Move masters away from D1 in eqiad DBA resolved 2017-04-27 2017-04-28
163912 Convert unique keys into primary keys for some wiki tables on s3-eqiad and s3-codfw DBA resolved 2017-04-27 2017-10-06
163979 Convert unique keys into primary keys for some wiki tables on s6-eqiad and codfw DBA resolved 2017-04-28 2017-09-27
164057 Decommission db1040 DBA resolved 2017-04-29 2017-05-09
164073 Puppetize Piwik's Database and set up periodical backups DBA resolved 2017-04-29 2017-07-18
164092 db1062 (s7 master eqiad) in a reboot cycle DBA resolved 2017-04-29 2017-04-29
164107 db1063 thermal issues (was: db1063 io (s5 master eqiad) performance is bad) DBA resolved 2017-04-29 2018-02-28
164173 Cache invalidations coming from the JobQueue are causing lag on several wikis DBA resolved 2017-05-01 2017-09-14
164185 Convert unique keys into primary keys for some wiki tables on s1, s2, s4, s5 and s7 (eqiad) DBA resolved 2017-05-02 2017-08-04
164399 Convert unique keys into primary keys for some wiki tables on all s* shards (codfw) DBA resolved 2017-05-04 2017-08-04
164488 Run pt-table-checksum on s3 DBA resolved 2017-05-05 2017-12-28
164530 Deploy uniqueness constraints on ores_classification table DBA resolved 2017-05-05 2017-06-28
164552 Slow query in API list=tags DBA resolved 2017-05-06 2017-05-05
164604 Document performance optimization of servermon and/or puppet reporting tools DBA resolved 2017-05-06 2017-05-18
164702 Decommission db1024 DBA resolved 2017-05-09 2017-12-05
164805 Investigate requirements for MySQL access in RESTBase DBA resolved 2017-05-09 2017-12-06
164815 db2057 replication broken due to delete on echo_notification and echo_event DBA resolved 2017-05-10 2017-05-09
164887 Drop Semantic Database tables from wikitech wikis DBA resolved 2017-05-10 2017-05-17
164949 Drop titlekey table from all wmf databases DBA resolved 2017-05-11 2017-06-20
164968 remove mira wikitech grants DBA resolved 2017-05-11 2017-05-11
164990 RfC: Reading List service DBA resolved 2017-06-27 2018-01-09
165033 dbstore2001 takes 3 hours to start MySQL (was: dbstore2001 takes 3 hours to start MySQL after a crash) DBA resolved 2017-05-12 2017-06-20
165246 Add term_full_entity_id column to wb_terms table on testwikidatawiki DBA resolved 2017-05-17 2017-05-16
165629 Degraded RAID on db2058 DBA resolved 2017-05-18 2017-05-22
165743 frwiktionary on s7 still needs fixing on the revision table DBA resolved 2017-05-20 2017-05-23
165836 Drop table MediaWikiInstallPingback_15732959 from EventLogging DB DBA resolved 2017-05-21 2017-05-22
165977 Create CoC committee private wiki DBA resolved 2017-05-22 2017-10-24
166028 Rename user "Mlpearc" to "FlightTime" on Central Auth DBA resolved 2017-06-08 2017-08-01
166060 beta cluster databases have almost full disks DBA resolved 2017-06-01 2017-05-31
166097 Drop Gather tables from wmf wikis DBA resolved 2017-05-23 2018-03-06
166141 db1046 BBU looks faulty DBA resolved 2017-05-24 2017-07-06
166184 Run updateRestrictions.php on WMF wikis DBA resolved 2017-05-25 2017-07-19
166194 Database replication issues with deleted pages (affecting Tool Labs and Analytics Store) DBA resolved 2017-05-25 2017-05-25
166204 Convert unique keys into primary keys for some wiki tables on s1 DBA resolved 2017-05-25 2017-08-04
166205 Convert unique keys into primary keys for some wiki tables on s2 DBA resolved 2017-05-25 2017-06-21
166206 Convert unique keys into primary keys for some wiki tables on s4 DBA resolved 2017-05-25 2017-06-19
166207 Convert unique keys into primary keys for some wiki tables on s5 DBA resolved 2017-05-25 2017-06-28
166208 Convert unique keys into primary keys for some wiki tables on s7 DBA resolved 2017-05-25 2017-07-06
166261 fixProofreadIndexPagesContentModel doDBUpdates method turns infinitely DBA resolved 2017-05-25 2017-05-24
166267 dbtree: dbtree.wikimedia.org currently returns a 500 error DBA resolved 2017-05-25 2017-05-24
166278 Unify revision table on s3 DBA resolved 2017-05-26 2017-06-06
166326 db2055 mariadb weirdness -lagging after reboot and upgrade DBA resolved 2017-05-26 2017-05-30
166422 Degraded RAID on db1046 DBA resolved 2017-05-28 2017-05-31
166452 db1047 has been restarted - needs another restart DBA resolved 2017-05-29 2017-06-06
166486 Decommission db1023 DBA resolved 2017-05-30 2017-10-12
166518 Degraded BBU on db1094 (was: Degraded RAID on db1094) DBA resolved 2017-05-30 2017-06-06
166546 Point labsdb1001 and labsdb1003 to db1095 and db1102 DBA resolved 2017-05-31 2017-08-09
166568 Error (Wikimedia\Rdbms\DBQueryError) when creating a SecurePoll poll on testwiki DBA resolved 2017-05-31 2017-06-03
166683 db2044 cannot install jessie - requires BIOS firmware upgrade DBA resolved 2017-06-01 2017-06-06
166853 Degraded RAID on db2049 DBA resolved 2017-06-03 2017-06-06
166933 db1089 - possibly RAID controller broken DBA resolved 2017-06-04 2017-06-03
166935 db1089: update RAID controller firwmare DBA resolved 2017-06-04 2017-06-13
167031 Global rename of Idh0854 → Garam: supervision needed DBA resolved 2017-06-06 2017-08-21
167567 Migrate parsercache hosts to file per table DBA resolved 2017-06-11 2017-06-19
167597 Renaming Neoalpha: supervision needed DBA resolved 2017-06-12 2017-12-05
167667 Degraded RAID on db2070 DBA resolved 2017-06-13 2017-06-16
167715 Prepare and check storage layer for atjwiki DBA resolved 2017-06-13 2017-06-20
167758 Need to empty "linter" database table on large wikis DBA resolved 2017-06-14 2017-06-13
167784 WMF ParserCache disk space exhaustion DBA resolved 2017-06-28 2018-02-11
167961 Terbium cronjobs attempting to connect to labstestweb2001 DBA resolved 2017-06-16 2017-06-19
167978 Truncate 'searchindex' table on all wikis DBA resolved 2017-06-16 2017-06-26
168010 Special:ShortPages does not load in Wikimedia Commons: "Read timeout is reached" DBA resolved 2017-06-17 2017-07-19
168021 setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 DBA resolved 2017-06-16 2017-06-20
168109 Global rename of Smuconlaw → Sgconlaw: supervision needed DBA resolved 2017-06-18 2017-08-01
168354 dbstore2001: s5 thread isn't able to catch up with the master DBA resolved 2017-06-21 2017-07-24
168356 Prepare mysql hosts for stretch DBA resolved 2017-06-21 2017-06-30
168409 Migrate dbstore2001 to multi instance DBA resolved 2017-06-21 2017-09-12
168584 Labsdb* servers need to be rebooted DBA resolved 2017-06-22 2017-11-02
168661 Apply schema change to add 3D filetype for STL files DBA resolved 2017-06-23 2018-03-20
168776 Global rename of Green Cardamom → GreenC: supervision needed DBA resolved 2017-06-25 2017-08-01
168788 Prepare and check storage layer for maiwikimedia DBA resolved 2017-06-26 2018-02-28
169038 SQL dewiki_p categorylinks cl_from index missing DBA resolved 2017-06-29 2017-06-28
169050 dbstore1001 mysql crashed with: semaphore wait has lasted > 600 seconds DBA resolved 2017-06-29 2017-08-16
169193 Prepare and check storage layer for dinwiki DBA resolved 2017-06-30 2017-07-14
169294 File space alert for db1028 DBA resolved 2017-07-01 2017-06-30
169336 slow master queries on Wikibase\Client\Usage\Sql\EntityUsageTable::getAffectedRowIds DBA resolved 2017-07-01 2017-07-05
169355 Degraded RAID on db1052 DBA resolved 2017-07-01 2017-06-30
169376 Truncate table "interwiki" on all wmf sites DBA resolved 2017-07-01 2017-10-23
169377 Truncate table "job" on all wmf sites DBA resolved 2017-07-01 2017-07-24
169396 Global rename of Markos90 → Mαρκος: supervision needed DBA resolved 2017-07-02 2017-07-04
169448 Degraded RAID on db1066 DBA resolved 2017-07-02 2017-07-26
169488 Drop ukwikimedia from labsdb hosts (was: ukwikimedia still present on replicas dbs on labs hosts) DBA resolved 2017-07-04 2017-07-10
169501 Move some masters away from B6 DBA resolved 2017-07-04 2018-01-16
169510 Setup dbstore2002 with 2 new mysql instances from production and enable GTID DBA resolved 2017-07-04 2017-08-07
169514 Refactor puppet mariadb class to support multi-instance hosts DBA resolved 2017-07-04 2017-10-02
169516 Implement cron-based mydumper backups on the dbstore role DBA resolved 2017-07-04 2017-09-29
169517 Research backup storage options and prepare a design document DBA resolved 2017-07-04 2017-09-30
169527 Global rename of Antero de Quintal → JMagalhães: supervision needed DBA resolved 2017-07-04 2017-08-01
169693 db2044: Disk on predictive failure DBA resolved 2017-07-06 2017-07-06
169928 Evaluate how hard would be to get aa(wikibooks|wiktionary) and howiki databases deleted DBA resolved 2017-07-07 2017-08-20
170066 MySQL password for research@analytics-store.eqiad.wmnet publicly revealed DBA resolved 2017-07-10 2017-07-12
170158 Setup database for dmarc service DBA resolved 2017-07-11 2017-07-25
170308 dbstore1002 crashed DBA resolved 2017-07-12 2017-07-17
170351 db2019 has performance issues, replace disk or switchover s4 master elsewhere DBA resolved 2017-07-12 2017-08-04
170503 Degraded RAID on db2019 DBA resolved 2017-07-13 2017-07-13
170657 labsdb1009 crashed while doing an alter table on templatelinks DBA resolved 2017-07-15 2017-07-17
170662 Productionize 22 new codfw database servers DBA resolved 2017-07-15 2018-04-09
170666 Refactor prometheus-mysqld-exporter to support multi-instance hosts DBA resolved 2017-07-15 2017-10-09
170927 Make wbqc_constraints table available on Quarry et al. DBA resolved 2017-07-19 2017-08-25
170941 Global rename of user Moros DBA resolved 2017-07-19 2017-07-19
170952 Inconsistent default charset for analytics slaves DBA resolved 2017-07-19 2017-08-30
171027 "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis DBA resolved 2017-09-23 2017-10-22
171321 Finish dbstore2002 migration to multi-instance DBA resolved 2017-07-22 2017-08-07
171429 Delete _image_new.ibd table if exists somewhere DBA resolved 2017-07-25 2017-07-24
171460 Populate term_full_entity_id on www.wikidata.org DBA resolved 2017-07-25 2017-10-17
171461 Populate term_full_entity_id on test.wikidata.org DBA resolved 2017-07-25 2017-07-25
171474 Global rename of Carrotkit → 胡蘿蔔: supervision needed DBA resolved 2017-07-25 2017-08-01
171723 Degraded RAID on db1068 DBA resolved 2017-07-27 2017-07-31
171829 Prepare and check storage layer for hi.wikiversity DBA resolved 2017-07-28 2017-08-25
172015 wikibase-addUsagesForPage doesn't batch properly DBA resolved 2017-07-29 2017-08-18
172020 Drop WikiGrok tables from production DBA resolved 2017-07-29 2017-08-16
172169 Compress InnoDB on db1102 DBA resolved 2017-08-02 2017-08-07
172207 flaggedrevs.fr_user is unindexed DBA resolved 2017-08-02 2017-11-10
172265 es2013 faulty BBU DBA resolved 2017-08-03 2017-08-10
172364 Remove cx_drafts table from production DBA resolved 2017-08-04 2017-08-22
172485 Convert unique keys into primary keys for some wiki tables on s3 (both eqiad and codfw) DBA resolved 2017-08-05 2017-08-04
172494 Change oom_adj for dedicated mysql server processes DBA resolved 2017-08-05 2017-10-11
172498 Switch databases to the future parser DBA resolved 2017-08-05 2017-10-24
172679 Productionize 11 new eqiad database servers DBA resolved 2017-08-08 2017-10-06
172784 Unable to drop tables on dewiki.labsdb/s51072__dwl_p DBA resolved 2017-08-09 2017-08-08
172996 Migrate s4 from db1095 to db1102 DBA resolved 2017-08-11 2017-08-22
173027 Prepare and check storage layer for hi.wikivoyage DBA resolved 2017-08-11 2017-09-28
173159 Global rename of Darwinius → DarwIn: supervision needed DBA resolved 2017-08-13 2017-08-22
173269 Wikidata.org currently very slow DBA resolved 2017-08-14 2017-08-22
173365 RAID crashed on db1078 DBA resolved 2017-08-16 2017-08-24
173570 Decommission db1015 DBA resolved 2017-08-19 2018-01-10
173647 Prepare and check storage layer for hif.wiktionary DBA resolved 2017-08-21 2017-11-30
173834 Global rename: Opdire657 → Sakiv; supervision needed DBA resolved 2017-08-23 2017-08-23
173859 Global rename supervision request: Papa1234 → Karl-Heinz Jansen DBA resolved 2017-08-23 2017-08-24
173891 Create core ip_changes view for replicas DBA resolved 2017-08-24 2017-11-28
173915 Decommission db1041 DBA resolved 2017-08-24 2017-11-14
174054 Test reliability of RAID configuration/database hosts on single disk failure DBA resolved 2017-08-25 2017-10-05
174076 Decommission db1033 and db1028 DBA resolved 2017-08-25 2017-11-14
174182 set up hiwikiversity on labsdb1010 DBA resolved 2017-08-26 2017-08-25
174265 BBU issues on db1055, RAID cache on WriteThrough DBA resolved 2017-08-28 2017-09-14
174370 Create elections committee private wiki DBA resolved 2017-08-29 2018-03-08
174385 Prepare and check storage layer for electcomwiki DBA resolved 2017-08-29 2017-09-05
174509 Drop now redundant indexes from pagelinks and templatelinks DBA resolved 2017-08-30 2017-11-08
174543 CHAR_LENGTH does not return the character count DBA resolved 2017-09-04 2017-09-05
174569 Schema change for refactored comment storage DBA resolved 2017-08-31 2018-03-02
174586 Remove ReaderFeedback tables from wikis DBA resolved 2017-08-31 2017-09-05
174763 Decommission db1026 DBA resolved 2017-09-02 2018-01-10
174764 db2044 HW RAID failure DBA resolved 2017-09-02 2017-10-03
174782 Drop pr_index from wikis where ProofreadPage isn't enabled DBA resolved 2017-09-02 2017-09-05
174801 Drop flaggedrevs tables on wikis where it is not enabled DBA resolved 2017-09-02 2018-05-14
174804 Truncate `updatelog` on all wikis DBA resolved 2017-09-02 2018-03-23
174806 Decommission db1045 DBA resolved 2017-09-02 2018-01-10
174857 Degraded RAID on db1059 DBA resolved 2017-09-03 2017-09-06
174902 Decommission db1037 DBA resolved 2017-09-05 2017-11-14
175002 db1009 (m5, used primarily for cloud services) unresponsive for minutes DBA resolved 2017-09-06 2017-09-26
175051 Drop "trackbacks" table on all wikis that have it DBA resolved 2017-09-06 2017-09-27
175086 Create and announce timeline for shutting down labsdb100[13] DBA resolved 2017-09-06 2017-10-31
175096 Identify tools hosting databases on labsdb100[13] and notify maintainers DBA resolved 2017-09-06 2018-02-05
175228 Degraded RAID on db2010 DBA resolved 2017-09-08 2017-09-19
175264 Decommission db1049 DBA resolved 2017-09-08 2018-01-10
175487 Significant replication lag for the s1, s2, and s4 wikis on labsdb100[13] DBA resolved 2017-09-11 2017-09-19
175679 Decommission db1048 (was Move m3 slave to db1059) DBA resolved 2017-09-13 2018-01-10
175685 Decommission db2010 and move m1 codfw to db2078 DBA resolved 2017-09-13 2017-11-03
175946 Global rename for RadioFan DBA resolved 2017-09-15 2017-09-21
175970 Lost access to x1-analytics-slave DBA resolved 2017-09-15 2017-10-20
175973 db1100 crashed DBA resolved 2017-09-16 2018-04-29
176043 Prepare and check storage layer for amwikimedia (including dropping s7 version of the wiki) DBA resolved 2017-09-17 2018-01-04
176055 Update of QueryPages failing on commons with "MASTER_POS_WAIT() or MASTER_GTID_WAIT() failed: MySQL server has gone away" DBA resolved 2017-09-19 2017-09-28
176215 decommission db1018 DBA resolved 2017-09-20 2017-12-05
176243 Decommission database hosts <= db2031 (tracking) DBA resolved 2017-09-20 2018-05-03
176311 decommission db1036 DBA resolved 2017-09-21 2017-12-05
176381 Grant tool "tool-db-usage" (s53508) ability to read full table_schema.TABLES contents DBA resolved 2017-09-22 2017-09-21
176557 High replag on s1, s3, s5 DBA resolved 2017-09-24 2017-09-24
176573 db2047 got rebooted DBA resolved 2017-09-25 2017-09-26
176760 Larger Wikidata entity dump queries DBA resolved 2017-09-27 2017-09-27
176883 Help communicate read-only time for Commons for schema change required by adding 3D filetype DBA resolved 2017-09-28 2017-11-29
176931 Decommission db1035 DBA resolved 2017-09-29 2017-11-14
176985 Global rename of CodeCat → Rua: supervision needed DBA resolved 2017-09-29 2017-09-28
177096 Some queries to new replica hosts are dramatically slower than labsdb; missing indexes? DBA resolved 2017-09-30 2017-10-09
177155 Find test wiki(s) for new description usage and enable there DBA resolved 2017-10-01 2017-10-18
177171 Degraded RAID on db1056 DBA resolved 2017-10-02 2017-10-04
177223 Determine schema differences between labsdb1001 and labsdb1009 DBA resolved 2017-10-03 2018-01-31
177264 Degraded RAID on db1092 DBA resolved 2017-10-04 2017-10-18
177274 Provide a new s8 master for sanitarium DBA resolved 2017-10-04 2018-01-09
177505 Replag of S1, S3, and S5 are over 2.75 hours DBA resolved 2017-10-06 2017-10-05
177570 Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs DBA resolved 2017-10-07 2017-12-15
177601 Deploy dropping wb_entity_per_page table DBA resolved 2017-10-07 2017-10-30
177627 check db1052 power supply redundancy DBA resolved 2017-10-07 2017-10-08
177628 check db1054 power supply redundancy DBA resolved 2017-10-07 2017-10-08
177630 check db1080 power supply redundancy DBA resolved 2017-10-07 2017-10-08
177720 db2038 two disks with predictive failure DBA resolved 2017-10-09 2017-10-11
177772 Purge 90% of rows from recentchanges (and posibly defragment) from commonswiki and ruwiki (the ones with source:wikidata) DBA resolved 2017-10-10 2018-03-16
177908 dbstore2001:s3 crashed while backups were running DBA resolved 2017-10-12 2017-10-11
177911 Decommission db1038 DBA resolved 2017-10-12 2017-11-14
178040 Purge html5-misnesting tags from the database DBA resolved 2017-10-13 2017-10-13
178052 pagetranslation log_type missing on replicas DBA resolved 2017-10-13 2018-01-09
178128 Access to raw database tables on labsdb* for wmcs-admin users DBA resolved 2017-10-13 2017-10-31
178135 ores_classification table corrupt on enwiki labs replica labsdb1001 DBA resolved 2017-10-17 2017-10-16
178140 db2081 unreachable DBA resolved 2017-10-14 2017-10-17
178162 Decommission db1050 DBA resolved 2017-10-14 2018-01-03
178237 Access to x1 broken on stat1006 DBA resolved 2017-10-20 2017-10-20
178272 labsdb1005's mysql crashed DBA resolved 2017-10-17 2017-10-17
178294 User log table creation on tools.labsdb failing intermittantly for IABot interactive UI DBA resolved 2017-10-17 2017-10-17
178359 Support multi-instance on core hosts DBA resolved 2017-10-18 2017-12-12
178383 db1101 crashed - memory errors DBA resolved 2017-10-18 2017-11-19
178460 db1082 storage crashed DBA resolved 2017-10-19 2017-10-24
178515 Enable description usage tracking on further wikis DBA resolved 2017-10-19 2017-10-24
179012 re-enable Wikidata Recent Changes integration on Russian Wikipedia DBA resolved 2017-10-26 2018-04-12
179041 Toolforge oursql connecting to enwiki.analytics.db.svc.eqiad.wmflabs raises error 1615 'Prepared statement needs to be re-prepared' but works fine on enwiki.labsdb DBA resolved 2017-10-26 2017-10-26
179106 Drop the "wb_terms.wb_terms_language" index DBA resolved 2017-10-27 2017-11-21
179244 labsdb1009 crashed - OOM DBA resolved 2017-10-30 2018-03-31
179614 Enable innodb-large-prefix on tools.db.svc.eqiad.wmflabs DBA resolved 2017-11-04 2018-04-27
179727 Degraded RAID on db1059 DBA resolved 2017-11-04 2017-11-08
179767 Database replica drift on web and analytics clusters DBA resolved 2017-11-05 2017-11-06
179793 Consider dropping the "wb_items_per_site.wb_ips_site_page" index DBA resolved 2017-11-06 2018-03-23
180045 Review and deploy schema change on dropping oresc_rev_predicted_model index DBA resolved 2017-11-09 2018-01-30
180141 Raise concurrent mysql connection limit for Quarry (or throttle application concurrency) DBA resolved 2017-11-10 2017-11-10
180164 Drop thread.thread_root_2 indexes DBA resolved 2017-11-10 2017-11-10
180165 Drop ca_repo_author index from mediawikiwiki.code_authors DBA resolved 2017-11-10 2017-11-10
180166 Drop ep_articles_course_id index DBA resolved 2017-11-10 2018-01-10
180196 db2059 storage crash DBA resolved 2017-11-11 2017-11-10
180380 "ERROR 2006 (HY000): MySQL server has gone away" failures for a variety of queries against Wiki Replica servers DBA resolved 2017-11-14 2018-03-16
180560 labsdb1004 replication broken: in memory tables from s51290__dpl_p DBA resolved 2017-11-16 2017-11-16
180700 Rack and setup db1109 and db1110 DBA resolved 2017-11-17 2017-11-21
180703 Global rename of Knud Winckelmann → KnudW: supervision needed DBA resolved 2017-11-17 2018-02-22
180714 s5 primary master db1063 crashed DBA resolved 2017-11-17 2018-02-05
180788 Rack and setup db1111 and db1112 DBA resolved 2017-11-18 2018-01-04
180793 Frequent "Wikimedia\\Rdbms\\DatabaseMysqlBase::lock failed to acquire lock" errors on WMF mediawiki logs DBA resolved 2017-11-18 2017-12-05
180903 Rename user "Arsog1985" to "Sigma'am" on Central Auth : supervision needed DBA resolved 2017-11-20 2017-11-20
180917 db1100 replication broken DBA resolved 2017-11-20 2017-11-20
180927 db2068 storage crash DBA resolved 2017-11-21 2017-11-22
180946 Global rename supervision request: Angr → Mahagaja DBA resolved 2017-11-21 2017-11-21
181170 Global rename of JeanBono → Rexcornot: supervision needed DBA resolved 2017-11-23 2017-11-27
181266 Degraded RAID on db2055 DBA resolved 2017-11-25 2017-11-29
181293 es2018 crashed DBA resolved 2017-11-25 2017-11-24
181345 Degraded RAID on db1051 DBA resolved 2017-11-26 2017-11-28
181378 Decommission db1021 DBA resolved 2017-11-28 2018-01-10
181696 Decommission db1044 DBA resolved 2017-12-01 2018-01-10
181775 db2044: RAID disk with predictive failure DBA resolved 2017-12-02 2017-12-01
181779 Degraded RAID on db2044 DBA resolved 2017-12-02 2017-12-07
182143 Renaming 분당선M: supervision needed DBA resolved 2017-12-06 2017-12-06
182288 Degraded RAID on db1068 DBA resolved 2017-12-08 2017-12-08
182450 db1102 (sanitarium) filling up (WAS: Clean up old binlogs from db1102 (sanitarium multi-instance)) DBA resolved 2017-12-09 2018-01-23
182556 Decommission db1034 DBA resolved 2017-12-12 2018-05-07
182653 Power supply error on db1055 DBA resolved 2017-12-13 2017-12-12
182853 Degraded RAID on db1059 DBA resolved 2017-12-15 2017-12-19
182892 I can't connect to DB replica on Toolforge due to TLS-related failure DBA resolved 2017-12-15 2017-12-14
182896 Rack and setup db1113 and db1114 DBA resolved 2017-12-15 2018-01-04
182916 Database error: Unable to connect to s7.web.db.svc.eqiad.wmflabs DBA resolved 2018-02-04 2018-03-19
183123 Precautionary backup needed for the log database on db1107 before applying regular purging/sanitization DBA resolved 2017-12-19 2017-12-18
183249 Reimage and upgrade to stretch all dbproxies DBA resolved 2017-12-20 2018-03-19
183252 Unbreak replication in beta cluster DBA resolved 2017-12-20 2017-12-21
183285 Global rename of Makki98 → Bluemoon2999: supervision needed DBA resolved 2017-12-20 2018-01-02
183419 Determine how to update old compressed ExternalStore entries for T181555 DBA resolved 2017-12-21 2018-01-12
183467 Global rename of ولاء → لا روسا: supervision needed DBA resolved 2017-12-22 2018-01-02
183469 Setup newer machines and replace all old misc (m*) and x1 eqiad machines DBA resolved 2017-12-22 2018-03-22
183470 Setup newer machines and replace all old misc (m*) and x1 codfw machines DBA resolved 2017-12-22 2018-03-09
183486 MCR schema migration stage 0: create tables DBA resolved 2018-01-09 2018-05-10
183566 Prepare storage layer for lfnwiki DBA resolved 2017-12-23 2018-05-06
183708 Degraded RAID on db1001 DBA resolved 2017-12-28 2018-01-03
183735 Check data consistency across production shards DBA resolved 2017-12-29 2018-03-19
183750 pc2005 crashed: CPU2 internal error DBA resolved 2017-12-29 2018-01-03
183758 Create backups of user tables from decommissioned database servers DBA resolved 2017-12-29 2018-01-31
183887 db2054: Disk with predictive failure DBA resolved 2018-01-03 2018-01-05
184090 Decommission db2016, db2017, db2018, db2019, db2023, db2028, db2029 DBA resolved 2018-01-04 2018-02-12
184160 db1059 BBU issues DBA resolved 2018-01-05 2018-01-11
184161 Productionize 2 new eqiad database servers DBA resolved 2018-01-05 2018-03-12
184247 Drop `external_user` from all databases DBA resolved 2018-01-06 2018-01-11
184262 Decommission db1039 DBA resolved 2018-01-06 2018-05-07
184285 Degraded RAID on db2055 DBA resolved 2018-01-06 2018-01-09
184375 Prepare and check storage layer for inhwiki DBA resolved 2018-01-08 2018-05-06
184397 Decommission db1030 DBA resolved 2018-01-09 2018-04-03
184401 db1011 possibly faulty BBU DBA resolved 2018-01-09 2018-01-11
184446 Configure Toolforge replica views and dumps for the new MCR tables DBA resolved 2018-01-09 2018-05-22
184464 Degraded RAID on db2060 DBA resolved 2018-01-09 2018-01-10
184599 s5 wikidatawiki database cleanup DBA resolved 2018-01-11 2018-02-12
184666 DBA review for GlobalPreferences schema DBA resolved 2018-01-12 2018-03-20
184680 Update duplicate handling in reading lists API DBA resolved 2018-02-01 2018-04-02
184696 Finish the database backups generation script to create consistent logical backups in CODFW DBA resolved 2018-01-12 2018-03-19
184697 Failover existing eqiad database backup system to the new codfw database logical backup system DBA resolved 2018-01-12 2018-03-19
184699 Generate consistent logical database backups in CODFW DBA resolved 2018-01-12 2018-03-19
184703 Decommission db1011 DBA resolved 2018-01-12 2018-04-03
184704 Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw DBA resolved 2018-01-12 2018-02-28
184797 Move mariadb_maintenance away from terbium/wasat (mediawiki_maintenance) DBA resolved 2018-01-13 2018-05-08
184836 Degraded RAID on db2036 DBA resolved 2018-01-13 2018-01-19
184888 Replace codfw x1 master (db2033) (WAS: Failed BBU on db2033 (x1 master)) DBA resolved 2018-01-16 2018-04-03
185128 Schema change to prepare for dropping archive.ar_text and archive.ar_flags DBA resolved 2018-01-18 2018-05-10
185294 db2036 storage issues? (mysql crashed, installer issues) DBA resolved 2018-01-20 2018-01-29
185646 High database error rates on s2 and s3 DBA resolved 2018-01-25 2018-03-23
185719 Global rename of Dick Laurent → Qehath: supervision needed DBA resolved 2018-01-26 2018-02-05
185788 Rack and setup db1115 (tendril replacement database) DBA resolved 2018-01-27 2018-02-21
185795 Global rename of Ks-M9 → Stïnger: supervision needed DBA resolved 2018-01-28 2018-02-15
186049 db1051 database host BBU issues DBA resolved 2018-01-31 2018-02-07
186123 rack/setup/install db2093 (WAS: rack/setup/install tendril2001) DBA resolved 2018-02-01 2018-02-21
186319 Degraded RAID on db1070 DBA resolved 2018-02-03 2018-02-02
186321 Prepare and indicate proper master db failover candidates for all eqiad database sections (s1-s8, x1) DBA resolved 2018-02-03 2018-03-05
186533 Degraded RAID on db2039 DBA resolved 2018-02-06 2018-02-07
186579 labsdb1010 crashed DBA resolved 2018-02-07 2018-03-02
186585 Review m5 backups DBA resolved 2018-02-15 2018-03-19
186596 dbstore1001 crashed: Multibit ECC errors were detected on the RAID controller. DBA resolved 2018-02-07 2018-04-05
186685 Remove deleted wikis from wikireplicas DBA resolved 2018-02-08 2018-02-08
186699 Move db1069 to A1 DBA resolved 2018-02-08 2018-03-05
186730 Change user u11106 to have max 1 open connection DBA resolved 2018-02-08 2018-02-07
186863 labsdb1004 replication broken: s51999__merge_candidates.image_candidates DBA resolved 2018-02-10 2018-02-09
186865 Drop flaggedrevs tables from mediawikiwiki DBA resolved 2018-02-10 2018-05-02
186867 Drop contest* tables from mediawikiwiki DBA resolved 2018-02-10 2018-04-02
186937 Global rename of Etienfr → Limotecariu: supervision needed DBA resolved 2018-02-11 2018-02-12
186973 DBA review of purgeOldLogIPData.php DBA resolved 2018-02-11 2018-02-12
187143 Upcoming phabricator upgrade requires unusually long database migrations DBA resolved 2018-02-14 2018-02-15
187419 Degraded RAID on db2048 DBA resolved 2018-02-16 2018-02-16
187425 test DBA resolved 2018-02-16 2018-02-15
187526 Disk #5 (count starts at #0) of db1111 has corrupted sectors DBA resolved 2018-02-17 2018-02-27
187542 Decommission db1043 DBA resolved 2018-02-17 2018-03-23
187543 Decommission db2012 DBA resolved 2018-02-17 2018-04-18
187629 Global rename of Darkweasel94 → Tokfo: supervision needed DBA resolved 2018-02-18 2018-03-01
187722 Replace db2030 from m5 with another host (WAS: Degraded RAID on db2030) DBA resolved 2018-02-20 2018-02-25
187768 Decommission db2030 DBA resolved 2018-02-21 2018-04-17
187774 Prepare and check storage layer for romdwikimedia DBA resolved 2018-02-21 2018-05-06
187886 Decommission db2011 DBA resolved 2018-02-22 2018-05-03
187983 db2048: RAID with predictive failure DBA resolved 2018-02-23 2018-02-27
187986 CX2: Register the version used to start a translation DBA resolved 2018-03-16 2018-03-20
188187 Degraded RAID on db1068 DBA resolved 2018-02-25 2018-03-06
188279 Investigate optimzing wb_terms DBA resolved 2018-04-04 2018-04-13
188286 Degraded RAID on db2048 DBA resolved 2018-02-27 2018-03-01
188490 Prepare and check storage layer for hiwikimedia DBA resolved 2018-03-01 2018-05-06
188505 Investigate why query killer didn't kill 1-hour long queries DBA resolved 2018-03-01 2018-03-06
188589 db1009 overloaded by idle connections to the nova database DBA resolved 2018-03-02 2018-03-16
188685 Degraded RAID on db1064 DBA resolved 2018-03-03 2018-03-08
189005 Failover m5 master from db1009 to db1073 DBA resolved 2018-03-07 2018-03-09
189112 Prepare and check storage layer for gorwiki DBA resolved 2018-03-08 2018-05-06
189204 High (2-3x) write and connection load on enwiki databases DBA resolved 2018-03-09 2018-03-23
189216 Decommission db1009 DBA resolved 2018-03-09 2018-03-23
189383 Faster backups by archiving and unarchiving databases in parallel DBA resolved 2018-03-11 2018-03-27
189384 Faster backups by starting mydumper execution of different sections in parallel when they are about to finish and are running on a limited number of threads DBA resolved 2018-03-11 2018-03-28
189403 Degraded RAID on db1073 DBA resolved 2018-03-12 2018-03-14
189466 Prepare storage layer for euwikisource DBA resolved 2018-03-13 2018-05-06
189655 Switchover m1 master from db1016 to db1063 DBA resolved 2018-03-15 2018-03-20
189656 Switchover m2 master from db1020 to db1051 DBA resolved 2018-03-15 2018-03-15
189773 Decommission db1020 DBA resolved 2018-03-16 2018-04-04
189799 Grant access to pmiazga to db1112 and db1112 (MCR tests hosts) DBA resolved 2018-03-16 2018-03-15
190035 Generate report of disk health for database masters and master candidates DBA resolved 2018-03-20 2018-03-21
190133 Deploy translation_cx_version schema change to production DBA resolved 2018-03-21 2018-03-27
190153 DROP unused 'slots' table (WAS: In the slots table, replace slot_inherited with slot_origin) DBA resolved 2018-03-21 2018-05-10
190179 Decommission db1016 DBA resolved 2018-03-21 2018-04-03
190262 Decommission db1001 DBA resolved 2018-03-22 2018-04-03
190299 db1061 (s6 master) disk with lots of predictive failure errors DBA resolved 2018-03-22 2018-03-22
190301 db1052 (s1 master) disks with lots of predictive failure errors DBA resolved 2018-03-22 2018-03-23
190302 db1054 (s2 master) disk with lots of predictive failure errors DBA resolved 2018-03-22 2018-03-22
190303 db1062 (s7 master) disk with lots of predictive failure errors DBA resolved 2018-03-22 2018-03-22
190446 Degraded RAID on db1052 DBA resolved 2018-03-23 2018-03-23
190488 Possibly a big update going to: u2815__p`.`all_articles DBA resolved 2018-03-24 2018-03-23
190780 Schema changes to site_stats DBA resolved 2018-03-28 2018-04-12
190960 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" DBA resolved 2018-03-29 2018-04-27
191001 labsdb1004: broken replication DBA resolved 2018-03-30 2018-03-29
191116 Wikimedia\Rdbms\Database::tableName: use of subqueries is not supported this way. DBA resolved 2018-03-31 2018-04-05
191129 msw-c6-codfw offline DBA resolved 2018-03-31 2018-04-03
191149 labsdb1009 crashed DBA resolved 2018-04-01 2018-04-02
191193 Move masters away from codfw C6 DBA resolved 2018-04-03 2018-04-18
191275 Prepare and indicate proper master db failover candidates for all codfw database sections (s1-s8, x1) DBA resolved 2018-04-04 2018-04-11
191720 Degraded RAID on db2069 DBA resolved 2018-04-09 2018-04-10
191792 Rack and setup db1116 - db1123 DBA resolved 2018-04-10 2018-04-25
191875 Deletion not working on English Wikipedia DBA resolved 2018-04-11 2018-04-10
191892 Reduce locking contention on deletion of pages DBA resolved 2018-04-11 2018-04-12
191977 remote ipmi doesn't work for es2013 DBA resolved 2018-04-12 2018-04-12
192056 Drop table linkscc DBA resolved 2018-04-13 2018-04-12
193325 db2081 crashed/rebooted, probably due to hardware failure DBA resolved 2018-04-29 2018-05-04
193331 db1098 crashed and got rebooted DBA resolved 2018-04-29 2018-05-04
193390 Drop flaggedrevs tables at metawiki DBA resolved 2018-05-01 2018-05-03
193676 Drop flaggedrevs tables at eswikibooks DBA resolved 2018-05-03 2018-05-03
193678 Drop flaggedrevs tables at eswiki DBA resolved 2018-05-03 2018-05-03
193747 Degraded RAID on db1063 DBA resolved 2018-05-04 2018-05-04
193835 Move db1067 to row C DBA resolved 2018-05-05 2018-05-16
193847 Move db1066 to row A DBA resolved 2018-05-05 2018-05-18
193894 Run listed queries for all wikis in order to unblock existing changes DBA resolved 2018-05-05 2018-05-14
194103 Degraded RAID on db2067 DBA resolved 2018-05-09 2018-05-18
194155 Degraded RAID on db1073 DBA resolved 2018-05-09 2018-05-08
194197 Degraded RAID on db1073 DBA resolved 2018-05-09 2018-05-09
194516 Bug on mariadb systemd unit on stretch for multi-instance hosts DBA resolved 2018-05-12 2018-05-16
194626 Grant access to anomie to db1112 and db1112 (MCR tests hosts) DBA resolved 2018-05-15 2018-05-14
194661 Drop long_run_profiling unused table DBA resolved 2018-05-15 2018-05-14
194663 Drop unused tables: msg_resource msg_resource_links DBA resolved 2018-05-15 2018-05-16
194698 Degraded RAID on db1065 DBA resolved 2018-05-15 2018-05-15
194766 https://tendril.wikimedia.org/ IPv6 doesn't work DBA resolved 2018-05-16 2018-05-15
194852 Possibly BBU issues on db1067 DBA resolved 2018-05-17 2018-05-22
194885 Degraded RAID on db1064 DBA resolved 2018-05-18 2018-05-17
194955 Degraded RAID on db1066 DBA resolved 2018-05-19 2018-05-18
144169 Flake8 for python files without extension in puppet repo Software Development open 2017-01-02 2018-01-08
148494 Add shell scripts CI validations Software Development open 2017-01-02 2017-09-14
150560 More verbose messages from service-checker-swagger Software Development open 2017-01-02 2017-05-23
152950 E901 SyntaxError: invalid syntax is wrongly raised on using python's abc by jenkins python CI linter Software Development open 2017-01-02 2017-03-27
154776 Puppet compiler: order resources for easy comparison between hosts Software Development open 2017-01-07 2017-01-06
155705 confctl: log to SAL even if the selection doesn't match any host Software Development open 2017-01-20 2017-01-19
157001 Puppet compiler: abort on git rebase conflict Software Development open 2017-02-03 2017-02-02
157002 Puppet compiler: re-add the concurrency option NUM_THREADS Software Development open 2017-02-03 2017-02-02
157133 Consider adding a --skip-conftool option to puppet-merge Software Development open 2017-02-04 2018-02-23
159045 Update Puppet repo code that uses maniphest.update and maniphest.createtask conduit api Software Development open 2017-02-26 2017-02-25
161545 Cumin: PuppetDB backend, allow to specify boolean values for resource parameters Software Development open 2017-03-28 2017-03-27
163363 Switchdc: improvements Software Development open 2017-04-20 2017-05-16
163365 Switchdc RO/RW: add check to test it editing a real wiki Software Development open 2017-04-20 2017-04-19
164587 cumin could use randomization/splay options Software Development open 2017-05-06 2017-05-05
167504 New tool to track package updates/status for hosts and images (debmonitor) Software Development open 2017-06-10 2018-04-04
169304 Cumin masters: simplify usage in case of emergency Software Development open 2017-07-01 2017-06-30
177385 Upgrade Cumin masters to stretch Software Development open 2017-10-05 2017-10-18
179816 Cumin: create external backend for WMCS Puppet API Software Development open 2017-11-07 2017-11-06
182028 DNS repo: add CI checks for obvious configuration errors Software Development open 2017-12-05 2017-12-04
184435 Puppet tox: properly lint both Py2 and Py3 files Software Development open 2018-01-09 2018-01-28
186818 Cumin: add --limit to randomly select N hosts Software Development open 2018-02-09 2018-02-13
187709 Cumin feature idea: Prometheus backend Software Development open 2018-02-20 2018-02-19
187751 wmf-auto-reimage: migrate script to Python3 Software Development open 2018-02-20 2018-02-21
187773 Cumin: upgrade it to 3.0.1 in production Software Development open 2018-02-21 2018-02-21
188112 cumin 3.0.1-1 is broken on labs master Software Development open 2018-02-24 2018-04-07
191298 Release and deploy Debmonitor (patch management software) [Technology Goal 2017-18_Q4] Software Development open 2018-04-04 2018-04-24
191299 Debmonitor: deploy the service in production Software Development open 2018-04-04 2018-05-03
191300 Debmonitor: deploy the agent across the fleet Software Development open 2018-04-04 2018-04-03
142085 Automate creation of Phab task for failed disks Software Development resolved 2017-01-02 2016-10-19
142097 Package Python phabricator module for both Ubuntu Precise and Debian Jessie Software Development resolved 2017-01-02 2016-11-14
143306 Evaluation of automation/orchestration tools Software Development resolved 2017-01-02 2016-10-07
143536 Upgrade all mw* servers to debian jessie Software Development resolved 2017-01-02 2017-08-29
145192 icinga-downtime script waiting forever if host already in downtime Software Development resolved 2017-01-02 2018-02-09
148273 Setup SSH AuthN/AuthZ for the Automation Framework Software Development resolved 2017-01-02 2017-03-30
148814 wmf-auto-reimage improvements Software Development resolved 2017-01-02 2017-09-21
148815 wmf-auto-reimage: improve messaging Software Development resolved 2017-01-02 2016-10-24
148816 wmf-auto-reimage: add --new CLI option for new hosts Software Development resolved 2017-01-02 2016-11-07
148817 wmf-auto-reimage: remove dependency on wmf-reimage Software Development resolved 2017-01-02 2017-09-21
149213 wmf-auto-reimage: handle multiple conftool services per host Software Development resolved 2017-01-02 2017-09-08
149229 raid_handler: Icinga ACK comment persist after recovery Software Development resolved 2017-01-02 2016-10-27
149230 wmf-auto-reimage: allow to specify the conftool state Software Development resolved 2017-01-02 2017-09-21
149913 Icinga raid_handler: add option to handle frack instances Software Development resolved 2017-01-02 2017-02-24
150448 wmf-reimage: allow to power cycle also if the server is off Software Development resolved 2017-01-02 2017-02-23
151043 Add support for physical disks to get-raid-status-megacli.py Software Development resolved 2017-01-02 2016-12-12
152549 tox-jessie is failing on operations/software Software Development resolved 2017-01-02 2017-01-24
152774 raid_handler: random failure to gather RAID status Software Development resolved 2017-01-02 2016-12-09
152977 conftool service removal bugs Software Development resolved 2017-01-02 2017-03-08
154588 Automation framework first version Software Development resolved 2017-01-05 2017-02-24
157052 Puppet compiler: sync newest facts only Software Development resolved 2017-02-03 2017-04-28
158659 Keyholder: add support for ED25519 keys Software Development resolved 2017-02-22 2017-02-28
158660 Keyholder accept passwordless keys Software Development resolved 2017-02-22 2017-02-28
158746 Cumin: use safer default for hosts regex matching Software Development resolved 2017-02-23 2017-02-24
158747 Cumin: better error message if no config file is available Software Development resolved 2017-02-23 2017-07-03
158748 Cumin: add support for 'not' operator for hosts Software Development resolved 2017-02-23 2017-02-24
158753 Cumin: authorize also IPv6 on the targets Software Development resolved 2017-02-23 2017-02-24
158758 Cumin: ignore urllib3 SubjectAltNameWarning in PuppetDB calls Software Development resolved 2017-02-23 2017-02-24
158773 Cumin: skip target installation on labs realm Software Development resolved 2017-02-23 2017-02-22
158964 Cumin: fill wikitech page with documentation Software Development resolved 2017-02-25 2017-03-30
158967 Cumin: fix first batch of potential issues reported by codacy Software Development resolved 2017-02-25 2017-02-25
159127 Cumin: fine tuning configuration Software Development resolved 2017-02-28 2017-09-08
159308 Cumin: add man page Software Development resolved 2017-03-02 2017-10-17
159968 Cumin: add support for batch execution Software Development resolved 2017-03-09 2017-03-13
159969 Cumin: add integration tests Software Development resolved 2017-03-09 2017-03-15
159970 Cumin: auto ucfirst puppetdb resources Software Development resolved 2017-03-09 2017-03-29
160621 Cumin: upgrade to v0.0.2 in prod Software Development resolved 2017-03-17 2017-03-16
161730 Cumin: do not auto-ucfirst when the query is a regex in PuppetDB backend Software Development resolved 2017-03-30 2017-05-16
161887 Cumin: CLI, allow to just query without specifying a command Software Development resolved 2017-04-01 2017-03-31
162151 Cumin: PuppetDB, fail better if regex are used on resource parameters Software Development resolved 2017-04-05 2017-05-16
163364 Switchdc dnsdisc: add check for IP records Software Development resolved 2017-04-20 2017-04-26
163367 Switchdc: make SAL messages more human-friendly Software Development resolved 2017-04-20 2017-04-26
163369 Switchdc warmup: be less verbose Software Development resolved 2017-04-20 2017-04-29
163371 Switchdc menu: add a Next option in the menu Software Development resolved 2017-04-20 2017-04-26
163372 Switchdc: add task to re-enable and run puppet Software Development resolved 2017-04-20 2017-04-26
163373 Switchdc: varnish switch, remove the manual confirmation Software Development resolved 2017-04-20 2017-04-29
163376 Switchdc dnsdisc: remove stale conftool files Software Development resolved 2017-04-20 2017-04-29
163398 Switchdc: mediawiki, use etcd-driven config Software Development resolved 2017-04-20 2017-05-03
164177 switchdc: Improve wgReadOnly message Software Development resolved 2017-05-01 2017-05-04
164396 Switchdc dnsdisc: add retries to check_record Software Development resolved 2017-05-04 2017-05-08
164400 Switchdc t05 traffic: be explicit on automatic validation Software Development resolved 2017-05-04 2017-05-08
164403 Switchdc t09 start maintenance: clear systemctl failed state Software Development resolved 2017-05-04 2017-05-08
164780 Sunset our use of Salt Software Development resolved 2017-05-09 2017-09-28
164817 Migrate debdeploy to cumin Software Development resolved 2017-05-10 2017-09-11
164824 Cumin: only last ssh_option is set Software Development resolved 2017-05-10 2017-05-16
164827 Cumin: get live output when matching a single host Software Development resolved 2017-05-10 2017-05-16
164833 Cumin: allow to specify successful exit codes Software Development resolved 2017-05-10 2017-05-29
164838 Cumin: allow to specify a timeout per command Software Development resolved 2017-05-10 2017-07-03
165583 Puppet compiler: sync facts from all workers Software Development resolved 2017-05-18 2017-05-23
165838 Cumin: add a simple interactive mode Software Development resolved 2017-05-21 2017-05-30
165842 Cumin: add a simple txt/json output Software Development resolved 2017-05-21 2017-05-30
166300 Remove Salt from wmf-auto-reimage / wmf-reimage Software Development resolved 2017-05-26 2017-09-25
166371 Monitoring: create an alert for daemonized puppet Software Development resolved 2017-05-27 2017-06-13
166397 Cumin fails on huge nodelists emitted by its own outputs Software Development resolved 2017-05-27 2018-01-18
166519 Raid handler: manage new alarms Software Development resolved 2017-05-30 2017-05-31
166570 Do something to better handle wmf-reimage runs cleanups/failures Software Development resolved 2017-05-31 2017-09-21
167392 Cumin: fix --success-percentage 0 Software Development resolved 2017-06-09 2017-06-12
167394 Cumin: fix ok_codes when set to empty list Software Development resolved 2017-06-09 2017-06-12
169555 puppet.service systemctl failures after reimage Software Development resolved 2017-07-04 2017-09-21
169619 Degraded RAID on ms-be2024 Software Development resolved 2017-07-05 2017-07-05
169640 Support aliases in cumin Software Development resolved 2017-07-05 2017-07-17
170394 Cumin: add multi-query support Software Development resolved 2017-07-13 2017-08-02
171679 Cumin: allow to ignore exit codes of executed commands Software Development resolved 2017-07-26 2017-08-02
171684 Cumin: improve target management Software Development resolved 2017-07-26 2017-09-08
174008 Cumin: setup.py installs data_files in wrong directory Software Development resolved 2017-08-25 2017-09-21
174854 Icinga raid handler improvements Software Development resolved 2017-09-03 2017-09-08
174911 Cumin: clustershell worker should return if no target are specified Software Development resolved 2017-09-05 2017-09-08
175711 Cumin: create backend for OpenStack Software Development resolved 2017-09-13 2017-09-21
175712 Install cumin in the WMCS infrastructure Software Development resolved 2017-09-13 2017-09-20
176314 Replace salt on integration and deployment-prep projects Software Development resolved 2017-09-21 2018-01-08
176609 Cumin: fine tune WMCS setup Software Development resolved 2017-09-26 2017-09-30
176632 Remove salt master (and related packages) from labcontrol1001 Software Development resolved 2017-09-26 2017-09-28
176955 wmf-auto-reimage: add support for renaming while re-imaging Software Development resolved 2017-09-29 2017-10-04
178279 Cumin: add support for role/profile shortcuts in PuppetDB backend Software Development resolved 2017-10-17 2017-11-06
178342 Cumin: add support for external backends Software Development resolved 2017-10-17 2017-11-06
179002 Cumin: improve logging Software Development resolved 2017-10-26 2017-12-11
179593 Cumin: upload generated documentation to doc.w.o Software Development resolved 2017-11-03 2017-12-21
181796 wmf-auto-reimage: fix screen/tmux detection Software Development resolved 2017-12-02 2017-12-07
181798 wmf-auto-reimage: -c option cannot be specified alone Software Development resolved 2017-12-02 2017-12-07
182096 wmf-auto-reimage: pick correctly CA master Software Development resolved 2017-12-06 2017-12-05
182575 Cumin: PuppetDB backend, add support for API v4 Software Development resolved 2017-12-12 2018-01-22
185967 Cumin: add custom backend to WMCS Software Development resolved 2018-01-30 2018-03-30
187185 Cumin: CLI, allow to specify percentage too in --batch-size Software Development resolved 2018-02-14 2018-02-14
188627 Cumin: "No such file or directory" when log_file has no directory Software Development resolved 2018-03-02 2018-03-02
139775 eqiad: add all spare network switches to hardware spares tracking Hardware Requests open 2017-01-02 2017-04-27
178392 Replacement hardware for cumin masters Hardware Requests open 2017-10-18 2018-05-03
181264 Refresh or replace oxygen Hardware Requests open 2017-11-24 2018-04-05
181936 Give misc dump crons their own host Hardware Requests open 2017-12-05 2018-05-10
76986 RESTBase production hardware Hardware Requests resolved 2017-01-02 2016-02-09
83340 decom arsenic: (was: reclaim arsenic as spare) Hardware Requests resolved 2017-05-04 2017-05-10
84958 eqiad: (3) servers for logstash service Hardware Requests resolved 2017-01-02 2016-07-29
86561 Wikidata Query Service hardware Hardware Requests resolved 2017-01-02 2017-07-14
86896 Procure rdb2001-2004 - onsite pending racking Hardware Requests resolved 2017-01-02 2016-01-20
87215 Re-image osmium and repurpose into VE testing host Hardware Requests resolved 2017-01-02 2015-01-24
88294 eqiad: Dedicated hardware for wikitech web server - silver allocated Hardware Requests resolved 2017-01-02 2015-02-18
89266 virt1000 memory upgrade Hardware Requests resolved 2017-01-02 2015-02-24
89400 Add another redis jobqueue server master and slave Hardware Requests resolved 2017-01-02 2015-11-24
89402 purchase 3 additional logstash nodes Hardware Requests resolved 2017-01-02 2016-07-29
89752 eqiad: (6) labs virt nodes Hardware Requests resolved 2017-01-02 2015-04-13
89869 Assign hardware for the zotero service Hardware Requests resolved 2017-01-02 2015-03-11
90626 Replace virt1000 with a newer warrantied server Hardware Requests resolved 2017-01-02 2015-04-14
90747 codfw/eqiad: (1) eventlogging node (per site) Hardware Requests resolved 2017-01-02 2015-03-19
90904 deploy eventlog1001 Hardware Requests resolved 2017-01-02 2015-04-07
90907 deploy eventlog2001 Hardware Requests resolved 2017-01-02 2015-10-30
91277 Hardware for Designate Hardware Requests resolved 2017-01-02 2015-03-12
91678 setup deployment server in codfw (tin equivalent) Hardware Requests resolved 2017-01-02 2016-01-20
91853 Hardware for HTML / zim dumps Hardware Requests resolved 2017-01-02 2017-08-09
93076 eqiad: 2 hardware access request for CI isolation on labsnet Hardware Requests resolved 2017-01-02 2016-01-11
93118 order new array for dataset1001 Hardware Requests resolved 2017-01-02 2015-05-20
93266 codfw: (2) poolcounter systems Hardware Requests resolved 2017-01-02 2015-03-20
93790 Expand RESTBase cluster capacity Hardware Requests resolved 2017-01-02 2016-12-08
94846 Buy fiber patches Hardware Requests resolved 2017-01-02 2015-11-03
95153 Decom/repurpose rbf* hosts Hardware Requests resolved 2017-01-02 2015-09-16
95243 Repurpose rhenium as "network insight" host Hardware Requests resolved 2017-01-02 2015-04-07
95760 eqiad: (1) allocate server to migrate Zuul server to Hardware Requests resolved 2017-01-02 2016-04-21
97049 CODFW Search Servers Hardware Requests resolved 2017-01-02 2015-08-18
98166 Eqiad: 1 hardware access request for puppetmaster service scale out Hardware Requests resolved 2017-01-02 2015-05-05
98740 labnet1002 Hardware Requests resolved 2017-01-02 2015-05-19
100442 Hadoop worker node procurement - 2015 Hardware Requests resolved 2017-01-02 2016-01-27
101256 Replace rubidium with radon for authdns (allocate radon, deallocate rubidium) Hardware Requests resolved 2017-01-02 2015-06-05
102005 Kill off virt1000 Hardware Requests resolved 2018-03-29 2018-03-28
105707 Request Elasticsearch hardware for secondary CirrusSearch in codfw Hardware Requests resolved 2017-01-02 2016-03-10
106112 server for wikimania video transcoding Hardware Requests resolved 2017-01-02 2015-07-22
106147 New server: labservices1001 Hardware Requests resolved 2017-01-02 2015-09-30
107287 Assign WMF5842, WMF5843 for service cluster expansion as scb1001, scb1002 Hardware Requests resolved 2017-01-02 2015-08-10
111053 Request three servers for Pageview API Hardware Requests resolved 2017-01-02 2015-09-21
111382 codfw 3x spares for cassandra encryption testing Hardware Requests resolved 2017-01-02 2015-09-15
111777 Refresh Parser cache servers pc1001-pc1003 Hardware Requests resolved 2017-01-02 2015-12-18
112163 eqiad: (1) hardware request for ElasticSearch replication to Labs - 4 weeks use Hardware Requests resolved 2017-01-02 2015-10-16
114191 Setup a 2 server Kafka instance in both eqiad and codfw for reliable purge streams Hardware Requests resolved 2017-01-02 2017-05-25
114435 Labs test cluster in codfw Hardware Requests resolved 2017-01-02 2015-11-02
115288 Allocate hardware for salt master in eqiad Hardware Requests resolved 2017-01-02 2015-11-09
116790 Decommission calcium Hardware Requests resolved 2017-01-02 2016-04-12
118213 Decommission rubidium Hardware Requests resolved 2017-01-02 2016-09-01
118588 Get Ops bare metal test server Hardware Requests resolved 2017-01-02 2015-12-03
118983 Hardware access request for yubico auth servers Hardware Requests resolved 2017-01-02 2015-12-16
119579 Additional diskspace of wdqs1001/wdqs1002 Hardware Requests resolved 2017-01-02 2016-04-26
119598 eqiad: (2) servers request for ORES Hardware Requests resolved 2017-01-02 2016-08-05
119698 spare swift disks order Hardware Requests resolved 2017-01-02 2015-12-22
119935 Upgrade restbase100[7-9] to match restbase100[1-6] hardware Hardware Requests resolved 2017-01-02 2016-02-26
120679 eqiad out of warranty spares to decommission - approval request Hardware Requests resolved 2017-01-02 2017-01-30
120752 decommission dickson Hardware Requests resolved 2017-01-02 2017-06-21
120842 EQIAD/CODFW: 2 hardware access request for monitoring Hardware Requests resolved 2017-01-02 2015-12-15
121007 Decommission and remove from racks out of warranty spares Hardware Requests resolved 2017-01-02 2015-12-17
121575 Expand SSD space in Cassandra cluster Hardware Requests resolved 2017-01-02 2016-04-28
121882 3 conf200x servers in codfw for zookeeper (and etcd?) Hardware Requests resolved 2017-01-02 2016-04-19
122050 Decomission/reclaim nembus/neptunium Hardware Requests resolved 2017-01-02 2015-12-22
122100 wipe disks and add nembus back to server spares Hardware Requests resolved 2017-01-02 2017-04-13
122101 Decommission neptunium Hardware Requests resolved 2017-01-02 2016-04-12
123132 Need spare server to upgrade/migrate gerrit Hardware Requests resolved 2017-01-02 2016-10-06
123531 check for memory upgrade for lead Hardware Requests resolved 2017-01-02 2016-01-14
123559 allocate hardware for salt master in codfw Hardware Requests resolved 2017-01-02 2016-02-08
124408 mw2173 has probably a broken disk, needs substitution and reimaging Hardware Requests resolved 2017-01-02 2016-04-19
124945 eqiad: New Hive / Oozie server node in eqiad Analytics VLAN Hardware Requests resolved 2017-01-02 2016-03-24
124947 eqiad: (3) AQS replacement nodes Hardware Requests resolved 2017-01-02 2016-05-03
124962 Decommission pc1001-1003 Hardware Requests resolved 2017-01-02 2016-08-11
125223 Codfw: ms-be2003 2TB order request Hardware Requests resolved 2017-01-02 2016-01-29
125827 Decom db2001-db2009 Hardware Requests resolved 2017-01-02 2017-03-29
125842 normalize eqiad restbase cluster - replace restbase1001-1006 Hardware Requests resolved 2017-01-02 2016-04-19
126089 new labstore hardware for eqiad Hardware Requests resolved 2017-01-02 2016-04-19
126253 additional graphite machines request, 1x per DC Hardware Requests resolved 2017-01-02 2016-04-21
126453 codfw: (2) servers for redis jobrunners Hardware Requests resolved 2017-01-02 2016-03-12
126626 3x additional SSD for restbase hp hardware Hardware Requests resolved 2017-01-02 2016-04-19
126987 MediaWiki maintenance host for codfw (terbium's equivalent) Hardware Requests resolved 2017-01-02 2016-03-25
126988 Log host for codfw (fluorine's equivalent) Hardware Requests resolved 2017-01-02 2016-12-12
128000 Refresh elastic10{01..16}.eqiad.wmnet servers Hardware Requests resolved 2017-01-02 2016-05-11
128475 codfw: (2+2) sca & scb service clusters Hardware Requests resolved 2017-01-02 2016-03-09
128807 eqiad: (3) nodes for Druid / analytics Hardware Requests resolved 2017-01-02 2016-05-03
128808 +1 'stat' type box for hadoop client usage Hardware Requests resolved 2017-01-02 2016-04-05
130752 reclaim restbase1001-1006 to spares Hardware Requests resolved 2017-01-02 2016-05-16
130760 Allocate 2 analytics machines to experiment with a jupyterhub notebook service Hardware Requests resolved 2017-01-02 2016-05-09
130845 update label on analytics1003/WMF4541 Hardware Requests resolved 2017-01-02 2016-03-31
130884 reclaim to spares: cp1056, cp1057, cp1069, cp1070 Hardware Requests resolved 2017-01-02 2016-04-12
131180 Maps back end hardware Hardware Requests resolved 2017-01-02 2016-04-26
131184 eqiad: (2) Relevance forge servers Hardware Requests resolved 2017-01-02 2016-08-08
132476 connect an external harddisk with >2TB space to stat1001 Hardware Requests resolved 2017-01-02 2016-04-20
132679 Evaluate replacing SATA disks on ganeti100X.eqiad.wmnet with SSDs Hardware Requests resolved 2017-01-02 2016-05-10
133614 reclaim or decom: cp1043 + cp1044 Hardware Requests resolved 2017-01-02 2016-09-14
135468 codfw old mw app server decomission Hardware Requests resolved 2017-01-02 2017-01-19
136340 Staging / Test environment(s) for RESTBase Hardware Requests resolved 2017-01-02 2016-12-21
136970 eqiad: spare allocation to replace labmon1001 Hardware Requests resolved 2017-01-02 2016-06-07
137117 Replace/refresh carbon Hardware Requests resolved 2017-01-02 2016-08-08
137724 eqiad: 1 hardware access request for labs graphite Hardware Requests resolved 2017-01-02 2016-06-22
138075 procure syslog hardware in codfw Hardware Requests resolved 2017-01-02 2017-02-14
138637 codfw: (2) wqds200[12] systems Hardware Requests resolved 2017-01-02 2016-09-20
139758 Decommission elastic1001-1016 Hardware Requests resolved 2017-01-02 2016-09-14
139776 codfw: add all spare network switches to hardware spares tracking Hardware Requests resolved 2017-01-02 2016-07-11
139961 9x or 15x additional Cassandra/RESTBase nodes Hardware Requests resolved 2017-01-02 2017-01-20
140372 decommission WMF3155-WMF3175 (old lsearchd) Hardware Requests resolved 2017-01-02 2016-07-21
140786 eqiad: audit cisco servers for return/decom Hardware Requests resolved 2017-01-02 2016-07-27
140787 codfw: audit cisco servers for return/decom Hardware Requests resolved 2017-01-02 2016-08-09
141415 decom ytterbium (datacenter) Hardware Requests resolved 2017-01-02 2017-05-09
141624 eqiad: (4) worker servers for kubernetes Hardware Requests resolved 2017-01-02 2017-03-29
141762 decommission snapshot1002, 1003, 1004 Hardware Requests resolved 2017-01-02 2016-10-13
142190 codfw: (2) servers request for ORES redis databases Hardware Requests resolved 2017-01-02 2017-03-15
142218 EQIAD: (2) hardware access request for PUPPET Hardware Requests resolved 2017-01-02 2016-10-11
142219 CODFW: (2) hardware access request for PUPPET Hardware Requests resolved 2017-01-02 2016-09-06
142412 reclaim WMF4724 to spares Hardware Requests resolved 2017-01-02 2016-09-15
142578 codfw/eqiad:(9+9) hardware access request for ORES Hardware Requests resolved 2017-01-02 2017-06-22
142581 Reclaim nobelium Hardware Requests resolved 2017-01-02 2018-04-03
144728 reallocate snapshot1001 for use as canary/testbed for dumps Hardware Requests resolved 2017-01-02 2016-10-26
145082 eqiad/codfw: 1+1 Kafka broker in main clusters in eqiad and codfw Hardware Requests resolved 2017-01-02 2017-05-25
146171 Return wmf4747/wmf4748/wmf4749/wmf4750 to spares Hardware Requests resolved 2017-01-02 2017-01-23
146455 Decommission labsdb1002 Hardware Requests resolved 2017-01-02 2017-03-03
147053 eqiad: (2) hardware access request for dedicated Labs puppetmasters Hardware Requests resolved 2017-01-02 2017-06-14
147309 Decommission db1019 Hardware Requests resolved 2017-01-02 2017-04-12
147313 Decommission analytics1026 and analytics1015 Hardware Requests resolved 2017-01-02 2017-03-03
147596 Allocate spare misc box in eqiad for gerrit replacement Hardware Requests resolved 2017-01-02 2016-10-12
147926 Reclaim aqs100[123] Hardware Requests resolved 2017-01-02 2017-01-05
148187 Requesting 1 spare misc box for Gerrit in codfw Hardware Requests resolved 2017-01-02 2017-02-09
148510 eqiad/codfw: swift frontend hardware refresh Hardware Requests resolved 2017-01-02 2017-01-20
148513 codfw/eqiad: 2x systems for prometheus Hardware Requests resolved 2017-01-02 2017-01-20
148559 Estimate hardware requirements for ordering new servers for Elasticsearch Hardware Requests resolved 2017-01-02 2016-11-17
148747 Estimate hardware requirements for WDQS upgrade Hardware Requests resolved 2017-01-02 2016-12-16
149089 elasticsearch new servers (5x eqiad / 12x codfw) Hardware Requests resolved 2017-01-02 2017-01-19
149102 decomission db2015 Hardware Requests resolved 2017-01-02 2017-02-06
149336 codfw/eqiad: 12x swift backend refresh Hardware Requests resolved 2017-01-02 2017-03-27
149920 Analytics AQS cluster expansion Hardware Requests resolved 2017-01-02 2017-02-06
150865 codfw: 1 hardware access request for continuous integration Hardware Requests resolved 2017-01-02 2016-12-15
150936 Phase out scandium.eqiad.wmnet Hardware Requests resolved 2017-02-11 2017-03-03
153008 eqiad: (1) Mediawiki log host to replace fluorine Hardware Requests resolved 2017-01-02 2016-12-15
154164 decommission stat1001 Hardware Requests resolved 2017-01-02 2017-03-03
154664 codfw: (2) hardware access request for labtest Hardware Requests resolved 2017-01-06 2017-06-26
154706 Codfw: (1) hardware access request for labtestneutron refresh Hardware Requests resolved 2017-01-06 2017-06-14
156040 hardware request for netmon1001 replacement Hardware Requests resolved 2017-01-24 2017-03-30
156506 Replace bast3001 Hardware Requests resolved 2017-01-28 2017-03-02
156631 Site: 2 hardware access request for SCB@CODFW Hardware Requests resolved 2017-01-31 2017-05-25
156970 replacement hardware for iridium (phabricator) Hardware Requests resolved 2017-02-02 2017-04-26
157675 Reclaim/Decommission old codfw mc2001->mc2016 hosts Hardware Requests resolved 2017-02-10 2017-03-06
157839 spare ex4200s - check on quantity for potential shipment to OIT Hardware Requests resolved 2017-02-11 2017-02-17
157840 decom install2001 Hardware Requests resolved 2017-02-11 2017-02-17
157991 decommission ms2001 & ms2002 Hardware Requests resolved 2017-02-14 2017-03-09
158020 decom carbon Hardware Requests resolved 2017-02-15 2017-03-03
158204 Eqiad: (2) hardware access request for labnet1003/1004 Hardware Requests resolved 2017-02-16 2017-05-19
158207 Eqiad: (2) hardware access request for labcontrol1003/1004 Hardware Requests resolved 2017-02-16 2017-05-19
159207 Create one oresrdb VM in codfw Hardware Requests resolved 2017-03-01 2017-03-02
159413 Decomission ms-fe2001-4 Hardware Requests resolved 2017-03-03 2017-05-10
159838 EQIAD: stat1002 replacement Hardware Requests resolved 2017-03-08 2017-05-15
159839 EQIAD: stat1003 replacement Hardware Requests resolved 2017-03-08 2017-05-15
159996 decom fluorine Hardware Requests resolved 2017-03-09 2018-04-17
160986 Decommission ms-fe100[1-4] Hardware Requests resolved 2017-03-22 2017-12-06
161265 Rename labtestmetal2001 Hardware Requests resolved 2017-03-24 2017-03-27
161311 Eqiad: Hardware request for labstore1006/7, dataset1002/3 Hardware Requests resolved 2017-03-25 2018-02-09
161606 Additional ram quote for Prometheus baremetal Hardware Requests resolved 2017-03-29 2017-04-20
161700 CODFW: (4) hardware access request for kubernetes Hardware Requests resolved 2017-03-30 2017-05-30
161750 eqiad: (1) hardware access request for dedicated labmon1002 Hardware Requests resolved 2017-03-30 2017-05-19
161752 eqiad: (2) hardware access request for californium and silver (labweb1001/1002) Hardware Requests resolved 2017-03-30 2017-06-14
161753 eqiad: (1) hardware access request for labnodepool1002 Hardware Requests resolved 2017-03-30 2017-06-27
161764 Codfw: (1) hardware access request for labtestnet2003 [region 2] Hardware Requests resolved 2017-03-30 2017-06-14
161765 Codfw: (1) hardware access request for labtestvirt2003 [region 2] Hardware Requests resolved 2017-03-30 2017-06-27
161766 Codfw: (2) hardware access request for labtest [region 2] Hardware Requests resolved 2017-03-30 2017-06-26
162257 EQIAD: 2 hardware access request for kubernetes-staging Hardware Requests resolved 2017-04-06 2017-05-12
162785 Decommission ms-be2001 - ms-be2012 Hardware Requests resolved 2017-04-13 2017-07-06
162897 spare pool allocation of WMF6406 to replace mira Hardware Requests resolved 2017-04-14 2017-04-13
162928 decommision nembus Hardware Requests resolved 2017-04-14 2017-05-10
164036 decommission gadolinium Hardware Requests resolved 2017-04-28 2017-05-09
164513 reclaim tempdb2001(WMF6407) to spares Hardware Requests resolved 2017-05-05 2017-05-11
164515 codfw: (1) labtest puppetmaster Hardware Requests resolved 2017-05-05 2017-06-14
164588 decom mira Hardware Requests resolved 2017-05-06 2017-07-06
164959 Decomission mw2098 Hardware Requests resolved 2017-05-11 2017-06-26
166489 Decommission ms-be1001 - ms-be1012 Hardware Requests resolved 2017-05-30 2017-10-25
168271 Decommission mw1170-mw1179 Hardware Requests resolved 2017-06-20 2017-09-21
168472 reclaim/decom tmh200[12] Hardware Requests resolved 2017-06-21 2017-07-27
169506 Decommission subra/suhail Hardware Requests resolved 2017-07-04 2017-07-26
170441 Decommission mw1196 Hardware Requests resolved 2017-07-13 2018-03-28
171018 decom netmon1001 Hardware Requests resolved 2017-07-20 2017-10-12
171179 Decommisson restbase-dev100[1-3] Hardware Requests resolved 2017-07-21 2018-02-16
172323 Decommission WMF3248 (old R510) Hardware Requests resolved 2017-08-03 2017-11-08
172487 decom iridium Hardware Requests resolved 2017-08-05 2018-03-29
174452 eqiad: replacement tin/deployment server Hardware Requests resolved 2017-08-30 2018-05-20
175089 decommission mw1163 Hardware Requests resolved 2017-09-06 2018-02-06
175093 Decommission osmium.eqiad.wmnet Hardware Requests resolved 2017-09-06 2018-04-10
175150 Decommission stat1003.eqiad.wmnet Hardware Requests resolved 2017-09-07 2018-04-24
175595 decommission wdqs100[12] Hardware Requests resolved 2017-09-12 2018-04-18
175830 decommission logstash100[1-3] Hardware Requests resolved 2017-09-14 2018-03-23
176472 New package builder host Hardware Requests resolved 2017-09-23 2017-09-28
181784 Decommission db104[67] Hardware Requests resolved 2017-12-02 2018-01-10
182805 decom spare server caesium Hardware Requests resolved 2017-12-14 2018-04-10
182955 Decommission kafka1018 Hardware Requests resolved 2017-12-16 2018-04-18
182991 New WDQS clusters eqiad + codfw Hardware Requests resolved 2017-12-16 2018-04-23
183209 decom uranium Hardware Requests resolved 2017-12-20 2018-05-14
184480 hardware request for bast1001 replacement Hardware Requests resolved 2018-01-09 2018-02-26
184481 hardware request for tin replacement Hardware Requests resolved 2018-01-09 2018-03-27
184551 EQIAD: (1) hardware request for eventlog1001 replacement - eventlog1002. Hardware Requests resolved 2018-01-10 2018-01-24
185226 Decommission host erbium Hardware Requests resolved 2018-01-19 2018-03-06
186755 Decommission restbase-test environment Hardware Requests resolved 2018-02-28 2018-03-28
187446 Decommission xenon, cerium, praseodymium Hardware Requests resolved 2018-02-16 2018-03-19
187447 Decommission restbase-test200[123] Hardware Requests resolved 2018-02-16 2018-04-12
187466 Decommission mw1259-mw1260 Hardware Requests resolved 2018-02-16 2018-03-23
188075 eqiad/codfw: (4)+(4) hardware access request for videoscalers Hardware Requests resolved 2018-02-24 2018-05-03
189111 decommission mw2097-mw2134 Hardware Requests resolved 2018-03-08 2018-03-27
189727 decommission elastic1021 Hardware Requests resolved 2018-03-15 2018-04-18
191359 decom spare server iodine Hardware Requests resolved 2018-04-04 2018-04-10
191363 decom spare server nobelium/wmf4543 Hardware Requests resolved 2018-04-04 2018-04-10
192185 request to assign spare systems as terbium equivalent Hardware Requests resolved 2018-04-14 2018-05-03
17000 Special:Import error: "Import failed: Could not open import file" Other Operations open 2017-01-02 2018-01-18
32716 Run our own Tor client for Tor block Other Operations open 2017-01-02 2017-06-19
40860 security@mediawiki.org : Create a public key and publish it on the public key servers Other Operations open 2017-01-02 2017-12-11
41785 Create a labs SMTP smarthost Other Operations open 2017-08-31 2018-01-06
45952 Incorrect "non-identical file already exists" error when undeleting file on Commons Other Operations open 2018-02-26 2018-02-25
46016 SVG fails to render properly due to several issues Other Operations open 2017-01-02 2016-07-07
46791 [[wikitech:Server_admin_log]] should not rely on freenode irc for logmsgbot entries Other Operations open 2017-01-02 2017-02-14
55457 setup a DB backed parser cache Other Operations open 2017-01-02 2017-05-20
56515 Apply editing rate limits for all users Other Operations open 2017-01-02 2018-04-12
56713 Non-NDA users cannot access graphite.wikimedia.org Other Operations open 2017-01-02 2016-06-10
64987 librsvg misinterpret quoted font family names that contain whitespaces Other Operations open 2017-01-02 2017-10-25
67270 Default license for operations/puppet Other Operations open 2017-01-02 2018-02-19
67394 [EPIC] Performance testing environment Other Operations open 2017-01-02 2016-03-11
76203 Make ircecho run as its own user Other Operations open 2017-01-02 2016-07-16
76306 Set warning thresholds for average cluster utilization Other Operations open 2017-01-02 2016-12-16
78135 Provide a pxe-bootable rescue image Other Operations open 2018-02-03 2018-03-29
82350 update exim::listserve::private::mailing_lists value in puppet Other Operations open 2017-09-19 2017-09-18
82937 re-create script for manual paging Other Operations open 2017-12-29 2018-05-14
83729 Fix monitoring of poolcounter service Other Operations open 2017-07-21 2018-01-09
84163 Fix CirrusSearch monitoring Other Operations open 2017-01-02 2017-01-22
84279 Admin module should allow group management of system users Other Operations open 2017-01-02 2015-01-30
84700 Setup management switch in OE12 Other Operations open 2017-01-02 2016-03-17
84845 improve cron spam visibility Other Operations open 2017-10-19 2017-10-18
85451 scale graphite deployment (tracking) Other Operations open 2017-01-02 2017-12-18
86546 graphite-web logs are not rotated Other Operations open 2017-01-02 2016-11-18
86552 Monitor and alarm on SMART attributes Other Operations open 2017-01-02 2018-02-12
87220 Minimize differences between beta and production (Tracking) Other Operations open 2017-01-02 2017-09-01
87790 decom amslvs1-4 (dc work) Other Operations open 2017-01-02 2018-01-30
88730 Nutcracker needs to automatically recover from MC failure - rebalancing issues Other Operations open 2017-01-02 2018-04-16
88997 Improve graphite failover Other Operations open 2017-02-24 2017-12-14
89808 wikitech instances list is blank Other Operations open 2017-06-08 2017-06-07
91404 Setup backups of elasticsearch indices Other Operations open 2017-01-02 2016-09-22
92471 enable authenticated access to Cassandra JMX Other Operations open 2017-01-02 2018-03-02
93138 Procure hardware for Sentry Other Operations open 2017-01-02 2016-01-06
93531 secure.wikimedia.org entries still showing up in Google search results Other Operations open 2017-01-02 2015-12-08
94215 decommission cp3001 & cp3002 Other Operations open 2017-01-02 2018-01-03
94329 secure Cassandra/RESTBase cluster Other Operations open 2017-01-02 2016-10-19
94457 Install nodejs, nginx and other dependencies on francium Other Operations open 2017-12-19 2017-12-18
94819 Audit racktables Other Operations open 2017-01-02 2016-03-17
94951 Enable the usage of `hhvm -m debug --debug-host ::1` from mw1017 so developers can step through code (think gdb) in production to see what is going wrong. Other Operations open 2017-01-02 2016-12-01
95052 Make ircecho much better Other Operations open 2017-01-02 2016-07-16
95053 ircecho should accept input via unix sockets Other Operations open 2017-01-02 2016-07-15
95054 Move ircecho config file to be YAML Other Operations open 2017-01-02 2016-07-15
95742 Decomission amssq31-62 (32 hosts) Other Operations open 2017-01-02 2018-05-03
95801 Allow customizing the alert message from graphite Other Operations open 2017-01-02 2016-11-18
97524 ocg alarm ocg_job_status_queue 'flapping' Other Operations open 2017-01-02 2016-10-03
97909 Upgrade jobrunners to redis 2.8 Other Operations open 2017-01-02 2016-02-13
98831 Honor DNT header for access logs & varnish logs Other Operations open 2017-01-02 2018-02-14
98984 Check power supply balance settings on cp3030+ Other Operations open 2017-01-02 2018-01-03
100777 expose hosts in maintenance state so we can prevent scap from running on them Other Operations open 2017-01-02 2018-02-01
101141 udp rcvbuferrors and inerrors on graphite1001 Other Operations open 2017-01-02 2017-07-10
101585 document redis upgrade/restart procedures Other Operations open 2017-01-02 2016-05-23
102575 document graphite failover/backfill procedures Other Operations open 2017-01-02 2016-05-23
103886 Translation cache exhaustion caused by changes to PHP code in file scope Other Operations open 2017-07-09 2018-03-19
104352 Make scap able to depool/repool servers via the conftool API Other Operations open 2017-01-02 2017-09-26
104671 Rename 'restricted' group? Other Operations open 2017-07-12 2017-07-11
105780 Create a doc explaining the SLA between services and the monitoring tool Other Operations open 2017-01-02 2017-07-11
106346 setup an alertable threshold for Cassandra heap dumps Other Operations open 2017-01-02 2017-07-11
106664 Set up role accounts and feedback loops (FBL) with all providers Other Operations open 2017-01-02 2016-10-04
106937 Monitor [[Special:ListFiles]] for non 200 HTTP statuses in thumbnails Other Operations open 2017-01-02 2017-08-03
107108 Flow notification links on mobile point to desktop Other Operations open 2017-01-02 2017-07-25
108985 Monitor MediaWiki sessions Other Operations open 2017-01-02 2015-12-02
109089 EPIC: Cultivating the Elasticsearch garden (operational lessons from 1.7.1 upgrade) Other Operations open 2017-01-02 2016-10-20
109090 Investigate the need for master only (non data nodes) in our ES cluster Other Operations open 2017-01-02 2016-12-08
109606 Re-evaluate Limesurvey Other Operations open 2017-01-02 2017-06-05
110169 Monitor redis memory/disk usage Other Operations open 2017-01-02 2016-10-03
110171 Alert when ES indexes are freezed for more than 30 minutes Other Operations open 2017-01-02 2016-12-08
110240 [Discussion] Consider validating JSON schemas when running x-ample tests? Other Operations open 2017-01-02 2016-10-12
111540 Clean up labs graphite datapoints Other Operations open 2017-01-02 2016-11-18
111595 Do not apply spam headers on email assessed NOT to be spam Other Operations open 2017-01-16 2017-01-15
111653 Encrypt all the things Other Operations open 2017-12-27 2017-12-26
111838 Some files had disappeared from Commons after renaming Other Operations open 2017-01-02 2017-07-25
111934 Nutcracker stats monitoring should only listen on localhost Other Operations open 2017-01-02 2016-12-01
112774 solve mtp panel issue for row uplinks Other Operations open 2017-01-02 2016-03-17
113104 Set up a service IP for logstash Other Operations open 2017-06-24 2017-06-23
113785 Make the Shinken IRC alert and icinga-wm bots use colors Other Operations open 2017-01-02 2015-11-02
113792 Change LDAP cn to something more useful (was Rename "Dzahn" to "Daniel Zahn" in Gerrit) Other Operations open 2017-01-02 2017-04-14
114337 Assign 3 more servers to video scaler duty Other Operations open 2017-01-02 2018-02-23
114446 move human users out of UID range for system accounts Other Operations open 2017-01-02 2017-12-06
114801 operations-apache-config-lint replacement doesn't check syntax Other Operations open 2017-01-02 2017-12-07
114849 Log lines on flourine overflow at 8092 bytes. Other Operations open 2017-06-24 2017-06-23
115757 document debian packaging guidelines Other Operations open 2017-01-02 2016-05-23
115899 Move scap target configuration to etcd Other Operations open 2017-01-02 2016-10-05
116063 Hardware Automation Workflow - Overall Tracking Other Operations open 2017-01-02 2018-01-08
116580 monitor postgresql replication status Other Operations open 2017-01-02 2015-10-26
116627 Include 5xx numbers in fluorine fatalmonitor Other Operations open 2017-01-02 2015-10-28
116742 Track amount of package updates on systems Other Operations open 2017-01-02 2016-08-08
116747 Meta task "Revamp user authentication" Other Operations open 2017-01-02 2016-08-13
116750 2FA for SSH access to the production cluster Other Operations open 2017-01-02 2015-12-18
116767 limit the impact of heavy/large graphite queries Other Operations open 2017-01-02 2017-01-27
116805 DomainKeys Identified Mail (DKIM) for phabricator.wikimedia.org Other Operations open 2017-01-02 2016-08-31
116951 Reprepro should bail if it can't read and sign using the root keys Other Operations open 2017-01-02 2015-10-28
117508 Make ops-l a list for humans again (no cheating) Other Operations open 2017-01-02 2015-11-02
117673 labs precise and jessie instance not accessible after provisioning Other Operations open 2017-01-02 2016-06-21
118331 Alert when used_memory gets too high for redis queues Other Operations open 2017-01-02 2015-11-11
118677 Nastaleeq font for Western Punjabi Other Operations open 2017-01-02 2018-02-06
118746 Goal: Strengthen Incident monitoring infrastructure Other Operations open 2017-01-02 2016-08-13
118812 Investigate mysterious_sysctl settings and figure out what to do with them Other Operations open 2017-01-02 2015-11-17
118829 Automate the provisioning and management of MediaWiki clusters Other Operations open 2017-01-02 2015-11-17
119274 Check incoming requests to secure.wm.o Other Operations open 2017-01-02 2015-12-08
119401 Untangle labs/production roles from labs/instance roles Other Operations open 2017-01-02 2015-11-23
119660 Set up LVS for labs dns recursors Other Operations open 2017-01-02 2016-08-15
119679 Rewrite http://download.wikimedia.org/mediawiki/ -> https://releases.wikimedia.org/mediawiki in less than 3 redirects Other Operations open 2017-01-02 2017-05-16
119718 Make it easier to ban misbehaving dashboards from graphite Other Operations open 2017-01-02 2016-11-18
119719 Enforce a minimum refresh period for grafana dashboards hitting graphite Other Operations open 2017-01-02 2016-11-18
119846 Redirect revisions from svn.wikimedia.org to https://phabricator.wikimedia.org/rSVN Other Operations open 2017-01-02 2015-11-30
120377 labmon1001 graphite instance archiver keeps archiving the same instances Other Operations open 2017-01-02 2016-11-18
120532 Use user-specific passwords for accessing EventLogging database Other Operations open 2017-01-02 2017-02-20
120585 Make l10nupdate user a system user Other Operations open 2017-01-02 2017-09-26
120856 Remove all out of warranty unused cp10xx's from A2 Other Operations open 2017-01-02 2017-07-20
121240 Network isolation for production and semi-production services Other Operations open 2017-01-02 2017-07-12
121610 system users with UIDs > 500 Other Operations open 2017-01-02 2015-12-16
122127 Translation of namespaces for Gilaki Other Operations open 2017-01-02 2018-02-06
122825 Service Ownership and Maintenance Other Operations open 2017-01-02 2018-03-07
122917 Provide a good download service of dumps from Wikimedia Other Operations open 2017-01-02 2016-07-19
123237 Provide production jessie image with node 4.2; use this for service-runner build command Other Operations open 2017-01-02 2016-10-12
123276 URL parameters do not work with pages that have "?" in their names Other Operations open 2017-04-29 2017-09-13
123560 investigate rsync between dcs with encryption Other Operations open 2017-01-02 2016-05-30
123809 Module uwsgi doesn't allow passing multiple config params of same name Other Operations open 2017-01-02 2016-01-15
123918 'swift' user/group IDs should be consistent across the fleet Other Operations open 2017-01-02 2017-04-10
124101 Specific revisions of multiple files missing from Swift - 404 Not Found returned Other Operations open 2017-01-02 2018-04-16
124179 Improve access to and control over incident and metrics monitoring infrastructure Other Operations open 2017-01-02 2017-03-08
124185 Evaluate alternative web interfaces to icinga 1 core Other Operations open 2017-01-02 2017-04-18
124413 confctl should provide tags information after writing data Other Operations open 2017-01-02 2016-01-22
124991 evaluate possibility for nscd use with useldap Other Operations open 2017-01-02 2017-07-24
125015 Requests to (hard) redirect pages return their target's contents but are counted as pageviews to the redirect page Other Operations open 2017-01-02 2016-01-28
125085 Split the API MediaWiki appserver pool into two external/internal pools Other Operations open 2017-01-02 2016-10-12
125411 Diamond load averages do not contain scaled versions Other Operations open 2017-01-02 2017-07-25
125442 es2009 degraded RAID Other Operations open 2017-01-02 2016-04-22
125976 Run mediawiki::maintenance scripts in Beta Cluster Other Operations open 2018-05-03 2018-05-02
126083 overhaul labstore setup [tracking] Other Operations open 2017-01-02 2017-07-23
126158 [RFC] Alert about *when* partitions will run out of space, not a percentage/absolute number Other Operations open 2017-01-02 2016-05-02
126295 Spike: What do we have to package to run the Programs and Events dashboard on production? Other Operations open 2017-01-02 2017-02-28
126574 puppet should try to mount all mountable swift filesystems Other Operations open 2017-01-02 2016-02-11
126989 MediaWiki logging & encryption Other Operations open 2017-01-02 2017-07-10
127054 pinentry-gtk2 pulls in a lot of unneeded Gnome/GTK libs Other Operations open 2017-01-02 2016-02-16
127797 document all puppet classes / defined types!? Other Operations open 2017-01-02 2016-11-28
127825 Re-add intel-microcode Other Operations open 2017-01-02 2018-01-05
128590 Cassandra uses default ip address for outbound packets while bootstrapping Other Operations open 2017-01-02 2016-12-22
128615 Get rid of Tool Labs home page check from shinken Other Operations open 2017-01-02 2016-12-04
128715 Add other Tools administrators to the Icinga notification group Other Operations open 2017-01-02 2016-12-04
128716 Make icinga-wm report Tools homepage check at #wikimedia-labs, too Other Operations open 2017-01-02 2016-05-27
129180 Preserve SSH host key when re-imaging hosts Other Operations open 2017-01-02 2016-03-07
129188 mw2212 unresponsive Other Operations open 2017-01-02 2016-03-08
129847 conftool-merge should report which node is setting attributes for Other Operations open 2017-01-02 2016-03-14
129963 Update memcached package and configuration options Other Operations open 2017-01-02 2018-04-16
130590 Have dedicated master nodes for elasticsearch Other Operations open 2017-01-02 2016-11-03
130593 investigate slapd memory leak Other Operations open 2017-01-02 2017-06-30
130617 Collect metrics on pool counter usage Other Operations open 2017-01-02 2016-11-03
130709 authoritative copy of 'root' files for upload.wikimedia.org is only in swift Other Operations open 2017-01-02 2017-03-29
130883 decom cp3011-22 (12 machines) Other Operations open 2017-01-02 2018-05-03
131326 smokeping config puppetization issue? Other Operations open 2017-01-02 2016-04-15
131748 Refresh the appservers puppet code/configs Other Operations open 2017-01-02 2016-04-04
131832 Unable to restore file that has a very large file size Other Operations open 2017-01-02 2017-07-25
131966 Default gateway unreachable on baham.wikimedia.org after reboot Other Operations open 2017-01-02 2017-07-20
132104 Consider moving policy.wikimedia.org away from WordPress.com Other Operations open 2017-01-02 2016-06-08
132216 Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm Other Operations open 2017-01-02 2016-05-09
132324 Tracking and Reducing cron-spam to root@ Other Operations open 2017-01-02 2018-01-18
132325 Weak digest algorithm (SHA1) used to sign InRelease on apt.wikimedia.org Other Operations open 2017-01-02 2017-04-03
132532 rsync module doesnt work on trusty Other Operations open 2017-01-02 2016-04-12
132632 puppetize turning off reserved space for cassandra /srv Other Operations open 2017-07-19 2017-07-18
133091 Highest SSTables / read thresholds Other Operations open 2017-01-02 2017-07-11
133093 Investigate idle appservers in codfw Other Operations open 2017-01-02 2016-04-19
133164 Document eqiad/codfw transition plan for OCG Other Operations open 2017-01-02 2016-05-23
133179 Redis monitoring needs to be improved Other Operations open 2017-01-02 2017-07-07
133318 High levels of PoolCounter errors should trigger alerts Other Operations open 2017-01-02 2018-01-09
133476 Proposal: Centralize OTRS login methodology Other Operations open 2017-01-02 2017-08-15
133656 Have a paging check for Nova API accessible Other Operations open 2017-01-02 2016-12-21
133674 HHVM is leaking memory on the API appservers Other Operations open 2017-01-02 2016-10-10
133744 Epic: switch Maps to production status Other Operations open 2017-01-02 2016-07-19