TechnicalOperations Unpointed Tasks

Help.

82038 create a test for multicast relay Network Operations open 2017-01-02 2017-06-27
83119 Netflow Collector Project Network Operations open 2017-01-02 2017-06-27
83992 Juniper monitoring Network Operations open 2017-01-02 2017-06-27
86541 setup wifi in codfw Network Operations open 2017-01-02 2017-06-27
98006 Anycast (Auth)DNS Network Operations open 2017-01-02 2017-07-24
106056 set up a looking glass for WMF ASes Network Operations open 2017-01-02 2017-06-27
120425 dumps.wikimedia.org seems to have poor throughput towards some destinations Network Operations open 2017-01-02 2017-06-27
122406 Consider renumbering Labs to separate address spaces Network Operations open 2017-01-02 2017-07-17
124843 Peer with SFMIX at ULSFO in 200 Paul Network Operations open 2017-01-02 2017-06-27
133387 Enabling IGMP snooping on QFX switches breaks IPv6 (HTCP purges flood across codfw) Network Operations open 2017-01-02 2017-08-03
150256 Re-setup lvs1007-lvs1012, replace lvs1001-lvs1006 Network Operations open 2017-01-02 2017-09-25
150264 Icinga check for VRRP Network Operations open 2017-01-02 2017-06-27
157435 Review ACLs for the Analytics VLAN Network Operations open 2017-02-08 2017-10-09
163674 Frequent RST returned by appservers to LVS hosts Network Operations open 2017-04-25 2017-07-12
166171 rack/setup/wire/deploy msw2-c1-eqiad Network Operations open 2017-05-24 2017-09-12
167299 Upgrade BIOS/RBSU/etc on lvs1007 Network Operations open 2017-06-08 2017-09-26
167306 ospf link-protection Network Operations open 2017-06-08 2017-06-27
167691 High amount of unexpected ICMP dest unreachable toward esams cache clusters Network Operations open 2017-06-13 2017-06-27
167840 Merge AS14907 with AS43821 Network Operations open 2017-06-14 2017-10-13
167841 Cleanup confed BGP peerings and policies Network Operations open 2017-06-14 2017-06-27
167842 Find a new PIM RP IP Network Operations open 2017-06-14 2017-10-20
169644 eqiad: rack frack refresh equipment Network Operations open 2017-07-05 2017-09-11
170131 Recurring varnish-be fetch failures in codfw Network Operations open 2017-07-11 2017-07-12
170144 Evaluate NetBox as a Racktables replacement & IPAM Network Operations open 2017-07-11 2017-09-17
171032 Investigate lvs IP pages during codfw row C switch upgrade Network Operations open 2017-07-20 2017-07-20
171962 bonded/redundant network connections for fundraising hosts Network Operations open 2017-07-29 2017-07-28
172459 eqiad row D switch upgrade Network Operations open 2017-08-04 2017-08-29
173489 pmacct should be upgraded to 1.6.2 on Stretch Network Operations open 2017-08-18 2017-10-17
173698 Backfill librenms data in graphite with historical RRDs Network Operations open 2017-08-22 2017-08-21
174596 dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour Network Operations open 2017-08-31 2017-08-30
174616 set up cr3-esams Network Operations open 2017-08-31 2017-08-30
174637 Setup esams atlas anchor Network Operations open 2017-09-01 2017-09-04
176175 connect second ethernet interface for fundraising codfw hosts Network Operations open 2017-09-19 2017-09-18
176337 esams: networking audit for support contract renewal Network Operations open 2017-09-21 2017-09-20
176338 eqiad: networking audit for support contract renewal Network Operations open 2017-09-21 2017-09-21
176427 unrack/decom pfw1-codfw and pfw2-codfw Network Operations open 2017-09-22 2017-09-22
176816 cr2-esams temperature warning Network Operations open 2017-09-27 2017-09-26
176975 connect second interface for each frack to opposite switch for each eqiad host Network Operations open 2017-09-29 2017-10-02
61115 Implement RPKI (Resource Public Key Infrastructure) Network Operations resolved 2017-01-02 2017-10-06
79755 Setup flow monitoring of *internal* network traffic Network Operations resolved 2017-01-02 2017-06-27
80253 Fix esams management network Network Operations resolved 2017-01-02 2015-09-08
80273 nagios monitor transit/peering links and alert on low/high traffic Network Operations resolved 2017-01-02 2017-05-09
80847 hook up and dns oob access Network Operations resolved 2017-01-02 2015-04-03
81345 Re-Label transcode1 and transcode2 Network Operations resolved 2017-01-02 2014-12-18
81440 mr1 asymetric routing Network Operations resolved 2017-01-02 2015-09-02
81989 Notify transits of new esams prefixes Network Operations resolved 2017-01-02 2016-06-17
82509 Replace asw-c5-eqiad or asw-c8-eqiad with EX4550 Network Operations resolved 2017-01-02 2015-02-11
83037 Filter outgoing BGP announcements on AS regex Network Operations resolved 2017-01-02 2017-06-27
83196 internal network packet loss alerting Network Operations resolved 2017-07-20 2017-07-19
83898 JSNMP flood of errors across multiple switches Network Operations resolved 2017-01-02 2017-05-23
84205 Add missing rack "locations" in Librenms Network Operations resolved 2017-01-02 2015-09-08
84502 interfaces that are down on mr1-eqiad Network Operations resolved 2017-01-02 2015-09-02
84775 Upgrade switch fabrics in cr2-codfw Network Operations resolved 2017-01-02 2015-08-04
86574 Connect Apple Airport to mr1-codfw Network Operations resolved 2017-01-02 2015-01-12
87600 labnet1001 should be able to send stats to tungsten Network Operations resolved 2017-01-02 2015-01-26
89227 Set up cr1-eqord & cr1-eqdfw Network Operations resolved 2017-01-02 2015-08-14
89294 Establish IPsec tunnel between codfw and eqiad pfw Network Operations resolved 2017-01-02 2015-07-09
89833 set up switch port for second ethernet interface for ms1001 Network Operations resolved 2017-01-02 2015-02-27
89999 cr1-eqiad Control Board error Network Operations resolved 2017-01-02 2016-07-19
90695 permit syslog from labs hosts subnets to lithium Network Operations resolved 2017-01-02 2015-06-08
92548 Network congestion between DTAG & eqiad Network Operations resolved 2017-01-02 2015-03-17
92616 cr1/cr2-codfw QSFP+ errors every second for qsfp-0/0/0 Network Operations resolved 2017-01-02 2015-11-10
92815 Codfw mediawiki appservers from any rows but row A can't communicate with the dhcp server Network Operations resolved 2017-01-02 2016-01-20
93932 determine networking for ganeti2001-2006 Network Operations resolved 2017-01-02 2015-03-26
95959 install/setup/deploy cobalt as replacement for gallium Network Operations resolved 2017-01-02 2016-06-08
96385 setup network switch ports / vlans for db2053-2070 Network Operations resolved 2017-01-02 2015-04-22
96569 pfw-eqiad JunOS upgrade Network Operations resolved 2017-01-02 2015-04-30
102411 Change the ip of labs-ns0 with our DNS registrar Network Operations resolved 2017-01-02 2015-06-17
104036 networking: adjust ACLs to allow analytics clusters to talk to new ganglia aggregator Network Operations resolved 2017-01-02 2015-07-01
104458 rack/setup new eqiad lvs machines Network Operations resolved 2017-01-02 2016-11-08
105984 Very slow downloads from Wikimedia sites in eqiad on Wikimania hotel network Network Operations resolved 2017-01-02 2015-07-27
107056 Puppetize a server with a role that sets up Cassandra on Analytics machines [13 pts] {slug} Network Operations resolved 2017-01-02 2016-03-16
107635 investigate ethernet errors: asw2-a5-eqiad port xe-0/0/36 Network Operations resolved 2017-01-02 2015-09-02
108715 smokeping loss of ping for codfw rows Network Operations resolved 2017-01-02 2015-08-11
110208 Requests from a specific network are blocked Network Operations resolved 2017-01-02 2015-09-22
110435 cr1-eqdfw PEM 0 failure Network Operations resolved 2017-01-02 2015-09-18
111274 Set up NTT transit @ eqdfw, eqord Network Operations resolved 2017-01-02 2015-09-18
112000 cr2-eqiad PEM 2 failure Network Operations resolved 2017-01-02 2015-09-24
112396 Wikimedia sites not reachable through CenturyLink ISP Network Operations resolved 2017-01-02 2015-09-13
112902 Various misc. sites (e.g. gerrit) load very slow or not at all Network Operations resolved 2017-01-02 2015-09-17
113640 Upgrade JunOS on cr1/cr2-codfw Network Operations resolved 2017-01-02 2016-01-21
113771 setup new equinix out of band mgmt access Network Operations resolved 2017-01-02 2015-10-27
115330 block labs IPs from sending data to prod ganglia Network Operations resolved 2017-01-02 2016-06-24
115491 Allocate labs subnet in dallas Network Operations resolved 2017-01-02 2016-04-28
115492 Allocate subnet for labs test cluster instances Network Operations resolved 2017-01-02 2016-04-28
115921 remove tele2(patchid 2953) from dmarc panel Network Operations resolved 2017-01-02 2015-10-20
115924 Return psw2-eqiad to spares Network Operations resolved 2017-01-02 2015-10-27
116766 attach mr1-codfw to new out of band mgmt link Network Operations resolved 2017-01-02 2015-10-27
116928 drain ULSFO of all traffic on 2015-11-02 before 0900 PST Network Operations resolved 2017-01-02 2015-11-30
118178 test new sfp-t Network Operations resolved 2017-01-02 2015-11-10
118256 Set up cr2-esams Network Operations resolved 2017-01-02 2016-05-30
118259 Figure out the source of QSFP+ errors with DAC + MX480 Network Operations resolved 2017-01-02 2015-11-25
118721 cr1-eqiad PEM 0 fan failed Network Operations resolved 2017-01-02 2015-11-20
119512 add new saltmaster (neodymium) to network exceptions for labvirt* etc hosts Network Operations resolved 2017-01-02 2015-11-30
119634 Fix edit permissions of the netops project Network Operations resolved 2017-01-02 2015-11-25
119759 Investigate why disabling an uplink port did not deprioritize VRRP on cr2-eqiad Network Operations resolved 2017-01-02 2017-01-09
120010 Add firewall exception to get to wdqs*:8888 from analytics cluster Network Operations resolved 2017-01-02 2017-02-08
122293 Orange S.A. searches a contact at WMF for tests Network Operations resolved 2017-01-02 2016-01-06
122885 turn-up/implement zayo wave (579171) for ulsfo-codfw Network Operations resolved 2017-01-02 2016-02-19
123732 Decommission nitrogen (IPv6 relay) Network Operations resolved 2017-01-02 2016-01-28
124804 All wikis under wikimedia.org (Meta, Commons, loginwiki, others) are using wikimedia.org VHost, including /wiki/ -> wikimediafoundation.org redirect Network Operations resolved 2017-01-02 2016-02-12
124955 stat1002 - redis can't connect to mira.codfw.wmnet Network Operations resolved 2017-01-02 2016-01-27
127295 mr1-ulsfo booted from backup JunOS Network Operations resolved 2017-01-02 2016-02-18
129136 Deploy libreNMS with scap3 Network Operations resolved 2017-01-02 2017-05-05
131218 Update the port description on the network switch for analytics 1017 and 1021 Network Operations resolved 2017-01-02 2016-03-30
131375 Make sure phab can talk to gearman and nodepool instances can talk to phabricator Network Operations resolved 2017-01-02 2016-06-21
133852 analytics hosts frequently tripping 'port utilization threshold' librenms alerts Network Operations resolved 2017-01-02 2017-05-23
134930 codfw-eqiad Zayo link is down (cr2-codfw:xe-5/0/1) Network Operations resolved 2017-01-02 2016-05-19
134932 cr2-codfw LUCHIP/trinity_pio error messages Network Operations resolved 2017-01-02 2016-06-01
136671 Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR Network Operations resolved 2017-01-02 2016-07-19
136707 Upgrade cr1-codfw/cr2-codfw FPC 0 firmware Network Operations resolved 2017-01-02 2016-06-01
136717 Turn up new eqiad-esams wave (Level3) Network Operations resolved 2017-01-02 2016-08-14
136829 'German Wikipedia Broken Weblinks Bot' is ill-behaved and in danger of getting all of Labs blacklisted Network Operations resolved 2017-01-02 2016-06-06
138609 Network ACL rules to allow traffic from Analytics to Production for port 9160 Network Operations resolved 2017-01-02 2016-08-31
140257 Allocate contint1001 to releng and allocate to a vlan Network Operations resolved 2017-01-02 2016-11-16
140764 cr1/cr2-eqiad: install new SCBs and linecards Network Operations resolved 2017-01-02 2016-08-18
140765 Replace cr1/2-eqiad PSUs/fantrays with high-capacity ones Network Operations resolved 2017-01-02 2016-08-04
140770 Upgrade cr1/cr2-eqiad JunOS Network Operations resolved 2017-01-02 2016-08-18
140935 audit network ports in a4-codfw Network Operations resolved 2017-01-02 2016-08-09
140991 put pfw1- ge-2/0/11 in the 'fundraising' vlan for new host frqueue1001 Network Operations resolved 2017-01-02 2016-07-27
141085 deploy elasticsearch/plugins to relforge1001-1002 servers Network Operations resolved 2017-01-02 2016-08-03
141898 cr2-eqiad temperature alerts ("system warm") Network Operations resolved 2017-01-02 2016-08-11
142211 Enable access to relforge clusters from virtual machines running on labs Network Operations resolved 2017-01-02 2016-08-10
143248 configure port for frdb1001 Network Operations resolved 2017-01-02 2016-09-08
143335 Open hole in analytics vlan firewall to allow MirrorMaker to talk to main Kafka clusters Network Operations resolved 2017-01-02 2016-08-18
143800 codfw: rack/setup/deploy puppetmaster200[12]switch configuration Network Operations resolved 2017-01-02 2016-08-24
143913 Upgrade cr1-esams & cr2-knams to JunOS 13.3 Network Operations resolved 2017-01-02 2016-10-05
143914 Upgrade cr1-ulsfo & cr2-ulsfo to JunOS 13.3 Network Operations resolved 2017-01-02 2016-10-05
143915 Fix static IP fallbacks to Pybal LVS routes Network Operations resolved 2017-01-02 2016-10-26
144263 Connection problems (from NZ to ULSFO) Network Operations resolved 2017-01-02 2016-08-30
145270 Telkom/8ta (South Africa) users cannot connect to wikimedia sites Network Operations resolved 2017-01-02 2016-09-19
146391 eeden ethernet outage Network Operations resolved 2017-01-02 2017-08-29
146474 Add firewall exception to get to wdqs*.codfw.wmnet:8888 from analytics cluster Network Operations resolved 2017-01-02 2017-02-08
147061 Give elukey/ema access to network devices Network Operations resolved 2017-01-02 2016-10-05
147320 Decomission palladium Network Operations resolved 2017-01-02 2016-10-28
148506 Rack and setup new eqiad row D switch stack (EX4300/QFX5100) Network Operations resolved 2017-01-02 2017-08-03
149170 Move dbproxy1010 and dbproxy1011 to labs-support network, rename them to labsdbproxy1001 and labsdbproxy1002 Network Operations resolved 2017-01-02 2016-11-23
149196 Migrate links from cr1-eqiad/cr2-eqiad fpc 5 to fpc 3 Network Operations resolved 2017-01-02 2017-08-25
149224 Decommission psw1-eqiad Network Operations resolved 2017-01-02 2016-10-27
149226 cr1-eqiad:ae4 is disabled due to VRRP issue Network Operations resolved 2017-01-02 2016-11-08
149253 Firewall rules production/labs for prometheus-node-exporter Network Operations resolved 2017-01-02 2016-11-03
149829 Move labsdb1008 to production, rename it back to db1095, use it as a temporary sanitarium Network Operations resolved 2017-01-02 2016-11-15
150092 Provide read-only access to OpenStack APIs from WMF IP space Network Operations resolved 2017-01-02 2016-12-20
150380 kafka2003 switch port configuration Network Operations resolved 2017-01-02 2016-11-10
150400 Spare pool servers switch configuration Network Operations resolved 2017-01-02 2016-11-10
150769 restbase201[0-2] switch port configuration Network Operations resolved 2017-01-02 2016-11-15
151357 prometheus200[1-2] switch port configuration Network Operations resolved 2017-01-02 2016-11-22
152182 asw2-d-eqiad.mgmt.eqiad - JNX_ALARMS CRITICAL - 2 red alarms, Network Operations resolved 2017-01-02 2017-01-19
152215 Move novaobserver (and novaadmin) users out of ldap Network Operations resolved 2017-01-02 2016-12-07
152627 ms-fe200[5-8] switch port configuration Network Operations resolved 2017-01-02 2016-12-13
153094 wdqs2003 switch port configuration Network Operations resolved 2017-01-02 2016-12-14
153273 asw-a2-eqiad PEM 0 not powered Network Operations resolved 2017-01-02 2016-12-15
154507 Slight packet loss observed on the network starting Nov 2016 Network Operations resolved 2017-01-04 2017-04-10
154577 cr2-esams<->cr2-eqiad link flaps Network Operations resolved 2017-01-05 2017-01-18
154605 codfw: elastic2025-elastic2036/switch port configuration Network Operations resolved 2017-01-05 2017-01-04
154758 asw-a7-codfw is down Network Operations resolved 2017-01-07 2017-01-25
154952 cr2-esams<->cr2-eqiad link down Network Operations resolved 2017-01-10 2017-01-18
155487 Open temporary access from analytics vlan to new-labsdb one Network Operations resolved 2017-01-18 2017-01-26
155875 asw-c2-eqiad reboots & fdb_mac_entry_mc_set() issues Network Operations resolved 2017-01-22 2017-02-17
155999 DBA plan to mitigate asw-c2-eqiad reboots Network Operations resolved 2017-01-24 2017-02-09
156109 netops: switch all subnets to use install1002/2002 as DHCP Network Operations resolved 2017-01-25 2017-02-10
156212 codfw: mc2019-mc2036/switch port configuration Network Operations resolved 2017-01-26 2017-01-26
156225 Move db1054 to A3 Network Operations resolved 2017-01-26 2017-01-25
156381 pfws not on librenms Network Operations resolved 2017-01-27 2017-01-27
156957 asw-d-codfw public1-vlan addition review (blocks gerrit2001) Network Operations resolved 2017-02-02 2017-02-02
157593 Add firewall exception to get to wdqs[12]003.(codfw|eqiad).wmnet:8888 from analytics cluster Network Operations resolved 2017-02-09 2017-02-10
158601 git clone over EQIAD (wmflabs) CODFW timeout due to low bandwidth (~250 KiB/s) Network Operations resolved 2017-02-21 2017-03-13
158647 cr2-knams<->asw-esams GBLX fiber down Network Operations resolved 2017-02-22 2017-04-07
158714 codfw:ms-be2028-ms-be2039 switch port configuration Network Operations resolved 2017-02-22 2017-03-06
159336 deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001) Network Operations resolved 2017-03-02 2017-03-15
159464 asw-a1-codfw spontaneous reboot Network Operations resolved 2017-03-03 2017-04-03
159757 netmon1002 networking setup Network Operations resolved 2017-03-07 2017-05-02
160055 Audit and cleanup border-in ACL on core routers Network Operations resolved 2017-03-10 2017-04-04
160087 codfw: oresrdb2002 switch port configuration Network Operations resolved 2017-03-10 2017-03-09
162152 esams higher than usual temperature Network Operations resolved 2017-04-05 2017-04-24
162199 Faulty optics on asw-b-eqiad:xe-1/1/2 Network Operations resolved 2017-04-05 2017-04-10
162239 cr2-esams FPC 0 is dead Network Operations resolved 2017-04-06 2017-04-12
162601 knams equipment move Network Operations resolved 2017-04-11 2017-04-13
162944 db20[7-9][0-9] switch ports configuration Network Operations resolved 2017-04-14 2017-05-02
163002 Spread eqiad analytics Kafka nodes to multiple racks ans rows Network Operations resolved 2017-04-15 2017-04-28
163243 ripe-atlas-eqiad is down Network Operations resolved 2017-04-19 2017-05-25
163312 lvs2001: intermittent packet loss from Icinga checks Network Operations resolved 2017-04-20 2017-05-18
163323 Interface errors on asw-c-codfw:xe-7/0/46 Network Operations resolved 2017-04-20 2017-04-20
163324 switchover icinga.wikimedia.org from einsteinium to tegmen Network Operations resolved 2017-04-20 2017-08-03
163326 switchover oresrdb.svc.eqiad.wmnet from oresrdb1001 to oresrdb1002 and back (after T148506) Network Operations resolved 2017-04-20 2017-08-03
163447 Interface errors on pfw-codfw:xe-15/0/0 Network Operations resolved 2017-04-21 2017-04-24
163542 Interface errors on cr2-eqiad:xe-4/3/1 Network Operations resolved 2017-04-22 2017-07-12
164495 Replace old Prometheus VMs addresses with baremetal in firewall configuration Network Operations resolved 2017-05-05 2017-05-04
164554 pfw-eqiad.wikimedia.org - 3 interfaces down (fundraising hosts) Network Operations resolved 2017-05-06 2017-05-05
164594 codfw: ganeti2007-ganeti2008 switch port configuration Network Operations resolved 2017-05-06 2017-05-08
164911 LibreNMS improvements Network Operations resolved 2017-05-11 2017-06-09
164970 mr1-ulsfo crashed Network Operations resolved 2017-05-11 2017-05-18
164988 codfw: kubernetes200[1-4] switch port configuration Network Operations resolved 2017-05-11 2017-05-10
165006 Zayo Circuit ulsfo<->codfw down Network Operations resolved 2017-05-12 2017-05-11
165008 Interface errors on asw-c-eqiad:xe-8/0/38 Network Operations resolved 2017-05-12 2017-05-17
165288 Report of esams unreachable from Fastweb/Init7 Network Operations resolved 2017-05-16 2017-05-23
165584 Deploy pybal with BGP MED support (for primary/backup) in production Network Operations resolved 2017-05-18 2017-10-20
165614 LLDP on cache hosts Network Operations resolved 2017-05-18 2017-06-09
166156 ores200[1-9] switch port configuration Network Operations resolved 2017-05-24 2017-05-23
166564 codfw: labtestvirt2002 swith port configuration Network Operations resolved 2017-05-31 2017-06-27
166663 Init7 routing loop Network Operations resolved 2017-06-01 2017-05-31
167261 Faulty link between cr2-codfw and cr1-eqdfw Network Operations resolved 2017-06-08 2017-06-15
167274 codfw row D switch upgrade Network Operations resolved 2017-06-08 2017-08-03
167281 Some BGP sessions to the SF Office down Network Operations resolved 2017-06-08 2017-06-29
167288 Rancid improvements Network Operations resolved 2017-06-08 2017-06-09
167321 codfw: labtestpuppetmaster2001 switch port configuration Network Operations resolved 2017-06-08 2017-07-11
167322 codfw:labtestnet2002 switch port configuration Network Operations resolved 2017-06-08 2017-06-12
167326 codfw: labtestneutron2002 switch port configuration Network Operations resolved 2017-06-08 2017-06-12
168462 codfw row A switch upgrade Network Operations resolved 2017-06-21 2017-08-03
168667 Temperature increase in esams Network Operations resolved 2017-06-23 2017-06-27
169345 codfw row B switch upgrade Network Operations resolved 2017-07-01 2017-08-03
169624 deploy diffscan2 Network Operations resolved 2017-07-05 2017-08-02
169637 zombie rstp configuration Network Operations resolved 2017-07-05 2017-07-05
169643 codfw: rack frack refresh equipment Network Operations resolved 2017-07-05 2017-10-19
170138 Replace cr1/2-eqiad air filters Network Operations resolved 2017-07-11 2017-07-20
170369 Remove unsecure SSH algorithms on network devices Network Operations resolved 2017-07-13 2017-07-13
170380 codfw row C switch upgrade Network Operations resolved 2017-07-13 2017-08-03
170622 pfw-codfw still logging to indium Network Operations resolved 2017-07-14 2017-07-17
171167 Evaluate LibreNMS' Graphite backend Network Operations resolved 2017-07-21 2017-08-29
171714 "MySQL server has gone away" from librenms logs Network Operations resolved 2017-07-27 2017-08-02
171823 Grafana dashboards for librenms graphite data Network Operations resolved 2017-07-28 2017-09-20
171970 Move codfw frack to new infra Network Operations resolved 2017-07-29 2017-09-19
172262 Poor conectivity (Vodafone/THUS in UK) Network Operations resolved 2017-08-03 2017-08-15
172478 git-ssh.wikimedia.org and IPv6 are broken after switch to phab1001 Network Operations resolved 2017-08-05 2017-08-04
174218 Move eqiad frack to new infra Network Operations resolved 2017-08-26 2017-10-05
174364 DDOS_PROTOCOL_VIOLATION_SET: Protocol Rejectv6:aggregate is violated Network Operations resolved 2017-08-29 2017-08-28
174366 Power alarm flap on asw-d-codfw:et-7/0/52 channel 3 Network Operations resolved 2017-08-29 2017-08-30
174397 Tracking task for network syslog messages Network Operations resolved 2017-08-29 2017-10-19
175737 two switches have same serial in racktables Network Operations resolved 2017-09-13 2017-09-14
176319 remove fundraising firewall rules related to ganglia Network Operations resolved 2017-09-21 2017-10-02
176350 connect new office link to asw-ulsfo Network Operations resolved 2017-09-21 2017-10-18
176492 move frav1001's to the frack-fundraising VLAN so we can use it for database testing Network Operations resolved 2017-09-23 2017-09-22
176972 reconfigure networking on frack-eqiad management interfaces Network Operations resolved 2017-09-29 2017-09-28
177332 vcp port down on fasw-c-codfw Network Operations resolved 2017-10-04 2017-10-05
177333 Interface error on fasw-c-eqiad:vcp-255/1/0 Network Operations resolved 2017-10-04 2017-10-05
177820 Allow syslog (-tls) from both wezen and lithium in labs Network Operations resolved 2017-10-11 2017-10-12
178608 Disable Jenkins autodiscovery system Network Operations resolved 2017-10-20 2017-10-20
23027 Requests with utf-8 in the URL return a outdated page revision Traffic open 2017-01-02 2016-10-04
26413 Consider DNSSec Traffic open 2017-01-02 2016-09-30
36670 Check all wikis for inclusions of http resources on https Traffic open 2017-01-02 2016-10-04
45250 Redo /beacon/impression system (formerly Special:RecordImpression) to remove extra round trips on all FR impressions (title was: S:RI should pyroperish) Traffic open 2017-01-02 2017-03-27
50133 ForeignAPIRepo wrongly returns non-protocol-relative URLs for original "thumbs" Traffic open 2017-01-02 2017-06-27
50835 Separate Cache-Control header for proxy and client Traffic open 2017-01-02 2017-05-30
54253 Protocol-relative URLs are poorly supported or unsupported by a number of HTTP clients Traffic open 2017-01-02 2017-03-28
56783 Respect X-Forwarded-For only from trustworthy sources Traffic open 2017-01-02 2016-10-04
63782 Add varnish logs to logstash Traffic open 2017-01-02 2017-02-14
66214 Define an official thumb API Traffic open 2017-01-02 2017-08-16
74186 Varnish: Mobile site redirect interferes with OAuth authorization process Traffic open 2017-01-02 2016-12-05
75944 Monitor Varnish caches on beta cluster have two varnishd process running Traffic open 2017-01-02 2016-09-30
78421 m.{project}.org portal/redirect consistency Traffic open 2017-01-02 2017-08-08
78963 Support ESI for ResourceLoader Traffic open 2017-01-02 2016-12-05
79730 Add pybal check to ensure service IP is bound Traffic open 2017-03-31 2017-03-30
81305 Make PyBal respect advertised BGP capabilities Traffic open 2017-03-31 2017-03-30
81605 No IPv6 addresses on Wikimedia nameservers ns(0-2).wikimedia.org Traffic open 2017-01-02 2016-09-30
82849 lvs servers report 'Memory allocation problem' on bootup Traffic open 2017-02-11 2017-10-06
83467 LVS testing needs to include internal services testing Traffic open 2017-01-02 2016-09-30
84543 more robust certificate chain creation in puppet Traffic open 2017-01-02 2016-12-05
86650 Add support for setting weight=0 when depooling Traffic open 2017-01-02 2017-07-31
86671 HTTPS RFC5077 session tickets encryption key rollovers Traffic open 2017-01-02 2016-09-30
86915 nan and minnan subdomain redirects are a mess Traffic open 2017-01-02 2017-08-24
88861 wikipedia.lol Traffic open 2017-01-02 2016-09-30
89838 Move proxy IP lists to META for Varnish XFF decoding Traffic open 2017-01-02 2017-02-06
91372 $wgMFAnonymousEditing = true is sometimes not respected: cache? Traffic open 2017-01-02 2016-05-04
91820 Create HTTP verb and sticky cookie DC routing in VCL Traffic open 2017-01-02 2017-05-05
92002 implement Public Key Pinning (HPKP) for Wikimedia domains Traffic open 2017-01-02 2017-06-26
92298 Investigate our mitigation strategy for HTTPS response length attacks Traffic open 2017-01-02 2016-11-15
94125 Central login notice appears on unencrypted API format=*fm pages, where reloading does not affect login status Traffic open 2017-01-02 2016-10-04
94697 implement better failure-scenario geoip mapping in gdnsd Traffic open 2017-01-02 2016-09-30
96499 dbtree loads third party resources (from jquery.com and google.com) Traffic open 2017-01-02 2016-10-04
96844 Update TLS/HTTP documentation on wikitech Traffic open 2017-01-02 2016-09-30
96852 Define 3-host infra cluster for traffic pops Traffic open 2017-01-02 2017-03-21
96853 Evaluate Apache Traffic Server Traffic open 2017-01-02 2017-06-21
97051 adding new languages to DNS langs.tmpl doesn't work until zone template is edited as well Traffic open 2017-01-02 2016-10-04
97333 Align prod + beta-cluster LVS usage Traffic open 2017-01-02 2016-09-30
98165 Figure out an etcd deploy strategy that includes multi DC failure scenarios. Traffic open 2017-01-02 2017-07-11
99216 Please set up a CNAME for videoserver.wikimedia.org to Video Editing Server Traffic open 2017-01-02 2016-09-30
99531 [Task] move wikiba.se webhosting to wikimedia misc-cluster Traffic open 2017-07-22 2017-10-17
101002 Use Upgrade Insecure Requests on Wikimedia wikis Traffic open 2017-01-02 2017-04-12
101048 Policy decisions for new (and current) DNS domains registered to the WMF Traffic open 2017-01-02 2016-09-30
101525 Set up LVS for current AuthDNS Traffic open 2017-01-02 2017-04-25
102178 Fix RESTBase support for wikitech.wikimedia.org Traffic open 2017-01-02 2017-07-11
102367 Migrate tools.wmflabs.org to https only (and set HSTS) Traffic open 2017-01-02 2017-10-15
102848 Split GeoIP into a new component Traffic open 2017-01-02 2017-04-04
104442 Investigate better DNS cache/lookup solutions Traffic open 2017-01-02 2017-07-26
104681 HTTPS Plans (tracking / high-level info) Traffic open 2017-01-02 2017-07-11
105657 Expires header for load.php should be relative to request time instead of cache time Traffic open 2017-01-02 2017-10-17
106517 upload.wikimedia.org returns HTTP status code 503 for truncated urls, not 404 Traffic open 2017-01-02 2016-09-30
107236 Switch port 80 to nginx on primary clusters Traffic open 2017-01-02 2016-09-30
108580 HTTPS for internal service traffic Traffic open 2017-01-02 2016-11-16
109321 Inbound TLS for tier-1 varnish backend caches Traffic open 2017-01-02 2016-09-30
109325 Outbound HTTPS for varnish backend instances Traffic open 2017-01-02 2016-09-30
109331 Deleted files sometimes remain visible to non-privileged users if permanently linked Traffic open 2017-01-02 2017-07-27
109776 Tilerator should purge Varnish cache Traffic open 2017-01-02 2017-03-08
111588 RFC: API-driven web front-end Traffic open 2017-01-02 2017-09-12
111899 Deprecate pybal SSH health checks Traffic open 2017-01-02 2016-09-30
112316 Configure varnish to use "Unconfigured domain" page for 404 Not Served (instead of generic error) Traffic open 2017-01-02 2016-09-30
112521 Track/notify cert expiries better Traffic open 2017-01-02 2016-09-30
112765 Phabricator needs to expose notification daemon (websocket) Traffic open 2017-01-02 2016-11-15
113597 pybal-related issue on host start can break service IPs... Traffic open 2017-01-02 2017-02-10
113817 Connect Hadoop records of the same request coming via different channels Traffic open 2017-01-02 2016-10-04
114104 pybal doesn't fully manage LVS table leaving stale services (on IP change) Traffic open 2017-03-29 2017-03-30
114979 Run IPVS in a separate network namespace Traffic open 2017-01-02 2017-09-18
116132 missing H2 coalesce for upload.wm.o for images ref'd in projects' page outputs Traffic open 2017-01-02 2016-11-15
117826 TEST: redirect small portion of unauthenticated desktop users to mobile web Traffic open 2017-01-02 2017-10-02
118181 Planning for phasing out non-Forward-Secret TLS ciphers Traffic open 2017-01-02 2017-05-08
118468 point wikilovesmonuments.org ns to wmf Traffic open 2017-01-02 2016-09-30
119038 Image cache issue when 'over-writing' an image on commons Traffic open 2017-01-02 2016-04-28
119366 Disable caching on the main page for anonymous users Traffic open 2017-02-13 2017-03-02
119372 Pybal IdleConnectionMonitor with TCP KeepAlive shows random fails if more than 100 servers are involved. Traffic open 2017-01-02 2017-03-30
119396 Create globally-unique varnish cache cluster port/instancename mappings Traffic open 2017-01-02 2016-10-04
120121 Improve Varnish XFF processing for trusted proxies Traffic open 2017-01-02 2016-10-04
120486 add a https-only option to dynamicproxy Traffic open 2017-01-02 2017-10-15
120509 Cache education dashboard pages Traffic open 2017-01-02 2016-09-30
120631 Security: Is it safe to enable Zero spoofing Traffic open 2017-01-02 2017-07-20
122867 Evaluate the feasibility of cache invalidation for the action API Traffic open 2017-01-02 2017-07-19
123854 Set up action API latency / error rate metrics & alerts Traffic open 2017-01-02 2017-07-11
124418 Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan Traffic open 2017-01-02 2017-09-14
124954 Decrease max object TTL in varnishes Traffic open 2017-01-02 2017-07-11
125170 Internal DNS resolver responds with NXDOMAIN for localhost AAAA Traffic open 2017-01-02 2016-09-30
125226 [feature request] Redirect root API path to docs page Traffic open 2017-06-15 2017-06-14
125938 PHP fatal errors causing Varnish to return 503 - "Junk after gzip data" Traffic open 2017-01-02 2017-05-16
127387 Split slash decoding from general percent normalization in Varnish VCL Traffic open 2017-01-02 2017-10-17
127482 Enable VCL source-DC switching via confd Traffic open 2017-01-02 2016-09-30
127485 Enable VCL applayer datacenter-switch via confd Traffic open 2017-01-02 2016-09-30
127573 wikiknihy.cz - transfer to Wikimedia Czech Republic? Traffic open 2017-01-02 2016-10-04
128182 Server certificate is classified as invalid on government computers Traffic open 2017-01-02 2016-09-30
128188 Make CI run Varnish VCL tests Traffic open 2017-01-02 2016-09-30
128358 Uploading 1.2GB ogv results in 503 Traffic open 2017-01-02 2016-12-13
128374 Sort out analytics service dependency issues for cp* cache hosts Traffic open 2017-01-02 2017-10-02
128409 Detect tools.wmflabs.org tools which are HTTP-only Traffic open 2017-01-02 2017-10-15
128559 store.wikimedia.org HTTPS issues Traffic open 2017-01-02 2017-07-11
129682 Look into solutions for replaying traffic to testing environment(s) Traffic open 2017-01-02 2017-07-12
129839 restrict upload cache access for private wikis Traffic open 2017-01-02 2016-09-30
130904 Host rewrite for /static/ not applied to purges Traffic open 2017-01-02 2016-09-30
131894 Collect Backend-Timing in Graphite (or Prometheus) Traffic open 2017-01-02 2017-09-20
131930 Set SPF (... -all) for toolserver.org Traffic open 2017-01-02 2017-06-14
132629 Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense Traffic open 2017-01-02 2017-07-21
133149 Move californium to an internal host? Traffic open 2017-01-02 2016-10-04
133178 RESTBase support for www.wikimedia.org missing Traffic open 2017-01-02 2017-07-11
133410 Deploy TemplateStyles to WMF production Traffic open 2017-01-02 2017-09-19
133548 Create a secure redirect service for large count of non-canonical / junk domains Traffic open 2017-01-02 2017-07-14
133717 Letsencrypt all the prod things we can - planning Traffic open 2017-01-02 2017-01-18
133821 Content purges are unreliable Traffic open 2017-01-02 2017-10-13
133895 Varnish configuration for mobile domains should be coherent with Apache configuration Traffic open 2017-01-02 2016-09-30
134323 confctl: give regexen more freedom Traffic open 2017-01-02 2016-10-04
134324 confctl select needs a -y flag? Traffic open 2017-01-02 2016-10-04
134404 Varnish support for active:active backend services Traffic open 2017-01-02 2016-10-03
134447 letsencrypt puppetization: upgrade for scalability Traffic open 2017-01-02 2016-09-30
134807 Replace test hostnames in datecenter-specific subdomains with dashed names Traffic open 2017-01-02 2016-10-03
134893 Unhandled pybal error causing services to be depooled in etcd but not in lvs Traffic open 2017-01-02 2017-01-26
135762 A/B Testing solid framework Traffic open 2017-01-02 2017-09-19
136703 Add LVS public endpoint checks that bypass caches Traffic open 2017-01-02 2016-09-30
136737 Fix lvs1001-6 storage Traffic open 2017-01-02 2017-07-19
136944 Set up LVS connection sync Traffic open 2017-01-02 2016-09-30
137252 Redirect phabricator.mediawiki.org to phabricator.wikimedia.org Traffic open 2017-01-02 2016-10-07
137747 Parametrization of VCL is inconsistent Traffic open 2017-01-02 2016-09-30
137979 Support brotli compression Traffic open 2017-01-02 2017-10-10
137990 Zero: Investigate removing the limit on carrier tagging to m-dot and zero-dot requests Traffic open 2016-06-17 2017-07-17
138093 Investigate query parameter normalization for MW/services Traffic open 2017-01-02 2017-07-12
138546 Backend naming in VCL needs to use fqdn+port Traffic open 2017-01-02 2016-09-30
138591 Backport iproute2 4.x from debian testing -> our jessie Traffic open 2017-01-02 2017-07-27
140365 Lower geodns TTLs from 600 to 300 Traffic open 2017-01-02 2016-09-30
141266 letsencrypt puppetization: add parallel rsa+ecdsa cert support Traffic open 2017-01-02 2016-09-30
141373 Age header reset to 0 after 24 hours on varnish frontends Traffic open 2017-01-02 2016-09-30
141480 mixed-content issues on planet.wikimedia.org Traffic open 2017-01-02 2016-10-04
143562 High number of failed inbound TFO connections in esams Mon-Fri Traffic open 2017-01-02 2016-10-04
144187 Better handling for one-hit-wonder objects Traffic open 2017-01-02 2016-09-30
144194 Varnish-triggered CN campaign about browser security Traffic open 2017-01-02 2016-10-13
144508 Point wikipedia.in to 205.147.101.160 instead of URL forward Traffic open 2017-01-02 2016-10-04
144626 Strong cipher preference ordering for cache terminators Traffic open 2017-01-02 2016-11-03
146332 Create short link for outreachdashboard.wmflabs.org Traffic open 2017-03-01 2017-02-28
146619 DNS domains registered to WMF no longer redirecting Traffic open 2017-01-02 2016-09-30
146832 Clarify caching to enable direct Wikidata Query Service access by <mapframe/link> Traffic open 2017-01-02 2016-12-15
147199 Removing support for DES-CBC3-SHA TLS cipher (drops IE8-on-XP support) Traffic open 2017-01-02 2017-10-19
147202 Removing support for AES128-SHA TLS cipher Traffic open 2017-01-02 2016-10-03
147209 etcd cluster has Raft Internal errors sporadically Traffic open 2017-01-02 2016-10-04
147648 Unexplained increase in thumbnail 500s Traffic open 2017-01-02 2016-11-18
147844 Standardize varnish applayer backend definitions Traffic open 2017-01-02 2016-10-11
147967 The WMF-Last-Access Set-Cookie header should follow RFC 2965 syntax rather than the pre-RFC Netscape format Traffic open 2017-01-02 2016-10-17
148131 Deploy redundant unified certs Traffic open 2017-01-02 2017-06-26
148134 OCSP Stapling for Intermediates Traffic open 2017-01-02 2016-11-01
148422 cp3009: memory scrubbing error Traffic open 2017-01-02 2017-10-03
148595 Visting [[c:File:FEZ_trial_gameplay_HD.webm]] in IE11 shows errors in developer console about insecure data:image/png;base64 "URL" Traffic open 2017-01-02 2016-10-19
148830 cache_upload: uncompressed images with Content-Encoding: gzip cause content decoding issues Traffic open 2017-01-02 2017-04-10
148976 Strongswan Icinga check: do not report issues about depooled hosts Traffic open 2017-01-02 2016-10-24
149847 Use content hash based image / thumb URLs Traffic open 2017-01-02 2017-10-13
150022 thumb_handler.php should not set CC:no-cache on renderer 404 responses? Traffic open 2017-01-02 2017-03-07
150479 Prometheus varnish metric churn due to VCL reloads Traffic open 2017-01-02 2017-03-16
150673 Thumb API: Varnish / CDN questions Traffic open 2017-01-02 2017-08-16
152091 Block hotlinking Traffic open 2017-01-02 2016-12-05
152622 Wikipedia.cz and other domains owned by WMCZ have invalid certificate Traffic open 2017-01-02 2017-07-14
152882 Many misc wikis lack mobile domains Traffic open 2017-01-02 2017-07-25
153468 Ferm/DNS library weirdness on deployment-mediawiki boxes Traffic open 2017-01-02 2017-09-08
153563 Consider switching to HTTPS for Wikidata query service links Traffic open 2017-01-02 2017-10-11
154015 announce breaking change: http > https for entities in rdf Traffic open 2017-01-02 2017-07-25
154017 compile number of http uses for http://www.wikidata.org/entity Traffic open 2017-01-02 2017-07-25
154702 Fix broken referer categorization for visits from Safari browsers Traffic open 2017-01-06 2017-09-01
154801 Investigate varnishd child crashes when multiple nodes get depooled/pooled concurrently Traffic open 2017-01-07 2017-02-07
155314 Varnish does not cache Action API responses when logged in Traffic open 2017-01-15 2017-07-19
155806 Add CAA records to our domains Traffic open 2017-01-21 2017-05-30
156026 Enable Service in Asia Cache DC Traffic open 2017-01-24 2017-01-23
156027 Configuration for Asia Cache DC hosts Traffic open 2017-01-24 2017-10-19
156031 Turn up network links for Asia Cache DC Traffic open 2017-01-24 2017-04-11
156032 Server hardware installation for Asia Cache DC Traffic open 2017-01-24 2017-01-23
156033 Server hardware purchasing for Asia Cache DC Traffic open 2017-01-24 2017-04-20
156256 Allocate address space for Singapore (APNIC) Traffic open 2017-01-26 2017-10-20
156320 $wgServer with initial https:// does not force HTTPS (wgSecureLogin) Traffic open 2017-01-27 2017-04-28
157786 Unhandled error stopping pybal: 'RunCommandMonitoringProtocol' object has no attribute 'checkCall' Traffic open 2017-02-11 2017-02-10
158599 Samsung Internet's desktop mode getting redirected to mobile site Traffic open 2017-02-21 2017-07-19
159056 cp2017 froze and stopped serving traffic Traffic open 2017-02-26 2017-02-27
159137 certspotter: Error retrieving STH from log Traffic open 2017-02-28 2017-02-28
159346 convert mail servers from GS to LE certificates Traffic open 2017-03-02 2017-03-02
159411 Uniform cluster nomenclature across puppet Traffic open 2017-03-03 2017-03-26
159412 Convert all of our site.pp/roles to the role/profile paradigm Traffic open 2017-03-03 2017-03-26
159429 Allow setting varnish connection timeouts in puppet Traffic open 2017-03-03 2017-03-07
161148 AuthDNS CM/CI refactor Traffic open 2017-03-23 2017-03-24
161256 multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert Traffic open 2017-03-24 2017-03-28
161360 404 loading images from Virgin Media Traffic open 2017-03-25 2017-03-28
161517 Allow anonymous users to change interface language on Commons with ULS Traffic open 2017-03-30 2017-07-25
162099 lvs2002 random shut down Traffic open 2017-04-04 2017-04-07
162683 Network hardware purchasing for Asia Cache DC Traffic open 2017-04-12 2017-04-17
162684 Network hardware configuration for Asia Cache DC Traffic open 2017-04-12 2017-04-11
162818 icinga alerts on nodejs services when a recdns server is depooled Traffic open 2017-04-13 2017-06-27
163141 dbtree: make wasat a working backend and become active-active Traffic open 2017-04-18 2017-07-25
163251 Communicate dropping IE8-on-XP support (a security change) to affected editors and other community members Traffic open 2017-04-19 2017-10-04
163541 cache hosts should auto-repool iff OCSP files are sane Traffic open 2017-04-22 2017-04-21
164259 Add VSL error counters to Varnishkafka stats Traffic open 2017-05-03 2017-08-04
164327 replace ulsfo aging servers Traffic open 2017-05-03 2017-10-19
164376 [Discuss] Split ORES scores in datacenters based on wiki Traffic open 2017-05-04 2017-05-11
164456 Build nginx without image filter support Traffic open 2017-05-05 2017-05-09
164609 Merge cache_misc into cache_text functionally Traffic open 2017-05-06 2017-06-27
164768 Explicitly limit varnishd transient storage Traffic open 2017-05-09 2017-06-20
164868 SSL error for https://wikispecies.org/ Traffic open 2017-05-10 2017-06-20
165252 cp1053 possible hardware issues Traffic open 2017-05-16 2017-07-20
165560 Artificial spike in offset of unique devices from November to February 6th on wikidata Traffic open 2017-05-18 2017-06-26
165651 Collect Google IPs pinging the load balancers Traffic open 2017-05-19 2017-05-18
165764 Fully-redundant LVS clusters using Pybal per-service MED feature Traffic open 2017-05-20 2017-09-19
165765 Refactor pybal/LVS config for shared failover Traffic open 2017-05-20 2017-06-01
166782 wikimediafoundation.org's language selector is confusing to most visitors who don't have accounts there Traffic open 2017-06-02 2017-08-24
166965 Degraded RAID on lvs3001 Traffic open 2017-06-05 2017-07-03
167060 en.wiki domain owned by us, but isn't hosted by us?? Traffic open 2017-06-15 2017-06-14
167400 Disable serving unpatrolled new files to Wikipedia Zero users Traffic open 2017-06-16 2017-09-02
167513 Redirect lzh.wikipedia to zh-classical.wikipedia Traffic open 2017-06-10 2017-06-12
167906 Make API usage limits easier to understand, implement, and more adaptive to varying request costs / concurrency limiting Traffic open 2017-06-15 2017-10-20
167972 Respect host header in RESTBase, and redirect /rest_v1 to /rest_v1/ Traffic open 2017-06-16 2017-06-20
168529 Upgrade to Varnish 5 Traffic open 2017-06-22 2017-10-10
168539 Unhandled pybal error: OpenSSL.SSL.Error - ssl handshake failure Traffic open 2017-06-22 2017-06-22
168699 Verify that the codfw lvs is configured correctly for Phabricator Traffic open 2017-06-24 2017-08-08
169765 pybal should automatically reconnect to etcd Traffic open 2017-07-06 2017-07-18
170518 Non zero rated LVS IPs Traffic open 2017-07-14 2017-08-14
170567 Support TLSv1.3 Traffic open 2017-07-14 2017-07-13
170605 ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION Traffic open 2017-07-14 2017-08-09
170847 Icinga check for pybal HTTP connections to etcd Traffic open 2017-07-18 2017-07-18
171168 cp1050 apparently stuck while "Initializing firmware interfaces..." Traffic open 2017-07-21 2017-07-25
171470 Monitor DNS delegations Traffic open 2017-07-25 2017-07-24
171498 Implement machine-local forwarding DNS caches Traffic open 2017-07-25 2017-09-13
171850 Backport ipvsadm Traffic open 2017-07-28 2017-10-11
171966 setup/install cp402[34] Traffic open 2017-07-29 2017-08-18
171967 setup/install cp4022 Traffic open 2017-07-29 2017-08-18
172103 IPVS issues with UDP services, pybal depooling strategy Traffic open 2017-08-01 2017-08-01
172116 Improve OCSP fetching and monitoring strategies Traffic open 2017-08-01 2017-08-01
172123 Determine how to upload Zim files to Swift infrastructure Traffic open 2017-08-01 2017-08-12
172124 PyBal Feature: progressive depooling strategy for monitored failures Traffic open 2017-08-01 2017-09-06
172148 Determine URL paths for Zim files Traffic open 2017-08-01 2017-09-12
172198 setup/install cp402[5-8].ulsfo.wmnet Traffic open 2017-08-02 2017-08-21
173966 Like nan.wikipedia.org, redirect other nan.*.org to the proper zh-min-nan.*.org domains Traffic open 2017-08-24 2017-08-23
174342 Missing IP addresses for Maroc Telecom Traffic open 2017-08-29 2017-09-21
174432 Unclear LVS bandwidth graph in "load balancers" dashboard Traffic open 2017-08-30 2017-08-30
174891 cp4024 kernel errors Traffic open 2017-09-04 2017-10-17
174960 Varnish does not vary elasticsearch query by request body Traffic open 2017-09-05 2017-09-07
175203 Implement stateless TCP balancing in our LVS servers Traffic open 2017-09-07 2017-09-07
175319 cp1066 unexplained 503 spikes Traffic open 2017-09-09 2017-09-19
175636 prometheus -> grafana stats for per-numa-node meminfo Traffic open 2017-09-12 2017-10-02
176366 Decom cp4005-8,13-16 (8 nodes) Traffic open 2017-09-21 2017-10-17
176388 pybal: race condition in alerts instrumentation Traffic open 2017-09-22 2017-09-28
176875 Allow access to wdqs.svc.eqiad.wmnet on port 8888 Traffic open 2017-09-28 2017-10-13
176905 Evaluate requesting a rate limit change from Letsencrypt Traffic open 2017-09-28 2017-09-28
177199 Add Prometheus client support for varnish/statsd metrics daemons Traffic open 2017-10-03 2017-10-10
177228 Multiple systems in esams OE10 showing PSU failures Traffic open 2017-10-03 2017-10-10
177927 Refactor kafka_config.rb and and kafka_cluster_name.rb in puppet to avoid explicit hiera calls Traffic open 2017-10-12 2017-10-13
177961 Upgrade LVS servers to stretch Traffic open 2017-10-12 2017-10-13
178151 Add UDP monitor for pybal Traffic open 2017-10-14 2017-10-13
178173 Renew unified certificates 2017 Traffic open 2017-10-14 2017-10-18
178423 rack/setup/install cp40(29|3[012]).ulsfo.wmnet Traffic open 2017-10-18 2017-10-19
178436 rack/setup/install lvs400[567].ulsfo.wmnet Traffic open 2017-10-18 2017-10-18
178535 decommission lvs400[1-4].ulsfo.wmnet Traffic open 2017-10-19 2017-10-18
178567 Server error (500) while trying to download files from Commons from PAWS Traffic open 2017-10-21 2017-10-20
178592 decommission/replace bast4001.wikimedia.org Traffic open 2017-10-20 2017-10-19
178778 Parsoid, VisualEditor not working with SSL / HTTPS Traffic open 2017-10-23 2017-10-22
1147 Project Proposal: Label style projects for common operations tools Traffic resolved 2017-01-02 2016-05-04
2225 Secure login on Wikimedia wikis via SSL Traffic resolved 2017-01-02 2016-08-05
18822 Provide SSL/HTTPS interface to upload.wikimedia.org and use it for SSL-served pages Traffic resolved 2017-01-02 2016-08-05
20496 Non-secure logos on secure pages Traffic resolved 2017-01-02 2016-08-05
22643 Serve SSL/HTTPS sites out of same domain names as HTTP access: https://en.wikipedia.org/ Traffic resolved 2017-01-02 2016-08-05
25559 Secure Server 404 Error Message: Did you mean URL is wrong Traffic resolved 2017-01-02 2016-08-05
26332 OpenSSL on secure.wikimedia.org is possibly vulnerable to CVE-2009-3555 Traffic resolved 2017-01-02 2016-08-05
28971 Thumbnail generation for private wikis on https://secure.wikimedia.org/ is broken Traffic resolved 2017-01-02 2016-08-05
29448 meta:Special:BannerLoader is loaded using http protocol when using secure server (https) Traffic resolved 2017-01-02 2016-08-05
29622 Setup internal wikis as https only Traffic resolved 2017-01-02 2016-08-05
29909 SSL RSA keys should be at least 2048 bits Traffic resolved 2017-01-02 2016-08-05
31014 Disable SSLv2 on secure.wikimedia.org Traffic resolved 2017-01-02 2016-08-05
31878 Resolution of protocol-independent wgServer for email messages should not be dependent on the performer's url protocol Traffic resolved 2017-01-02 2016-08-05
31981 Location: redirects on https sites always point to http sites Traffic resolved 2017-01-02 2016-08-05
32329 SSL termination gateway can't handle large-ish uploads Traffic resolved 2017-01-02 2016-08-05
32560 Bugzilla links to Wikipedia and to MW revisions should use secure server Traffic resolved 2017-01-02 2016-08-05
33323 Mandate only-SSL for accounts with access to private information Traffic resolved 2017-01-02 2016-08-05
33333 No SSL for MobileFrontend m. domain: https://en.m.wikipedia.org/ does not respond Traffic resolved 2017-01-02 2016-08-05
33334 Secure https://en.wikipedia.org/ redirects to insecure http://en.m.wikipedia.org/ on mobile Traffic resolved 2017-01-02 2016-08-05
33335 Rename wikis with multiple subdomains Traffic resolved 2017-01-02 2017-08-23
33539 Use protocol relative URLs on wgUploadNavigationUrl Traffic resolved 2017-01-02 2016-08-05
33563 https://secure.wikimedia.org should redirect to the new same-domain SSL sites Traffic resolved 2017-01-02 2016-08-05
34144 When user is logging out via HTTPS, insecure HTTP cookies keeping logged in state should be cleared as well Traffic resolved 2017-01-02 2016-05-31
34769 Notification emails should link to https, not http Traffic resolved 2017-01-02 2016-08-05
34796 status.wikimedia.org has no (valid) HTTPS Traffic resolved 2017-01-02 2016-06-02
35890 Support SPDY Traffic resolved 2017-01-02 2016-08-05
38456 Native "force https" option Traffic resolved 2017-01-02 2016-08-05
39790 shop.wikimedia.org should be HTTPS only Traffic resolved 2017-01-02 2016-03-02
40516 Enable HSTS on Wikimedia sites Traffic resolved 2017-01-02 2016-06-14
41380 Set $wgSecureLogin = true; on Wikimedia wikis Traffic resolved 2017-01-02 2016-08-05
45466 when serving Uzbek Wikipedia, make HTTPS canonical Traffic resolved 2017-01-02 2016-05-29
49832 Force all Wikimedia cluster traffic to be over SSL for all users (logged-in and anon) Traffic resolved 2017-01-02 2016-08-05
50402 rel=canonical of https pages should point to http Traffic resolved 2017-01-02 2016-08-05
51467 Server at http://upload.wikimedia.org isn't HTTP/1.1 compliant: doesn't allow absolute URI in request Traffic resolved 2017-01-02 2015-09-22
51698 When uploading from Flickr, unencrypted XHRs are sent from client to Flickr despite being on https:// Traffic resolved 2017-01-02 2016-08-05
53002 Point rel=canonical to HTTPS on every page Traffic resolved 2017-01-02 2016-08-05
54302 Varnish vagrant role Traffic resolved 2017-01-02 2016-05-04
55526 Recent Changes IRC feed should link to the HTTPS diff Traffic resolved 2017-01-02 2016-08-05
63443 Enable gzip transfer encoding for png and ico files in /static Traffic resolved 2017-01-02 2016-05-04
63528 shop switches HTTPS -> HTTP when showing login prompt (on clicking checkout) Traffic resolved 2017-01-02 2016-04-04
70528 stream.wikimedia.org - redirect http(s) to docs Traffic resolved 2017-01-02 2017-07-19
70861 Performance review #2 of Hovercards (Popups extension) Traffic resolved 2017-01-02 2017-05-10
73267 Enable IPv6 on donate.wikimedia.org Traffic resolved 2017-01-02 2015-09-22
74072 Disable SSL 3.0 on Wikimedia sites to mitigate POODLE attack (CVE-2014-3566) Traffic resolved 2017-06-08 2017-06-07
74514 links.email.donate.wikimedia.org should offer HTTPS Traffic resolved 2017-01-02 2016-03-18
75943 Investigate Varnish 3.0.5plus~x-wm7 throwing on beta cluster "Could not mmap SILO" Traffic resolved 2017-01-02 2016-05-04
76562 Chrome warns about insecure certificate on gerrit.wikimedia.org Traffic resolved 2017-01-02 2016-08-05
82096 update the multicast purging documentation Traffic resolved 2017-01-02 2016-01-26
82747 pybal health checks are ipv4 even for ipv6 vips Traffic resolved 2017-01-02 2017-07-18
84980 Varnish: the lower the Age value, the slower the request Traffic resolved 2017-01-02 2016-05-31
86063 HTTPS phabricator project(s) Traffic resolved 2017-01-02 2016-07-18
86654 Switch to ECDSA hybrid certificates Traffic resolved 2017-01-02 2015-07-13
86664 HTTPS performance & UA adoption metrics Traffic resolved 2017-01-02 2015-08-27
86666 HTTPS performance tuning Traffic resolved 2017-01-02 2015-08-07
86718 Upgrade eqiad-misc varnish cluster from 2 to 4 systems. Traffic resolved 2017-01-02 2015-08-07
86993 Static image files from en.m.wikipedia.org are served with cache-suppressing headers Traffic resolved 2017-01-02 2016-01-13
87276 Set an explicit "Origin When Cross-Origin" referer policy via the meta referrer tag Traffic resolved 2017-01-02 2017-05-16
87465 create a project for tasks related to WMF domain names Traffic resolved 2017-01-02 2016-07-18
88366 Some traffic is not identified as Zero in Varnish Traffic resolved 2017-01-02 2016-05-31
88862 Enforce HTTPS for authenticated public connections Traffic resolved 2017-01-02 2016-07-12
88873 register .wiki gTLD domains Traffic resolved 2017-01-02 2017-06-14
89120 Upgrade eqiad LVS to 10G Traffic resolved 2017-01-02 2015-07-07
89688 Varnish GeoIP is broken for HTTPS+IPv6 traffic Traffic resolved 2017-01-02 2016-08-18
91347 EventLogging query strings are truncated to 1014 bytes by ?(varnishncsa? or udp packet size?) Traffic resolved 2017-01-02 2016-06-14
91612 acquire SSL certificate for w.wiki Traffic resolved 2017-01-02 2015-12-11
92435 Code changes in VCL to add a Last-Access cookie to track "top domain" uniques [13 pts] {bear} Traffic resolved 2017-01-02 2015-04-27
92438 Changing the URL for the Wikimedia Shop Traffic resolved 2017-06-08 2017-06-07
92602 Secure inter-datacenter web request log (Kafka) traffic Traffic resolved 2017-01-02 2016-01-13
92709 wikitech.wikimedia.org SSL certificate considered "outdated security" in Chrome Traffic resolved 2017-06-08 2017-06-07
93927 Make OCSP Stapling support more generic and robust Traffic resolved 2017-01-02 2016-11-15
94896 Optimize prod's resource domains for SPDY/HTTP2 Traffic resolved 2017-01-02 2017-03-27
95229 Set up an API base path for REST and action APIs Traffic resolved 2017-01-02 2015-08-05
95448 Move bits traffic to text/mobile clusters Traffic resolved 2017-01-02 2017-03-27
96375 Upgrade codfw,ulsfo,esams LVS to jessie Traffic resolved 2017-01-02 2015-09-24
96688 Proxying new services through RESTBase Traffic resolved 2017-01-02 2016-10-12
96847 Refactor varnish puppet config Traffic resolved 2017-01-02 2016-02-19
96848 Support HTTP/2 Traffic resolved 2017-01-02 2017-03-27
96850 Test then switch to openssl 1.0.2 + nginx 1.9.2 Traffic resolved 2017-01-02 2015-06-28
96854 Reboot caches for kernel 3.19.6 globally Traffic resolved 2017-01-02 2015-05-26
96873 reinstall/rename dysprosium as cp1099 (upload eqiad) Traffic resolved 2017-01-02 2015-05-27
97029 integrate (pybal|varnish)->varnish backend config/state with etcd or similar Traffic resolved 2017-01-02 2015-11-30
97206 RFC: Re-evaluate varnish-level request-restart behavior on 5xx Traffic resolved 2017-01-02 2016-05-22
97316 review/rebase/merge the final sslcert patch... Traffic resolved 2017-01-02 2015-06-02
97351 Fix annoying varnishncsa+initsystem issues on jessie Traffic resolved 2017-01-02 2016-05-31
97411 Build a non-trunk 3.19 kernel for jessie Traffic resolved 2017-01-02 2015-06-03
97506 Use OCSP Stapling on misc cluster Traffic resolved 2017-01-02 2015-05-20
97750 wmgUseBits = false still loads favicon from bits.wikimedia.org Traffic resolved 2017-01-02 2015-05-03
97970 Package a modern version of etcd for jessie, trusty Traffic resolved 2017-01-02 2015-08-07
97972 Figure out a security model for etcd Traffic resolved 2017-01-02 2016-01-13
97973 Create an etcd puppet module + find suitable servers for deployment Traffic resolved 2017-01-02 2015-06-11
97974 Create a confd puppet module Traffic resolved 2017-01-02 2015-06-16
97975 Integrate confd into the varnish configuration to generate the list of active backends Traffic resolved 2017-01-02 2015-07-01
97978 Create a tool to sync static configuration from a repository to the consistent k/v store Traffic resolved 2017-01-02 2015-06-16
98003 Upgrade prod DNS daemons to gdnsd 2.2.0 Traffic resolved 2017-01-02 2015-08-07
98203 Fix cpufrequtils issues on jessie Traffic resolved 2017-01-02 2015-05-26
98309 Mobile site broken Traffic resolved 2017-01-02 2015-05-06
98373 Replace bits URL in Firefox app, if possible Traffic resolved 2017-01-02 2016-08-03
98758 Upgrade beta-cluster caches to jessie Traffic resolved 2017-01-02 2016-01-11
98803 Varnish does not honor Cache-Control for Graphoid Traffic resolved 2017-01-02 2015-05-14
99096 Make Varnish cache for /static/$wmfbranch/ expire when resources change within branch lifetime Traffic resolved 2017-01-02 2017-04-05
99226 Switch Varnish's GeoIP code to libmaxminddb/GeoIP2 Traffic resolved 2017-01-02 2016-08-24
99372 Graphoid access logs are missing from Hadoop Traffic resolved 2017-01-02 2016-08-11
99839 Sanitize varnish director-level retries Traffic resolved 2017-01-02 2015-05-21
100519 Phabricator needs to expose ssh Traffic resolved 2017-01-02 2016-08-18
100902 Get rid of geoiplookup service Traffic resolved 2017-01-02 2017-08-19
101060 wikipedia.es Traffic resolved 2017-01-02 2016-09-02
101204 setup/install/deploy cp2001-cp2026 Traffic resolved 2017-01-02 2015-06-19
101206 rack & on-site setup of cp2001-cp2026 Traffic resolved 2017-01-02 2015-06-19
101435 SPDY traffic isn't gzipped for Firefox Traffic resolved 2017-01-02 2015-06-08
101858 Create a confd template for pybal files that will work with our etcd schema. Traffic resolved 2017-01-02 2015-06-29
101892 Ensure {text-domain}/w/load.php requests do not bypass cache for session cookies Traffic resolved 2017-01-02 2015-06-28
102393 Make pybal accept 30[12] for ProxyFetch Traffic resolved 2017-01-02 2015-11-30
102394 Implement pybal pool state monitoring and alerting via icinga Traffic resolved 2017-01-02 2016-02-09
102457 Make Magnus tools on tools.wmflabs.org work in HTTPS Traffic resolved 2017-01-02 2017-01-29
102814 Decom old multiple-subdomain wikis in wikipedia.org Traffic resolved 2017-01-02 2015-09-14
102815 Decom www.$lang hostnames/redirects Traffic resolved 2017-01-02 2015-09-14
102824 Clean up DNS/redirects for TLS Traffic resolved 2017-01-02 2016-07-27
102826 Fix/decom multiple-subdomain wikis in wikimedia.org Traffic resolved 2017-01-02 2015-09-22
102827 Decide what to do with *.donate.wikimedia.org subdomain + TLS Traffic resolved 2017-01-02 2016-04-12
102991 Verify traffic to static resources from past branches does indeed drain Traffic resolved 2017-01-02 2016-01-13
103226 eqiad: investigate thermal issues with some cp10xx machines Traffic resolved 2017-01-02 2016-01-29
103660 Migrate Parsoid cache from Trusty to Jessie Traffic resolved 2017-01-02 2016-05-04
103773 check if services behind misc-web enforce http->https redirect or not Traffic resolved 2017-01-02 2015-06-28
103919 let all services on misc-web enforce http->https redirects Traffic resolved 2017-01-02 2016-04-13
103921 pybal DNS lookup issues causing outage risks Traffic resolved 2017-01-02 2017-07-04
104004 Can't download large datasets from datasets.wikimedia.org Traffic resolved 2017-01-02 2015-12-11
104076 Puppet failing on deployment-prep caches Traffic resolved 2017-01-02 2015-09-12
104143 openssl 1.0.2 packaging for jessie Traffic resolved 2017-01-02 2015-07-01
104211 Renew toolserver.org SSL certificate Traffic resolved 2017-06-08 2017-06-07
104225 enwiki Main_Page timeouts Traffic resolved 2017-01-02 2017-07-22
104244 Preload HSTS Traffic resolved 2017-01-02 2016-06-06
104274 Switch to explicit ciphersuite Traffic resolved 2017-01-02 2015-06-30
104281 Sort out DHE for Forward Secrecy w/ older clients Traffic resolved 2017-01-02 2015-07-07
104359 https://wikitech.wikimedia.org/beacon/statsv 404 Not Found Traffic resolved 2017-01-02 2016-05-04
104532 /static generates (and caches!) redirect loops on cache-miss Traffic resolved 2017-01-02 2016-05-04
104728 make blog links from wmfwiki front page use HTTPS links Traffic resolved 2017-01-02 2016-08-19
104936 move annual report from zirconium to bromine Traffic resolved 2017-01-02 2015-09-24
104937 move transparency report from zirconium to bromine Traffic resolved 2017-01-02 2015-07-13
104942 TLS and *.wap/*.mobile multi-level subdomains of wikipedia.org Traffic resolved 2017-01-02 2015-08-14
105076 Set up standard HTTPS Termination -> 2layer caching for maps service Traffic resolved 2017-01-02 2015-08-27
105455 Xbox 360 Internet Explorer unable to view Wikipedia Traffic resolved 2017-01-02 2016-09-16
105734 Varnish caches Grafana dashboard configuration too strongly Traffic resolved 2017-01-02 2015-09-22
105794 Insecure POST traffic Traffic resolved 2017-01-02 2016-09-23
105905 Switch blog to HTTPS-only Traffic resolved 2017-01-02 2017-04-06
105999 Redirect yue.wikipedia.org to zh-yue.wikipedia.org Traffic resolved 2017-01-02 2017-06-09
106620 Figure out purging of static logos for updates Traffic resolved 2017-01-02 2016-05-04
106662 deployment-parsoidcache02 fails varnish VCL compilation Traffic resolved 2017-01-02 2016-05-04
107030 Restbase insecure POST requests to MW api.php Traffic resolved 2017-01-02 2015-08-07
107060 DNS request for wikimedia.org Traffic resolved 2017-01-02 2015-08-07
107086 Provide an API listing at /api/ Traffic resolved 2017-01-02 2016-02-29
107430 Decom bits.wikimedia.org hostname Traffic resolved 2017-01-02 2017-04-06
107575 download.wiki[mp]edia.org are using an invalid certificate Traffic resolved 2017-01-21 2017-01-20
107602 Set up a public interface to the wikidata query service Traffic resolved 2017-01-02 2015-09-22
107627 Quarry should be HTTPS-only Traffic resolved 2017-01-02 2016-05-03
107940 DNS request for wikimedia.org (let 3rd party send mail as wikimedia.org) Traffic resolved 2017-01-02 2015-08-27
108579 Getting ssl_error_inappropriate_fallback_alert very rarely Traffic resolved 2017-01-02 2016-05-31
108612 Fix Varnish TTLs across the board Traffic resolved 2017-01-02 2015-09-22
108649 Set up "w.wiki" domain for usage with UrlShortener Traffic resolved 2017-01-02 2015-12-11
108827 Investigate TCP Fast Open for tlsproxy Traffic resolved 2017-01-02 2016-08-15
109162 Set up proper edge Varnish caching for maps cluster Traffic resolved 2017-01-02 2016-04-26
109226 Sometimes apache error 503s redirect to /503.html and this redirect gets cached Traffic resolved 2017-01-02 2016-10-25
109286 Merge mobile cache into text cache Traffic resolved 2017-01-02 2016-02-08
109712 Invalid OCSP signing certificate in OCSP response : can't read Wikimedia websites on Firefox Traffic resolved 2017-01-02 2015-08-27
109737 ocsp updater: handle openssl "trylater" and similar more-gracefully Traffic resolved 2017-01-02 2016-10-03
109738 ocsp updater: validate the signature expiry lifetime Traffic resolved 2017-01-02 2016-10-03
109740 ocsp updater: re-enable automatic updates Traffic resolved 2017-01-02 2015-08-27
110065 Switch codfw caches to tier2, begin pushing some traffic through them to test Traffic resolved 2017-01-02 2015-09-22
110197 SSL certificate for policy.wikimedia.org Traffic resolved 2017-01-02 2016-07-28
110472 Decom parsoidcache cluster Traffic resolved 2017-01-02 2016-04-19
110475 Remove restbase from parsoidcache Traffic resolved 2017-01-02 2016-01-29
110476 Remove citoid from parsoidcache Traffic resolved 2017-01-02 2016-01-29
110477 Remove graphoid from parsoidcache Traffic resolved 2017-01-02 2016-01-28
110478 Remove cxserver from parsoidcache cluster Traffic resolved 2017-01-02 2016-01-29
110530 Fix ethernet startup race on HP LVS w/ jessie Traffic resolved 2017-01-02 2015-12-11
110899 figure out what's wrong with confd + varnish currently Traffic resolved 2017-01-02 2015-09-22
110954 Upgrade Pybal to 1.08 Traffic resolved 2017-01-02 2017-03-08
111045 zerofetcher in production is getting throttled for API logins Traffic resolved 2017-01-02 2015-11-30
111715 ru.wikinews.org Parsoid backend is down Traffic resolved 2017-01-02 2015-09-22
112195 Beta giving Error: 403, Insecure POST Forbidden Traffic resolved 2017-01-02 2015-09-22
112836 allow maps cluster Varnish cache purging Traffic resolved 2017-01-02 2016-05-26
113007 [Research] Investigate 30% load.php reqs increase since 2015-07-30 Traffic resolved 2017-01-02 2016-08-19
113122 maps: support wikivoyages in incubator Traffic resolved 2017-01-02 2015-10-26
113151 pybal fails to detect dead servers under production lb IPs for port 80 Traffic resolved 2017-01-02 2016-02-09
113154 LVS HTTPS IPv6 on mobile-lb.eqiad alert occasionally flapping Traffic resolved 2017-01-02 2015-10-05
113184 cp1046 Varnish backend panic Traffic resolved 2017-01-02 2015-09-25
113223 Varnish not letting through RESTBase back-end service responses for rest.wm.org Traffic resolved 2017-01-02 2016-05-04
113319 Saving preferences or blocking (and probably various other things) give 403 error Traffic resolved 2017-01-02 2015-09-22
113591 Enable caching for the Mobile Content Service's RESTBase public endpoints Traffic resolved 2017-01-02 2016-05-04
113639 cp1046 is crashing and becoming unresponsive Traffic resolved 2017-01-02 2015-10-26
114059 ssl expiry tracking in icinga - we don't monitor that many domains Traffic resolved 2017-01-02 2016-03-23
114370 Flag in x-analytics in varnish any request that comes with no cookies whatsoever {bear} [5 pts] Traffic resolved 2017-01-02 2015-10-26
114659 Re-balance North American traffic in DNS for codfw Traffic resolved 2017-01-02 2016-09-14
114922 Transfer of domain names to WMF servers Traffic resolved 2017-01-02 2016-02-24
114995 [Bug] Varnish cache invalidation not working for m.wikidata.org Traffic resolved 2017-01-02 2016-05-04
115028 SSL cert needed for benefactorevents.wikimedia.org Traffic resolved 2017-01-02 2016-03-09
115191 ml.wikipedia.org not redirecting to mobile site while accessing from a mobile device; many "Error: Module not found" errors Traffic resolved 2017-01-02 2016-12-24
116752 Split HTCP multicast addresses Traffic resolved 2017-01-02 2015-11-30
116806 phabricator.wikimedia.org has no SPF record Traffic resolved 2017-01-02 2016-05-10
116898 On beta cluster varnish stats process points to production statsd Traffic resolved 2017-01-02 2016-08-29
117015 Investigate PyBal dead on lvs1007-12 Traffic resolved 2017-01-02 2015-11-30
117746 cp4007 crashed Traffic resolved 2017-01-02 2015-11-30
118078 Clean up nginx / nginx::ssl classes and usage Traffic resolved 2017-01-02 2016-05-31
118365 Increase request limits for GETs to /api/rest_v1/ Traffic resolved 2017-01-02 2017-06-14
118388 configure https for www.wikilovesmonuments.org Traffic resolved 2017-01-02 2016-09-08
118394 cache_upload should give an informative 404 rather than 403 on req.http.host != upload.wikimedia.org Traffic resolved 2017-01-02 2016-10-04
118428 mobile [[w:en:GrandCentral]] was not purged on edit Traffic resolved 2017-01-02 2016-05-04
118787 releases.wikimedia.org should be https only and have hsts set Traffic resolved 2017-01-02 2017-06-07
118790 Decom observium.wikimedia.org Traffic resolved 2017-01-02 2017-06-07
118830 Backport etcd 2.2 to jessie Traffic resolved 2017-01-02 2016-01-13
118831 Upgrade the production etcd cluster to 2.2 Traffic resolved 2017-01-02 2016-01-13
118833 Upgrade conftool to support credentials form a config file Traffic resolved 2017-01-02 2016-01-13
118834 Upgrade python-etcd to 0.4.2+ Traffic resolved 2017-01-02 2016-01-13
118892 Update CP cookie VCL once HTTP/2 support lands Traffic resolved 2017-01-02 2016-08-05
118948 Pybal 1.12 has issues with executing ipvsadm commands Traffic resolved 2017-01-02 2016-01-13
119144 EventLogging sees too few distinct client IPs {oryx} [8 pts] Traffic resolved 2017-01-02 2017-06-07
119394 Convert misc cluster to 2-layer Traffic resolved 2017-01-02 2015-11-30
119515 Upgrade LVS servers to a 4.3+ kernel Traffic resolved 2017-01-02 2016-02-11
119517 upgrade lvs1001-3 to jessie Traffic resolved 2017-01-02 2015-12-11
119576 Mark cookies from varnish as secure Traffic resolved 2017-01-02 2016-04-19
119917 Set up backend per-IP limits on varnish for WDQS Traffic resolved 2017-01-02 2017-02-22
120143 Investigate if www.m.wikipedia.org needs to stay around Traffic resolved 2017-01-02 2016-02-22
120151 Improve handling of mobile variants in Varnish Traffic resolved 2017-01-02 2016-06-15
121135 Banners fail to show up occassionally on Russian Wikivoyage Traffic resolved 2017-01-02 2016-04-28
122319 ssl certificate replacement: tendril.wikimedia.org (expires 2016-02-15) Traffic resolved 2017-01-02 2016-05-03
122321 ssl certificate replacement: dumps.wikimedia.org (expires 2016-02-26) Traffic resolved 2017-01-02 2016-02-23
122453 Wikimedia servers won't serve images Traffic resolved 2017-01-02 2016-01-13
122455 Varnish leaks memory Traffic resolved 2017-01-02 2016-05-31
122462 Varnish Assert error in VGZ_Ibuf() Traffic resolved 2017-01-02 2016-05-31
122673 Varnish apparently unconditionally varies on session cookies Traffic resolved 2017-01-02 2016-01-13
122880 Evaluate and Test Limited Deployment of Varnish 4 Traffic resolved 2017-01-02 2016-08-01
122881 Install XKey vmod Traffic resolved 2017-01-02 2016-11-28
123431 Investigate if login.m.wikimedia.org needs to stay around Traffic resolved 2017-01-02 2016-03-28
124165 Fix mobile purging Traffic resolved 2017-01-02 2016-01-22
124166 Fix varnish handling of mobile hostname rewriting Traffic resolved 2017-01-02 2016-01-20
124277 Forward-port Varnish 3 patches to Varnish 4 Traffic resolved 2017-01-02 2016-01-29
124279 Forward-port maps VCL to Varnish 4 Traffic resolved 2017-01-02 2016-03-31
124281 Create separate packages for required vmods Traffic resolved 2017-01-02 2016-04-01
124356 Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) Traffic resolved 2017-01-02 2017-07-27
124482 Use Text IP for Mobile hostnames to gain SPDY/H2 coalesce between the two Traffic resolved 2017-01-02 2016-05-04
124711 Varnish failed to provision Traffic resolved 2017-01-02 2016-05-04
124835 Automate and/or better-document varnish ban procedure for operations staff, so it can be accomplished with more speed and confidence in outage conditions Traffic resolved 2017-01-02 2016-01-27
125397 pybal etcd coroutine crashed Traffic resolved 2017-01-02 2016-02-09
125401 3 Varnish cache_upload servers crashed in a short time window Traffic resolved 2017-01-02 2016-05-31
125485 esams cache cluster re-arrangements, early 2016 Traffic resolved 2017-01-02 2016-03-24
125486 eqiad cache cluster re-arrangements Traffic resolved 2017-01-02 2016-03-24
125510 Traffic Infrastructure support for Mar 2016 codfw rollout Traffic resolved 2017-01-02 2016-03-23
125841 Purge pages cached with mobile editlinks Traffic resolved 2017-01-02 2016-04-06
125979 Disable SPDY on cache_text for a week Traffic resolved 2017-01-02 2016-08-05
126015 Understand and improve streaming behaviour from Varnish Traffic resolved 2017-01-02 2016-05-09
126053 Send X-Carrier + X-Carrier-Meta headers on all responses Traffic resolved 2017-01-02 2016-02-29
126062 cp30[34]x hw/firmware/BMC issues Traffic resolved 2017-01-02 2016-05-31
126206 Upgrade to Varnish 4: things to remember Traffic resolved 2017-01-02 2017-04-18
126321 lvs2002 Embedded Flash/SD-CARD iLO errors Traffic resolved 2017-01-02 2017-04-07
126616 openssl-1.0.2f introduced minor bug with nginx Traffic resolved 2017-01-02 2016-02-29
127014 Empty result on a tree query Traffic resolved 2017-01-02 2017-07-14
127021 Stop default redirecting Samsung Smart TVs to mobile web Traffic resolved 2017-01-02 2016-05-04
127094 ulsfo possible downtime - PDU swaps in both cabinets Traffic resolved 2017-01-02 2016-02-18
127294 https://integration.wikimedia.org/ci/api/json is corrupted when required more than one time in a raw Traffic resolved 2017-01-02 2016-02-18
127370 Varnish not purging RESTBase URIs Traffic resolved 2017-01-02 2017-10-17
127436 Logging out of a wiki leaves an XXwikiSession= Cookie behind Traffic resolved 2017-01-02 2016-08-15
127481 Refactor VCL for 3-tier capabilities and source-DC switching Traffic resolved 2017-01-02 2016-03-04
127484 Refactor VCL for applayer datacenter-switching Traffic resolved 2017-01-02 2016-03-07
127492 Switch ulsfo to backend to codfw rather than eqiad Traffic resolved 2017-01-02 2016-03-08
127931 Number of 5xx (mainly 503) errors surge from text caches since February 17th Traffic resolved 2017-01-02 2016-02-24
128085 figure out if we can park wicipediacymraeg.org Traffic resolved 2017-01-02 2016-03-01
128199 confctl: improve/upgrade --tags/--find Traffic resolved 2017-01-02 2016-04-28
128587 http://dumps.wikimedia.org should redirect to https:// Traffic resolved 2017-01-02 2016-04-08
128709 Segment Navigation Timing data by continent Traffic resolved 2017-01-02 2017-05-26
128788 Port varnishlog.py to new VSL API Traffic resolved 2017-01-02 2016-03-31
128813 cache_misc's misc_fetch_large_objects has issues Traffic resolved 2017-01-02 2016-04-24
128961 Images not showing up at Commons Traffic resolved 2017-01-02 2016-04-20
128968 Register nlwikipedia.org to prevent squatting Traffic resolved 2017-01-02 2016-11-30
129270 Fix puppet on deployment-cache* hosts in beta cluster Traffic resolved 2017-01-02 2016-03-25
129344 varnishkafka logrotate cronspam Traffic resolved 2017-01-02 2016-04-14
129424 Varnish support for shutting users out of a DC Traffic resolved 2017-01-02 2016-04-01
130414 delete links.email.donate.wikimedia.org (and all other email.donate.*?) from DNS Traffic resolved 2017-01-02 2016-04-22
130449 Do something better than an "Unauthorized" error page at https://upload.wikimedia.org/ Traffic resolved 2017-01-02 2016-03-23
130647 alias /apple-app-site-association and /.well-known/apple-app-site-association Traffic resolved 2017-01-02 2017-01-17
130937 Updating DNS records for Major Gifts subdomain (benefactors.wikimedia.org) Traffic resolved 2017-01-02 2017-06-26
130981 irc.wikimedia.org talks HTTP but not HTTPS Traffic resolved 2017-01-02 2016-03-31
131053 investigate radon crash Traffic resolved 2017-01-02 2016-04-11
131131 Canonical URL in Store points to HTTP address, should be HTTPS Traffic resolved 2017-01-02 2017-06-26
131163 Improve varnish stop for backend instances Traffic resolved 2017-01-02 2016-03-29
131353 Port remaining scripts depending on varnishlog.py to new VSL API Traffic resolved 2017-01-02 2016-10-21
131499 Upgrade all cache clusters to Varnish 4 Traffic resolved 2017-01-02 2016-11-24
131501 Convert misc cluster to Varnish 4 Traffic resolved 2017-01-02 2016-07-27
131502 Convert upload cluster to Varnish 4 Traffic resolved 2017-01-02 2016-11-02
131503 Convert text cluster to Varnish 4 Traffic resolved 2017-01-02 2016-11-24
131760 Add icinga monitoring for varnish statistics daemons Traffic resolved 2017-01-02 2016-04-13
131761 Solve large-object/stream/pass/chunked in upload cluster better Traffic resolved 2017-01-02 2016-08-19
131775 codfw: (1) phabricator host (backup node) Traffic resolved 2017-01-02 2016-06-14
131830 Varnish 4 panic log registered on cp1044 Traffic resolved 2017-01-02 2016-04-08
131880 codfw/eqiad/esams/ulsfo: (4) servers for maps caching cluster Traffic resolved 2017-01-02 2016-04-26
131908 Support TLS chacha20-poly1305 AEAD ciphers Traffic resolved 2017-01-02 2016-09-02
132337 MediaWiki-Vagrant varnish role fails to provision Traffic resolved 2017-01-02 2016-05-04
132346 cronspam from cpXXXX hosts related to varnishkafka non existent processes Traffic resolved 2017-01-02 2016-04-27
132374 Generate a list of junk CN cookies being sent by clients Traffic resolved 2017-01-02 2016-07-20
132430 varnishstatsd crashes with ValueError in vsl_callback without being restarted by systemd Traffic resolved 2017-01-02 2016-05-11
132450 enable https for (ubuntu|apt|mirrors).wikimedia.org Traffic resolved 2017-01-02 2016-04-26
132452 HSTS preload for wmfusercontent.org Traffic resolved 2017-01-02 2016-04-19
132457 Move wdqs to an LVS service Traffic resolved 2017-01-02 2016-10-12
132458 Move logstash.wikimedia.org (kibana) to an LVS service Traffic resolved 2017-01-02 2016-10-21
132459 HTTPS redirects for config-master.wikimedia.org Traffic resolved 2017-01-02 2016-04-13
132460 HTTPS redirects for git.wikimedia.org Traffic resolved 2017-01-02 2016-04-13
132461 HTTPS redirects for graphite.wikimedia.org Traffic resolved 2017-01-02 2016-04-13
132462 HTTPS redirects for parsoid-tests.wikimedia.org Traffic resolved 2017-01-02 2016-04-13
132463 HTTPS redirects for datasets.wikimedia.org Traffic resolved 2017-01-02 2016-04-25
132464 HTTPS redirects for transparency.wikimedia.org Traffic resolved 2017-01-02 2016-04-13
132465 HTTPS redirects for stats.wikimedia.org Traffic resolved 2017-01-02 2016-04-13
132474 varnishmedia: repeated calls to flush_stats() Traffic resolved 2017-01-02 2016-06-17
132521 Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination Traffic resolved 2017-01-02 2017-07-11
132543 enable HSTS on *.planet.wikimedia.org Traffic resolved 2017-01-02 2016-04-13
132685 Preload STS for wikimedia.org Traffic resolved 2017-01-02 2016-06-17
132812 Sort out letsencrypt puppetization for simple public hosts Traffic resolved 2017-01-02 2016-04-27
132835 cronspam from cpXXXX hosts due to update-ocsp-all and zero_fetch Traffic resolved 2017-01-02 2016-06-17
132969 Wikipedia app hits loads.php on bits.wikimedia.org Traffic resolved 2017-01-02 2016-08-18
133217 Fix apache-2.4 + DHE ciphersuites issue Traffic resolved 2017-01-02 2016-04-26
133360 Fix wikitech-static TLS config Traffic resolved 2017-01-02 2016-04-21
133485 Configure apache rules for UrlShortener Traffic resolved 2017-01-02 2016-06-23
133490 Wikidata Query Service REST endpoint returns truncated results Traffic resolved 2017-01-02 2016-05-09
133791 check_dns needs to be rewritten Traffic resolved 2017-01-02 2017-10-13
133866 Varnish seems to sometimes mangle uncompressed API results Traffic resolved 2017-01-02 2016-05-10
133915 lvs1012 eth1 NIC Link flapping Traffic resolved 2017-01-02 2016-04-28
133936 Beta cluster varnish sets overly broad domain on GeoIP cookie Traffic resolved 2017-01-02 2016-08-18
133988 Questions about map tile cache performance Traffic resolved 2017-01-02 2016-04-29
134346 varnish text on beta is unreachable / stuck Traffic resolved 2017-01-02 2016-06-30
134362 beta cluster varnish cache can't apt-get upgrade nginx-full: nginx: [emerg] unknown "spdy" variable Traffic resolved 2017-01-02 2016-06-17
134542 Graph results are not being cached in Varnish Traffic resolved 2017-01-02 2016-06-17
134869 Secure connection failed when attempting to send POST request using HTTP/2 (if connection has been idle for a certain time) Traffic resolved 2017-01-02 2016-07-27
134870 Support websockets in cache_misc Traffic resolved 2017-01-02 2016-06-17
134871 Move stream.wikimedia.org (rcstream) behind cache_misc Traffic resolved 2017-01-02 2016-08-11
134989 WDQS empty response - transfer clsoed with 15042 bytes remaining to read Traffic resolved 2017-01-02 2016-07-27
135227 varnish.clients graphite metric spammed with jsessionid Traffic resolved 2017-01-02 2016-06-17
135384 Raise cache frontend memory sizes significantly Traffic resolved 2017-01-02 2016-08-25
135470 Lots of Title::purgeExpiredRestriction from API DELETE FROM `page_restrictions` WHERE (pr_expiry < '20160517063108') without batching/throttling potentially causing lag on s5-api Traffic resolved 2017-01-02 2016-05-31
135480 arbcom-nl.wikipedia.org doesn't have a functioning mobile website Traffic resolved 2017-01-02 2017-09-09
135515 graphite.wikimedia.org 503s on some css/js resources Traffic resolved 2017-01-02 2016-05-18
135696 Sort out vcl_deliver vs vcl_synth mess with v4 VCL Traffic resolved 2017-01-02 2016-06-17
135700 Varnishlog doesn't properly rotates logs, varnish.log is empty since forever (was: deployment-cache-upload04 (m1.medium) / is almost full) Traffic resolved 2017-01-02 2016-06-17
136067 Ashburn servers almost unreachable from part of Europe Traffic resolved 2017-01-02 2016-05-24
136557 Create moon.wikimedia.org and redirect it to https://meta.wikimedia.org/wiki/Wikipedia_to_the_Moon Traffic resolved 2017-01-02 2016-06-17
136674 Help contact bot owners about the end of HTTP access to the API Traffic resolved 2017-01-02 2016-07-19
137106 californium and gallium ferm rules should disallow public access to port 80 Traffic resolved 2017-01-02 2016-06-17
137114 Scripts depending on varnishlog.py maxing out CPU usage on cache_misc Traffic resolved 2017-01-02 2016-06-17
137161 Fix nits in HTTPS/HSTS configs in externally-hosted fundraising domains Traffic resolved 2017-01-02 2017-07-11
137238 Tune WDQS caching headers Traffic resolved 2017-01-02 2016-08-24
137258 policy.wikimedia.org SSL vulnerability Traffic resolved 2017-01-02 2016-06-08
137609 Make upload.wikimedia.org cookieless Traffic resolved 2017-01-02 2017-03-07
137620 Send traffic to new maps200? servers Traffic resolved 2017-01-02 2016-06-17
137707 Antigng-bot improper non-api http requests Traffic resolved 2017-01-02 2016-07-14
137848 Remove referrer check from varnish for maps cluster Traffic resolved 2017-01-02 2016-06-17
138010 Investigate Android app API request latency regression Traffic resolved 2016-06-17 2016-06-22
138084 unix domain socket listening for varnish4 Traffic resolved 2017-01-02 2017-04-18
138682 https://ores.wikimedia.org redirects me to HTTP when I don't include a trailing slash Traffic resolved 2017-01-02 2016-08-12
140658 Make sure we're not relying on HTTP_PROXY headers Traffic resolved 2017-01-02 2017-04-18
141170 Strip query parameters from w.wiki domain Traffic resolved 2017-01-02 2016-08-03
141454 piuparts fail with WARN: Broken symlinks: /etc/systemd/system... Traffic resolved 2017-01-02 2016-11-18
141687 Cached outdated revisions served to logged-out users Traffic resolved 2017-01-02 2016-08-11
141786 TLS stats regression related to Chrome/41 on Windows Traffic resolved 2017-01-02 2017-09-13
142076 Analyze Range requests on cache_upload frontend Traffic resolved 2017-01-02 2016-08-15
142233 Varnish 4 stalls with two consecutive Range requests using HTTP persistent connections Traffic resolved 2017-01-02 2016-08-24
142410 Correct cache_status field on webrequest dataset Traffic resolved 2017-01-02 2016-08-11
142810 varnishd: Assert error in smp_oc_getobj(), storage/storage_persistent_silo.c line 417 Traffic resolved 2017-01-02 2016-10-03
142848 Stop using persistent storage in our backend varnish layers. Traffic resolved 2017-01-02 2016-10-03
142944 Performance and caching considerations for article placeholders accesses Traffic resolved 2017-01-02 2017-01-31
143078 Unhandled pybal ValueError: need more than 1 value to unpack Traffic resolved 2017-01-02 2017-04-03
143132 Expose the PDF rendering service via RESTBase Traffic resolved 2017-01-02 2016-12-05
143270 ULS GeoIP should not use meta.wm.o/geoiplookup Traffic resolved 2017-01-02 2016-08-26
143694 Preliminary Design document for A/B testing Traffic resolved 2017-01-02 2017-03-22
144158 varnishkafka frequently disconnects from kafka servers Traffic resolved 2017-01-02 2016-08-30
144293 adjust ssl certificate montioring to differentiate between standard and LE certificates. Traffic resolved 2017-01-02 2016-11-16
144523 OpenSSL 1.1 deployment for cache clusters Traffic resolved 2017-01-02 2016-11-10
145120 update *.wmflabs.org certificate (existing expires on 2016-09-16) Traffic resolved 2017-01-02 2016-09-12
145244 rcstream support defaults to stream.wikimedia.org:80 Traffic resolved 2017-01-02 2016-09-10
145661 varnish backends start returning 503s after ~6 days uptime Traffic resolved 2017-01-02 2017-10-10
145758 Maps - move traffic to eqiad instead of codfw Traffic resolved 2017-01-02 2016-10-24
146451 repeated 503 errors for 90 minutes now Traffic resolved 2017-01-02 2016-11-26
146638 Remove "GeoIP lookup" service from https://status.wikimedia.org Traffic resolved 2017-01-02 2016-09-27
147116 Upload cache in beta is broken Traffic resolved 2017-01-02 2016-10-04
147424 Port varnish metrics from ganglia to prometheus Traffic resolved 2017-01-02 2016-11-14
147426 Port gdnsd statistics from ganglia to prometheus Traffic resolved 2017-01-02 2017-10-13
147429 Port vhtcpd statistics from ganglia to prometheus Traffic resolved 2017-01-02 2016-12-06
147480 Upgrade conftool to 0.3.1 Traffic resolved 2017-01-02 2016-10-24
147569 Evaluate/Deploy TCP BBR when available (kernel 4.9+) Traffic resolved 2017-01-02 2017-05-30
147784 Mediawiki thumbnail requests for 0px should result in http 400 not 500 Traffic resolved 2017-01-02 2016-12-15
147814 Favicon broken on doc.wikimedia.org and integration.wikimedia.org (HTTP 500) Traffic resolved 2017-01-02 2016-10-11
147845 Move rcstream to an LVS service Traffic resolved 2017-01-02 2016-12-19
147847 Move pybal_config to an LVS service Traffic resolved 2017-01-02 2016-10-19
147848 Use hostnames (not IPs) in deployment-prep varnish app_directors Traffic resolved 2017-01-02 2016-10-12
148045 Windows 10 & MacOS Sierra Certificate errors due to GlobalSign Traffic resolved 2017-01-02 2017-06-12
148069 GlobalSign intermediate updates for one-offs Traffic resolved 2017-01-02 2016-10-14
148076 OCG failing with new GlobalSign intermediate workaround Traffic resolved 2017-01-02 2016-10-13
148490 Extend check_sslxnn to check OCSP Stapling Traffic resolved 2017-01-02 2016-10-31
148497 Some JPGs are being served as text Traffic resolved 2017-01-02 2017-01-27
148723 reimage cp1047 Traffic resolved 2017-01-02 2016-11-02
148780 mobile-safari has very few internally-referred pageviews Traffic resolved 2017-01-02 2017-02-15
148891 cp1052 ethernet link down 2016-10-22 14:11 Traffic resolved 2017-01-02 2017-04-04
148893 nginx SSL_do_handshake spam filling disks Traffic resolved 2017-01-02 2016-10-31
148917 Thumbnails failing to render sporadically (ERR_CONNECTION_CLOSED or ERR_SSL_BAD_RECORD_MAC_ALERT) Traffic resolved 2017-01-02 2016-11-15
149217 cp1066.mgmt.eqiad.wmnet is unreachable Traffic resolved 2017-01-02 2017-04-27
149295 Restbase redirects with cors not working on Android 4 native browser Traffic resolved 2017-01-02 2016-11-10
149784 varnish-backend: weekly cron restart for all clusters Traffic resolved 2017-01-02 2016-11-24
149843 reimage cp4016 and cp1055 Traffic resolved 2017-01-02 2016-11-07
149865 503 errors for users connecting to esams Traffic resolved 2017-01-02 2016-11-11
149881 varnish-be not restarting correctly because of disk space issues Traffic resolved 2017-01-02 2016-11-15
150168 Make our docker registry public Traffic resolved 2017-01-02 2016-11-08
150247 Varnish4 is unexpectedly retrying certain applayer failure cases Traffic resolved 2017-01-02 2017-03-07
150503 Failure to save recent changes Traffic resolved 2017-01-02 2016-11-21
150561 Extra RTT on TLS handshakes Traffic resolved 2017-01-02 2017-01-05
150660 Post Varnish 4 migration cleanup Traffic resolved 2017-01-02 2016-11-24
150879 Cannot connect to cp3039.mgmt.esams.wmnet:22 Traffic resolved 2017-01-02 2016-11-17
151078 Investigate major HTTP 500 spike since 2016-09-23 Traffic resolved 2017-01-02 2016-11-23
151444 Huge increase in cache_upload 404s due to buggy client-side code from graphiq.com Traffic resolved 2017-01-02 2017-04-20
151561 varnishapi.py AttributeError: VSM_Close Traffic resolved 2017-01-02 2016-11-28
151643 python-varnishapi daemons seeing "Log overrun" constantly Traffic resolved 2017-01-02 2017-07-24
151686 several 502 Bad Gateway Traffic resolved 2017-01-02 2016-11-30
152093 Ganglia varnishkafka python module crashing repeatedly Traffic resolved 2017-01-02 2016-12-06
152690 Refreshing mobile-sections does not purge mobile-sections-lead Traffic resolved 2017-01-02 2016-12-13
152870 OTRS-Wiki link to mobile website Traffic resolved 2017-01-02 2016-12-12
153042 Enable TLS termination on the MediaWiki clusters Traffic resolved 2017-01-02 2016-12-16
153336 Strip query string in varnish upload Traffic resolved 2017-01-02 2016-12-19
154558 Periodic 500s from piwik.wikimedia.org Traffic resolved 2017-01-05 2017-04-21
154704 Rate-limit browsers without referers Traffic resolved 2017-01-31 2017-06-29
154759 Pybal not happy with DNS delays Traffic resolved 2017-01-07 2017-07-18
154780 Cache and media (images) issues on all Wikimedia wikis - creates problems on upload, display and generation of thumbnails and files Traffic resolved 2017-01-07 2017-08-27
154826 Donate wiktionary.pl to the Foundation Traffic resolved 2017-01-07 2017-03-07
154864 Plot number of cached objects on a per-server per-DC basis Traffic resolved 2017-01-09 2017-01-24
154913 convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24) Traffic resolved 2017-01-10 2017-01-17
154917 convert lists.wikimedia.org certificate to LetsEncrypt (deadline:2017-03-02) Traffic resolved 2017-01-10 2017-03-01
154919 convert librenms.wikimedia.org from GS to LE cert (expires: 2017-02-11) Traffic resolved 2017-01-10 2017-01-18
154938 convert tendril to use Letsencrypt for SSL cert (deadline 2017-03-17) Traffic resolved 2017-01-10 2017-04-14
154939 convert ganglia to use Letsencrypt for SSL cert (deadline: 2017-02-07) Traffic resolved 2017-01-10 2017-01-10
154940 convert dumps to use Letsencrypt for SSL cert (deadline: 2017-04-26) Traffic resolved 2017-01-10 2017-01-19
154942 convert archiva to use Letsencrypt for SSL cert (deadline 2017-05-08) Traffic resolved 2017-01-10 2017-01-26
155504 Fix universal link support in iOS when the OS requests the site association file from m.wikipedia.org Traffic resolved 2017-01-18 2017-01-27
155807 Monitor Certificate Transparency (CT) logs Traffic resolved 2017-01-21 2017-02-13
156028 Name Asia Cache DC site Traffic resolved 2017-01-24 2017-09-26
156029 Select location for Asia Cache DC Traffic resolved 2017-01-24 2017-04-11
156030 Select site vendor for Asia Cache Datacenter Traffic resolved 2017-01-24 2017-09-26
157353 prometheus-vhtcpd-stats cronspamming if vhtcpd is not running yet Traffic resolved 2017-02-07 2017-07-19
157398 Book collections communicate with pediapress using http: Traffic resolved 2017-02-08 2017-02-15
157430 varnish-be returning 503s upon VCL reload Traffic resolved 2017-02-08 2017-02-07
157794 diamond crashing on hosts using systemd-timesyncd Traffic resolved 2017-02-11 2017-03-13
158638 Using wikimedia.ee mail address as Google account Traffic resolved 2017-02-22 2017-04-07
159424 Upgrade text and upload cache clusters to varnish 4.1.5 Traffic resolved 2017-03-03 2017-03-08
159574 LDF endpoint ordering is not stable between servers when paging Traffic resolved 2017-03-09 2017-07-14
159870 baham (ns1) CPU-related issues Traffic resolved 2017-03-08 2017-04-18
159956 Prepare pywikibot for http -> https switch in entity uri Traffic resolved 2017-03-15 2017-04-30
159995 Separate subdomain for parsoid visual diff test service on ruthenium Traffic resolved 2017-03-09 2017-03-17
160109 What happened 2017-03-09 04:00 - 06:00 UTC Traffic resolved 2017-03-14 2017-04-03
160156 Add node_exporter ipvs ipv6 support Traffic resolved 2017-03-11 2017-05-24
160405 pybal stops logging Traffic resolved 2017-03-15 2017-03-15
160433 Upgrade twisted on load balancers to 16.2.0 Traffic resolved 2017-03-15 2017-03-30
160616 Enable HTTPS for swift clients Traffic resolved 2017-03-17 2017-07-05
161101 Performance impact evaluation of enabling nginx-lua and nginx-lua-prometheus on tlsproxy Traffic resolved 2017-03-23 2017-07-13
161819 Investigate 502 errors from nginx when backend returns 302 Traffic resolved 2017-03-31 2017-04-10
162035 Some PNG thumbnails and JPEG originals delivered as [text/html] content-type and hence not rendered in browser Traffic resolved 2017-04-04 2017-04-20
162073 Ops Onboarding for Arzhel Younsi Traffic resolved 2017-04-04 2017-05-04
162111 Make WDQS active / active Traffic resolved 2017-04-05 2017-05-03
162129 Someone is parsing all enwiki pages using the action api at a rate of ~2M pages/hour Traffic resolved 2017-04-05 2017-04-17
162348 swift-object-server 1.13.1: Wrong Content-Type returned on 304 Not Modified responses Traffic resolved 2017-04-07 2017-04-25
162362 Make maps active / active Traffic resolved 2017-04-07 2017-09-22
162976 dbtree broken (for some users?) Traffic resolved 2017-04-15 2017-04-17
163233 Implement Varnish-level rough ratelimiting Traffic resolved 2017-04-19 2017-09-01
164579 Investigate nginx reload behavior Traffic resolved 2017-05-06 2017-07-24
164608 Merge cache_maps into cache_upload functionally Traffic resolved 2017-05-06 2017-06-08
164610 Unprovision cache_misc @ ulsfo Traffic resolved 2017-05-06 2017-06-27
165063 varnish frontend transient memory usage keeps growing Traffic resolved 2017-05-12 2017-05-15
165324 Can't upload large files with X-Wikimedia-Debug turned on Traffic resolved 2017-05-16 2017-05-29
165736 Update Varnishkafka to support TLS encryption/authentication Traffic resolved 2017-05-20 2017-08-08
166120 phab.wmfusercontent.org "homepage" yields a 500 Traffic resolved 2017-05-25 2017-06-27
166229 Mediawiki replies with 500 on wrongly formatted CSP report Traffic resolved 2017-05-25 2017-06-06
166695 Substantive HTTP and mediawiki/database traffic coming from a single ip Traffic resolved 2017-06-01 2017-09-04
166758 cp3032 ethernet link down (bnx2x dump in the dmesg) Traffic resolved 2017-06-02 2017-10-03
167492 Accessing zh-classical.wikipedia.org on a mobile device does not redirect to zh-classical.m.wikipedia.org Traffic resolved 2017-06-10 2017-06-14
167920 Impending load test Traffic resolved 2017-06-15 2017-06-19
168013 Remove disableImages handling from VCL Traffic resolved 2017-06-16 2017-07-04
168019 runUpdate.sh script in wikidata stand-alone has abruptly started incurring numerous 429 errors. Traffic resolved 2017-06-17 2017-06-20
168919 stream.wikimedia.org: remove legacy rcstream/socket.io HTTPS redirect hole punches Traffic resolved 2017-06-27 2017-07-11
169600 Enable diamond PowerDNSRecursor collector on dnsrecursors Traffic resolved 2017-07-05 2017-10-13
169893 pybal should reset the etcdindex it's looking at after losing a connection Traffic resolved 2017-07-07 2017-07-18
170140 revoke benefactorevents.wikimedia.org SSL certificate Traffic resolved 2017-07-11 2017-07-11
170192 remove eventdonations.wikimedia.org CNAME Traffic resolved 2017-07-11 2017-07-11
170193 revoke eventdonations.wikimedia.org SSL cert if there is one... Traffic resolved 2017-07-11 2017-07-11
170295 remove benefactorevents.wikimedia.org cname from DNS Traffic resolved 2017-07-12 2017-07-11
170598 Extending our HSTS value beyond ~1y Traffic resolved 2017-07-14 2017-09-13
170843 Determine where to host zim files for the Android app Traffic resolved 2017-07-18 2017-08-09
171028 Degraded RAID on cp1008 Traffic resolved 2017-07-20 2017-08-29
171145 cp3048 down, mgmt console not reachable Traffic resolved 2017-07-21 2017-07-20
171318 logster should not resolve statsd's IP every time it sends a metric Traffic resolved 2017-07-22 2017-07-26
171421 503 error for certain JPG thumbnail: "Backend fetch failed" Traffic resolved 2017-07-26 2017-07-28
171710 pybal: add prometheus metrics Traffic resolved 2017-07-27 2017-10-06
172101 OCSP update failed for /etc/update-ocsp.d/globalsign-2016-ecdsa-unified.conf Traffic resolved 2017-08-01 2017-07-31
172417 Create wikimedia.org/resources redirect for Wikimedia Resource Center Traffic resolved 2017-08-04 2017-08-09
172418 Get translations for "IE8 on XP won't work" Traffic resolved 2017-08-04 2017-08-21
173028 pybal vs firewall failover - BGP session down Traffic resolved 2017-09-22 2017-09-28
173422 Investigate the increase in the number of requests to Swift after the Page Previews deploy Traffic resolved 2017-08-17 2017-08-23
173506 cp3036 crashed Traffic resolved 2017-08-18 2017-08-17
173787 setup CAA record for policy.wikimedia.org for namecheap (used by WP VIP GO) due by 2017-09-27 Traffic resolved 2017-08-22 2017-08-25
174932 Recurrent 'mailbox lag' critical alerts and 500s Traffic resolved 2017-09-05 2017-10-10
175473 Multiple 503 Errors Traffic resolved 2017-09-11 2017-09-11
175585 cp4021 memory hardware issue - DIMM B1 Traffic resolved 2017-09-12 2017-09-25
175803 Text eqiad varnish 503 spikes Traffic resolved 2017-09-14 2017-10-10
175864 CI: operations-dns-lint broken due to missing Maxmind DB file Traffic resolved 2017-09-14 2017-09-26
176386 upload@ulsfo strange ethernet / power / switch issues, etc... Traffic resolved 2017-09-22 2017-10-19
177233 Upgrade cache_misc to Varnish 5 Traffic resolved 2017-10-03 2017-10-17
177815 Alerts on LVS services with one single realserver Traffic resolved 2017-10-11 2017-10-16
178011 cp4026 memory error Traffic resolved 2017-10-12 2017-10-17
178078 RESTBase logs disappeared from logstash Traffic resolved 2017-10-13 2017-10-19
178149 RunCommandMonitoringProtocol throws an exception if runcommand.arguments is not specified Traffic resolved 2017-10-14 2017-10-19
2323 edits where the rev_user_text and ar_user_text fields contain underscores, initial lower-case letters or consecutive spaces, which could occur in the Phase I and II software, are inaccessible using Special:Contributions DBA open 2017-01-02 2015-12-05
3268 Database replication lag issues (tracking) DBA open 2017-01-02 2017-10-09
17441 Some tables lack unique or primary keys, may allow confusing duplicate data DBA open 2017-01-02 2017-09-05
18660 Database table cleanup (tracking) DBA open 2017-01-02 2017-10-09
29112 Select of revisions for stub history files does not explicitly order revisions DBA open 2017-01-02 2016-07-18
33863 Fix use of DB schema so RenameUser is trivial DBA open 2017-01-02 2017-09-12
39591 History of a page was lost after several move actions on zhwp DBA open 2017-01-02 2016-12-29
48047 Database error on es.wikipedia.org in function "Revision::insertOn": "Duplicate entry for key 'PRIMARY'" DBA open 2017-01-02 2015-12-05
48098 Disable miser mode ($wgMiserMode) on small wikis (wikis in small.dblist) DBA open 2017-02-14 2017-03-27
50694 Show replication lags in Graphite DBA open 2017-01-02 2016-03-25
50930 Database replication problems - production and labs (tracking) DBA open 2017-01-02 2017-03-23
51190 Truncate SHA-1 indexes DBA open 2017-01-02 2017-04-26
51191 Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases DBA open 2017-01-02 2017-04-28
51195 Drop fj_new_sha1 field DBA open 2017-01-02 2017-04-26
51593 Rename two indexes in the Echo extension DBA open 2017-01-02 2017-04-26
54921 Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) DBA open 2017-01-02 2017-09-01
54932 Drop *_old database tables from Wikimedia wikis DBA open 2017-01-02 2016-12-30
56401 Backups systems for userDBs hosted on replicas DBA open 2017-01-02 2017-01-19
57385 Investigate dropping "edit_page_tracking" database table from Wikimedia wikis after archiving it DBA open 2017-01-02 2016-07-07
57455 Provide dynamic report of differences between replica databases and production databases DBA open 2017-01-02 2016-11-15
59176 ApiQueryExtLinksUsage::run query has crazy limit DBA open 2017-01-02 2017-04-27
59186 blob_tracking indexes apparently unused DBA open 2017-01-02 2016-07-20
59617 Make watchlist table available as curated foo_p.watchlist_count on labsdb DBA open 2017-01-02 2017-09-29
60674 Drop page.page_restrictions column from Wikimedia wikis DBA open 2017-01-02 2017-04-26
67448 Dropping rc_cur_time on wmf databases DBA open 2017-01-02 2017-04-26
68025 [Story] Monitor size of some Wikidata database tables DBA open 2017-01-02 2016-05-11
68961 Logging needs an index to optimize searching by log_title DBA open 2017-02-16 2017-08-07
71088 Queries of commonswiki_p.filearchive for fa_sha1 are slow DBA open 2017-01-02 2017-10-16
71127 Discrepancies with logging table on different wikis DBA open 2017-01-02 2017-08-07
71182 Database upgrade MariaDB 10: Metadata access in INFORMATION_SCHEMA causes complete blocks DBA open 2017-01-02 2015-09-23
71222 list=logevents slow for users with last log action long time ago DBA open 2017-01-02 2016-05-23
72005 Apply enum changes to (img|oi|fa)_major_mime on production DBA open 2017-01-02 2017-04-28
79922 Set up backup strategy for es clusters DBA open 2017-01-02 2017-04-12
83609 script & docs to rename wiki databases DBA open 2017-01-02 2017-10-14
85266 Look into Maria 10 parallel-replication DBA open 2017-01-02 2017-03-02
85756 Make (redacted) log_search table available on ToolLabs DBA open 2017-02-09 2017-02-08
85757 Dropping user.user_options on wmf databases DBA open 2017-01-02 2017-04-26
86338 Dropping page.page_counter on wmf databases DBA open 2017-01-02 2017-04-26
86339 Dropping site_stats.ss_total_views on wmf databases DBA open 2017-01-02 2017-04-26
86342 Dropping page.page_no_title_convert on wmf databases DBA open 2017-01-02 2017-04-26
89737 Make several mediawiki table fields unsigned ints on wmf databases DBA open 2017-01-02 2017-09-07
89741 Expose ar_content_format and ar_content_model columns of archive table on Labs replicas DBA open 2017-03-27 2017-08-23
92739 Remove AFT tables from the analytics slaves DBA open 2017-01-02 2016-12-14
95799 wfWaitForSlaves in JobRunner can massively slow down run rate if just a single slave is lagged DBA open 2017-01-02 2017-09-13
97414 [Task] Use an own load section for Wikibase DB queries DBA open 2017-01-02 2016-02-05
98110 Pagelinks table contains a row having pl_from = 0 DBA open 2017-01-02 2015-10-13
100501 mysql user and group should be a system user/group DBA open 2017-01-02 2017-06-22
101502 Slow API list=recentchanges query (DBError: Lost connection to MySQL server during query) DBA open 2017-01-02 2016-12-20
101631 rev_len should be available also for deleted revisions in database replicas DBA open 2017-01-02 2017-10-16
104459 Automatize the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves DBA open 2017-01-02 2017-09-14
104699 Firewall configurations for database hosts DBA open 2017-01-02 2016-11-10
105395 UserMerge using SELECT img_name FROM image WHERE img_user = ... but img_user doesn't have an index DBA open 2017-01-02 2015-07-09
105728 MySQL field aliases in select() do not use any quoting DBA open 2017-01-02 2015-08-14
106151 write irc bot to report high replag of s{1,2,3}.labsdb on #wikimedia-labsdb DBA open 2017-01-02 2016-06-22
106854 Duplicate key errors (tracking) DBA open 2017-01-02 2016-10-12
107610 Setup separate logical External Store for Flow in production DBA open 2015-10-02 2016-12-16
107676 Undiscoverable log entries DBA open 2017-01-02 2015-11-05
108255 Enable MariaDB/MySQL's Strict Mode DBA open 2017-01-02 2017-07-21
108434 latin1 encoding in sql databases DBA open 2017-01-02 2015-08-14
108782 Special:Contributions: "contributions page filtered for namespace or RevisionDeleted edits" error DBA open 2017-01-02 2016-07-11
108850 Set up auto-purging after 90 days {tick} DBA open 2017-01-02 2017-02-01
108968 prop=revisions with rvdir=newer gives internal_api_error_DBQueryError for page with many version DBA open 2017-01-02 2017-02-06
109179 Migrate MySQLs to use ROW-based replication DBA open 2017-01-02 2017-06-20
109964 Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections. DBA open 2017-01-02 2015-09-29
110115 Possible to run writes (e.g. UPDATE) on Beta Cluster replica DBA open 2017-01-02 2016-10-15
111929 Puppetize grants for mysql backups on dbstore hosts DBA open 2017-01-02 2017-04-12
112282 Multiple pages with no revisions DBA open 2017-01-02 2016-09-23
112473 Better mysql monitoring for number of connections and processlist strange patterns DBA open 2017-01-02 2017-08-04
113842 `pr_index`to be replicated to Labs public databases DBA open 2017-09-01 2017-10-06
114117 Drop externallinks.el_from_namespace on wmf databases DBA open 2017-01-02 2017-04-28
114476 Puppetize grants for mysql analytics servers DBA open 2017-01-02 2017-02-20
114902 Remove numeric entity IDs from database schema DBA open 2017-01-02 2016-08-13
114903 Migrate wb_terms to using prefixed entity IDs instead of numeric IDs DBA open 2017-01-02 2017-07-24
114904 Migrate wb_items_per_site to using prefixed entity IDs instead of numeric IDs DBA open 2017-01-02 2017-10-18
115081 Pages where page.page_latest = 0 on the English Wikipedia DBA open 2017-01-02 2015-12-05
115148 Consider dropping fr_page_qal_rev/fr_page_rev indexes DBA open 2017-01-02 2015-10-09
115303 Expunge old AFTv5 log entries on WMF wikis DBA open 2017-01-02 2016-08-12
116793 Investigate slow query logging/digest for Beta Cluster DBA open 2017-01-02 2016-08-05
117272 Wikidata: Possibility to add comments with more than 255 bytes, potential db issue DBA open 2017-05-05 2017-05-04
118434 Reduce Flow DB queries on Special:Contributions DBA open 2015-11-13 2016-04-19
118764 Drop old devwikiinternal and rel13testwiki from s3 DBA open 2017-01-02 2015-12-05
118859 Drop pre_ tables DBA open 2017-01-02 2017-05-31
119154 Move echo tables from local wiki databases onto extension1 cluster for mediawikiwiki, metawiki, and officewiki DBA open 2015-11-21 2017-02-09
119371 Enable MariaDB/MySQL strict mode on CI slaves DBA open 2017-01-02 2017-05-08
119479 Create a script that'll allow easier swatting of queries that're overloading labsdb DBA open 2017-01-02 2015-11-24
119601 Certain tools users create multiple long running queries that take all memory from labsdb hosts, slowing it down and potentially crashing (tracking) DBA open 2017-01-02 2017-06-20
119626 Eliminate SPOF at the main database infrastructure DBA open 2017-01-02 2017-05-23
119847 Replicate ContentTranslation databases on Labs DBA open 2017-01-02 2017-03-03
121094 Throttling linkwatcher tool user as it is consuming 100% CPU DBA open 2017-01-02 2017-04-19
121161 CentralAuthCreateLocalAccountJob failing on meta due to Echo deadlocks DBA open 2016-07-14 2017-01-10
121306 Error: 2013 Lost connection to MySQL server during query on IndexPager::buildQueryInfo (LogPager) DBA open 2017-01-02 2017-05-08
121857 Implement a system to automatically deploy schema changes without needing DBA intervention DBA open 2017-01-02 2016-11-10
123557 Database query error (internal_api_error_DBQueryError) while getting list=allrevisions DBA open 2017-01-02 2017-02-09
124214 Allow filtering based on tag on Special:NewFiles DBA open 2017-01-02 2017-06-27
126218 Replicate wikitech wikis to analytics-store.eqiad.wmnet DBA open 2017-01-02 2016-02-11
126251 Get HA db support for labs internal services DBA open 2017-01-02 2017-02-14
126252 Populate the wikishared db on all dbstores DBA open 2017-01-02 2016-02-09
127363 mysqlbinlog doesn't recognize the ssl-* options in [client] DBA open 2017-01-02 2017-01-06
127570 Rename be_x_oldwiki database to be_taraskwiki DBA open 2017-01-02 2017-10-14
127636 MariaDB missing logrotate for error and slow logs DBA open 2017-01-02 2016-06-08
127792 Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace DBA open 2016-02-24 2017-08-10
128151 Migrate all old DB rows from windows-1252 to UTF-8 on enwiki DBA open 2017-01-02 2017-10-07
128152 Migrate all old DB rows from windows-1252 to UTF-8 on dawiki DBA open 2017-01-02 2017-10-07
128153 Migrate all old DB rows from windows-1252 to UTF-8 on svwiki DBA open 2017-01-02 2017-10-07
128154 Migrate all old DB rows from windows-1252 to UTF-8 on nlwiki DBA open 2017-01-02 2017-10-07
128155 Migrate all old DB rows from windows-1252 to UTF-8 on dawiktionary DBA open 2017-01-02 2017-10-07
128156 Migrate all old DB rows from windows-1252 to UTF-8 on svwiktionary DBA open 2017-01-02 2017-10-07
128623 Purge all Schema:Echo data after 90 days DBA open 2017-01-02 2016-03-07
129093 SHOW SLAVE STATUS as a health check should have a low timeout DBA open 2017-01-02 2016-03-08
129830 large s2 database traffic DBA open 2017-01-02 2016-03-18
131065 API: Contributions: Database query error DBA open 2017-01-02 2017-05-08
131266 Querying the logging table on labs is slow DBA open 2017-01-02 2016-12-04
131956 Disabling general.confirmeduser from dbreports for using up too much db resources DBA open 2017-01-02 2016-12-04
132838 Missing / Dropped databases? DBA open 2017-01-02 2017-07-06
133185 Database error while saving an article (1205 Lock wait exceeded in Title::invalidateCache) DBA open 2017-01-02 2016-04-25
133251 archive/archive_userindex is not filled in eswiki_p DBA open 2017-01-02 2016-04-21
133333 Audit MySQL configurations DBA open 2017-01-02 2016-11-10
133523 [RFC] improve parsercache replication and sharding handling DBA open 2017-01-02 2017-07-05
133705 Multiple concurrent long running queries from s51434 overloading labsdb1003/labsdb1009 DBA open 2017-01-02 2017-10-22
133906 memcached on db1011/tendril not puppetised DBA open 2017-01-02 2016-04-29
134476 Decommission old coredb machines (<=db1050) DBA open 2017-01-02 2017-10-13
134809 Apache <=> mariadb SSL/TLS for cross-datacenter writes DBA open 2017-01-02 2017-09-28
135414 Entries in revision table where rev_len is NULL DBA open 2017-01-02 2017-01-10
135851 Preserve InnoDB table auto_increment on restart DBA open 2017-01-02 2017-05-08
136335 Allow self-serve database credential and permissions management for Tool Labs projects DBA open 2017-01-02 2017-06-08
137654 Tendril DB graphs say 24hr but at 12hr DBA open 2017-01-02 2016-06-14
138517 mysqldump is timing out preventing all tables from being included in the dump DBA open 2017-01-02 2017-04-20
138562 Improve regular production database backups handling DBA open 2017-01-02 2017-07-03
138915 OTRS database is "too large" DBA open 2017-01-02 2017-09-18
140302 plwiki API request is excessively slow when including a badrevid DBA open 2017-01-02 2016-08-24
140609 Add pp_propname/pp_value index to Labs replica DBA open 2017-01-02 2017-10-17
140751 Create a logstash input filter to preprocess mysqld syslog messages DBA open 2017-01-02 2016-08-05
140788 Labs databases rearchitecture (tracking) DBA open 2017-01-02 2017-09-05
140832 Investigate moving labsdb (replicas) user credential management to 'Striker' (codename) DBA open 2017-01-02 2017-08-22
141095 Explore 'Analyze' statement as substitute for Explain DBA open 2017-01-02 2016-12-04
141255 Separate host lookup from the sql shell script DBA open 2017-01-02 2016-07-27
141547 Setup automatic failover for misc database servers DBA open 2017-01-02 2016-08-17
141765 Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) DBA open 2017-01-02 2017-05-08
141968 Display lag on grafana (prometheus) and dbtree from pt-heartbeat instead (or in addition) of Seconds_Behind_Master DBA open 2017-01-02 2016-08-25
142807 Migrate all users to new Wiki Replica cluster and decommission old hardware DBA open 2017-01-02 2017-10-18
143366 Removal of {{#coordinates:}} leaves DB entries behind DBA open 2017-07-13 2017-07-13
143870 Some dump hosts are accessing main traffic servers DBA open 2017-01-02 2016-08-31
143896 MySQL metrics monitoring DBA open 2017-01-02 2017-10-09
144010 Drop eu_touched in production DBA open 2017-01-02 2017-10-05
144382 Spiky write pattern on core db masters DBA open 2017-01-02 2016-09-08
145072 Create a script to regenerate prometheus mysqld exporter listing that works with puppetdb DBA open 2017-01-02 2017-07-10
146055 Improve privilege separation for phabricator's config files and mysql credentials DBA open 2017-03-16 2017-03-28
147148 Wikipedia requires a patch to load its data from the dumps with mwdumper DBA open 2017-01-02 2017-05-08
148078 Decommission db1015, db1035, db1044 and db1038 DBA open 2017-01-02 2017-09-28
148955 Puppetize tendril web user creation DBA open 2017-01-02 2016-10-24
149077 Certain ApiQueryRecentChanges::run api query is too slow, slowing down dewiki DBA open 2017-01-02 2017-06-16
149643 Review Icinga alarms with disabled notifications DBA open 2017-01-02 2016-11-02
149936 Drop flow_subscription table DBA open 2016-11-04 2017-05-25
150438 Meta ticket: Deploy InnoDB compression where possible DBA open 2017-01-02 2017-07-03
150547 Add visitingwatchers to watchlist_count DBA open 2017-01-02 2016-11-12
150548 Reduce watchlist_count threshold DBA open 2017-01-02 2017-02-22
150850 Decouple roles from mariadb.pp into their own file DBA open 2017-01-02 2017-02-09
150949 Set barracuda InnoDB file format as the default configuration everywhere DBA open 2017-01-02 2017-05-05
151491 Icinga MariaDB disk space check on silver checks the wrong partition DBA open 2017-01-02 2016-11-24
151583 Use tls for dump backup generation DBA open 2017-01-02 2017-04-12
151681 DispatchChanges: Avoid long-lasting connections to the master DB DBA open 2017-01-02 2017-07-21
151717 Usage tracking: record which statement group is used DBA open 2017-03-08 2017-10-20
151999 Create script to monitor db dumps for backups are successful (and if not, old backups are not deleted) DBA open 2017-01-02 2017-04-12
152385 66 rows from external storage (dewiki) gave duplicate key errors on master failover DBA open 2017-01-02 2016-12-05
152427 Create a check/calendar alert for MariaDB TLS certs DBA open 2017-01-02 2017-02-21
153058 LabsDB infrastructure pending work DBA open 2017-01-02 2017-08-11
153182 Perform schema change to add externallinks.el_index_60 to all wikis DBA open 2017-01-02 2017-06-06
153440 Create a full backup of all external storage records that would be easy to restore/setup a temporary delayed slave DBA open 2017-01-02 2016-12-16
153638 Drop echo tables from local wiki databases DBA open 2017-01-02 2017-07-12
154490 Delete prefstats tables DBA open 2017-01-04 2017-01-03
156461 [META ticket] Automation for our DBs tracking task DBA open 2017-01-28 2017-01-27
156462 Framework to transfer files over the LAN DBA open 2017-01-28 2017-01-27
156465 For switchovers: A way to check if slaves are up to date DBA open 2017-01-28 2017-01-27
156844 Prep to decommission old dbstore hosts (db1046, db1047) DBA open 2017-02-01 2017-10-19
157359 labsdb1006/1007 (postgresql) maintenance DBA open 2017-02-07 2017-06-22
157668 Automate dataset backup recovery DBA open 2017-02-10 2017-04-12
157702 Followup for TLS MariaDB server roll-out DBA open 2017-02-10 2017-02-21
158166 Discuss labsdb visibility of rev_text_id and ar_comment DBA open 2017-02-16 2017-03-13
158724 Increase size of categorylinks.cl_collation column DBA open 2017-02-23 2017-04-10
159423 Meta ticket: Migrate multi-source database hosts to multi-instance DBA open 2017-03-03 2017-07-03
159430 convert dbstore1001 to InnoDB compressed by importing db shards to it DBA open 2017-03-03 2017-04-05
160062 Allow setting the watchlist table to read-only on a per-wiki basis DBA open 2017-03-10 2017-10-17
160731 Decom db1048 (BBU Faulty - slave lagging) DBA open 2017-03-18 2017-09-13
160983 Optimize SpecialAllPages::showChunk for large wikis DBA open 2017-03-22 2017-10-19
161066 Merge pagelinks, templatelinks and imagelinks tables DBA open 2017-05-02 2017-05-24
161088 Migrate some s4 hosts to file per table DBA open 2017-03-23 2017-07-14
161294 run pt-tablechecksum on s5 DBA open 2017-03-25 2017-10-05
161754 eqiad: (2) hardware access request for labsdb1004 & 5 refresh DBA open 2017-03-30 2017-04-07
161755 eqiad: (2) hardware access request for labsdb1006 & 7 refresh DBA open 2017-03-30 2017-04-07
162070 Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases DBA open 2017-04-04 2017-04-03
162558 Use memcached (or something similar) to keep the latest chd_seen state, only flush to table every once in a while DBA open 2017-04-10 2017-05-05
162562 Choose a field length for entity id columns and stick to it DBA open 2017-04-10 2017-05-05
162699 Decomissions old s2 eqiad hosts (db1018, db1021, db1024, db1036) DBA open 2017-04-12 2017-09-20
162789 Create less overhead on bacula jobs when dumping production databases DBA open 2017-04-13 2017-10-04
162807 Run pt-table-checksum on s1 (enwiki) DBA open 2017-04-13 2017-07-17
163190 Run pt-table-checksum on s7 DBA open 2017-04-19 2017-10-05
163339 pdu phase inbalances: ps1-a3-codfw, ps1-c6-codfw, & ps1-d6-codfw DBA open 2017-04-20 2017-05-08
163681 Use a random timestamp on Special:NewFiles on Commons doesn't loads the page (504 Gateway Timeout) DBA open 2017-08-02 2017-08-01
164167 change_tag table needs redesign DBA open 2017-05-01 2017-06-07
164382 Evaluate the need for FORCE INDEX (ls_field_val) [now IGNORE INDEX (ls_log_id)], delete the index hint if not needed anymore DBA open 2017-05-04 2017-05-08
164488 Run pt-table-checksum on s3 DBA open 2017-05-05 2017-10-16
164504 Cleanup x1 database connection patterns DBA open 2017-05-09 2017-05-08
164805 Investigate requirements for MySQL access in RESTBase DBA open 2017-05-09 2017-06-15
164834 In some database hosts, performance schema loses digest statistics DBA open 2017-05-10 2017-10-04
165358 Set up and package wmfmariadbpy helper scripts so they can easily be deployed to all database server and client hosts DBA open 2017-05-16 2017-05-15
165625 Evaluate future of wmf puppet module "mysql" DBA open 2017-05-18 2017-09-12
165674 Investigate slow servermon updating queries on db1016 DBA open 2017-05-19 2017-06-14
165677 Create a backend check for pybal to monitor the MySQL protocol being up DBA open 2017-05-19 2017-05-25
165756 Create summary templates on Wikitech wiki to stop writing the same things everywhere, everytime DBA open 2017-05-20 2017-09-12
166108 x1 master db1031: Faulty BBU DBA open 2017-05-24 2017-06-27
166344 db1016 m1 master: Possibly faulty BBU DBA open 2017-05-26 2017-10-05
166798 Decide whether back-compat views for upcoming major schema changes will be provided in the Labs replicas DBA open 2017-06-02 2017-09-07
167973 Move wikitech and labstestwiki to s3 DBA open 2017-06-16 2017-07-25
168107 Review new Echo table for user group expiration DBA open 2017-06-17 2017-07-20
168349 enwiki_p logging vs logging_userindex returning dramatically different results DBA open 2017-06-21 2017-09-12
168414 Purge all old data from EventLogging master DBA open 2017-06-21 2017-10-02
169501 Move some masters away from B6 DBA open 2017-07-04 2017-10-09
170508 The "show ip" action should also provide a distinct list of user-agents for each IP DBA open 2017-07-20 2017-09-12
170662 Productionize 22 new codfw database servers DBA open 2017-07-15 2017-10-17
171071 Perform testing for TLS effect on connection rate DBA open 2017-07-20 2017-09-12
171570 Rename database labswiki to wikitech DBA open 2017-07-26 2017-10-14
171643 Clean up skin properties DBA open 2017-07-26 2017-07-31
171898 Deleting large watchlist takes > 4 seconds causing rollback due to write time limit DBA open 2017-07-29 2017-09-10
172207 flaggedrevs.fr_user is unindexed DBA open 2017-08-02 2017-09-25
172489 Monitor read_only variable and/or uptime on database masters, make it page DBA open 2017-08-05 2017-10-09
172490 Monitor swap/memory usage on databases DBA open 2017-08-05 2017-08-07
172492 Improve database alerting (tracking) DBA open 2017-08-05 2017-10-09
172497 Change pt-heartbeat model to not use super-user, avoid SPOF and switch automatically to the real master without puppet dependency DBA open 2017-08-05 2017-08-07
172498 Switch databases to the future parser DBA open 2017-08-05 2017-10-16
173511 Implement technical details and process for "datasets_p" on wikireplica hosts DBA open 2017-08-18 2017-10-03
173570 Decommission db1015 DBA open 2017-08-19 2017-10-18
173647 Prepare and check storage layer for hif.wiktionary DBA open 2017-08-21 2017-09-14
173915 Decommission db1041 DBA open 2017-08-24 2017-10-18
174044 Deploy MCR storage layer DBA open 2017-08-25 2017-10-13
174076 Decommission db1033 and db1028 DBA open 2017-08-25 2017-10-18
174370 Create elections committee private wiki DBA open 2017-08-29 2017-10-09
174509 Drop now redundant indexes from pagelinks and templatelinks DBA open 2017-08-30 2017-10-17
174569 Schema change for refactored comment storage DBA open 2017-08-31 2017-10-19
174648 Create MW Schema Diff maintenance script DBA open 2017-09-01 2017-09-12
174763 Decommission db1026 DBA open 2017-09-02 2017-10-18
174801 Drop flaggedrevs tables on wikis where it is not enabled DBA open 2017-09-02 2017-09-01
174802 Drop education program (ep_*) tables on wikis where it is not enabled DBA open 2017-09-02 2017-09-01
174804 Truncate `updatelog` on all wikis DBA open 2017-09-02 2017-09-01
174806 Decommission db1045 DBA open 2017-09-02 2017-10-18
174902 Decommission db1037 DBA open 2017-09-05 2017-10-18
175096 Identify tools hosting databases on labsdb100[13] and notify maintainers DBA open 2017-09-06 2017-10-12
175264 Decommission db1049 DBA open 2017-09-08 2017-10-18
175672 Make apache/maintenance hosts TLS connections to mariadb work DBA open 2017-09-13 2017-10-17
175679 Decommission db1048 (was Move m3 slave to db1059) DBA open 2017-09-13 2017-10-18
175685 Decommission db2010 and move m1 codfw to db2078 DBA open 2017-09-13 2017-10-10
176043 Prepare and check storage layer for amwikimedia (including dropping s7 version of the wiki) DBA open 2017-09-17 2017-10-16
176215 decommission db1018 DBA open 2017-09-20 2017-10-16
176243 Decommission database hosts < db2030 (tracking) DBA open 2017-09-20 2017-09-25
176277 Provision a separate DB shard for wbc_entity_usage DBA open 2017-09-20 2017-09-25
176311 decommission db1036 DBA open 2017-09-21 2017-10-04
176532 Gerrit is failing to connect to db on gerrit2001 thus preventing systemd from working DBA open 2017-09-24 2017-10-15
176931 Decommission db1035 DBA open 2017-09-29 2017-10-11
177223 Determine schema differences between labsdb1001 and labsdb1009 DBA open 2017-10-03 2017-10-18
177274 Provide a new sanitarium host for s8 DBA open 2017-10-04 2017-10-12
177601 Deploy dropping wb_entity_per_page table DBA open 2017-10-07 2017-10-20
177778 Improve database application performance monitoring visibility DBA open 2017-10-10 2017-10-13
177779 Generate instance list of database hosts to be monitored automatically from exported resources DBA open 2017-10-10 2017-10-10
177782 Reduce false positives on database pages DBA open 2017-10-10 2017-10-13
177911 Decommission db1038 DBA open 2017-10-12 2017-10-11
178128 Access to raw database tables on labsdb* for wmcs-admin users DBA open 2017-10-13 2017-10-18
178162 Decommission db1050 DBA open 2017-10-14 2017-10-18
178290 Check recentchanges table and query errors on wikis other than commonswiki and ruwiki DBA open 2017-10-17 2017-10-16
178359 Support multi-instance on core hosts DBA open 2017-10-18 2017-10-19
178383 db1101 crashed - memory errors DBA open 2017-10-18 2017-10-18
178460 db1082 storage crashed DBA open 2017-10-19 2017-10-18
178515 Enable description usage tracking on further wikis DBA open 2017-10-19 2017-10-20
178661 Drop wb_entity_per_page views in Wiki Replicas DBA open 2017-10-21 2017-10-22
178752 gblrename log_type missing on replicas DBA open 2017-10-23 2017-10-22
12817 Filtering Special:NewPages by page size DBA resolved 2017-01-02 2016-11-10
26675 Revision 186704908 on en.wikipedia.org, Fatal exception: unknown "cluster16" DBA resolved 2017-01-02 2016-03-09
32824 Integrate Facebooks "Online Schema Change for MySQL" into MediaWiki DBA resolved 2017-05-24 2017-09-04
42867 Update change tag indexes DBA resolved 2017-01-02 2017-10-09
52422 Replicate the Phabricator database to labsdb DBA resolved 2017-01-02 2015-10-22
54924 Drop PovWatch extension-related database tables from Wikimedia wikis DBA resolved 2017-01-02 2016-12-14
54927 Drop database table "hashs" from Wikimedia wikis DBA resolved 2017-01-02 2017-04-27
54929 Drop database table "namespaces" from Wikimedia wikis DBA resolved 2017-01-02 2015-06-09
54934 Drop database table "optin_survey" from Wikimedia wikis DBA resolved 2017-01-02 2015-08-26
57676 Drop database table "email_capture" from Wikimedia wikis DBA resolved 2017-01-02 2016-10-10
59185 Archive and drop all article feedback related tables from all wikis DBA resolved 2017-01-02 2016-08-12
62539 [Task] Convert wb_terms term_row_id from INT to BIGINT on wikidatawiki DBA resolved 2017-01-02 2016-02-25
69223 Schema change for page content language DBA resolved 2017-01-02 2017-06-08
70942 Database upgrade MariaDB 10: Engine / Option mismatch on table `user_properties` DBA resolved 2017-01-02 2017-02-24
71463 Create a table in labs with replication lag data DBA resolved 2017-01-02 2016-04-22
72221 internal_api_error_DBQueryError on page edit; db1024 DBA resolved 2017-01-02 2015-12-07
72893 meta_p.wiki and meta_p.legacy tables not filled on all servers DBA resolved 2017-01-02 2015-06-09
73176 Discrepancy between enwiki_p.pagelinks on labs and production DBA resolved 2017-01-02 2015-06-26
73563 *_minor_mime are varbinary(32) on WMF sites, out of sync with varbinary(100) in MW core DBA resolved 2017-01-02 2017-04-20
74413 Different results with queries in labs versus production DBA resolved 2017-01-02 2016-06-29
74908 Replication behind or missing records DBA resolved 2017-01-02 2015-08-17
75047 Replicate flowdb from X1 to analytics-store DBA resolved 2017-01-02 2015-11-02
78276 allpages filterlanglinks DBQueryError DBA resolved 2017-01-02 2016-09-11
84178 investigate RAID BBU auto-learn on db hosts DBA resolved 2017-01-02 2015-09-02
85813 "sites" table missing from some wikis DBA resolved 2017-01-02 2015-11-16
86340 Dropping hitcounter table on wmf databases DBA resolved 2017-01-02 2017-01-19
86482 puppet stopped mysqld using orphan pid file from puppet agent DBA resolved 2017-01-02 2017-06-23
86958 install/deploy dbproxy1003 through dbproxy1011 DBA resolved 2017-01-02 2016-12-02
88718 Make sure tools-db is replicated somewhere DBA resolved 2017-01-02 2015-10-05
88770 sleeper database connection surges during outage DBA resolved 2017-01-02 2016-07-29
88986 [Task] Avoid inconsistencies in Wikidata editing due to slave lag DBA resolved 2017-01-02 2016-09-07
89689 Missing record in replica DBA resolved 2017-01-02 2015-09-26
89986 Update GeoData schema DBA resolved 2017-01-02 2015-10-23
90679 contacts.wikimedia.org drupal unpuppetized / retire contacts DBA resolved 2017-01-02 2015-06-01
91797 mariadb puppet module doesn't start mysql service in labs (possibly anywhere) DBA resolved 2017-01-02 2016-12-01
91859 Update diagram of core database tables to MediaWiki 1.24 DBA resolved 2017-01-02 2015-04-06
92693 Get Labs openstack service dbs on a proper db server DBA resolved 2017-01-02 2015-06-19
92841 Database replicas: replicate user.user_touched DBA resolved 2017-01-02 2015-11-29
93211 Grant access to the databases to codfw appserver networks DBA resolved 2017-01-02 2016-01-20
93708 Can't access x1-analytics-slave DBA resolved 2017-01-02 2015-03-26
94512 Bug: DB gather_list.gl_id autonumber is always odd DBA resolved 2017-01-02 2015-04-29
95421 Database error when trying to connect to NL wikipedia DBA resolved 2017-01-02 2015-04-08
95835 Better backup coverage for X1 database cluster DBA resolved 2017-01-02 2015-05-02
95871 Use External Store on Beta Cluster DBA resolved 2015-10-02 2016-08-12
96383 install/setup/deploy db2043-db2070 DBA resolved 2017-01-02 2015-08-28
96638 wikimania2016wiki not recognised by sql command DBA resolved 2017-01-02 2015-06-26
98682 labswiki DB is inaccessible from tin, terbium, etc. DBA resolved 2017-01-02 2015-08-20
98706 internal_api_error_DBQueryError: Database query error while (mass) deleting file over api DBA resolved 2017-01-02 2015-07-02
98816 move tendril to gerrit repo and puppetize cloning DBA resolved 2017-01-02 2015-07-14
98998 Database connection failure issues on s7 shard DBA resolved 2017-01-02 2015-05-15
99459 ips_site_page is too short to store some (full) page titles DBA resolved 2017-01-02 2016-08-11
99485 implement performance_schema for mysql monitoring DBA resolved 2017-01-02 2017-02-14
99520 On a maintenance window, upgrade db1063 to 14.04 and its MariaDB package to 10.0.16 DBA resolved 2017-01-02 2015-05-26
100441 Santitize recent wikis: wikimania 2016 and cn.wikimedia.org at labs dbs DBA resolved 2017-01-02 2015-06-26
100527 Document/fix scripts for echo x1 DB requirements DBA resolved 2015-10-02 2016-10-09
100846 Add Story points (from Sprint Extension) to the phabricator data dump DBA resolved 2017-01-02 2015-06-18
101009 Enabling automatic buffer pool dumping on start/stop (puppet) for all servers DBA resolved 2017-01-02 2015-07-03
101084 es[12]00[123] maintenance and upgrade DBA resolved 2017-01-02 2015-06-25
101182 Permission problem on parsercache db servers DBA resolved 2017-01-02 2015-06-16
101347 need 'spaces' database for phabricator DBA resolved 2017-01-02 2015-06-04
101516 Upgrade db1022, which has an older kernel DBA resolved 2017-01-02 2017-05-04
101561 labsdb1002.eqiad.wmnet seems to have stopped replicating DBA resolved 2017-01-02 2015-07-02
101567 Rebuild s6 and s7 on labsdb1002 DBA resolved 2017-01-02 2015-06-12
101735 db1058 (s5 master) degraded RAID DBA resolved 2017-01-02 2015-07-12
101750 Delete dbname = 'centralauth' from meta_p.wikis table in labs replicas DBA resolved 2017-01-02 2015-10-02
101937 setup logging for #wikimedia-databases DBA resolved 2017-01-02 2016-11-25
102484 Tokudb requires disabling huge pages; puppetize on labsdb* hosts DBA resolved 2017-01-02 2015-06-15
102647 Add Wikipedia Northern Luri and Wikipedia Goan Konkani to labs replicas DBA resolved 2017-01-02 2015-07-23
102915 Special:GlobalUsers varies between claiming a user is or isn't attached DBA resolved 2017-01-02 2016-12-15
103230 Disk issue on db1028 DBA resolved 2017-01-02 2015-09-03
103417 Change character set on etherpad- lite database to utf8mb4_bin DBA resolved 2017-01-02 2015-06-23
103435 BadMethodCallException in sql object cache DBA resolved 2017-01-02 2015-07-06
103611 Update Gather DB schema for flagging backend DBA resolved 2017-01-02 2015-07-06
103843 Faulty memory on es2004 (purchase one module) DBA resolved 2017-01-02 2015-07-23
103936 review eqiad database server quantities / warranties / service(s) DBA resolved 2017-01-02 2015-12-22
104310 Create a whitelist of tables to checksum on all wikis DBA resolved 2017-01-02 2015-09-29
104313 Revision Deletion page won't load for too many (10) revisions DBA resolved 2017-01-02 2016-10-01
104471 Tables created on the s7 master have not been replicated to dbstore2001 and dbstore2002. Replication issue? DBA resolved 2017-01-02 2015-07-23
104476 Provision a labsdb useraccount that can be used to run replica-addusers.pl DBA resolved 2017-01-02 2015-12-01
104573 codfw frontends cannot connect to mysql at db2029 DBA resolved 2017-01-02 2015-07-06
104670 recentchanges deadlocks for dewiki (db1058) DBA resolved 2017-01-02 2015-07-26
104743 Check wikilove_log table exists on all wikis and create it where it doesn't DBA resolved 2017-01-02 2015-07-09
104748 db1046 innodb signal 6 abort and restart DBA resolved 2017-01-02 2016-01-25
104922 [Bug] Deadlock in EntityUsageTable::addUsages DBA resolved 2017-01-02 2015-09-21
105135 Implement mariadb 10.0 masters DBA resolved 2017-01-02 2016-05-02
105469 Page deletion fails when MathSearch is enabled (due to a foreign key restriction) DBA resolved 2017-01-02 2015-07-11
105503 Tables corrupted or impossible to work with them DBA resolved 2017-01-02 2015-07-13
105698 Uncommitted DB writes in GeoData::getAllCoordinates() DBA resolved 2017-01-02 2016-11-07
105699 Uncommitted DB writes in GlobalUsage::getLinksFromPage() DBA resolved 2017-01-02 2016-11-07
105768 Remove db1002-db1007 from production DBA resolved 2017-01-02 2017-02-10
105843 new external storage cluster(s) DBA resolved 2017-01-02 2016-01-27
105879 db1022 duplicate key errors DBA resolved 2017-01-02 2015-12-29
105911 tools.tools-info credentials are not functioning DBA resolved 2017-01-02 2016-01-27
105964 HELP! Database is getting Slow: A test which took less than 10 min, now it takes 3 hours. I cannot do my experiments. DBA resolved 2017-01-02 2017-08-02
106312 m1-master switch from db1001 to db1016 DBA resolved 2017-01-02 2016-06-22
106470 Tool Labs enwiki_p replicated database missing rows DBA resolved 2017-01-02 2016-06-29
106647 mariadb multi-source replication glitch with site_identifiers DBA resolved 2017-01-02 2015-08-15
106682 SpecialWhatLinksHere::showIndirectLinks - Lost connection to MySQL server during query (10.64.16.28) DBA resolved 2017-01-02 2016-08-29
106798 Special:AbuseFilter on all large Wikipedias is returning errors DBA resolved 2017-01-02 2015-07-30
107029 Lost connection to MySQL server during query DBA resolved 2017-01-02 2015-08-02
107070 Measure capacity and utilization of labsdb*** boxes DBA resolved 2017-01-02 2016-04-01
107072 Spikes of job runner new connection errors to mysql "Error connecting to 10.64.32.24: Can't connect to MySQL server on '10.64.32.24' (4)" - mainly on db1035 DBA resolved 2017-01-02 2015-12-16
107282 Reduce memory commitment on database hosts with many objects, specially s3, dbstore/research and labs DBA resolved 2017-01-02 2017-02-14
107319 [Bug] EntityUsageTable::touchUsages slow query DBA resolved 2017-01-02 2015-09-08
107753 Puppet failure on all hosts with Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Mysql::Error: Out of range value for column 'id' at row 1: INSERT INTO `fact_values` (`updated_at`, `host_id`, `created_at`, `fact_name_id`, `value`) VALUES('date', etc) DBA resolved 2017-01-02 2016-05-01
107923 Watchlist shows pages still bold when "mark all pages as read" button clicked DBA resolved 2017-01-02 2015-09-04
108033 duplicate key error on db1056 DBA resolved 2017-01-02 2015-08-18
108316 huge (140GB) decrease on available disk space on db1042 DBA resolved 2017-01-02 2015-08-14
108648 Migrate OAuth data from mediawikiwiki to metawiki DBA resolved 2017-01-02 2016-01-06
108856 Set up bucketization of editCount fields {tick} DBA resolved 2017-01-02 2017-05-03
108857 Delete obsolete schemas {tick} [5 pts] DBA resolved 2017-01-02 2015-12-01
109069 TokuDB crashes frequently -consider upgrade it or search for alternative engines with similar features DBA resolved 2017-01-02 2016-08-25
109279 Phabricator creates MySQL connection spikes: Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections. DBA resolved 2017-01-02 2017-09-26
109282 db2034 crashed DBA resolved 2017-01-02 2015-11-05
109401 dbtree shows 0 lag for db1047 DBA resolved 2017-01-02 2016-05-30
109676 Add artificial primary key to flow_wiki_ref and flow_ext_ref DBA resolved 2015-10-02 2016-03-01
109777 decom ishmael DBA resolved 2017-01-02 2015-10-06
109943 Long running query from LinksUpdate::incrTableUpdate job causing general lag DBA resolved 2017-01-02 2016-05-13
109972 tools.citationhunt can't access databases DBA resolved 2017-01-02 2016-04-08
110504 defragment db1015, db1035 and db1027 DBA resolved 2017-01-02 2016-12-21
110693 MySQL database for Nodepool DBA resolved 2017-01-02 2015-09-11
110963 Create CentralNotice campaign mixin tables DBA resolved 2017-01-02 2015-10-21
111084 Do backfill and final schema change for "Segregate Reference objects by source wiki." DBA resolved 2015-10-02 2015-10-22
111200 Grant puppet script access to "phabricator_project" DB DBA resolved 2017-01-02 2015-09-03
111254 Run "patch-remove_unique_ref_indices.sql" and phase 1 of "Segregate Reference objects by source wiki." on officewiki DBA resolved 2015-10-02 2015-09-07
111266 Make LoadBalancer slave lag check and read-only mode more robust (for example, using pt-heartbeat) DBA resolved 2017-01-02 2016-08-08
111333 wikishared database has latin1 as the default encoding DBA resolved 2017-01-02 2015-09-10
111371 Potential templatelinks data integrity issue on Tool Labs' enwiki_p DBA resolved 2017-01-02 2015-09-07
111547 phabricator metrics script should use slave, not master DBA resolved 2017-01-02 2015-09-10
111654 Set up TLS for MariaDB replication DBA resolved 2017-01-02 2017-07-11
111656 Replica database tables may need to be ANALYZEd again DBA resolved 2017-01-02 2015-09-07
111766 Add icinga check for all MySQL/MariaDB hosts to check they have the right read_only value DBA resolved 2017-01-02 2016-10-01
111992 Physical location SPOF because of database server distribution on a single rack (D1) DBA resolved 2017-01-02 2016-06-16
112041 New database for designate pool manager DBA resolved 2017-01-02 2015-09-10
112079 Upgrade x1 cluster DBA resolved 2017-01-02 2016-06-17
112080 the bounce_records table in x1 wikishared has latin1 encoding DBA resolved 2017-01-02 2016-01-29
112103 fix m5 shard mysql issues with default encoding binary DBA resolved 2017-01-02 2016-12-16
112193 phabricator dump script should use slave db, not master DBA resolved 2017-01-02 2015-10-07
112478 Upgrade db1055 mysql version and configuration, and reduce its pool weight DBA resolved 2017-01-02 2015-10-08
112479 Check, test and tune pool-of-connections and max_connections configuration DBA resolved 2017-01-02 2015-10-29
112573 Drop phlegal_* databases from m3 DBA resolved 2017-01-02 2016-07-06
112732 Truncate table cx_suggestions, cx_lists DBA resolved 2017-01-02 2015-10-01
112835 Document stricter MySQL policy and settings for MediaWiki DBA resolved 2017-01-02 2015-11-05
113901 Odd plan for simple query on revisions + page DBA resolved 2017-01-02 2016-04-22
114068 Clean up search user preferences in user_properties table on Wikimedia wikis DBA resolved 2017-01-02 2015-09-29
114208 User skin preference for MonoBook changed to default (Vector) on small and medium wikis (about 95% of wikis) DBA resolved 2017-01-02 2016-12-02
114396 Grant 'show view' permissions on s1-analytics-slave/jmorgan to user jmorgan DBA resolved 2017-01-02 2015-10-02
114513 Slow running query on cawiki.labsdb; incomplete/missing table stats? DBA resolved 2017-01-02 2015-10-02
114519 Another slow running query on - lvwiki.labsdb this time. DBA resolved 2017-01-02 2015-10-02
114559 s51053 (tools.jackbot) is abusing resources on labsdbs, throttle his grants DBA resolved 2017-01-02 2017-06-20
114752 Puppetize pt-heartbeat on MariaDB10 masters and its corresponding checks on the several monitoring backends DBA resolved 2017-01-02 2016-08-03
115275 Replicate Echo tables to analytics-store DBA resolved 2016-04-13 2017-10-20
115517 Data missing from June 11/12 on s3.labsdb DBA resolved 2017-01-02 2016-12-21
115682 hewiki's categorylinks shown as not empty though it is; purging does not help DBA resolved 2017-01-02 2017-05-20
115711 Drop `user_daily_contribs` table from all production wikis DBA resolved 2017-01-02 2015-12-11
115825 action=query&list=pagepropnames really slow on a big wiki, got error with ppnlimit=500 function: /* ApiQueryPagePropNames::execute */ DBA resolved 2017-01-02 2017-04-27
115982 Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas DBA resolved 2017-01-02 2017-08-02
116425 Rename user creates lag on enwiki DBA resolved 2017-01-02 2016-08-04
116557 AFComputedVariable::compute query timeouts DBA resolved 2017-01-02 2017-04-28
116599 Eventlogging tables not replicating from master to slave DBA resolved 2017-01-02 2015-10-27
116903 Adapt wmf-mariadb101 package for stretch and adapt its service to systemd DBA resolved 2017-01-02 2017-06-22
117089 Deadlock found when trying to get lock; try restarting transaction DBA resolved 2017-01-02 2016-08-11
117398 number of database updates multiplied x3 since 29 October DBA resolved 2017-01-02 2016-08-08
117783 Add ref_id to flow_wiki_ref and flow_ext_ref (DBA phase 1) DBA resolved 2015-11-05 2015-11-05
117785 Deploy artificial primary key reference change to all production wikis simultaneously DBA resolved 2015-11-05 2015-11-22
117786 Run maintenance/FlowPopulateRefId.php on all production wikis DBA resolved 2015-11-05 2015-12-17
117787 Make ref_id of flow_wiki_ref and flow_ext_ref not null and PK (DBA phase 2) DBA resolved 2015-11-05 2015-12-22
117881 Several hosts on virt1002 are down DBA resolved 2017-01-02 2015-11-05
118079 Bug on MariaDB use_stat_tables DBA resolved 2017-01-02 2016-04-22
118159 dbtree fails to render correctly on a new server (mw1152) both with zend php and hhvm DBA resolved 2017-01-02 2015-11-10
118162 Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections DBA resolved 2017-01-02 2015-11-19
118816 mysql privs: restrict access to racktables to krypton DBA resolved 2017-01-02 2016-01-29
119056 External Storage on codfw (es2005-2010) is consuming 100-90GB of disk space per server and per month and it has 370GB available DBA resolved 2017-01-02 2016-03-10
119206 Requests to globally reset a user's skin preferences DBA resolved 2017-01-02 2016-01-26
119305 s2/s51072__dwl_p is stuck DBA resolved 2017-01-02 2015-11-23
119315 labsdb1002 and labsdb1003 crashed, affecting replication DBA resolved 2017-01-02 2016-05-22
119370 Send deployment-db1 deployment-db2 syslog to beta cluster logstash DBA resolved 2017-01-02 2016-07-19
119380 db1046 running out of disk space DBA resolved 2017-01-02 2016-02-08
119461 Evaluate security concerns of logging beta cluster db queries on tendril DBA resolved 2017-01-02 2015-11-26
119488 dbstore1002.mgmt.eqiad.wmnet: "No more sessions are available for this type of connection!" DBA resolved 2017-01-02 2016-07-21
119509 Cleanup ptwikibooks conversion DBA resolved 2015-11-25 2017-03-14
119525 Create "echo_unread_wikis" table on wikishared database for Echo cross-wiki notifications DBA resolved 2017-01-02 2016-05-18
119567 Run Flow External Store migration in dry-run mode on Beta DBA resolved 2015-11-25 2016-06-22
119577 Client loses connection to database replica and cannot connect to it any further DBA resolved 2017-01-02 2015-11-27
119604 Faebot is crashing labsdb1002 DBA resolved 2017-01-02 2016-01-08
119619 Decide storage backend for performance schema monitoring stats DBA resolved 2017-01-02 2017-02-14
119642 Create a Master-master topology between datacenters for easier failover (setup circular replication dallas -> eqiad for mysql databases) DBA resolved 2017-01-02 2016-03-16
119648 Fix getLagFromPtHeartbeat() to use the pt entry for the root master, not the immediate one DBA resolved 2017-01-02 2016-08-05
119695 tools.joanjoc is executing the same >1h query every 5 minutes DBA resolved 2017-01-02 2015-11-26
119704 HHVM default configuration kills queries after one minute, disable that DBA resolved 2017-01-02 2015-11-26
119742 Database error at uk:Сетакон page DBA resolved 2017-01-02 2015-12-04
119751 Rename Database to DBA, create blocked-by-schema-change DBA resolved 2017-01-02 2016-09-12
119752 Review schema changes for T119495 DBA resolved 2017-01-02 2016-09-12
119811 Drop localisation and localisation_file_hash tables, l10nwiki databases too DBA resolved 2017-01-02 2017-07-12
119841 labs db inconsistent data DBA resolved 2017-01-02 2015-12-11
120119 Query runs extremely slow on analytics-store DBA resolved 2017-01-02 2015-12-04
120122 Perform a rolling restart of all MySQL slaves (masters too for those services with low traffic) DBA resolved 2017-01-02 2016-04-22
120187 Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) DBA resolved 2017-01-02 2016-02-04
120513 Apply recentchanges special partitioning to a redundant slave at least on s1, s4, and s5 DBA resolved 2017-01-02 2017-03-29
120674 Drop tables MobileWebClickTracking_* from eventlogging db DBA resolved 2017-01-02 2015-12-15
120967 Make EL create tables using TokuDB by default [5] DBA resolved 2017-01-02 2015-12-14
121120 2 hour outage to update mysql on EL slaves {oryx} [3 pts] DBA resolved 2017-01-02 2015-12-15
121207 Implement slave_run_triggers_for_rbr at sanitarium for labs filtering DBA resolved 2017-01-02 2017-07-20
121335 Create oauth_registered_consumer.oarc_owner_only column on metawiki DBA resolved 2017-01-02 2015-12-18
121509 Correct the schema to make revision id columns of cx_translations nullable DBA resolved 2017-01-02 2015-12-21
122048 db1024 (s2 master) will run out of disk space in ~4 months DBA resolved 2017-01-02 2016-02-25
122111 ref_target should be NOT NULL DBA resolved 2015-12-23 2016-02-25
122148 DBs provided on Tool Labs should have utf8 charset and collation by default DBA resolved 2017-01-02 2017-05-15
122338 Rename user database DBA resolved 2017-01-02 2015-12-24
122429 Batch updates create slave lag on s3 over WAN DBA resolved 2017-01-02 2016-02-05
122658 MySQL connections die in less than 30 seconds using tools-login tunnels DBA resolved 2017-01-02 2016-01-18
122998 setup/install/deploy db2033 DBA resolved 2017-01-02 2016-04-29
123108 Unthrottle faebot DBA resolved 2017-01-02 2016-01-19
123130 Labs: High Replication Lag (s6 on c1, s1 and s5 on c2) DBA resolved 2017-01-02 2016-01-18
123509 "db1047/eventlogging_sync processes" icinga alert is flaky since at least early January DBA resolved 2017-01-02 2017-04-12
123634 EL replication having issues since at least January 11th DBA resolved 2017-01-02 2016-02-08
123731 Migrate labsdb1005/1006/1007 to jessie DBA resolved 2017-01-02 2017-03-31
123985 Inconsistencies between labs replicas DBA resolved 2017-01-02 2017-01-01
124002 Untangle wikitech/labtestwikitech and s7 DBs and networking and mysql grants DBA resolved 2017-01-02 2016-01-20
124205 Error: 2013 Lost connection to MySQL server during query on NewFilesPager DBA resolved 2017-01-02 2016-08-01
124406 s7 master is executing 54K transactions/s DBA resolved 2017-01-02 2016-12-15
124443 (Potential) ORES database schema improvements DBA resolved 2017-01-02 2016-09-16
124464 db1069 is running low on space DBA resolved 2017-01-02 2016-02-13
124649 "Lock wait timeout exceeded" errors regarding module_deps DBA resolved 2017-01-02 2016-07-27
124681 Architecture decision to solve the need larger serves (for better capacity and consolidation) vs. more, smaller servers (for high availability) DBA resolved 2017-01-02 2016-08-12
124697 Stress-test mediawiki application servers at codfw (specially to figure out db weights configuration) and basic buffer warming DBA resolved 2017-01-02 2016-04-01
124699 Change configuration to make codfw db masters as the masters of all datacenters DBA resolved 2017-01-02 2016-04-20
124704 puppetize mysql m5-master access on parsoid-test server DBA resolved 2017-01-02 2016-01-27
124705 New database for access from stat1003 (project_illustration) DBA resolved 2017-01-02 2016-01-25
124795 codfw is in read only according to mediawiki DBA resolved 2017-01-02 2016-03-10
124805 What would be the preferred way to run a series of read queries on DB replica, each on two *_p databases ? DBA resolved 2017-01-02 2017-10-06
125027 upgrade dbproxy1001/1002 to jessie DBA resolved 2017-01-02 2016-07-28
125028 reimage or decom db servers on precise DBA resolved 2017-01-02 2016-07-22
125215 Prepare db1018 and s2-slaves for s2 master failover DBA resolved 2017-01-02 2016-02-10
125386 Clarify how mysql dumps will be architectured during codfw failover DBA resolved 2017-01-02 2016-02-08
125435 Grant mysql client access to testreduce_vd and testreduce_0715 databases DBA resolved 2017-01-02 2016-02-08
125591 Set up additional filters for Echo tables DBA resolved 2017-01-02 2016-02-04
125927 labs copy of nlwiktionary db corrupted DBA resolved 2017-01-02 2016-02-05
126096 Replicate wikimania2017wiki to labs DBA resolved 2017-01-02 2016-11-02
126209 Reimage db2012 DBA resolved 2017-01-02 2016-02-11
126832 Recreate a wiki for Wikimedia Portugal DBA resolved 2017-01-02 2017-08-23
126942 labs -Database is offline DBA resolved 2017-01-02 2017-09-22
126946 disk failure on labsdb1002 DBA resolved 2017-01-02 2016-07-28
126996 Reimage db1021 database DBA resolved 2017-01-02 2016-02-25
127040 Beta cluster update database is broken because of Flow db patch db_patches/patch-ref_target_not_null.sql DBA resolved 2016-02-17 2016-02-17
127107 Copy user database to c1 DBA resolved 2017-01-02 2016-02-17
127164 Some databases cannot be backed up/replicated on toolsdb DBA resolved 2017-01-02 2017-06-23
127219 Drop wikilove_image_log table from Wikimedia wikis DBA resolved 2017-01-02 2017-06-26
127228 tools.ptwikis throttled for excessive usage of labs db replica resources DBA resolved 2017-01-02 2016-03-11
127266 Tool Labs queries die DBA resolved 2017-01-02 2016-12-21
127330 Migration from es2001-es2010 to es2011-es2019 DBA resolved 2017-01-02 2016-03-10
127385 Add pigz package to all wmf mariadbs DBA resolved 2017-01-02 2016-02-22
127638 Email spam from some MariaDB's logrotate DBA resolved 2017-01-02 2016-06-08
127769 es2010 controller issue DBA resolved 2017-01-02 2016-05-24
127811 MariaDB package improvements DBA resolved 2017-01-02 2016-11-17
128185 Prepare mysql account and options for prometheus DBA resolved 2017-01-02 2016-08-03
128353 Switchover to new s3 master DBA resolved 2017-01-02 2016-04-20
128355 s52490 created 338 simultanous conections to the same server executing the same query DBA resolved 2017-01-02 2016-04-04
128737 Move labs pdns database off of m5-master DBA resolved 2017-01-02 2016-06-17
129236 Improve documentation about database switchover DBA resolved 2017-01-02 2016-04-28
129350 es2014 revert data compression DBA resolved 2017-01-02 2017-03-24
130067 Add wl_id to watchlist tables on production dbs DBA resolved 2017-01-02 2017-06-12
130098 Create a new x1 slave in codfw (just in case) DBA resolved 2017-01-02 2016-03-23
130128 Fix dbstore2001 and dbstore2002 DBA resolved 2017-01-02 2017-03-10
130351 Configure and provision db1074-db1078 DBA resolved 2017-01-02 2016-03-24
130692 Add new indexes from eec016ece6d2b30addcdf3d3efcc2ba59b10e858 to production databases DBA resolved 2017-01-02 2016-06-17
130700 Create central OATHAuth table for CentralAuth wikis DBA resolved 2017-01-02 2016-12-15
130702 Several es20XX servers keep crashing (es2017, es2019, es2015, es2014) since 23 March DBA resolved 2017-01-02 2017-06-06
130819 Fix cluster name for databases that go into the misc group DBA resolved 2017-01-02 2016-04-04
131146 Analyze memory usage on db1011 DBA resolved 2017-01-02 2017-10-06
131236 Identical EventLogging queries give different results on db1047 and dbstore1002 DBA resolved 2017-01-02 2017-03-17
131630 Tgr unable to login on Horizon DBA resolved 2017-01-02 2016-05-30
131705 dbstore1001 low available space DBA resolved 2017-01-02 2016-06-30
131753 Semi-synchronous replication status in all MySQL production clusters DBA resolved 2017-01-02 2017-03-23
131897 High load on tools labsdb1005 DBA resolved 2017-01-02 2016-04-07
131933 BaGLAMa2 has very long running queries DBA resolved 2017-01-02 2016-04-07
131937 Throttling Cyberbot tool user as it is consuming most of the CPU DBA resolved 2017-01-02 2016-05-24
132011 Reimage db2047 - check for hardware errors DBA resolved 2017-01-02 2017-09-24
132279 db1053 transient replica lag during dumping DBA resolved 2017-01-02 2016-08-02
132318 Job Queue growing and then running a lot of jobs at once on commonswiki DBA resolved 2017-01-02 2016-04-11
132416 Rampant differences in indexes on enwiki.revision across the DB cluster DBA resolved 2017-01-02 2017-08-07
132431 labsdb1001 and labsdb1003 short on available space DBA resolved 2017-01-02 2017-04-27
132515 db1070, db1071 and db1065 overheating problems DBA resolved 2017-01-02 2016-04-27
132539 Dumps for misc triggers unnecessary pages DBA resolved 2017-01-02 2016-12-14
132696 DBA-related homes on tin DBA resolved 2017-01-02 2017-06-23
132837 hitcounter and _counter tables are on the cluster but were deleted/unsused? DBA resolved 2017-01-02 2016-12-13
133086 Killed very long transaction that was blocking the replica on labsdb1001 DBA resolved 2017-01-02 2017-07-18
133122 Backfill data into db1065 and db1066 DBA resolved 2017-01-02 2016-04-20
133160 Jobqueue increase activity on refreshlinks for frwiktionary DBA resolved 2017-01-02 2016-04-22
133205 Switchback to eqiad DBA resolved 2017-01-02 2016-04-21
133265 Improve external storage MySQL configuration for connection spikes DBA resolved 2017-01-02 2016-04-21
133309 db1041 regular mysql stalls creates connection failures DBA resolved 2017-01-02 2016-04-22
133321 `s51187__xtools_tmp` database using 272G on labsdb1001 and 118G on labsdb1003 DBA resolved 2017-01-02 2017-07-10
133322 u3532__ (=marcmiquel) table using 64G on labsdb1001 and 108 GB on labsdb1003 DBA resolved 2017-01-02 2017-03-28
133323 u2815__old_p (dispenser) database using 20G on labsdb1001 (enwiki) DBA resolved 2017-01-02 2016-04-21
133325 s51127__dewiki_lists (merlbot) database using 13G on labsdb1001 (enwiki) DBA resolved 2017-01-02 2017-06-21
133326 p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) DBA resolved 2017-01-02 2017-07-21
133337 Automate database datacenter switchover steps DBA resolved 2017-01-02 2016-07-14
133338 Create salt database groups DBA resolved 2017-01-02 2016-05-10
133339 Improve lag detection mechanism's reliability and agility DBA resolved 2017-01-02 2016-05-18
133385 Implement GTID replication on MariaDB 10 servers DBA resolved 2017-01-02 2016-06-06
133386 Change dbstore1001 delayed slave to be a direct slave of the eqiad masters DBA resolved 2017-01-02 2016-12-09
133398 Install, configure and provision recently arrived db core machines DBA resolved 2017-01-02 2017-03-14
133584 Reduce stress caused by IABot on the DB DBA resolved 2017-01-02 2016-08-05
133588 Gerrit 285208 broke eventlogging_sync.sh DBA resolved 2017-01-02 2016-11-10
133715 Missing data on labs replica database DBA resolved 2017-01-02 2016-12-21
133780 Multiple Puppet class make MySQL load /etc/my.cnf twice DBA resolved 2017-01-02 2016-10-13
133801 SELECT /* CategoryMembershipChangeJob::run 127.0.0.1 */ GET_LOCK('CategoryMembershipUpdates:XXXX', 10) AS lockstatus DBA resolved 2017-01-02 2016-10-09
134114 Investigate db1040 for overload DBA resolved 2017-01-02 2016-05-03
134203 enwiki_p replica on s1 is corrupted DBA resolved 2017-01-02 2017-01-25
134349 Upgrade db1069 DBA resolved 2017-01-02 2016-05-10
134392 Add shared edit rights between eranbot and plagiabot tools DBA resolved 2017-01-02 2016-06-14
134480 Improve replication lag detection for multi-dc environment DBA resolved 2017-01-02 2016-05-24
134481 Refactor $master in mariadb::core Puppet class DBA resolved 2017-01-02 2016-06-17
134555 db1033 (old s7 master) needs backup and reimage DBA resolved 2017-01-02 2016-05-20
134755 Decommission es2005-es2010 DBA resolved 2017-01-02 2016-09-23
134977 Throttle mysql backups on dbstore1001 in order to not saturate the node DBA resolved 2017-01-02 2017-01-03
135027 MASTER_POS_WAIT() alternative that works cross-DC DBA resolved 2017-01-02 2017-04-03
135029 adywiki and jamwiki are missing the associated *_p databases with appropriate views DBA resolved 2017-01-02 2016-11-25
135100 High replication lag to dewiki DBA resolved 2017-01-02 2016-05-26
135278 Import page restrictions to Cassandra restriction table DBA resolved 2017-01-02 2016-10-18
135405 Replicate CentralNotice tables to Labs DBA resolved 2017-01-02 2017-08-31
135471 Hundreds of concurrent SELECT MASTER_POS_WAIT('db1049-bin.002927', 354219766, 10) without debugging information (no user, not function) on s5 API node DBA resolved 2017-01-02 2016-05-26
135690 DBReplicationWaitError: Could not wait for slaves to catch up to 10.64.0.7 DBA resolved 2017-01-02 2016-07-20
135699 Schema changes for Echo moderation DBA resolved 2016-05-20 2016-06-06
135944 db1034 was killed 22-05-16 at 14:17:06 DBA resolved 2017-01-02 2016-06-06
135947 Missing grants on tools.labsdb DBA resolved 2017-01-02 2016-05-31
135971 labsdb1001 crashed yesterday at 21:48:07 DBA resolved 2017-01-02 2016-05-31
135973 Upgrade m1 db servers DBA resolved 2017-01-02 2016-06-27
136135 DB backup plan for ptwikibooks Flow cleanup DBA resolved 2017-01-02 2016-05-26
136136 db1056 BBU Failed DBA resolved 2017-01-02 2016-05-25
136214 Query from stat1003 brought down db1047 DBA resolved 2017-01-02 2017-07-03
136319 Labs users missing grants on replicas DBA resolved 2017-01-02 2016-07-02
136333 dbstore1002 crashed DBA resolved 2017-01-02 2016-06-06
136398 Empty sql-mode in my.cnf DBA resolved 2017-01-02 2016-06-17
136583 db2034 degraded RAID DBA resolved 2017-01-02 2016-07-14
136598 Wikidata master database connection issue DBA resolved 2017-01-02 2016-06-16
136618 Wrong page title in labs database replica enwiki page table DBA resolved 2017-01-02 2017-03-03
136789 High replag on db1069 DBA resolved 2017-01-02 2016-06-02
137191 Defragment db1070, db1082, db1087, db1092 DBA resolved 2017-01-02 2017-03-24
137567 Review ORES extension for data leaks DBA resolved 2017-01-02 2016-06-10
137691 Add shared edit rights between eranbot and copypatrol tools DBA resolved 2017-01-02 2016-06-15
137895 hidenondamaging is slow DBA resolved 2017-01-02 2016-08-12
138283 Some users use persistent connections that are idle, wasting memory and other resources that could be used for other users DBA resolved 2017-01-02 2017-04-11
138450 maintain-replicas.pl unmaintained, unmaintainable DBA resolved 2017-01-02 2016-10-18
138460 Upgrade m3 (phabricator) db servers DBA resolved 2017-01-02 2016-08-25
138516 etherpad database issues DBA resolved 2017-01-02 2016-07-07
138559 Correct m1 backups, that were being done supposing m1-master was still a slave DBA resolved 2017-01-02 2016-06-24
138778 Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 DBA resolved 2017-01-02 2016-10-01
138967 Labs database replica drift DBA resolved 2017-01-02 2017-10-18
139055 Test InnoDB compression DBA resolved 2017-01-02 2016-12-01
139056 API database error on continue with generator=categorymembers and prop=linkshere DBA resolved 2017-01-02 2017-04-12
139090 Deploy I2b042685 to all databases DBA resolved 2017-01-02 2016-11-08
139280 db1034 decommission DBA resolved 2017-01-02 2016-10-13
139283 pc2006 down DBA resolved 2017-01-02 2016-07-08
139342 DROP OAI-related tables DBA resolved 2017-01-02 2017-06-14
139346 All s4 (commons) slaves failed replication DBA resolved 2017-01-02 2016-07-08
139552 Create tables for PageAssessments in enwiki database DBA resolved 2017-01-02 2016-08-04
139618 Petscan is being used with excesive parallelism by a user on Wikidata DBA resolved 2017-01-02 2016-07-11
139714 BIOS upgrade on certain codfw machines DBA resolved 2017-01-02 2016-08-03
139755 Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades DBA resolved 2017-01-02 2016-07-08
140108 ApiQueryRecentChanges::run is spiking, nuking API servers DBA resolved 2017-01-02 2016-10-25
140337 dbstore1002 disk errors DBA resolved 2017-01-02 2016-07-22
140341 dbstore1002 lag on s7 DBA resolved 2017-01-02 2016-07-14
140452 Initial setup and provision of labsdb1009, labsdb1010 and labsdb1011 DBA resolved 2017-01-02 2016-12-14
140537 Query for Special:Contributions?contribs=newbie&dir=prev is slow on enwiki DBA resolved 2017-01-02 2016-07-19
140650 Special:Block request makes the database timeout on db1055 DBA resolved 2017-01-02 2016-10-20
140803 Remove oresc_rev index DBA resolved 2017-01-02 2017-07-03
140983 dbproxy1002 down DBA resolved 2017-01-02 2016-07-28
141090 Investigate increase in "readonly" saving errors in Content Translation DBA resolved 2017-01-02 2016-10-09
141096 Having lots of accounts with separate grants makes auditing difficult. DBA resolved 2017-01-02 2017-02-14
141097 Implement a frontend failover solution for labsdb replicas DBA resolved 2017-01-02 2016-12-21
141252 icinga hp raid check timeout on busy ms-be and db machines DBA resolved 2017-01-02 2017-08-16
141601 db2069 crashed due to RAID controller DBA resolved 2017-01-02 2016-07-29
141988 After operation aborted, on-wiki log is missing, but it was reported to irc.wikimedia DBA resolved 2017-01-02 2016-08-16
142223 Enable access to Wikipedia Tulu (tcywiki) on labs replicas DBA resolved 2017-01-02 2016-12-17
142231 The Optimistic Guy -> RainFall global rename completed without log entry DBA resolved 2017-01-02 2016-12-15
142273 dbstore2002 stopped providing mysql service despite the process being running DBA resolved 2017-01-02 2016-08-09
142305 Cannot drop table DBA resolved 2017-01-02 2016-08-06
142310 Replication seems to be halted for multiple databases DBA resolved 2017-01-02 2016-08-12
142545 Create production database and users for Striker DBA resolved 2017-01-02 2017-03-04
142725 Clean up revision UNIQUE indexes DBA resolved 2017-01-02 2017-06-06
143220 es2004 has a dead disk, but it is not under warranty DBA resolved 2017-01-02 2016-08-24
143862 s3 throughput tripled since 24 august DBA resolved 2017-01-02 2016-08-25
143932 MySQL chooses poor query plan for link counting query DBA resolved 2017-01-02 2016-11-17
143934 s2 replag currently 8 hours DBA resolved 2017-01-02 2016-08-26
144180 u13367 is running 2 inefficient 9-day-long queries, causing high cpu usage DBA resolved 2017-01-02 2016-08-29
144278 Queries on PageContentSaveComplete are starting to pileup DBA resolved 2017-01-02 2016-09-01
144661 Add section for long-running tasks on the Deployment page (specially for database maintenance) DBA resolved 2017-01-02 2016-09-20
144723 Reimage & upgrade db1064 DBA resolved 2017-01-02 2016-09-07
145077 mysqld process hang in db1069 - S2 mysql instance DBA resolved 2017-01-02 2016-10-04
145079 Investigate slow transcludedin query DBA resolved 2017-01-02 2016-10-25
145356 Ensure ORES data violating constraints do not affect production DBA resolved 2017-01-02 2016-10-11
145366 Create and populate babel database table on Wikimedia wikis DBA resolved 2017-01-02 2017-08-01
145487 Investigate (and if possible, drop) _counters DBA resolved 2017-01-02 2016-10-17
145533 Investigate db1082 crash DBA resolved 2017-01-02 2017-10-18
145607 db1082 hardware check DBA resolved 2017-01-02 2016-09-15
145630 Cannot delete two pages with large histories even having the appropriate permissions to do so DBA resolved 2017-01-02 2016-09-14
145844 db2017 failed disk (degraded RAID) DBA resolved 2017-01-02 2016-10-05
146018 Investigate db1061 crash DBA resolved 2017-01-02 2017-04-05
146079 s5-master contention caused (?) by refreshlinksprioritized/addUsagesForPage jobs running for all wikis DBA resolved 2017-01-02 2016-09-20
146111 hidenondamaging=1 query is extremely slow on enwiki DBA resolved 2017-01-02 2016-10-11
146121 db1069: convert user_groups table to InnoDB across all the wikis DBA resolved 2017-01-02 2016-10-03
146261 Reimage dbstore2001 as jessie DBA resolved 2017-01-02 2017-02-14
146265 db1019: Decommission DBA resolved 2017-01-02 2017-02-16
146449 db1060: Degraded RAID DBA resolved 2017-01-02 2016-09-26
146529 5-10 times more rows are currently read from the s5 master than from all other masters together DBA resolved 2017-01-02 2016-09-28
146570 Give user_properties a primary key DBA resolved 2017-08-05 2017-08-29
146673 Contention on search phabricator database creating full phabricator outages DBA resolved 2017-01-02 2016-10-14
146718 [Discuss] Hosting the monthly article quality dataset on labsDB DBA resolved 2017-01-02 2017-08-17
147051 Provide at least 2 separate service endpoints: one for slow, long running queries; and another for quick, web requests DBA resolved 2017-01-02 2016-12-21
147052 Provision with data the new labsdb servers and provide replica service with at least 1 shard from a sanitized copy from production DBA resolved 2017-01-02 2016-12-21
147166 Apply change_tag and tag_summary primary key schema change to Wikimedia wikis DBA resolved 2017-01-02 2017-05-16
147172 db1055: degraded array DBA resolved 2017-01-02 2016-10-06
147296 img_metadata queries for PDF files saturates s4 slaves DBA resolved 2017-01-02 2016-12-02
147302 Prepare and check production and labs-side filtering for olowiki DBA resolved 2017-01-02 2016-11-02
147305 Unify commonswiki.revision DBA resolved 2017-01-02 2017-06-02
147396 db1065: Degraded RAID DBA resolved 2017-01-02 2016-10-10
147734 Review and deploy 309825 DBA resolved 2017-01-02 2016-10-21
147769 es2015 crashed with no os logs (kernel logs or other software ones) - it shuddenly went down DBA resolved 2017-01-02 2017-03-13
147774 Physically move db1053 to a different rack DBA resolved 2017-01-02 2016-10-13
147957 CirrusSearch SQL query for locating pages for reindex performs poorly DBA resolved 2017-01-02 2017-03-24
148373 db2037: Disk in predictive failure DBA resolved 2017-01-02 2016-10-20
148434 es2014 raid controler temporary failure DBA resolved 2017-01-02 2016-10-18
148500 Implement proxysql both for labs and for later production usage DBA resolved 2017-01-02 2016-10-26
148560 Create maintain-views user for labsdb1001 and labsdb1003 DBA resolved 2017-01-02 2016-11-02
148561 Replicate ores_classification and ores_model tables in labs DBA resolved 2017-01-02 2017-02-22
148609 Review and deploy Linter extension to Wikimedia wikis DBA resolved 2017-04-04 2017-10-13
148633 Degraded RAID on db1046 DBA resolved 2017-01-02 2016-10-20
148790 apply events killing long running queries to db1070; any other production server DBA resolved 2017-01-02 2016-10-28
148866 DBA review of Linter extension DBA resolved 2017-01-02 2017-07-04
148967 Fix PK on S5 dewiki.revision DBA resolved 2017-01-02 2017-06-02
148988 Cognate DB review DBA resolved 2017-01-02 2017-06-07
149079 codfw: Fix S4 commonswiki.templatelinks partitions DBA resolved 2017-01-02 2017-04-21
149099 db2011 disk media errors DBA resolved 2017-01-02 2016-11-03
149186 Check wikitech has the right db grants DBA resolved 2017-01-02 2016-11-03
149357 dbtree broken DBA resolved 2017-01-02 2016-11-02
149377 Degraded RAID on db2052 DBA resolved 2017-01-02 2016-11-03
149418 Deploy gtid_domain_id flag in our mysql hosts DBA resolved 2017-01-02 2017-03-07
149422 Prepare for mariadb 10.1 DBA resolved 2017-01-02 2016-11-03
149457 install new disks into dbstore2001 DBA resolved 2017-01-02 2016-11-21
149509 Degraded RAID on db1050 DBA resolved 2017-01-02 2016-10-31
149526 es2019 crashed again DBA resolved 2017-01-02 2017-05-09
149553 db2034: investigate its crash and reimage DBA resolved 2017-01-02 2017-04-11
149624 Labsdbs for WMF tools and contributors: get more data, faster DBA resolved 2017-01-02 2017-05-12
149633 db1065 paged for NRPE timeout DBA resolved 2017-01-02 2016-11-03
149637 Implement functionality for RC page 'Experience level' filters DBA resolved 2016-11-01 2017-03-05
149728 Multiple hardware issues on db1073 DBA resolved 2017-01-02 2016-12-16
149793 Decommission db1042 DBA resolved 2017-01-02 2017-04-12
149819 Add primary keys to remaining Flow tables DBA resolved 2016-11-03 2017-01-25
149844 Decide between proxysql and haproxy for labsdbproxy service DBA resolved 2017-01-02 2016-11-14
149933 Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too DBA resolved 2017-01-02 2017-03-27
150017 Reimage dbstore2002 DBA resolved 2017-01-02 2016-11-04
150211 labsdb1009 boot issues (power supply and controller?) DBA resolved 2017-01-02 2016-11-22
150233 db2034 crashes meta ticket DBA resolved 2017-01-02 2017-01-16
150306 truncate l10n_cache table on WMF wikis DBA resolved 2017-01-02 2017-09-07
150334 db2042 reimage DBA resolved 2017-01-02 2016-11-16
150474 Under high load, there is replication check pile-ups on coredbs, specially enwiki API servers DBA resolved 2017-01-02 2017-02-14
150511 db2035: RAID disk about to fail DBA resolved 2017-01-02 2016-11-22
150518 Import S5 to dbstore2001 and dbstore2002 + compression DBA resolved 2017-01-02 2016-11-18
150553 High replication activity filled up labsdb1004 with binlogs DBA resolved 2017-01-02 2017-02-14
150644 Wikidatawiki revision table needs unification DBA resolved 2017-01-02 2017-02-27
150747 Compress InnoDB on S5 (wikidatawiki, dewiki) DBA resolved 2017-01-02 2016-11-18
150802 Provision db1095 with at least 1 shard, sanitize and test slave-side triggers DBA resolved 2017-01-02 2016-12-07
150832 Replicate page_assessments and page_assessments_projects tables on Labs DBA resolved 2017-01-02 2016-12-07
150851 Moving backup and otrs role into their own .pp DBA resolved 2017-01-02 2016-12-01
150876 db2049 overheated and restarted DBA resolved 2017-01-02 2017-06-06
150960 Initial data tests for db1095 (temporary db1069 - sanitarium replacement) DBA resolved 2017-01-02 2016-12-07
150965 phabricator close to saturate its database connections DBA resolved 2017-01-02 2016-11-17
150974 db2042 disk predictive failure DBA resolved 2017-01-02 2016-12-07
151014 Provisioning MySQL replica users fails on tool labs DBA resolved 2017-01-02 2017-01-20
151029 duplicate key problems DBA resolved 2017-01-02 2017-08-14
151039 Unknown cause is creating lag on db1048 under write load (but not on the other m3 slaves) DBA resolved 2017-01-02 2016-12-15
151076 labsdbadmin needs to be able to drop users and update passwords DBA resolved 2017-01-02 2016-11-18
151203 db2041: Disk RAID predictive failure DBA resolved 2017-01-02 2016-12-02
151228 Editing a recurring event overrides all past instances DBA resolved 2017-01-02 2016-12-09
151272 db1092 crash DBA resolved 2017-01-02 2016-11-28
151356 Wikibase\Repo\Store\Sql\SqlEntityIdPager::fetchIds query slow DBA resolved 2017-01-02 2016-11-28
151384 phabricator_conduit.conduit_methodcalllog failed replicating on dbstore1002, probably m3 needs a reload on that server DBA resolved 2017-01-02 2016-11-23
151465 Degraded RAID on db1053 DBA resolved 2017-01-02 2017-03-03
151552 Import S2,S6,S7,m3 and x1 to dbstore2001 and dbstore2002 DBA resolved 2017-01-02 2017-03-06
151607 Rebuild old timestamp format tables DBA resolved 2017-01-02 2017-09-12
151746 fstrim: Operation not supported on Labs DBs DBA resolved 2017-01-02 2017-04-12
151752 Prepare and check storage layer for the future private wiki arbcom-cs.wikipedia.org DBA resolved 2017-01-02 2016-12-11
151756 Prepare and check storage layer for new fi.wikivoyage.org DBA resolved 2017-01-02 2016-12-01
151763 Degraded RAID on db2068 DBA resolved 2017-01-02 2016-12-07
151993 Implement ChangeDispatchCoordinator based on RedisLockManager DBA resolved 2017-01-02 2017-04-09
151995 Rolling restart of external storage servers for TLS certificate update DBA resolved 2017-01-02 2016-12-01
152029 Rolling restart of parsercache servers for TLS certificate update DBA resolved 2017-01-02 2016-12-01
152043 Tool Labs: Add skin, language, and variant to user_properties_anon DBA resolved 2017-01-02 2017-04-12
152081 Moving eventlogging mariadb role into its own .pp DBA resolved 2017-01-02 2017-07-14
152188 Restart pending mysql hosts with old TLS cert DBA resolved 2017-01-02 2017-02-08
152194 Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 DBA resolved 2017-01-02 2016-12-21
152214 Degraded RAID on db1033 DBA resolved 2017-01-02 2017-01-04
152364 db1047 out of disk space, eventlogging_sync spam DBA resolved 2017-01-02 2016-12-07
152377 Create labsdb_accounts db on m5 to store state about labsdb accounts DBA resolved 2017-01-02 2016-12-05
152411 Degraded RAID on db1048 DBA resolved 2017-01-02 2017-01-04
152585 rcshow=oresreview is slow DBA resolved 2017-01-02 2017-03-16
152761 db1028 increased lag after extensions/CentralAuth/maintenance/populateLocalAndGlobalIds.php DBA resolved 2017-01-02 2016-12-15
152766 dbstore1001 duplicate entry on s4 commonswiki.recentchanges DBA resolved 2017-01-02 2016-12-12
152894 high s1 replag to labs DBA resolved 2017-01-02 2016-12-11
153033 Drop MoodBar tables from all wikis DBA resolved 2017-01-02 2017-10-11
153184 s3 database resource usage and contention increased 2-10x times DBA resolved 2017-01-02 2016-12-15
153194 Remove partitions from metawiki.pagelinks DBA resolved 2017-01-02 2016-12-15
153213 Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" DBA resolved 2017-01-02 2017-06-26
153300 Remove partitions from metawiki.pagelinks in s7 DBA resolved 2017-01-02 2017-04-27
153634 db1047 paged low on disk space DBA resolved 2017-01-02 2016-12-20
153680 Create a cronjob/check to run check_private_data data script and report back DBA resolved 2017-01-02 2017-04-27
153739 Defragment db1015 DBA resolved 2017-01-02 2017-01-26
153740 Degraded RAID on db2011 DBA resolved 2017-01-02 2017-01-30
153743 Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts DBA resolved 2017-01-02 2017-09-01
153768 Install and reimage dbstore1001 as jessie DBA resolved 2017-01-02 2017-03-13
153826 Defragment db1044 DBA resolved 2017-01-02 2017-01-18
154031 db2060 crashed (RAID controller) DBA resolved 2017-01-02 2017-01-24
154097 Remove partitions from enwiktionary.templatelinks in s2 DBA resolved 2017-01-02 2017-03-28
154189 db2035 reset DBA resolved 2017-01-02 2017-01-03
154327 Gerrit: Schedule downtime for T154205 (To do with data loss) DBA resolved 2017-01-02 2017-01-06
154355 page_lang column of the page table is not replicated to Labs DBA resolved 2017-01-02 2017-03-22
154465 Defragment db1038 DBA resolved 2017-01-04 2017-01-20
154485 run pt-table-checksum on s2 (WAS: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038) DBA resolved 2017-01-04 2017-05-05
154497 Spatial database for tool-labs DBA resolved 2017-01-04 2017-01-12
154685 Sqoop doesn't run anymore - Seem related to a DB change (analytics store) DBA resolved 2017-01-06 2017-01-18
154929 db1026 (s5) needs some compression DBA resolved 2017-01-10 2017-01-19
155041 Prepare and check storage layer for wikimania2018wiki DBA resolved 2017-01-11 2017-08-07
155042 Increase quota for wikidata-dev project DBA resolved 2017-01-11 2017-01-24
155091 "Invalid DB key" errors on various special pages DBA resolved 2017-01-12 2017-10-13
155099 Database maintenance scheduled while eqiad datacenter is non primary (after the DC switchover) DBA resolved 2017-01-12 2017-05-03
155102 Review capacity on codfw DBA resolved 2017-01-12 2017-03-02
155190 Build an environment to test change dispatching using Redis-based locking DBA resolved 2017-01-13 2017-03-07
155399 db1045 disk space - compression needed DBA resolved 2017-01-17 2017-02-22
155500 Fatal exception of type "DBQueryError" on sorting ORES contributions DBA resolved 2017-01-18 2017-01-24
155605 Schema changes for expiring user groups DBA resolved 2017-01-19 2017-04-25
155769 Spurious completely empty `image` table row on commonswiki DBA resolved 2017-01-20 2017-04-26
155902 Reset password for database user p50380g50491 DBA resolved 2017-01-22 2017-01-25
156004 Move db1051 to row B3 DBA resolved 2017-01-24 2017-01-24
156005 Reimage db1065 and db1066 DBA resolved 2017-01-24 2017-01-25
156006 Move db1052 to row B3 DBA resolved 2017-01-24 2017-01-24
156008 Switchover s1 master db1057 -> db1052 DBA resolved 2017-01-24 2017-01-31
156126 Move db1073 to B3 DBA resolved 2017-01-25 2017-02-09
156161 db2060 not accessible DBA resolved 2017-01-25 2017-02-14
156166 db1072 change_tag schema and dataset is not consistent DBA resolved 2017-01-25 2017-01-31
156226 Reimage and clone db1072 DBA resolved 2017-01-26 2017-02-06
156285 Labs users reporting timouts when connecting to labsdb-web.eqiad.wmnet DBA resolved 2017-01-26 2017-01-25
156305 Update page_assessments_projects schema for subprojects in production DBA resolved 2017-01-26 2017-02-14
156373 During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time DBA resolved 2017-01-27 2017-02-01
156478 Change rack for servers in s1 in codfw DBA resolved 2017-01-28 2017-02-17
156560 lab's templatelinks contain possible garbage rows DBA resolved 2017-01-29 2017-01-28
156758 Drop m3 from dbstore servers DBA resolved 2017-02-01 2017-08-10
156905 Phabricator master and slave crashed DBA resolved 2017-02-02 2017-02-14
157348 Have search recognise STL files as a new kind of media file ('type:3d' or whatever) DBA resolved 2017-05-19 2017-07-29
157358 labsdb1005 (mysql) maintenance for reimage DBA resolved 2017-02-07 2017-05-15
157714 Better mysql command prompt info for Beta DBA resolved 2017-02-10 2017-06-14
157931 s5: db1070 not using file per table DBA resolved 2017-02-14 2017-03-17
158188 db1082 MySQL crashed DBA resolved 2017-02-16 2017-10-18
158193 Reimage db1060 due to pysical disk corruption (was: Degraded RAID on db1060) DBA resolved 2017-02-16 2017-03-08
158194 Replace BBU for db1060 DBA resolved 2017-02-16 2017-02-22
158246 codfw: switch ports clean up DBA resolved 2017-02-16 2017-02-16
158446 fundraising database tuning DBA resolved 2017-02-18 2017-04-17
158761 Degraded RAID on db1049 DBA resolved 2017-02-23 2017-02-22
158764 m1 slaves all broke replication due to bacula.DelCandidates temporary table DBA resolved 2017-02-23 2017-03-02
158893 dbstore1001 troubleshoot IPMI issue DBA resolved 2017-02-24 2017-07-10
158969 Degraded RAID on db1070 DBA resolved 2017-02-25 2017-03-17
159023 Possible labs imagelinks drift DBA resolved 2017-02-26 2017-03-06
159266 db1047 BBU RAID issues (was: Investigate db1047 replication lag) DBA resolved 2017-03-01 2017-06-19
159410 Degraded RAID on db1056 DBA resolved 2017-03-03 2017-03-03
159414 Rampant differences in indexes and PK on s6 (frwiki, jawiki, ruwiki) for revision table DBA resolved 2017-03-03 2017-03-14
159493 Data integrity issue with enwiki_p user_groups on Wikimedia Tool Labs (missing rows) DBA resolved 2017-03-03 2017-03-05
159572 labsdb1004 MySQL crash DBA resolved 2017-03-04 2017-03-08
159665 Degraded RAID on db2044 DBA resolved 2017-03-07 2017-03-08
159666 Degraded RAID on db2048 DBA resolved 2017-03-07 2017-03-08
159707 Import x1 on dbstore2001 DBA resolved 2017-03-07 2017-03-13
159718 Evaluate how to best add a column for full entity ID to wb_terms without affecting wikidata.org users DBA resolved 2017-03-07 2017-05-05
159753 Concerns about ores_classification table size on enwiki DBA resolved 2017-03-08 2017-10-17
159826 Use redis-based lock manager in dispatch changes in production DBA resolved 2017-03-08 2017-05-08
159828 Use redis-based lock manager for dispatchChanges on test.wikidata.org DBA resolved 2017-03-08 2017-04-12
160242 es2015 crashed on 2017-03-11 DBA resolved 2017-03-12 2017-03-23
160390 Unify revision table on s7 DBA resolved 2017-03-15 2017-05-19
160392 Reset db1070 idrac DBA resolved 2017-03-15 2017-07-10
160407 Apply wikishared.cx_translations index change DBA resolved 2017-03-15 2017-03-15
160415 Review schema changes for T125071 - Add index to image table on all wikis DBA resolved 2017-03-16 2017-04-12
160435 db1057 does not react to powercycle/powerdown/powerup commands DBA resolved 2017-03-16 2017-04-04
160509 run pt-tablechecksum on s6 DBA resolved 2017-03-16 2017-10-13
160611 Make "linter" table available on Labs DBA resolved 2017-03-27 2017-05-09
160686 ug_expiry column of the user_groups table is not present on Labs DBA resolved 2017-03-17 2017-04-24
160691 Refreshing testreduce_vd database on ruthenium DBA resolved 2017-03-17 2017-03-17
160713 Replicate babel db table on Labs DBA resolved 2017-03-17 2017-07-07
160832 db1094 crash DBA resolved 2017-03-19 2017-03-22
160869 Prepare and check storage layer for kbp.wikipedia.org DBA resolved 2017-03-20 2017-07-04
160887 Wikibase\Repo\Store\Sql\SqlEntitiesWithoutTermFinder::getEntitiesWithoutTerm can take 19 hours to execute and it is run by the web requests user DBA resolved 2017-03-21 2017-06-25
160984 Reduce max execution time of interactive queries or a better detection and killing of bad query patterns DBA resolved 2017-03-22 2017-08-07
160991 Check OATHAuth tables are in each private or fishbowl wiki DBA resolved 2017-03-22 2017-03-21
161007 Decouple Mariadb semi-sync replication from $::mw_primary DBA resolved 2017-03-22 2017-05-03
161024 dbstore2001 creates bad query plans for wikidata.wb_changes DBA resolved 2017-03-22 2017-03-22
161159 Cannot access the database: Can't connect to MySQL server on '10.192.48.41' (111) (10.192.48.41) DBA resolved 2017-03-23 2017-03-23
161232 Bump MediaWiki's minimum supported MySQL Version to 5.5.8 DBA resolved 2017-03-24 2017-10-20
161524 Implements ContentHandler abstraction for ProofreadPage Index: pages DBA resolved 2017-03-28 2017-08-02
161600 Degraded RAID on db1067 DBA resolved 2017-03-29 2017-03-29
161631 [Task] Remove "all" option for Special:EntitiesWithout*" DBA resolved 2017-03-29 2017-04-17
161683 Remove partitioning from db2019 (codfw master) commonswiki.templatelinks DBA resolved 2017-03-30 2017-04-05
161712 codfw: (1) spare pool system for temp allocation as database failover DBA resolved 2017-03-30 2017-05-04
161781 s2 on db1069 stuck on a query for trwiki flaggedrevs_tracking table DBA resolved 2017-03-31 2017-04-18
162133 Replace some masters in eqiad while it is not active DBA resolved 2017-04-05 2017-04-26
162135 Decommission db1057 DBA resolved 2017-04-05 2017-05-03
162138 How many revision comments are exactly the same? Get some stats. DBA resolved 2017-04-05 2017-04-07
162159 codfw rack/setup 22 DB servers DBA resolved 2017-04-05 2017-07-14
162183 tendril cert expiry alerts on dbmonitor hosts DBA resolved 2017-04-05 2017-04-14
162212 dbstore1002 in bad shape DBA resolved 2017-04-05 2017-04-05
162233 eqiad rack/setup 11 new DB servers DBA resolved 2017-04-06 2017-08-07
162252 Create SQL database and Tables for Cognate extension to be used on Wiktionaries DBA resolved 2017-04-06 2017-04-13
162290 setup tempdb2001(WMF6407) DBA resolved 2017-04-06 2017-04-10
162519 s51362 has been rate limited to 2 concurrent connections for creating hundreds of 1400-second queries to labsdb1001 and labsdb1003 every 10 seconds DBA resolved 2017-04-09 2017-07-31
162539 Deploy schema change for adding term_full_entity_id column to wb_terms table DBA resolved 2017-04-27 2017-07-24
162556 Consider only updating wb_changes_dispatch after a successful run DBA resolved 2017-04-10 2017-06-29
162557 Use replica for reading the last dispatch position (chd_seen) DBA resolved 2017-04-10 2017-06-29
162593 Run pt-table-checksum on s4 (commonswiki) DBA resolved 2017-04-11 2017-09-12
162611 Unify revision table on s2 DBA resolved 2017-04-11 2017-05-25
162681 Network maintenance on row D (databases) DBA resolved 2017-04-12 2017-04-28
162774 FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks DBA resolved 2017-04-13 2017-08-29
163023 S1 replag at 3 hours DBA resolved 2017-04-15 2017-04-17
163109 Reclone db1063 to become a slave in s5 DBA resolved 2017-04-18 2017-04-21
163110 Reclone db1068 to become master in s4 DBA resolved 2017-04-18 2017-04-20
163143 dbtree: don't return 200 on error pages DBA resolved 2017-04-18 2017-09-07
163183 db1069 s7 replication thread stuck on fawiki.flaggedrevs_tracking DBA resolved 2017-04-19 2017-04-18
163351 codfw API slaves overloaded during the 2017-04-19 codfw switch DBA resolved 2017-04-20 2017-04-21
163413 Pool new server db2071 DBA resolved 2017-04-21 2017-04-21
163450 I99pema reset watchlist DBA resolved 2017-04-21 2017-04-20
163548 Drop the useless wb_terms keys "wb_terms_entity_type" and "wb_terms_type" on "wb_terms" table DBA resolved 2017-04-22 2017-05-11
163665 Reclone db1062 from db1041 (s7 master) DBA resolved 2017-04-25 2017-04-25
163778 Decommission db1022 (Was: db1022 broke while changing topology on s6- evaluate if to fix or directly decommission) DBA resolved 2017-04-26 2017-10-16
163895 Move masters away from D1 in eqiad DBA resolved 2017-04-27 2017-04-28
163912 Convert unique keys into primary keys for some wiki tables on s3-eqiad and s3-codfw DBA resolved 2017-04-27 2017-10-06
163979 Convert unique keys into primary keys for some wiki tables on s6-eqiad and codfw DBA resolved 2017-04-28 2017-09-27
164057 Decommission db1040 DBA resolved 2017-04-29 2017-05-09
164073 Puppetize Piwik's Database and set up periodical backups DBA resolved 2017-04-29 2017-07-18
164092 db1062 (s7 master eqiad) in a reboot cycle DBA resolved 2017-04-29 2017-04-29
164107 db1063 thermal issues (was: db1063 io (s5 master eqiad) performance is bad) DBA resolved 2017-04-29 2017-05-26
164173 Cache invalidations coming from the JobQueue are causing lag on several wikis DBA resolved 2017-05-01 2017-09-14
164185 Convert unique keys into primary keys for some wiki tables on s1, s2, s4, s5 and s7 (eqiad) DBA resolved 2017-05-02 2017-08-04
164399 Convert unique keys into primary keys for some wiki tables on all s* shards (codfw) DBA resolved 2017-05-04 2017-08-04
164530 Deploy uniqueness constraints on ores_classification table DBA resolved 2017-05-05 2017-06-28
164552 Slow query in API list=tags DBA resolved 2017-05-06 2017-05-05
164604 Document performance optimization of servermon and/or puppet reporting tools DBA resolved 2017-05-06 2017-05-18
164702 Decommission db1024 DBA resolved 2017-05-09 2017-10-12
164815 db2057 replication broken due to delete on echo_notification and echo_event DBA resolved 2017-05-10 2017-05-09
164887 Drop Semantic Database tables from wikitech wikis DBA resolved 2017-05-10 2017-05-17
164949 Drop titlekey table from all wmf databases DBA resolved 2017-05-11 2017-06-20
164968 remove mira wikitech grants DBA resolved 2017-05-11 2017-05-11
164990 RfC: Reading List service DBA resolved 2017-05-11 2017-10-05
165033 dbstore2001 takes 3 hours to start MySQL (was: dbstore2001 takes 3 hours to start MySQL after a crash) DBA resolved 2017-05-12 2017-06-20
165246 Add term_full_entity_id column to wb_terms table on testwikidatawiki DBA resolved 2017-05-17 2017-05-16
165629 Degraded RAID on db2058 DBA resolved 2017-05-18 2017-05-22
165743 frwiktionary on s7 still needs fixing on the revision table DBA resolved 2017-05-20 2017-05-23
165836 Drop table MediaWikiInstallPingback_15732959 from EventLogging DB DBA resolved 2017-05-21 2017-05-22
165977 Create CoC committee private wiki DBA resolved 2017-05-22 2017-08-23
166028 Rename user "Mlpearc" to "FlightTime" on Central Auth DBA resolved 2017-06-08 2017-08-01
166060 beta cluster databases have almost full disks DBA resolved 2017-05-23 2017-05-31
166097 Drop Gather tables from wmf wikis DBA resolved 2017-05-23 2017-05-29
166141 db1046 BBU looks faulty DBA resolved 2017-05-24 2017-07-06
166184 Run updateRestrictions.php on WMF wikis DBA resolved 2017-05-25 2017-07-19
166194 Database replication issues with deleted pages (affecting Tool Labs and Analytics Store) DBA resolved 2017-05-25 2017-05-25
166204 Convert unique keys into primary keys for some wiki tables on s1 DBA resolved 2017-05-25 2017-08-04
166205 Convert unique keys into primary keys for some wiki tables on s2 DBA resolved 2017-05-25 2017-06-21
166206 Convert unique keys into primary keys for some wiki tables on s4 DBA resolved 2017-05-25 2017-06-19
166207 Convert unique keys into primary keys for some wiki tables on s5 DBA resolved 2017-05-25 2017-06-28
166208 Convert unique keys into primary keys for some wiki tables on s7 DBA resolved 2017-05-25 2017-07-06
166261 fixProofreadIndexPagesContentModel doDBUpdates method turns infinitely DBA resolved 2017-05-25 2017-05-24
166267 dbtree: dbtree.wikimedia.org currently returns a 500 error DBA resolved 2017-05-25 2017-05-24
166278 Unify revision table on s3 DBA resolved 2017-05-26 2017-06-06
166326 db2055 mariadb weirdness -lagging after reboot and upgrade DBA resolved 2017-05-26 2017-05-30
166422 Degraded RAID on db1046 DBA resolved 2017-05-28 2017-05-31
166452 db1047 has been restarted - needs another restart DBA resolved 2017-05-29 2017-06-06
166486 Decommission db1023 DBA resolved 2017-05-30 2017-10-12
166518 Degraded BBU on db1094 (was: Degraded RAID on db1094) DBA resolved 2017-05-30 2017-06-06
166546 Point labsdb1001 and labsdb1003 to db1095 and db1102 DBA resolved 2017-05-31 2017-08-09
166568 Error (Wikimedia\Rdbms\DBQueryError) when creating a SecurePoll poll on testwiki DBA resolved 2017-05-31 2017-06-03
166683 db2044 cannot install jessie - requires BIOS firmware upgrade DBA resolved 2017-06-01 2017-06-06
166853 Degraded RAID on db2049 DBA resolved 2017-06-03 2017-06-06
166933 db1089 - possibly RAID controller broken DBA resolved 2017-06-04 2017-06-03
166935 db1089: update RAID controller firwmare DBA resolved 2017-06-04 2017-06-13
167031 Global rename of Idh0854 → Garam: supervision needed DBA resolved 2017-06-06 2017-08-21
167567 Migrate parsercache hosts to file per table DBA resolved 2017-06-11 2017-06-19
167597 Renaming Neoalpha: supervision needed DBA resolved 2017-06-12 2017-08-01
167667 Degraded RAID on db2070 DBA resolved 2017-06-13 2017-06-16
167715 Prepare and check storage layer for atjwiki DBA resolved 2017-06-13 2017-06-20
167758 Need to empty "linter" database table on large wikis DBA resolved 2017-06-14 2017-06-13
167784 WMF ParserCache disk space exhaustion DBA resolved 2017-06-15 2017-10-13
167961 Terbium cronjobs attempting to connect to labstestweb2001 DBA resolved 2017-06-16 2017-06-19
167978 Truncate 'searchindex' table on all wikis DBA resolved 2017-06-16 2017-06-26
168010 Special:ShortPages does not load in Wikimedia Commons: "Read timeout is reached" DBA resolved 2017-06-17 2017-07-19
168021 setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 DBA resolved 2017-06-16 2017-06-20
168109 Global rename of Smuconlaw → Sgconlaw: supervision needed DBA resolved 2017-06-18 2017-08-01
168354 dbstore2001: s5 thread isn't able to catch up with the master DBA resolved 2017-06-21 2017-07-24
168356 Prepare mysql hosts for stretch DBA resolved 2017-06-21 2017-06-30
168409 Migrate dbstore2001 to multi instance DBA resolved 2017-06-21 2017-09-12
168584 Labsdb* servers need to be rebooted DBA resolved 2017-06-22 2017-09-26
168661 Apply schema change to add 3D filetype for STL files DBA resolved 2017-06-23 2017-10-11
168776 Global rename of Green Cardamom → GreenC: supervision needed DBA resolved 2017-06-25 2017-08-01
168788 Prepare and check storage layer for maiwikimedia DBA resolved 2017-06-26 2017-07-18
169038 SQL dewiki_p categorylinks cl_from index missing DBA resolved 2017-06-29 2017-06-28
169050 dbstore1001 mysql crashed with: semaphore wait has lasted > 600 seconds DBA resolved 2017-06-29 2017-08-16
169193 Prepare and check storage layer for dinwiki DBA resolved 2017-06-30 2017-07-14
169294 File space alert for db1028 DBA resolved 2017-07-01 2017-06-30
169336 slow master queries on Wikibase\Client\Usage\Sql\EntityUsageTable::getAffectedRowIds DBA resolved 2017-07-01 2017-07-05
169355 Degraded RAID on db1052 DBA resolved 2017-07-01 2017-06-30
169376 Truncate table "interwiki" on all wmf sites DBA resolved 2017-07-01 2017-07-19
169377 Truncate table "job" on all wmf sites DBA resolved 2017-07-01 2017-07-24
169396 Global rename of Markos90 → Mαρκος: supervision needed DBA resolved 2017-07-02 2017-07-04
169448 Degraded RAID on db1066 DBA resolved 2017-07-02 2017-07-26
169488 Drop ukwikimedia from labsdb hosts (was: ukwikimedia still present on replicas dbs on labs hosts) DBA resolved 2017-07-04 2017-07-10
169510 Setup dbstore2002 with 2 new mysql instances from production and enable GTID DBA resolved 2017-07-04 2017-08-07
169514 Refactor puppet mariadb class to support multi-instance hosts DBA resolved 2017-07-04 2017-10-02
169516 Implement cron-based mydumper backups on the dbstore role DBA resolved 2017-07-04 2017-09-29
169517 Research backup storage options and prepare a design document DBA resolved 2017-07-04 2017-09-30
169527 Global rename of Antero de Quintal → JMagalhães: supervision needed DBA resolved 2017-07-04 2017-08-01
169693 db2044: Disk on predictive failure DBA resolved 2017-07-06 2017-07-06
169928 Evaluate how hard would be to get aa(wikibooks|wiktionary) and howiki databases deleted DBA resolved 2017-07-07 2017-08-20
170066 MySQL password for research@analytics-store.eqiad.wmnet publicly revealed DBA resolved 2017-07-10 2017-07-12
170158 Setup database for dmarc service DBA resolved 2017-07-11 2017-07-25
170308 dbstore1002 crashed DBA resolved 2017-07-12 2017-07-17
170351 db2019 has performance issues, replace disk or switchover s4 master elsewhere DBA resolved 2017-07-12 2017-08-04
170503 Degraded RAID on db2019 DBA resolved 2017-07-13 2017-07-13
170657 labsdb1009 crashed while doing an alter table on templatelinks DBA resolved 2017-07-15 2017-07-17
170666 Refactor prometheus-mysqld-exporter to support multi-instance hosts DBA resolved 2017-07-15 2017-10-09
170927 Make wbqc_constraints table available on Quarry et al. DBA resolved 2017-07-19 2017-08-25
170941 Global rename of user Moros DBA resolved 2017-07-19 2017-07-19
170952 Inconsistent default charset for analytics slaves DBA resolved 2017-07-19 2017-08-30
171027 "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis DBA resolved 2017-09-23 2017-10-22
171321 Finish dbstore2002 migration to multi-instance DBA resolved 2017-07-22 2017-08-07
171429 Delete _image_new.ibd table if exists somewhere DBA resolved 2017-07-25 2017-07-24
171460 Populate term_full_entity_id on www.wikidata.org DBA resolved 2017-07-25 2017-10-17
171461 Populate term_full_entity_id on test.wikidata.org DBA resolved 2017-07-25 2017-07-25
171474 Global rename of Carrotkit → 胡蘿蔔: supervision needed DBA resolved 2017-07-25 2017-08-01
171723 Degraded RAID on db1068 DBA resolved 2017-07-27 2017-07-31
171829 Prepare and check storage layer for hi.wikiversity DBA resolved 2017-07-28 2017-08-25
172015 wikibase-addUsagesForPage doesn't batch properly DBA resolved 2017-07-29 2017-08-18
172020 Drop WikiGrok tables from production DBA resolved 2017-07-29 2017-08-16
172169 Compress InnoDB on db1102 DBA resolved 2017-08-02 2017-08-07
172265 es2013 faulty BBU DBA resolved 2017-08-03 2017-08-10
172364 Remove cx_drafts table from production DBA resolved 2017-08-04 2017-08-22
172485 Convert unique keys into primary keys for some wiki tables on s3 (both eqiad and codfw) DBA resolved 2017-08-05 2017-08-04
172494 Change oom_adj for dedicated mysql server processes DBA resolved 2017-08-05 2017-10-11
172679 Productionize 11 new eqiad database servers DBA resolved 2017-08-08 2017-10-06
172784 Unable to drop tables on dewiki.labsdb/s51072__dwl_p DBA resolved 2017-08-09 2017-08-08
172996 Migrate s4 from db1095 to db1102 DBA resolved 2017-08-11 2017-08-22
173027 Prepare and check storage layer for hi.wikivoyage DBA resolved 2017-08-11 2017-09-28
173159 Global rename of Darwinius → DarwIn: supervision needed DBA resolved 2017-08-13 2017-08-22
173269 Wikidata.org currently very slow DBA resolved 2017-08-14 2017-08-22
173365 RAID crashed on db1078 DBA resolved 2017-08-16 2017-08-24
173834 Global rename: Opdire657 → Sakiv; supervision needed DBA resolved 2017-08-23 2017-08-23
173859 Global rename supervision request: Papa1234 → Karl-Heinz Jansen DBA resolved 2017-08-23 2017-08-24
173891 Create core ip_changes view for replicas DBA resolved 2017-08-24 2017-10-18
174054 Test reliability of RAID configuration/database hosts on single disk failure DBA resolved 2017-08-25 2017-10-05
174182 set up hiwikiversity on labsdb1010 DBA resolved 2017-08-26 2017-08-25
174265 BBU issues on db1055, RAID cache on WriteThrough DBA resolved 2017-08-28 2017-09-14
174385 Prepare and check storage layer for electcomwiki DBA resolved 2017-08-29 2017-09-05
174543 CHAR_LENGTH does not return the character count DBA resolved 2017-09-04 2017-09-05
174586 Remove ReaderFeedback tables from wikis DBA resolved 2017-08-31 2017-09-05
174764 db2044 HW RAID failure DBA resolved 2017-09-02 2017-10-03
174782 Drop pr_index from wikis where ProofreadPage isn't enabled DBA resolved 2017-09-02 2017-09-05
174857 Degraded RAID on db1059 DBA resolved 2017-09-03 2017-09-06
175002 db1009 (m5, used primarily for cloud services) unresponsive for minutes DBA resolved 2017-09-06 2017-09-26
175051 Drop "trackbacks" table on all wikis that have it DBA resolved 2017-09-06 2017-09-27
175086 Create and announce timeline for shutting down labsdb100[13] DBA resolved 2017-09-06 2017-10-19
175228 Degraded RAID on db2010 DBA resolved 2017-09-08 2017-09-19
175487 Significant replication lag for the s1, s2, and s4 wikis on labsdb100[13] DBA resolved 2017-09-11 2017-09-19
175946 Global rename for RadioFan DBA resolved 2017-09-15 2017-09-21
175970 Lost access to x1-analytics-slave DBA resolved 2017-09-16 2017-10-20
175973 db1100 crashed DBA resolved 2017-09-16 2017-10-17
176055 Update of QueryPages failing on commons with "MASTER_POS_WAIT() or MASTER_GTID_WAIT() failed: MySQL server has gone away" DBA resolved 2017-09-19 2017-09-28
176381 Grant tool "tool-db-usage" (s53508) ability to read full table_schema.TABLES contents DBA resolved 2017-09-22 2017-09-21
176557 High replag on s1, s3, s5 DBA resolved 2017-09-24 2017-09-24
176573 db2047 got rebooted DBA resolved 2017-09-25 2017-09-26
176760 Larger Wikidata entity dump queries DBA resolved 2017-09-27 2017-09-27
176883 Help communicate read-only time for Commons for schema change required by adding 3D filetype DBA resolved 2017-09-28 2017-10-11
176985 Global rename of CodeCat → Rua: supervision needed DBA resolved 2017-09-29 2017-09-28
177096 Some queries to new replica hosts are dramatically slower than labsdb; missing indexes? DBA resolved 2017-09-30 2017-10-09
177155 Find test wiki(s) for new description usage and enable there DBA resolved 2017-10-01 2017-10-18
177171 Degraded RAID on db1056 DBA resolved 2017-10-02 2017-10-04
177264 Degraded RAID on db1092 DBA resolved 2017-10-04 2017-10-18
177505 Replag of S1, S3, and S5 are over 2.75 hours DBA resolved 2017-10-06 2017-10-05
177570 Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs DBA resolved 2017-10-07 2017-10-17
177627 check db1052 power supply redundancy DBA resolved 2017-10-07 2017-10-08
177628 check db1054 power supply redundancy DBA resolved 2017-10-07 2017-10-08
177630 check db1080 power supply redundancy DBA resolved 2017-10-07 2017-10-08
177720 db2038 two disks with predictive failure DBA resolved 2017-10-09 2017-10-11
177772 Purge 90% of rows from recentchanges (and posibly defragment) from commonswiki and ruwiki (the ones with source:wikidata) DBA resolved 2017-10-10 2017-10-19
177908 dbstore2001:s3 crashed while backups were running DBA resolved 2017-10-12 2017-10-11
178040 Purge html5-misnesting tags from the database DBA resolved 2017-10-13 2017-10-13
178052 pagetranslation log_type missing on replicas DBA resolved 2017-10-13 2017-10-22
178135 ores_classification table corrupt on enwiki labs replica labsdb1001 DBA resolved 2017-10-17 2017-10-16
178140 db2081 unreachable DBA resolved 2017-10-14 2017-10-17
178237 Access to x1 broken on stat1006 DBA resolved 2017-10-21 2017-10-20
178272 labsdb1005's mysql crashed DBA resolved 2017-10-17 2017-10-17
178294 User log table creation on tools.labsdb failing intermittantly for IABot interactive UI DBA resolved 2017-10-17 2017-10-17
144169 Flake8 for python files without extension in puppet repo Software Development open 2017-01-02 2017-06-05
145192 icinga-downtime script waiting forever if host already in downtime Software Development open 2017-01-02 2016-09-09
148494 Add shell scripts CI validations Software Development open 2017-01-02 2017-09-14
150560 More verbose messages from service-checker-swagger Software Development open 2017-01-02 2017-05-23
152950 E901 SyntaxError: invalid syntax is wrongly raised on using python's abc by jenkins python CI linter Software Development open 2017-01-02 2017-03-27
154776 Puppet compiler: order resources for easy comparison between hosts Software Development open 2017-01-07 2017-01-06
155705 confctl: log to SAL even if the selection doesn't match any host Software Development open 2017-01-20 2017-01-19
157001 Puppet compiler: abort on git rebase conflict Software Development open 2017-02-03 2017-02-02
157002 Puppet compiler: re-add the concurrency option NUM_THREADS Software Development open 2017-02-03 2017-02-02
157133 Consider adding a --skip-conftool option to puppet-merge Software Development open 2017-02-04 2017-10-09
159045 Update Puppet repo code that uses maniphest.update and maniphest.createtask conduit api Software Development open 2017-02-26 2017-02-25
161545 Cumin: PuppetDB backend, allow to specify boolean values for resource parameters Software Development open 2017-03-28 2017-03-27
163363 Switchdc: improvements Software Development open 2017-04-20 2017-05-16
163365 Switchdc RO/RW: add check to test it editing a real wiki Software Development open 2017-04-20 2017-04-19
164587 cumin could use randomization/splay options Software Development open 2017-05-06 2017-05-05
167504 New tool to track package updates/status for hosts and images (debmonitor) Software Development open 2017-06-10 2017-10-07
169304 Cumin masters: simplify usage in case of emergency Software Development open 2017-07-01 2017-06-30
177385 Upgrade Cumin masters to stretch Software Development open 2017-10-05 2017-10-18
178279 Cumin: add support for role/profile shortcuts in PuppetDB backend Software Development open 2017-10-17 2017-10-16
178342 Cumin: add support for external backends Software Development open 2017-10-17 2017-10-17
142085 Automate creation of Phab task for failed disks Software Development resolved 2017-01-02 2016-10-19
142097 Package Python phabricator module for both Ubuntu Precise and Debian Jessie Software Development resolved 2017-01-02 2016-11-14
143306 Evaluation of automation/orchestration tools Software Development resolved 2017-01-02 2016-10-07
143536 Upgrade all mw* servers to debian jessie Software Development resolved 2017-01-02 2017-08-29
148273 Setup SSH AuthN/AuthZ for the Automation Framework Software Development resolved 2017-01-02 2017-03-30
148814 wmf-auto-reimage improvements Software Development resolved 2017-01-02 2017-09-21
148815 wmf-auto-reimage: improve messaging Software Development resolved 2017-01-02 2016-10-24
148816 wmf-auto-reimage: add --new CLI option for new hosts Software Development resolved 2017-01-02 2016-11-07
148817 wmf-auto-reimage: remove dependency on wmf-reimage Software Development resolved 2017-01-02 2017-09-21
149213 wmf-auto-reimage: handle multiple conftool services per host Software Development resolved 2017-01-02 2017-09-08
149229 raid_handler: Icinga ACK comment persist after recovery Software Development resolved 2017-01-02 2016-10-27
149230 wmf-auto-reimage: allow to specify the conftool state Software Development resolved 2017-01-02 2017-09-21
149913 Icinga raid_handler: add option to handle frack instances Software Development resolved 2017-01-02 2017-02-24
150448 wmf-reimage: allow to power cycle also if the server is off Software Development resolved 2017-01-02 2017-02-23
151043 Add support for physical disks to get-raid-status-megacli.py Software Development resolved 2017-01-02 2016-12-12
152549 tox-jessie is failing on operations/software Software Development resolved 2017-01-02 2017-01-24
152774 raid_handler: random failure to gather RAID status Software Development resolved 2017-01-02 2016-12-09
152977 conftool service removal bugs Software Development resolved 2017-01-02 2017-03-08
154588 Automation framework first version Software Development resolved 2017-01-05 2017-02-24
157052 Puppet compiler: sync newest facts only Software Development resolved 2017-02-03 2017-04-28
158659 Keyholder: add support for ED25519 keys Software Development resolved 2017-02-22 2017-02-28
158660 Keyholder accept passwordless keys Software Development resolved 2017-02-22 2017-02-28
158746 Cumin: use safer default for hosts regex matching Software Development resolved 2017-02-23 2017-02-24
158747 Cumin: better error message if no config file is available Software Development resolved 2017-02-23 2017-07-03
158748 Cumin: add support for 'not' operator for hosts Software Development resolved 2017-02-23 2017-02-24
158753 Cumin: authorize also IPv6 on the targets Software Development resolved 2017-02-23 2017-02-24
158758 Cumin: ignore urllib3 SubjectAltNameWarning in PuppetDB calls Software Development resolved 2017-02-23 2017-02-24
158773 Cumin: skip target installation on labs realm Software Development resolved 2017-02-23 2017-02-22
158964 Cumin: fill wikitech page with documentation Software Development resolved 2017-02-25 2017-03-30
158967 Cumin: fix first batch of potential issues reported by codacy Software Development resolved 2017-02-25 2017-02-25
159127 Cumin: fine tuning configuration Software Development resolved 2017-02-28 2017-09-08
159308 Cumin: add man page Software Development resolved 2017-03-02 2017-10-17
159968 Cumin: add support for batch execution Software Development resolved 2017-03-09 2017-03-13
159969 Cumin: add integration tests Software Development resolved 2017-03-09 2017-03-15
159970 Cumin: auto ucfirst puppetdb resources Software Development resolved 2017-03-09 2017-03-29
160621 Cumin: upgrade to v0.0.2 in prod Software Development resolved 2017-03-17 2017-03-16
161730 Cumin: do not auto-ucfirst when the query is a regex in PuppetDB backend Software Development resolved 2017-03-30 2017-05-16
161887 Cumin: CLI, allow to just query without specifying a command Software Development resolved 2017-04-01 2017-03-31
162151 Cumin: PuppetDB, fail better if regex are used on resource parameters Software Development resolved 2017-04-05 2017-05-16
163364 Switchdc dnsdisc: add check for IP records Software Development resolved 2017-04-20 2017-04-26
163367 Switchdc: make SAL messages more human-friendly Software Development resolved 2017-04-20 2017-04-26
163369 Switchdc warmup: be less verbose Software Development resolved 2017-04-20 2017-04-29
163371 Switchdc menu: add a Next option in the menu Software Development resolved 2017-04-20 2017-04-26
163372 Switchdc: add task to re-enable and run puppet Software Development resolved 2017-04-20 2017-04-26
163373 Switchdc: varnish switch, remove the manual confirmation Software Development resolved 2017-04-20 2017-04-29
163376 Switchdc dnsdisc: remove stale conftool files Software Development resolved 2017-04-20 2017-04-29
163398 Switchdc: mediawiki, use etcd-driven config Software Development resolved 2017-04-20 2017-05-03
164177 switchdc: Improve wgReadOnly message Software Development resolved 2017-05-01 2017-05-04
164396 Switchdc dnsdisc: add retries to check_record Software Development resolved 2017-05-04 2017-05-08
164400 Switchdc t05 traffic: be explicit on automatic validation Software Development resolved 2017-05-04 2017-05-08
164403 Switchdc t09 start maintenance: clear systemctl failed state Software Development resolved 2017-05-04 2017-05-08
164780 Sunset our use of Salt Software Development resolved 2017-05-09 2017-09-28
164817 Migrate debdeploy to cumin Software Development resolved 2017-05-10 2017-09-11
164824 Cumin: only last ssh_option is set Software Development resolved 2017-05-10 2017-05-16
164827 Cumin: get live output when matching a single host Software Development resolved 2017-05-10 2017-05-16
164833 Cumin: allow to specify successful exit codes Software Development resolved 2017-05-10 2017-05-29
164838 Cumin: allow to specify a timeout per command Software Development resolved 2017-05-10 2017-07-03
165583 Puppet compiler: sync facts from all workers Software Development resolved 2017-05-18 2017-05-23
165838 Cumin: add a simple interactive mode Software Development resolved 2017-05-21 2017-05-30
165842 Cumin: add a simple txt/json output Software Development resolved 2017-05-21 2017-05-30
166300 Remove Salt from wmf-auto-reimage / wmf-reimage Software Development resolved 2017-05-26 2017-09-25
166371 Monitoring: create an alert for daemonized puppet Software Development resolved 2017-05-27 2017-06-13
166397 Cumin fails on huge nodelists emitted by its own outputs Software Development resolved 2017-05-27 2017-09-28
166519 Raid handler: manage new alarms Software Development resolved 2017-05-30 2017-05-31
166570 Do something to better handle wmf-reimage runs cleanups/failures Software Development resolved 2017-05-31 2017-09-21
167392 Cumin: fix --success-percentage 0 Software Development resolved 2017-06-09 2017-06-12
167394 Cumin: fix ok_codes when set to empty list Software Development resolved 2017-06-09 2017-06-12
169555 puppet.service systemctl failures after reimage Software Development resolved 2017-07-04 2017-09-21
169619 Degraded RAID on ms-be2024 Software Development resolved 2017-07-05 2017-07-05
169640 Support aliases in cumin Software Development resolved 2017-07-05 2017-07-17
170394 Cumin: add multi-query support Software Development resolved 2017-07-13 2017-08-02
171679 Cumin: allow to ignore exit codes of executed commands Software Development resolved 2017-07-26 2017-08-02
171684 Cumin: improve target management Software Development resolved 2017-07-26 2017-09-08
174008 Cumin: setup.py installs data_files in wrong directory Software Development resolved 2017-08-25 2017-09-21
174854 Icinga raid handler improvements Software Development resolved 2017-09-03 2017-09-08
174911 Cumin: clustershell worker should return if no target are specified Software Development resolved 2017-09-05 2017-09-08
175711 Cumin: create backend for OpenStack Software Development resolved 2017-09-13 2017-09-21
175712 Install cumin in the WMCS infrastructure Software Development resolved 2017-09-13 2017-09-20
176314 Replace salt on integration and deployment-prep projects Software Development resolved 2017-09-21 2017-10-18
176609 Cumin: fine tune WMCS setup Software Development resolved 2017-09-26 2017-09-30
176632 Remove salt master (and related packages) from labcontrol1001 Software Development resolved 2017-09-26 2017-09-28
176955 wmf-auto-reimage: add support for renaming while re-imaging Software Development resolved 2017-09-29 2017-10-04
95742 Decomission amssq31-62 (32 hosts) Hardware Requests open 2017-01-02 2017-08-29
128821 reclaim and return all cisco servers Hardware Requests open 2017-01-02 2016-09-23
130883 decom cp3011-22 (12 machines) Hardware Requests open 2017-01-02 2017-10-03
139752 Find and rack 2 EX4200s in rack c1-eqiad Hardware Requests open 2017-01-02 2017-08-29
139775 eqiad: add all spare network switches to hardware spares tracking Hardware Requests open 2017-01-02 2017-04-27
159480 Decommission bast3001 Hardware Requests open 2017-03-03 2017-08-29
159996 decom fluorine Hardware Requests open 2017-03-09 2017-07-20
160986 Decommission ms-fe100[1-4] Hardware Requests open 2017-03-22 2017-09-20
166341 SSDs for main Kafka clusters Hardware Requests open 2017-05-26 2017-05-29
166489 Decommission ms-be1001 - ms-be1012 Hardware Requests open 2017-05-30 2017-09-19
167376 Decommission cp300[3456] Hardware Requests open 2017-06-09 2017-08-29
167377 Decommission cp4011, cp4012, cp4019, cp4020 Hardware Requests open 2017-06-09 2017-08-01
168559 decom silver (was silver has trouble rebooting) Hardware Requests open 2017-06-22 2017-08-11
169020 Decommission cp400[1-4] Hardware Requests open 2017-06-28 2017-08-04
170441 Decommission mw1196 Hardware Requests open 2017-07-13 2017-07-20
172323 Decommission WMF3248 (old R510) Hardware Requests open 2017-08-04 2017-09-19
172487 decom iridium Hardware Requests open 2017-08-05 2017-08-17
173097 Decommission stat1002.eqiad.wmnet Hardware Requests open 2017-08-12 2017-09-28
175089 decommission mw1163 Hardware Requests open 2017-09-06 2017-09-05
175093 Decommission osmium.eqiad.wmnet Hardware Requests open 2017-09-06 2017-10-20
175595 decommission wdqs100[12] Hardware Requests open 2017-09-12 2017-10-04
177958 Decommission ocg1001-3 Hardware Requests open 2017-10-12 2017-10-18
178392 Replacement hardware for cumin masters Hardware Requests open 2017-10-18 2017-10-18
76986 RESTBase production hardware Hardware Requests resolved 2017-01-02 2016-02-09
83340 decom arsenic: (was: reclaim arsenic as spare) Hardware Requests resolved 2017-05-04 2017-05-10
84958 eqiad: (3) servers for logstash service Hardware Requests resolved 2017-01-02 2016-07-29
86561 Wikidata Query Service hardware Hardware Requests resolved 2017-01-02 2017-07-14
86896 Procure rdb2001-2004 - onsite pending racking Hardware Requests resolved 2017-01-02 2016-01-20
87215 Re-image osmium and repurpose into VE testing host Hardware Requests resolved 2017-01-02 2015-01-24
88294 eqiad: Dedicated hardware for wikitech web server - silver allocated Hardware Requests resolved 2017-01-02 2015-02-18
89266 virt1000 memory upgrade Hardware Requests resolved 2017-01-02 2015-02-24
89400 Add another redis jobqueue server master and slave Hardware Requests resolved 2017-01-02 2015-11-24
89402 purchase 3 additional logstash nodes Hardware Requests resolved 2017-01-02 2016-07-29
89752 eqiad: (6) labs virt nodes Hardware Requests resolved 2017-01-02 2015-04-13
89869 Assign hardware for the zotero service Hardware Requests resolved 2017-01-02 2015-03-11
90626 Replace virt1000 with a newer warrantied server Hardware Requests resolved 2017-01-02 2015-04-14
90747 codfw/eqiad: (1) eventlogging node (per site) Hardware Requests resolved 2017-01-02 2015-03-19
90904 deploy eventlog1001 Hardware Requests resolved 2017-01-02 2015-04-07
90907 deploy eventlog2001 Hardware Requests resolved 2017-01-02 2015-10-30
91277 Hardware for Designate Hardware Requests resolved 2017-01-02 2015-03-12
91678 setup deployment server in codfw (tin equivalent) Hardware Requests resolved 2017-01-02 2016-01-20
91853 Hardware for HTML / zim dumps Hardware Requests resolved 2017-01-02 2017-08-09
93076 eqiad: 2 hardware access request for CI isolation on labsnet Hardware Requests resolved 2017-01-02 2016-01-11
93118 order new array for dataset1001 Hardware Requests resolved 2017-01-02 2015-05-20
93266 codfw: (2) poolcounter systems Hardware Requests resolved 2017-01-02 2015-03-20
93790 Expand RESTBase cluster capacity Hardware Requests resolved 2017-01-02 2016-12-08
94846 Buy fiber patches Hardware Requests resolved 2017-01-02 2015-11-03
95153 Decom/repurpose rbf* hosts Hardware Requests resolved 2017-01-02 2015-09-16
95243 Repurpose rhenium as "network insight" host Hardware Requests resolved 2017-01-02 2015-04-07
95760 eqiad: (1) allocate server to migrate Zuul server to Hardware Requests resolved 2017-01-02 2016-04-21
97049 CODFW Search Servers Hardware Requests resolved 2017-01-02 2015-08-18
98166 Eqiad: 1 hardware access request for puppetmaster service scale out Hardware Requests resolved 2017-01-02 2015-05-05
98740 labnet1002 Hardware Requests resolved 2017-01-02 2015-05-19
100442 Hadoop worker node procurement - 2015 Hardware Requests resolved 2017-01-02 2016-01-27
101256 Replace rubidium with radon for authdns (allocate radon, deallocate rubidium) Hardware Requests resolved 2017-01-02 2015-06-05
105707 Request Elasticsearch hardware for secondary CirrusSearch in codfw Hardware Requests resolved 2017-01-02 2016-03-10
106112 server for wikimania video transcoding Hardware Requests resolved 2017-01-02 2015-07-22
106147 New server: labservices1001 Hardware Requests resolved 2017-01-02 2015-09-30
107287 Assign WMF5842, WMF5843 for service cluster expansion as scb1001, scb1002 Hardware Requests resolved 2017-01-02 2015-08-10
111053 Request three servers for Pageview API Hardware Requests resolved 2017-01-02 2015-09-21
111382 codfw 3x spares for cassandra encryption testing Hardware Requests resolved 2017-01-02 2015-09-15
111777 Refresh Parser cache servers pc1001-pc1003 Hardware Requests resolved 2017-01-02 2015-12-18
112163 eqiad: (1) hardware request for ElasticSearch replication to Labs - 4 weeks use Hardware Requests resolved 2017-01-02 2015-10-16
114191 Setup a 2 server Kafka instance in both eqiad and codfw for reliable purge streams Hardware Requests resolved 2017-01-02 2017-05-25
114435 Labs test cluster in codfw Hardware Requests resolved 2017-01-02 2015-11-02
115288 Allocate hardware for salt master in eqiad Hardware Requests resolved 2017-01-02 2015-11-09
116790 Decommission calcium Hardware Requests resolved 2017-01-02 2016-04-12
118213 Decommission rubidium Hardware Requests resolved 2017-01-02 2016-09-01
118588 Get Ops bare metal test server Hardware Requests resolved 2017-01-02 2015-12-03
118983 Hardware access request for yubico auth servers Hardware Requests resolved 2017-01-02 2015-12-16
119579 Additional diskspace of wdqs1001/wdqs1002 Hardware Requests resolved 2017-01-02 2016-04-26
119598 eqiad: (2) servers request for ORES Hardware Requests resolved 2017-01-02 2016-08-05
119698 spare swift disks order Hardware Requests resolved 2017-01-02 2015-12-22
119935 Upgrade restbase100[7-9] to match restbase100[1-6] hardware Hardware Requests resolved 2017-01-02 2016-02-26
120679 eqiad out of warranty spares to decommission - approval request Hardware Requests resolved 2017-01-02 2017-01-30
120752 decommission dickson Hardware Requests resolved 2017-01-02 2017-06-21
120842 EQIAD/CODFW: 2 hardware access request for monitoring Hardware Requests resolved 2017-01-02 2015-12-15
121007 Decommission and remove from racks out of warranty spares Hardware Requests resolved 2017-01-02 2015-12-17
121575 Expand SSD space in Cassandra cluster Hardware Requests resolved 2017-01-02 2016-04-28
121882 3 conf200x servers in codfw for zookeeper (and etcd?) Hardware Requests resolved 2017-01-02 2016-04-19
122050 Decomission/reclaim nembus/neptunium Hardware Requests resolved 2017-01-02 2015-12-22
122100 wipe disks and add nembus back to server spares Hardware Requests resolved 2017-01-02 2017-04-13
122101 Decommission neptunium Hardware Requests resolved 2017-01-02 2016-04-12
123132 Need spare server to upgrade/migrate gerrit Hardware Requests resolved 2017-01-02 2016-10-06
123531 check for memory upgrade for lead Hardware Requests resolved 2017-01-02 2016-01-14
123559 allocate hardware for salt master in codfw Hardware Requests resolved 2017-01-02 2016-02-08
124408 mw2173 has probably a broken disk, needs substitution and reimaging Hardware Requests resolved 2017-01-02 2016-04-19
124945 eqiad: New Hive / Oozie server node in eqiad Analytics VLAN Hardware Requests resolved 2017-01-02 2016-03-24
124947 eqiad: (3) AQS replacement nodes Hardware Requests resolved 2017-01-02 2016-05-03
124962 Decommission pc1001-1003 Hardware Requests resolved 2017-01-02 2016-08-11
125223 Codfw: ms-be2003 2TB order request Hardware Requests resolved 2017-01-02 2016-01-29
125827 Decom db2001-db2009 Hardware Requests resolved 2017-01-02 2017-03-29
125842 normalize eqiad restbase cluster - replace restbase1001-1006 Hardware Requests resolved 2017-01-02 2016-04-19
126089 new labstore hardware for eqiad Hardware Requests resolved 2017-01-02 2016-04-19
126253 additional graphite machines request, 1x per DC Hardware Requests resolved 2017-01-02 2016-04-21
126453 codfw: (2) servers for redis jobrunners Hardware Requests resolved 2017-01-02 2016-03-12
126626 3x additional SSD for restbase hp hardware Hardware Requests resolved 2017-01-02 2016-04-19
126987 MediaWiki maintenance host for codfw (terbium's equivalent) Hardware Requests resolved 2017-01-02 2016-03-25
126988 Log host for codfw (fluorine's equivalent) Hardware Requests resolved 2017-01-02 2016-12-12
128000 Refresh elastic10{01..16}.eqiad.wmnet servers Hardware Requests resolved 2017-01-02 2016-05-11
128475 codfw: (2+2) sca & scb service clusters Hardware Requests resolved 2017-01-02 2016-03-09
128807 eqiad: (3) nodes for Druid / analytics Hardware Requests resolved 2017-01-02 2016-05-03
128808 +1 'stat' type box for hadoop client usage Hardware Requests resolved 2017-01-02 2016-04-05
130752 reclaim restbase1001-1006 to spares Hardware Requests resolved 2017-01-02 2016-05-16
130760 Allocate 2 analytics machines to experiment with a jupyterhub notebook service Hardware Requests resolved 2017-01-02 2016-05-09
130845 update label on analytics1003/WMF4541 Hardware Requests resolved 2017-01-02 2016-03-31
130884 reclaim to spares: cp1056, cp1057, cp1069, cp1070 Hardware Requests resolved 2017-01-02 2016-04-12
131180 Maps back end hardware Hardware Requests resolved 2017-01-02 2016-04-26
131184 eqiad: (2) Relevance forge servers Hardware Requests resolved 2017-01-02 2016-08-08
132476 connect an external harddisk with >2TB space to stat1001 Hardware Requests resolved 2017-01-02 2016-04-20
132679 Evaluate replacing SATA disks on ganeti100X.eqiad.wmnet with SSDs Hardware Requests resolved 2017-01-02 2016-05-10
133614 reclaim or decom: cp1043 + cp1044 Hardware Requests resolved 2017-01-02 2016-09-14
135468 codfw old mw app server decomission Hardware Requests resolved 2017-01-02 2017-01-19
136340 Staging / Test environment(s) for RESTBase Hardware Requests resolved 2017-01-02 2016-12-21
136970 eqiad: spare allocation to replace labmon1001 Hardware Requests resolved 2017-01-02 2016-06-07
137117 Replace/refresh carbon Hardware Requests resolved 2017-01-02 2016-08-08
137724 eqiad: 1 hardware access request for labs graphite Hardware Requests resolved 2017-01-02 2016-06-22
138075 procure syslog hardware in codfw Hardware Requests resolved 2017-01-02 2017-02-14
138637 codfw: (2) wqds200[12] systems Hardware Requests resolved 2017-01-02 2016-09-20
139758 Decommission elastic1001-1016 Hardware Requests resolved 2017-01-02 2016-09-14
139776 codfw: add all spare network switches to hardware spares tracking Hardware Requests resolved 2017-01-02 2016-07-11
139961 9x or 15x additional Cassandra/RESTBase nodes Hardware Requests resolved 2017-01-02 2017-01-20
140372 decommission WMF3155-WMF3175 (old lsearchd) Hardware Requests resolved 2017-01-02 2016-07-21
140786 eqiad: audit cisco servers for return/decom Hardware Requests resolved 2017-01-02 2016-07-27
140787 codfw: audit cisco servers for return/decom Hardware Requests resolved 2017-01-02 2016-08-09
141415 decom ytterbium (datacenter) Hardware Requests resolved 2017-01-02 2017-05-09
141624 eqiad: (4) worker servers for kubernetes Hardware Requests resolved 2017-01-02 2017-03-29
141762 decommission snapshot1002, 1003, 1004 Hardware Requests resolved 2017-01-02 2016-10-13
142190 codfw: (2) servers request for ORES redis databases Hardware Requests resolved 2017-01-02 2017-03-15
142218 EQIAD: (2) hardware access request for PUPPET Hardware Requests resolved 2017-01-02 2016-10-11
142219 CODFW: (2) hardware access request for PUPPET Hardware Requests resolved 2017-01-02 2016-09-06
142412 reclaim WMF4724 to spares Hardware Requests resolved 2017-01-02 2016-09-15
142578 codfw/eqiad:(9+9) hardware access request for ORES Hardware Requests resolved 2017-01-02 2017-06-22
142581 Reclaim nobelium Hardware Requests resolved 2017-01-02 2017-01-25
144728 reallocate snapshot1001 for use as canary/testbed for dumps Hardware Requests resolved 2017-01-02 2016-10-26
145082 eqiad/codfw: 1+1 Kafka broker in main clusters in eqiad and codfw Hardware Requests resolved 2017-01-02 2017-05-25
146171 Return wmf4747/wmf4748/wmf4749/wmf4750 to spares Hardware Requests resolved 2017-01-02 2017-01-23
146455 Decommission labsdb1002 Hardware Requests resolved 2017-01-02 2017-03-03
147053 eqiad: (2) hardware access request for dedicated Labs puppetmasters Hardware Requests resolved 2017-01-02 2017-06-14
147309 Decommission db1019 Hardware Requests resolved 2017-01-02 2017-04-12
147313 Decommission analytics1026 and analytics1015 Hardware Requests resolved 2017-01-02 2017-03-03
147596 Allocate spare misc box in eqiad for gerrit replacement Hardware Requests resolved 2017-01-02 2016-10-12
147926 Reclaim aqs100[123] Hardware Requests resolved 2017-01-02 2017-01-05
148187 Requesting 1 spare misc box for Gerrit in codfw Hardware Requests resolved 2017-01-02 2017-02-09
148510 eqiad/codfw: swift frontend hardware refresh Hardware Requests resolved 2017-01-02 2017-01-20
148513 codfw/eqiad: 2x systems for prometheus Hardware Requests resolved 2017-01-02 2017-01-20
148559 Estimate hardware requirements for ordering new servers for Elasticsearch Hardware Requests resolved 2017-01-02 2016-11-17
148747 Estimate hardware requirements for WDQS upgrade Hardware Requests resolved 2017-01-02 2016-12-16
149089 elasticsearch new servers (5x eqiad / 12x codfw) Hardware Requests resolved 2017-01-02 2017-01-19
149102 decomission db2015 Hardware Requests resolved 2017-01-02 2017-02-06
149336 codfw/eqiad: 12x swift backend refresh Hardware Requests resolved 2017-01-02 2017-03-27
149920 Analytics AQS cluster expansion Hardware Requests resolved 2017-01-02 2017-02-06
150865 codfw: 1 hardware access request for continuous integration Hardware Requests resolved 2017-01-02 2016-12-15
150936 Phase out scandium.eqiad.wmnet Hardware Requests resolved 2017-01-02 2017-03-03
153008 eqiad: (1) Mediawiki log host to replace fluorine Hardware Requests resolved 2017-01-02 2016-12-15
154164 decommission stat1001 Hardware Requests resolved 2017-01-02 2017-03-03
154664 codfw: (2) hardware access request for labtest Hardware Requests resolved 2017-01-06 2017-06-26
154706 Codfw: (1) hardware access request for labtestneutron refresh Hardware Requests resolved 2017-01-06 2017-06-14
156040 hardware request for netmon1001 replacement Hardware Requests resolved 2017-01-24 2017-03-30
156506 Replace bast3001 Hardware Requests resolved 2017-01-28 2017-03-02
156631 Site: 2 hardware access request for SCB@CODFW Hardware Requests resolved 2017-01-31 2017-05-25
156970 replacement hardware for iridium (phabricator) Hardware Requests resolved 2017-02-02 2017-04-26
157675 Reclaim/Decommission old codfw mc2001->mc2016 hosts Hardware Requests resolved 2017-02-10 2017-03-06
157839 spare ex4200s - check on quantity for potential shipment to OIT Hardware Requests resolved 2017-02-11 2017-02-17
157840 decom install2001 Hardware Requests resolved 2017-02-11 2017-02-17
157991 decommission ms2001 & ms2002 Hardware Requests resolved 2017-02-14 2017-03-09
158020 decom carbon Hardware Requests resolved 2017-02-15 2017-03-03
158204 Eqiad: (2) hardware access request for labnet1003/1004 Hardware Requests resolved 2017-02-16 2017-05-19
158207 Eqiad: (2) hardware access request for labcontrol1003/1004 Hardware Requests resolved 2017-02-16 2017-05-19
159207 Create one oresrdb VM in codfw Hardware Requests resolved 2017-03-01 2017-03-02
159413 Decomission ms-fe2001-4 Hardware Requests resolved 2017-03-03 2017-05-10
159838 EQIAD: stat1002 replacement Hardware Requests resolved 2017-03-08 2017-05-15
159839 EQIAD: stat1003 replacement Hardware Requests resolved 2017-03-08 2017-05-15
161265 Rename labtestmetal2001 Hardware Requests resolved 2017-03-24 2017-03-27
161311 Eqiad: Hardware request for labstore1006/7, dataset1002/3 Hardware Requests resolved 2017-03-25 2017-07-06
161606 Additional ram quote for Prometheus baremetal Hardware Requests resolved 2017-03-29 2017-04-20
161700 CODFW: (4) hardware access request for kubernetes Hardware Requests resolved 2017-03-30 2017-05-30
161750 eqiad: (1) hardware access request for dedicated labmon1002 Hardware Requests resolved 2017-03-30 2017-05-19
161752 eqiad: (2) hardware access request for californium and silver (labweb1001/1002) Hardware Requests resolved 2017-03-30 2017-06-14
161753 eqiad: (1) hardware access request for labnodepool1002 Hardware Requests resolved 2017-03-30 2017-06-27
161764 Codfw: (1) hardware access request for labtestnet2003 [region 2] Hardware Requests resolved 2017-03-30 2017-06-14
161765 Codfw: (1) hardware access request for labtestvirt2003 [region 2] Hardware Requests resolved 2017-03-30 2017-06-27
161766 Codfw: (2) hardware access request for labtest [region 2] Hardware Requests resolved 2017-03-30 2017-06-26
162257 EQIAD: 2 hardware access request for kubernetes-staging Hardware Requests resolved 2017-04-06 2017-05-12
162785 Decommission ms-be2001 - ms-be2012 Hardware Requests resolved 2017-04-13 2017-07-06
162897 spare pool allocation of WMF6406 to replace mira Hardware Requests resolved 2017-04-14 2017-04-13
162928 decommision nembus Hardware Requests resolved 2017-04-14 2017-05-10
164036 decommission gadolinium Hardware Requests resolved 2017-04-28 2017-05-09
164513 reclaim tempdb2001(WMF6407) to spares Hardware Requests resolved 2017-05-05 2017-05-11
164515 codfw: (1) labtest puppetmaster Hardware Requests resolved 2017-05-05 2017-06-14
164588 decom mira Hardware Requests resolved 2017-05-06 2017-07-06
164959 Decomission mw2098 Hardware Requests resolved 2017-05-16 2017-06-26
168271 Decommission mw1170-mw1179 Hardware Requests resolved 2017-06-20 2017-09-21
168472 reclaim/decom tmh200[12] Hardware Requests resolved 2017-06-21 2017-07-27
169506 Decommission subra/suhail Hardware Requests resolved 2017-07-04 2017-07-26
171018 decom netmon1001 Hardware Requests resolved 2017-07-20 2017-10-12
171179 Decommisson restbase-dev100[1-3] Hardware Requests resolved 2017-07-21 2017-10-18
174452 eqiad: replacement tin/deployment server Hardware Requests resolved 2017-08-30 2017-09-07
176472 New package builder host Hardware Requests resolved 2017-09-23 2017-09-28
123147 Wikipedia.com warns about bad certificate Other Operations 2017-01-02 2016-01-09
17000 Special:Import error: "Import failed: Could not open import file" Other Operations open 2017-01-02 2017-05-25
32716 Run our own Tor client for Tor block Other Operations open 2017-01-02 2017-06-19
40860 security@mediawiki.org : Create a public key and publish it on the public key servers Other Operations open 2017-01-02 2016-03-13
41785 Create a labs SMTP smarthost Other Operations open 2017-08-31 2017-09-07
46016 SVG fails to render properly due to several issues Other Operations open 2017-01-02 2016-07-07
46791 [[wikitech:Server_admin_log]] should not rely on freenode irc for logmsgbot entries Other Operations open 2017-01-02 2017-02-14
50029 crackling at start of OGG renditions of MIDI files (fixed in TiMidity++ 2.14.0) Other Operations open 2017-08-06 2017-08-05
55457 setup a DB backed parser cache Other Operations open 2017-01-02 2017-05-20
56515 Apply editing rate limits for all users Other Operations open 2017-01-02 2017-01-29
56713 Non-NDA users cannot access graphite.wikimedia.org Other Operations open 2017-01-02 2016-06-10
64987 librsvg misinterpret quoted font family names that contain whitespaces Other Operations open 2017-01-02 2017-05-21
67270 Default license for operations/puppet Other Operations open 2017-01-02 2017-02-07
67394 [EPIC] Performance testing environment Other Operations open 2017-01-02 2016-03-11
69015 m.wikipedia.org incorrectly redirects to en.m.wikipedia.org Other Operations open 2017-01-02 2016-12-24
76203 Make ircecho run as its own user Other Operations open 2017-01-02 2016-07-16
76306 Set warning thresholds for average cluster utilization Other Operations open 2017-01-02 2016-12-16
82350 update exim::listserve::private::mailing_lists value in puppet Other Operations open 2017-09-19 2017-09-18
83729 Fix monitoring of poolcounter service Other Operations open 2017-07-21 2017-10-13
84163 Fix CirrusSearch monitoring Other Operations open 2017-01-02 2017-01-22
84700 Setup management switch in OE12 Other Operations open 2017-01-02 2016-03-17
84845 improve cron spam visibility Other Operations open 2017-10-19 2017-10-18
85451 scale graphite deployment (tracking) Other Operations open 2017-01-02 2017-05-02
86546 graphite-web logs are not rotated Other Operations open 2017-01-02 2016-11-18
86552 Monitor and alarm on SMART attributes Other Operations open 2017-01-02 2017-09-14
87220 Minimize differences between beta and production (Tracking) Other Operations open 2017-01-02 2017-09-01
87790 decom amslvs1-4 (dc work) Other Operations open 2017-01-02 2017-08-29
88730 Nutcracker needs to automatically recover from MC failure - rebalancing issues Other Operations open 2017-01-02 2017-08-04
88997 Improve graphite failover Other Operations open 2017-02-24 2017-07-20
89808 wikitech instances list is blank Other Operations open 2017-06-08 2017-06-07
89829 bond eth interfaces on ms1001 Other Operations open 2017-01-02 2016-01-05
89887 Clean up permissions for privatedata files on stat1002 - they should be group readable by statistics-privatedata-users Other Operations open 2017-07-12 2017-07-13
91404 Setup backups of elasticsearch indices Other Operations open 2017-01-02 2016-09-22
92471 enable authenticated access to Cassandra JMX Other Operations open 2017-01-02 2016-11-29
93138 Procure hardware for Sentry Other Operations open 2017-01-02 2016-01-06
93531 secure.wikimedia.org entries still showing up in Google search results Other Operations open 2017-01-02 2015-12-08
94215 decommission cp3001 & cp3002 Other Operations open 2017-01-02 2017-08-29
94277 Convert snapshot hosts to use HHVM and trusty Other Operations open 2017-01-02 2017-10-18
94329 secure Cassandra/RESTBase cluster Other Operations open 2017-01-02 2016-10-19
94819 Audit racktables Other Operations open 2017-01-02 2016-03-17
94951 Enable the usage of `hhvm -m debug --debug-host ::1` from mw1017 so developers can step through code (think gdb) in production to see what is going wrong. Other Operations open 2017-01-02 2016-12-01
95052 Make ircecho much better Other Operations open 2017-01-02 2016-07-16
95053 ircecho should accept input via unix sockets Other Operations open 2017-01-02 2016-07-15
95054 Move ircecho config file to be YAML Other Operations open 2017-01-02 2016-07-15
95801 Allow customizing the alert message from graphite Other Operations open 2017-01-02 2016-11-18
97204 RFC: Request timeouts and retries Other Operations open 2017-01-02 2017-03-31
97524 ocg alarm ocg_job_status_queue 'flapping' Other Operations open 2017-01-02 2016-10-03
97909 Upgrade jobrunners to redis 2.8 Other Operations open 2017-01-02 2016-02-13
98831 Honor DNT header for access logs & varnish logs Other Operations open 2017-01-02 2017-05-16
98984 Check power supply balance settings on cp3030+ Other Operations open 2017-01-02 2016-03-17
100777 expose hosts in maintenance state so we can prevent scap from running on them Other Operations open 2017-01-02 2017-02-10
101141 udp rcvbuferrors and inerrors on graphite1001 Other Operations open 2017-01-02 2017-07-10
101585 document redis upgrade/restart procedures Other Operations open 2017-01-02 2016-05-23
102575 document graphite failover/backfill procedures Other Operations open 2017-01-02 2016-05-23
103886 Translation cache exhaustion caused by changes to PHP code in file scope Other Operations open 2017-01-02 2017-09-26
104352 Make scap able to depool/repool servers via the conftool API Other Operations open 2017-01-02 2017-09-26
104671 Rename 'restricted' group? Other Operations open 2017-07-12 2017-07-11
105780 Create a doc explaining the SLA between services and the monitoring tool Other Operations open 2017-01-02 2017-07-11
106346 setup an alertable threshold for Cassandra heap dumps Other Operations open 2017-01-02 2017-07-11
106664 Set up role accounts and feedback loops (FBL) with all providers Other Operations open 2017-01-02 2016-10-04
106937 Monitor [[Special:ListFiles]] for non 200 HTTP statuses in thumbnails Other Operations open 2017-01-02 2017-08-03
107108 Flow notification links on mobile point to desktop Other Operations open 2015-10-02 2017-07-25
108985 Monitor MediaWiki sessions Other Operations open 2017-01-02 2015-12-02
109089 EPIC: Cultivating the Elasticsearch garden (operational lessons from 1.7.1 upgrade) Other Operations open 2017-01-02 2016-10-20
109090 Investigate the need for master only (non data nodes) in our ES cluster Other Operations open 2017-01-02 2016-12-08
109606 Re-evaluate Limesurvey Other Operations open 2017-01-02 2017-06-05
110169 Monitor redis memory/disk usage Other Operations open 2017-01-02 2016-10-03
110171 Alert when ES indexes are freezed for more than 30 minutes Other Operations open 2017-01-02 2016-12-08
110240 [Discussion] Consider validating JSON schemas when running x-ample tests? Other Operations open 2017-01-02 2016-10-12
111540 Clean up labs graphite datapoints Other Operations open 2017-01-02 2016-11-18
111595 Do not apply spam headers on email assessed NOT to be spam Other Operations open 2017-01-16 2017-01-15
111838 Some files had disappeared from Commons after renaming Other Operations open 2017-01-02 2017-07-25
111934 Nutcracker stats monitoring should only listen on localhost Other Operations open 2017-01-02 2016-12-01
112257 rename cassandra cluster Other Operations open 2017-01-02 2016-10-13
112774 solve mtp panel issue for row uplinks Other Operations open 2017-01-02 2016-03-17
113104 Set up a service IP for logstash Other Operations open 2017-06-24 2017-06-23
113785 Make the Shinken IRC alert and icinga-wm bots use colors Other Operations open 2017-01-02 2015-11-02
113792 Change LDAP cn to something more useful (was Rename "Dzahn" to "Daniel Zahn" in Gerrit) Other Operations open 2017-01-02 2017-04-14
114337 Assign 3 more servers to video scaler duty Other Operations open 2017-01-02 2017-02-14
114446 move human users out of UID range for system accounts Other Operations open 2017-01-02 2016-04-15
114801 operations-apache-config-lint replacement doesn't check syntax Other Operations open 2017-01-02 2016-07-28
114849 Log lines on flourine overflow at 8092 bytes. Other Operations open 2017-06-24 2017-06-23
115757 document debian packaging guidelines Other Operations open 2017-01-02 2016-05-23
115899 Move scap target configuration to etcd Other Operations open 2017-01-02 2016-10-05
116063 Hardware Automation Workflow - Overall Tracking Other Operations open 2017-01-02 2017-02-28
116288 Install mailman-api for internal use Other Operations open 2017-01-02 2015-10-23
116580 monitor postgresql replication status Other Operations open 2017-01-02 2015-10-26
116627 Include 5xx numbers in fluorine fatalmonitor Other Operations open 2017-01-02 2015-10-28
116742 Track amount of package updates on systems Other Operations open 2017-01-02 2016-08-08
116747 Meta task "Revamp user authentication" Other Operations open 2017-01-02 2016-08-13
116750 2FA for SSH access to the production cluster Other Operations open 2017-01-02 2015-12-18
116767 limit the impact of heavy/large graphite queries Other Operations open 2017-01-02 2017-01-27
116805 DomainKeys Identified Mail (DKIM) for phabricator.wikimedia.org Other Operations open 2017-01-02 2016-08-31
116951 Reprepro should bail if it can't read and sign using the root keys Other Operations open 2017-01-02 2015-10-28
117508 Make ops-l a list for humans again (no cheating) Other Operations open 2017-01-02 2015-11-02
117673 labs precise and jessie instance not accessible after provisioning Other Operations open 2017-01-02 2016-06-21
118154 determine hardware needs for dumps in eqiad and codfw Other Operations open 2017-01-02 2017-03-24
118331 Alert when used_memory gets too high for redis queues Other Operations open 2017-01-02 2015-11-11
118746 Goal: Strengthen Incident monitoring infrastructure Other Operations open 2017-01-02 2016-08-13
118812 Investigate mysterious_sysctl settings and figure out what to do with them Other Operations open 2017-01-02 2015-11-17
118829 Automate the provisioning and management of MediaWiki clusters Other Operations open 2017-01-02 2015-11-17
119274 Check incoming requests to secure.wm.o Other Operations open 2017-01-02 2015-12-08
119401 Untangle labs/production roles from labs/instance roles Other Operations open 2017-01-02 2015-11-23
119660 Set up LVS for labs dns recursors Other Operations open 2017-01-02 2016-08-15
119679 Rewrite http://download.wikimedia.org/mediawiki/ -> https://releases.wikimedia.org/mediawiki in less than 3 redirects Other Operations open 2017-01-02 2017-05-16
119718 Make it easier to ban misbehaving dashboards from graphite Other Operations open 2017-01-02 2016-11-18
119719 Enforce a minimum refresh period for grafana dashboards hitting graphite Other Operations open 2017-01-02 2016-11-18
119846 Redirect revisions from svn.wikimedia.org to https://phabricator.wikimedia.org/rSVN Other Operations open 2017-01-02 2015-11-30
120159 Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module Other Operations open 2017-01-02 2016-10-04
120165 Implement role based hiera lookups for labs Other Operations open 2017-01-02 2017-08-18
120377 labmon1001 graphite instance archiver keeps archiving the same instances Other Operations open 2017-01-02 2016-11-18
120532 Use user-specific passwords for accessing EventLogging database Other Operations open 2017-01-02 2017-02-20
120585 Make l10nupdate user a system user Other Operations open 2017-01-02 2017-09-26
120856 Remove all out of warranty unused cp10xx's from A2 Other Operations open 2017-01-02 2017-07-20
121105 Mails from MediaWiki seem to get (partially) lost Other Operations open 2017-01-02 2016-03-08
121240 Network isolation for production and semi-production services Other Operations open 2017-01-02 2017-07-12
121610 system users with UIDs > 500 Other Operations open 2017-01-02 2015-12-16
122144 Move most (all?) exim personal aliases to OIT Other Operations open 2017-01-02 2017-07-12
122210 Security audit for tftp on Carbon Other Operations open 2017-01-02 2016-03-29
122825 Service Ownership and Maintenance Other Operations open 2017-01-02 2017-08-08
122917 Provide a good download service of dumps from Wikimedia Other Operations open 2017-01-02 2016-07-19
123106 PNG thumbnail preview of SVG misses some text Other Operations open 2017-01-02 2017-01-06
123237 Provide production jessie image with node 4.2; use this for service-runner build command Other Operations open 2017-01-02 2016-10-12
123276 URL parameters do not work with pages that have "?" in their names Other Operations open 2017-04-29 2017-09-13
123560 investigate rsync between dcs with encryption Other Operations open 2017-01-02 2016-05-30
123809 Module uwsgi doesn't allow passing multiple config params of same name Other Operations open 2017-01-02 2016-01-15
123818 setup YubiHSM and laptop at office Other Operations open 2017-01-02 2016-01-25
123918 'swift' user/group IDs should be consistent across the fleet Other Operations open 2017-01-02 2017-04-10
124101 Specific revisions of multiple files missing from Swift - 404 Not Found returned Other Operations open 2017-01-02 2017-07-25
124179 Improve access to and control over incident and metrics monitoring infrastructure Other Operations open 2017-01-02 2017-03-08
124185 Evaluate alternative web interfaces to icinga 1 core Other Operations open 2017-01-02 2017-04-18
124413 confctl should provide tags information after writing data Other Operations open 2017-01-02 2016-01-22
124991 evaluate possibility for nscd use with useldap Other Operations open 2017-01-02 2017-07-24
125015 Requests to (hard) redirect pages return their target's contents but are counted as pageviews to the redirect page Other Operations open 2017-01-02 2016-01-28
125085 Split the API MediaWiki appserver pool into two external/internal pools Other Operations open 2017-01-02 2016-10-12
125411 Diamond load averages do not contain scaled versions Other Operations open 2017-01-02 2017-07-25
125442 es2009 degraded RAID Other Operations open 2017-01-02 2016-04-22
126083 overhaul labstore setup [tracking] Other Operations open 2017-01-02 2017-07-23
126158 [RFC] Alert about *when* partitions will run out of space, not a percentage/absolute number Other Operations open 2017-01-02 2016-05-02
126281 [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") Other Operations open 2017-01-02 2016-08-08
126295 Spike: What do we have to package to run the Programs and Events dashboard on production? Other Operations open 2017-01-02 2017-02-28
126574 puppet should try to mount all mountable swift filesystems Other Operations open 2017-01-02 2016-02-11
126619 cassandra slow streaming during (de)commission Other Operations open 2017-01-02 2017-07-11
126989 MediaWiki logging & encryption Other Operations open 2017-01-02 2017-07-10
127054 pinentry-gtk2 pulls in a lot of unneeded Gnome/GTK libs Other Operations open 2017-01-02 2016-02-16
127549 move travel related aliases to OIT Other Operations open 2017-01-02 2016-11-05
127797 document all puppet classes / defined types!? Other Operations open 2017-01-02 2016-11-28
127825 Re-add intel-microcode Other Operations open 2017-01-02 2016-09-25
128590 Cassandra uses default ip address for outbound packets while bootstrapping Other Operations open 2017-01-02 2016-12-22
128615 Get rid of Tool Labs home page check from shinken Other Operations open 2017-01-02 2016-12-04
128715 Add other Tools administrators to the Icinga notification group Other Operations open 2017-01-02 2016-12-04
128716 Make icinga-wm report Tools homepage check at #wikimedia-labs, too Other Operations open 2017-01-02 2016-05-27
129180 Preserve SSH host key when re-imaging hosts Other Operations open 2017-01-02 2016-03-07
129188 mw2212 unresponsive Other Operations open 2017-01-02 2016-03-08
129621 "internal_api_error_MWException: [dbf916b7] Exception Caught: Could not acquire lock for" for some uploads (during upload with Pywikibot OAuth) Other Operations open 2017-01-02 2016-07-20
129847 conftool-merge should report which node is setting attributes for Other Operations open 2017-01-02 2016-03-14
129963 Update memcached package and configuration options Other Operations open 2017-01-02 2017-08-16
130209 Collect threaddumps from elasticsearch at regular intervals Other Operations open 2017-01-02 2016-03-17
130512 Grafana: Job Queue Health: Panel is displayed incorrectly Other Operations open 2017-01-02 2016-03-20
130590 Have dedicated master nodes for elasticsearch Other Operations open 2017-01-02 2016-11-03
130593 investigate slapd memory leak Other Operations open 2017-01-02 2017-06-30
130617 Collect metrics on pool counter usage Other Operations open 2017-01-02 2016-11-03
130709 authoritative copy of 'root' files for upload.wikimedia.org is only in swift Other Operations open 2017-01-02 2017-03-29
131326 smokeping config puppetization issue? Other Operations open 2017-01-02 2016-04-15
131748 Refresh the appservers puppet code/configs Other Operations open 2017-01-02 2016-04-04
131832 Unable to restore file that has a very large file size Other Operations open 2017-01-02 2017-07-25
131928 Upgrade jessie systems from Linux 3.19 to 4.4 Other Operations open 2017-01-02 2016-08-23
131966 Default gateway unreachable on baham.wikimedia.org after reboot Other Operations open 2017-01-02 2017-07-20
132104 Consider moving policy.wikimedia.org away from WordPress.com Other Operations open 2017-01-02 2016-06-08
132216 Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm Other Operations open 2017-01-02 2016-05-09
132256 Analytics hosts showed high temperature alarms Other Operations open 2017-01-02 2017-10-06
132324 Tracking and Reducing cron-spam from root@ Other Operations open 2017-01-02 2017-06-15
132325 Weak digest algorithm (SHA1) used to sign InRelease on apt.wikimedia.org Other Operations open 2017-01-02 2017-04-03
132532 rsync module doesnt work on trusty Other Operations open 2017-01-02 2016-04-12
132632 puppetize turning off reserved space for cassandra /srv Other Operations open 2017-01-02 2017-07-18
132856 Write documentation on how / when to use custom Diamond metrics collectors Other Operations open 2017-01-02 2016-04-16
133091 Highest SSTables / read thresholds Other Operations open 2017-01-02 2017-07-11
133093 Investigate idle appservers in codfw Other Operations open 2017-01-02 2016-04-19
133164 Document eqiad/codfw transition plan for OCG Other Operations open 2017-01-02 2016-05-23
133179 Redis monitoring needs to be improved Other Operations open 2017-01-02 2017-07-07
133318 High levels of PoolCounter errors should trigger alerts Other Operations open 2017-01-02 2017-10-13
133392 save grafana dashboards in revision control / puppet Other Operations open 2017-01-02 2017-07-24
133476 Proposal: Centralize OTRS login methodology Other Operations open 2017-01-02 2017-08-15
133643 Upstream our Diamond PowerDNSRecursorCollector Other Operations open 2017-01-02 2016-04-26
133656 Have a paging check for Nova API accessible Other Operations open 2017-01-02 2016-12-21
133674 HHVM is leaking memory on the API appservers Other Operations open 2017-01-02 2016-10-10
133744 Epic: switch Maps to production status Other Operations open 2017-01-02 2016-07-19
133844 Improve Elasticsearch icinga alerting Other Operations open 2017-01-02 2017-05-04
133913 Completely port l10nupdate to scap Other Operations open 2017-01-02 2017-09-26
134237 Graphoid returns a 400 on MW API time-out Other Operations open 2017-01-02 2016-10-12
134271 Replace ircd-ratbox with something newer/maintained Other Operations open 2017-01-02 2016-06-17
134326 udpmxircecho should write stats of messages processed and we should alert when that drops to zero Other Operations open 2017-01-02 2016-05-03
134458 status.wikimedia.org should use some Wikimedia favicon if possible Other Operations open 2017-01-02 2016-05-05
134551 Create functional cluster checks for all services (and have them page!) Other Operations open 2017-01-02 2017-08-15
134811 Consider REST with SSL (HyperSwitch/Cassandra) for session storage Other Operations open 2017-01-02 2017-07-12
134875 udpmxircecho spam/not working if unable to connect to irc server Other Operations open 2017-01-02 2016-07-15
135113 Rationalize our jobqueues redis topology Other Operations open 2017-01-02 2017-07-07
135122 Reduce etcd technical debt Other Operations open 2017-01-02 2016-05-12
135124 Deploy etcddump (or another etcd dump & load tool) to production Other Operations open 2017-01-02 2016-05-12
135125 Install a second etcd cluster in codfw Other Operations open 2017-01-02 2016-05-12
135128 Turn on etcd TLS for intra-cluster communications Other Operations open 2017-01-02 2016-05-12
135318 Document how to handle 'inconsistent state within the internal storage backends' issues Other Operations open 2017-01-02 2016-07-31
135338 On Trusty and Jessie PHP yields: PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/20-xhprof.ini on line 2 Other Operations open 2017-01-02 2017-07-01
135385 investigate carbon-c-relay stalls/drops towards graphite2002 Other Operations open 2017-01-02 2016-11-18
135595 mod_deflate + mod_uwsgi causing mangled apache responses Other Operations open 2017-01-02 2016-05-18
135991 Automated service restarts for common low-level system services Other Operations open 2017-01-02 2016-06-21
136094 Race condition in setting net.netfilter.nf_conntrack_tcp_timeout_time_wait Other Operations open 2017-01-02 2017-08-09
136311 Monitor the BMC's event log for hardware errors Other Operations open 2017-01-02 2016-05-26
136312 Encrypt syslog traffic Other Operations open 2017-01-02 2017-10-10
136403 Move cp3030+ from OE14 to OE13 in racktables Other Operations open 2017-01-02 2016-06-21
136562 Audit/fix hosts with no RAID configured Other Operations open 2017-01-02 2016-08-24
136603 Update limit.sh to support systemd-based cgroup management Other Operations open 2017-01-02 2017-10-11
136702 Increase time before alert for elasticsearch disk space issues Other Operations open 2017-01-02 2016-11-11
136735 create endowment.wm.org microsite Other Operations open 2017-01-02 2017-10-12
137176 catch-all apache vhost on the cluster should return 404 for non-existing sites Other Operations open 2017-01-02 2016-11-24
137217 Clean up apt:pin of python modules used for Nodepool Other Operations open 2017-01-02 2016-11-16
137229 Tune thread for osm2pgsql / postgres max connections for Maps Other Operations open 2017-01-02 2016-12-15
137397 revisit swift (sys)logging Other Operations open 2017-01-02 2016-07-06
137616 Epic: cultivating the Maps garden Other Operations open 2017-01-02 2016-12-15
137791 libcglib3-java replaces libcglib-java in Jessie Other Operations open 2017-01-02 2016-06-14
137939 Increase frequency of OSM replication Other Operations open 2017-01-02 2017-06-06
138017 Improve automation around Maps servers Other Operations open 2017-01-02 2016-12-15
138136 MB Lateefi Fonts for Sindhi Wikipedia. Other Operations open 2017-01-02 2016-12-10
138496 bring swift eqiad to one zone per row Other Operations open 2017-01-02 2016-06-23
138685 notebook1001 shown as DOWN in icinga, due to firewall rules Other Operations open 2017-01-02 2016-06-26
138799 Create a simple puppet role for setting up a singlenode kubernetes install Other Operations open 2017-01-02 2016-06-27
138821 extend existing graphite whisper files retention to five years Other Operations open 2017-01-02 2016-11-18
138866 Update & standardize Platform-specific_documentation for HP servers Other Operations open 2017-01-02 2016-06-28
139971 access_new_install role vs. Labs vs. the future Other Operations open 2017-01-02 2016-07-11
140075 investigate swift used space spikes since June 2016 Other Operations open 2017-01-02 2016-07-12
140141 Install mscorefonts on scaling servers for SVG rendering Other Operations open 2017-01-02 2017-07-25
140270 Determine a core set or a checklist of permissions for deployment purpose Other Operations open 2017-01-02 2017-05-16
140316 Add granularity limiter (g=) to wikimedia.org DKIM record(s) Other Operations open 2017-01-02 2016-08-05
140442 reinstall rdb100[56] with RAID Other Operations open 2017-01-02 2017-04-18
140536 Thumbnails of some specific images show unwanted black lines Other Operations open 2017-01-02 2016-07-19
140594 svn.wikimedia.org redirects to Diffusion main page, hence hard to find e.g. "flexbisonparse" Other Operations open 2017-01-02 2016-07-18
140813 Protect sensitive user-related information with a UserData / auth / session service Other Operations open 2017-01-02 2017-06-08
140879 503 error raises again while trying to load a Wikidata page Other Operations open 2017-01-02 2017-06-23
140942 Tracking: Monitoring and alerts for "business" metrics Other Operations open 2017-01-02 2017-06-21
141038 implement icinga paging for non-ops teams Other Operations open 2017-01-02 2016-07-21
141128 determine/process/document bios firmware tracking/updating policies Other Operations open 2017-01-02 2016-07-22
141520 "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) Other Operations open 2017-01-02 2017-05-20
141524 eventbus should send statsd in batches Other Operations open 2017-01-02 2016-11-18
141704 Storage backend errors on commons when deleting/restoring pages Other Operations open 2017-01-02 2017-07-25
141756 audit / test / upgrade hp smartarray P840 firmware Other Operations open 2017-01-02 2017-06-06
141783 Add monitoring for detecting when logstash services are down Other Operations open 2017-01-02 2016-08-01
141897 Review new service 'pre-deployment to production' checklist Other Operations open 2017-01-02 2017-07-12
141959 Moving network::external to hiera broke much of labs Other Operations open 2017-01-02 2016-08-03
142002 Clean up puppet & configs for ORES Other Operations open 2017-01-02 2016-09-22
142205 use granularity (g=) restrictions for wikimedia.org fundraising DKIM records Other Operations open 2017-01-02 2016-08-05
142815 Enhance account handling (meta bug) Other Operations open 2017-01-02 2017-03-28
142821 Synchronise groups defined in data.yaml to LDAP Other Operations open 2017-01-02 2016-08-12
142827 Enforce reference to Phabricator task for all commits to modules/admin/data/data.yaml Other Operations open 2017-01-02 2017-03-10
142984 Review lists of config/sysctl recommendations by "kernel self-protection project" Other Operations open 2017-01-02 2016-08-15
142991 Enable "upload by url" feature at zhwiki Other Operations open 2017-01-02 2017-06-08
143552 Make elasticsearch configuration more robust to loss of network connectivity Other Operations open 2017-01-02 2016-08-30
143556 Setting up grafana should also setup Anonymous read-only access for the default org Other Operations open 2017-01-02 2016-11-29
143931 Update ICU version to 55.1 Other Operations open 2017-01-02 2016-08-25
144006 Move the MW Beta appservers to Debian Other Operations open 2017-01-02 2017-08-29
144431 RESTBase k-r-v as Cassandra anti-pattern Other Operations open 2017-01-02 2017-03-17
144539 Remove /srv/deployment/wdqs/wdqs/rules.log symlink Other Operations open 2017-01-02 2016-10-11
144933 Cleanup debconf handling in mailman puppet setup Other Operations open 2017-01-02 2016-11-29
145065 Decrease time required to fully restart the Cirrus elasticsearch clusters Other Operations open 2017-01-02 2016-11-30
146090 High failure rate of account creation should trigger an alarm / page people Other Operations open 2017-01-02 2017-05-20
146285 Switch mwscript from Zend PHP5 to default php alternative (e.g. HHVM or PHP7) Other Operations open 2017-01-02 2017-10-18
146355 Replace etcd internal auth mechanism with a frontend proxy Other Operations open 2017-01-02 2016-09-22
146627 Make deployment-prep puppetmaster more similar to Production puppetmaster Other Operations open 2017-01-02 2016-10-05
146657 create notifications about user accounts that have not been used for a long time Other Operations open 2017-01-02 2016-09-26
146664 Limit resources used by ORES Other Operations open 2017-01-02 2017-05-01
146841 Reach out to Google about @yahoo.com emails not reaching gmail inboxes (when sent to mailing lists) Other Operations open 2017-01-02 2017-07-18
146968 OTRS spam classification methods and systems Other Operations open 2017-01-02 2017-10-11
147040 Two recently uploaded files have disappeared (404) Other Operations open 2017-01-02 2017-03-11
147204 Update confd package Other Operations open 2017-01-02 2017-09-14
147366 Setup automated topk wide row reporting Other Operations open 2017-01-02 2017-07-18
147872 Rename rhodium to puppetmaster1003 Other Operations open 2017-01-02 2016-10-11
147905 investigate lead hardware issue Other Operations open 2017-01-02 2017-04-28
147923 Extract metrics from logs Other Operations open 2017-01-02 2017-09-20
148017 lvs2002 repeated usb connect/disconnect message Other Operations open 2017-01-02 2017-04-04
148048 Store Wikimedia unified account name (SUL) in LDAP directory Other Operations open 2017-01-02 2017-06-20
148061 Feasibility of hosting podcast setup on Wikimedia servers Other Operations open 2017-01-02 2016-10-13
148478 Investigate seemingly random Gerrit slow-downs Other Operations open 2017-01-02 2017-08-17
148567 Restrict outgoing network connections from Electron render service Other Operations open 2017-01-02 2017-02-18
148614 Icinga check for Tor Other Operations open 2017-01-02 2017-06-19
148637 Port redis statistics to Prometheus Other Operations open 2017-01-02 2017-10-19
148647 refresh swift hardware in codfw/eqiad Other Operations open 2017-01-02 2017-03-16
148693 Deploy IDS rendering engine to production Other Operations open 2017-01-02 2017-08-26
148843 GPU upgrade for stats machine Other Operations open 2017-01-02 2017-08-06
148968 Build Kubernetes for production use Other Operations open 2017-01-02 2017-06-05
148986 Firewall sets not being loaded post-reboot due to a @resolve race on jessie Other Operations open 2017-01-02 2017-05-31
149057 Designate seems very slow to delete records? Other Operations open 2017-01-02 2016-10-24
149287 Heating alerts for mw servers in eqiad Other Operations open 2017-01-02 2017-05-25
149421 Long running mediawiki web requests impacts service availability, specially databases Other Operations open 2017-01-02 2017-04-05
149543 Setup PAWS internal experimentally on notebook* nodes Other Operations open 2017-01-02 2016-10-31
149589 Puppet tab in Horizon unusably slow Other Operations open 2017-01-21 2017-02-02
149617 Integrating MediaWiki (and other services) with dynamic configuration Other Operations open 2017-01-02 2017-10-07
149804 Review of ferm services without srange Other Operations open 2017-01-02 2016-11-02
149845 Something is wrong with installer root disk stuff Other Operations open 2017-01-02 2017-01-20
149885 Investigate Swift as a storage backend for maps tiles Other Operations open 2017-01-02 2016-12-09
150020 Refactor puppet-postgresql module to use custom types Other Operations open 2017-01-02 2016-12-09
150185 Deploy ElectronPdfService Extension to production Other Operations open 2017-01-02 2017-10-19
150300 icinga notification if elevated writing to badpass.log Other Operations open 2017-01-02 2017-09-07
150356 Wikidata Query Service is overly verbose toward logstash Other Operations open 2017-01-02 2016-12-16
150396 Phabricator leaving old files in /tmp Other Operations open 2017-01-02 2017-01-03
150460 Configure maps cluster to send statsd metrics to the statsd endpoint in the same datacenter Other Operations open 2017-01-02 2016-11-10
150466 publish kartotherian / tilerator metrics by cluster Other Operations open 2017-01-02 2016-11-10
150486 Deploy federation for Prometheus Other Operations open 2017-01-02 2016-11-15
150532 Upgrade qemu on ganeti clusters to 2.7 Other Operations open 2017-01-02 2017-05-16
150651 Information missing from racktables Other Operations open 2017-01-02 2017-07-13
150672 Provide a /parsoid directory on releases.wikimedia.org Other Operations open 2017-01-02 2017-05-20
150771 Secondary production Jenkins for CI Other Operations open 2017-01-02 2017-10-12
150811 Evaluate ScyllaDB as a near-term replacement to Cassandra Other Operations open 2017-01-02 2017-01-25
150822 Internal PKI for secure communication - Barcelona Ops offsite 2016 Other Operations open 2017-01-02 2017-02-22
150823 Puppet CA rollover Other Operations open 2017-01-02 2016-11-29
150871 [EPIC] (Proposal) Replicate core OCG features and sunset OCG service Other Operations open 2017-01-02 2017-10-10
150872 Replace OCG in collection extension with Electron Other Operations open 2017-01-02 2017-10-02
150875 Confirm attribution needs Other Operations open 2017-01-02 2017-06-29
150912 Class 'Memcached' not found when running mwscript eval.php on debug servers Other Operations open 2017-02-01 2017-03-07
150917 Remove deprecated features from book creator UI Other Operations open 2017-01-02 2017-10-02
151009 Provide authenticated access to Prometheus native web interface Other Operations open 2017-01-02 2017-09-11
151045 Extending Yubico 2FA for production use (meta bug) Other Operations open 2017-01-02 2016-11-18
151046 Fully puppetise yubikey-val Other Operations open 2017-01-02 2016-11-18
151047 Integrate Yubikey into data.yaml Other Operations open 2017-01-02 2016-11-18
151048 Icinga monitoring for Yubikey components Other Operations open 2017-01-02 2017-09-07
151049 Run systematic availability tests Other Operations open 2017-01-02 2016-11-18
151050 Proper documentation for Yubico 2FA for production use Other Operations open 2017-01-02 2016-11-18
151273 lvs4002 power supply failure Other Operations open 2017-01-02 2017-10-18
151275 cp4008 and cp4012 running on single PSU Other Operations open 2017-01-02 2017-10-18
151304 tmpreaper possible race condition Other Operations open 2017-01-02 2016-11-22
151310 create-dbusers service failing on labstore1004 Other Operations open 2017-01-02 2016-11-22
151314 logrotate failing with $FILE.1.gz: File exists Other Operations open 2017-01-02 2016-11-22
151317 stat user crontab on stat hosts for old file removal Other Operations open 2017-01-02 2016-11-22
151486 Silver anomalies Other Operations open 2017-01-02 2016-12-14
151489 silver: /dev/md2 mounted twice Other Operations open 2017-01-02 2016-12-14
151493 silver: / partition low on space Other Operations open 2017-01-02 2016-11-24
151702 API cluster failure / OOM Other Operations open 2017-01-02 2017-05-16
152073 Check concurrency/retry/timeout limits and syncronize those between services Other Operations open 2017-01-02 2016-12-30
152078 Load balancing "external" traffic to the Kubernetes cluster in production Other Operations open 2017-01-02 2017-06-05
152100 should we make privatewiki list available to puppet without maintaining two lists? Other Operations open 2017-01-02 2016-12-07
152439 cronspam from labtestservices2001 /etc/dns-floating-ip-updater.py > /dev/null Other Operations open 2017-01-02 2016-12-05
152445 Move prometheus entry point off port 80 Other Operations open 2017-01-02 2017-07-10
152632 Explore hosting the multimedia commons use case Other Operations open 2017-01-02 2016-12-07
152724 Current state and next steps for RESTBase storage Other Operations open 2017-01-02 2017-07-12
152767 Missing Labs hiera entry in labs-private repo Other Operations open 2017-01-02 2017-08-30
152782 Kibana functionality missing after upgrade: histograms Other Operations open 2017-01-02 2017-05-20
153068 Consider mounting labs NFS labstore1003.eqiad.wmnet:/scratch for server-side uploads Other Operations open 2017-01-02 2016-12-13
153246 Puppet failures with "Attempt to assign to a reserved variable name: 'trusted'" Other Operations open 2017-01-02 2017-09-11
153279 labnet/ labtestnet2001 - disk space - nova-api.log needs rotation Other Operations open 2017-01-02 2016-12-15
153416 docker-engine pulled into our repositories only keeps the latest version Other Operations open 2017-01-02 2017-01-04
153703 Option: Consider switching back to leveled compaction (LCS) Other Operations open 2017-01-02 2016-12-19
153816 apache::static_site is not working Other Operations open 2017-01-02 2016-12-22
153940 Logrotate fails for: "$FILE No such file or directory" Other Operations open 2017-01-02 2017-03-14
154026 On mobile, http://wikipedia.org/wiki/Foo redirects to https://www.m.wikipedia.org/wiki/Foo which does not exist Other Operations open 2017-07-26 2017-07-27
154619 Export ipsec counters as Prometheus metrics Other Operations open 2017-08-02 2017-08-01
154627 Production error message (when servers are down) points users to donate link which is likely to produce the same error message Other Operations open 2017-01-05 2017-04-29
154665 Look into behaviour of /etc/exim4/update-exim4.conf.conf related to updates Other Operations open 2017-01-06 2017-01-05
154915 Get rid of "import realm.pp" in manifests/site.pp Other Operations open 2017-01-10 2017-07-06
155129 Create prometheus nutcracker exporter Other Operations open 2017-01-12 2017-08-11
155209 Increase $wgHTTPImportTimeout to a higher value on WMF wikis Other Operations open 2017-01-14 2017-01-13
155401 Integrate jessie 8.7 point release Other Operations open 2017-01-17 2017-04-05
155761 DNS repo: add Jenkins job to ensure there are no duplicates Other Operations open 2017-01-20 2017-01-19
155929 Create /community-beacon alternative entry point Other Operations open 2017-01-23 2017-05-17
156136 Increase swift replication factor for accounts Other Operations open 2017-01-25 2017-01-24
156140 Lots of hosts with hyperthreading disabled Other Operations open 2017-01-25 2017-05-25
156143 High CPU usage from swift-proxy on frontend machines Other Operations open 2017-01-25 2017-01-24
156232 confctl SubjectAltNameWarning after python-urllib3 upgrade Other Operations open 2017-01-26 2017-02-22
156398 Decommission or repair old asw-c2-eqiad Other Operations open 2017-01-27 2017-07-20
156475 Investigate spike in 500s during asw-c2-eqiad replacement Other Operations open 2017-01-28 2017-02-09
156544 Create backups of Wikimedia content in diverse geographic places Other Operations open 2017-01-29 2017-01-28
156570 Investigate issues with wikitech-static.wikimedia.org Other Operations open 2017-01-30 2017-01-29
156924 Allow integration of data from etcd into the MediaWiki configuration Other Operations open 2017-02-02 2017-10-19
156937 Provide cross-dc redundancy (active-active or active-passive) to all important misc services Other Operations open 2017-02-02 2017-07-28
156955 Standardizing our partman recipes Other Operations open 2017-02-02 2017-05-24
157038 Make it possible to run the mediawiki testsuite against a staging repo of apt.wikimedia.org Other Operations open 2017-02-03 2017-06-13
157306 Fix config file handling for /etc/hhvm/php.ini Other Operations open 2017-02-07 2017-02-07
157972 Puppet fails only once when restarting ferm is not successful Other Operations open 2017-02-14 2017-02-13
158022 make apt.wikimedia.org HA Other Operations open 2017-02-15 2017-02-27
158196 Reimage labstore1001 and labstore1002 for DRBD storage setup Other Operations open 2017-02-16 2017-07-24
158288 Unclean stop of jobrunner service via puppet Other Operations open 2017-02-17 2017-02-16
158429 Switch to predictable network interface names? Other Operations open 2017-02-18 2017-06-07
158434 Phabricator: Make sure phabricator works properly including our puppet roles on jessie Other Operations open 2017-02-18 2017-05-20
158562 Manage apt sources via puppet? Other Operations open 2017-02-21 2017-02-20
158583 Restructure our internal repositories further Other Operations open 2017-02-21 2017-07-19
158757 Puppet certificate missing subjectAltName Other Operations open 2017-02-23 2017-03-22
158837 Consolidate performance website and related software Other Operations open 2017-02-24 2017-10-19
158915 Make sure replying to emails in gerrit 2.14 works Other Operations open 2017-02-24 2017-06-14
159242 Segmentation fault creating thumbnail Other Operations open 2017-03-01 2017-02-28
159354 Move coal from graphite machine(s) Other Operations open 2017-03-02 2017-09-12
159524 backup space is used unwisely Other Operations open 2017-03-04 2017-04-12
159536 Puppet constantly trying to stop the already stopped puppetmaster process on Trusty Other Operations open 2017-03-04 2017-03-03
159661 Improve Terbium (and wasat) userland to process server side uploads Other Operations open 2017-03-07 2017-03-06
159687 etcd switchover/enhancements Other Operations open 2017-03-07 2017-05-15
159750 E-mail for people in different OIT LDAP object unit Other Operations open 2017-03-09 2017-09-25
159830 Sanity check global-multiwrite logs for ConfirmEdit usage Other Operations open 2017-03-08 2017-03-07
160060 Icinga check for sysctl settings Other Operations open 2017-03-10 2017-09-07
160071 Add slabinfo prometheus exporter Other Operations open 2017-03-31 2017-03-30
160101 Upgrade php5-json .deb to at least 1.3.8 Other Operations open 2017-03-10 2017-03-09
160146 jobrunner/jobchron services fail in codfw Other Operations open 2017-03-11 2017-03-10
160158 Make disabled accounts visible in the corp mirror LDAP replica Other Operations open 2017-03-11 2017-03-28
160229 Back up of Commons files Other Operations open 2017-03-12 2017-05-14
160412 Add lock_wait_timeout to maintain_views and maintain-meta_p Other Operations open 2017-03-15 2017-03-26
160529 Sender email spoofing Other Operations open 2017-03-29 2017-07-28
160644 Eventstreams graphite disk usage Other Operations open 2017-04-02 2017-08-29
160677 Effects on adjusting Prometheus retention Other Operations open 2017-03-17 2017-07-10
160941 Improve SSH access information in onboarding documentation Other Operations open 2017-03-31 2017-03-30
161003 Cross-check disabled accounts from corp LDAP against data.yaml Other Operations open 2017-03-22 2017-03-21
161004 Remove disabled users from internal mailing lists Other Operations open 2017-03-22 2017-09-07
161096 confctl no longer logs a non-changing state change Other Operations open 2017-03-23 2017-03-22
161145 Fix the general problem of randomly-bad puppet agent cron timings within redundant clusters Other Operations open 2017-03-23 2017-07-21
161296 Upgrade mysqld_exporter to 0.10.0 Other Operations open 2017-03-25 2017-05-15
161528 incident 20170323-wikibase did not trigger Icinga paging Other Operations open 2017-03-28 2017-03-27
161598 Monitor HHVM bytecode cache depletion on mediawiki app servers Other Operations open 2017-03-29 2017-06-15
161834 Undo special tools-home and tools-project share definitions for NFS Other Operations open 2017-03-31 2017-07-23
161835 Convert labstore cluster configuration to hiera and profiles Other Operations open 2017-03-31 2017-07-23
161864 404 error while accessing some images files e.g. djvu and jpg Other Operations open 2017-03-31 2017-03-30
161899 End self-service new Trusty instance creation in Cloud VPS; standardize on Debian base images Other Operations open 2017-04-01 2017-10-05
161904 decommission backup4001 Other Operations open 2017-04-01 2017-07-18
161918 videoscalers (mw1168, mw1169) - high load / overheating Other Operations open 2017-04-01 2017-04-01
161920 logrotate for ruthenium Other Operations open 2017-04-01 2017-05-10
162013 etcd cluster in codfw has raft consensus issues Other Operations open 2017-04-04 2017-09-14
162029 Migrate all jessie hosts to Linux 4.9 Other Operations open 2017-04-04 2017-04-13
162037 Use SSL certificates with discovery entry for elasticsearch Other Operations open 2017-04-04 2017-10-19
162090 Investigate alternative RAID strategies for labstore1001/2 Other Operations open 2017-04-04 2017-06-06
162122 Swiftrepl was stuck in an infinite loop since days Other Operations open 2017-04-05 2017-04-12
162123 Running swiftrepl is not puppetized Other Operations open 2017-04-05 2017-09-28
162245 Enable GC for HHVM CLI (at least for dump runners) Other Operations open 2017-04-06 2017-07-27
162612 codfw/eqiad hosts occasionally spend > 3 minutes starting networking.service with linux 4.9 Other Operations open 2017-04-11 2017-07-20
162850 CPU throttling on DELL PowerEdge R320 Other Operations open 2017-04-13 2017-09-13
162857 Some Core availability Catchpoint tests might be more expensive than they need to be Other Operations open 2017-04-14 2017-10-11
162955 rebuild tools-grid-master as a large instance Other Operations open 2017-04-14 2017-06-08
163033 Create grafana dashboard for video scaler job runners Other Operations open 2017-05-24 2017-05-23
163068 More missing 'original' files on Commons Other Operations open 2017-04-17 2017-07-25
163288 Decide on /var/lib vs /home as locations of homedir for l10nupdate Other Operations open 2017-04-20 2017-04-19
163336 kube-proxy pulls in docker and starts service even when it isnt needed Other Operations open 2017-04-20 2017-04-19
163354 Find a way to verify mediawiki-config IPs ahead of datacenter switchovers Other Operations open 2017-04-20 2017-04-19
163362 audit all codfw pdu tower draws Other Operations open 2017-04-20 2017-04-19
163393 Determine appropriate proxy_read_timeout setting for Tools Proxy Other Operations open 2017-04-20 2017-04-19
163402 Ensure we can survive a loss of labservices1001 Other Operations open 2017-04-20 2017-08-03
163507 Intermittent DB connectivity problem on phabricator, needs investigation Other Operations open 2017-04-21 2017-05-24
163667 Fix UIDs for deployment server users Other Operations open 2017-04-25 2017-04-24
163673 Some swift disks wrongly mounted on 5 ms-be hosts Other Operations open 2017-04-25 2017-08-22
163698 Add flood protection to the ircecho bot (icinga-wm) Other Operations open 2017-04-25 2017-04-24
163823 During labservices1001 failover fqdn changed from foo.project.eqiad.wmflabs to foo.eqiad.wmflabs Other Operations open 2017-04-26 2017-04-25
163996 Icinga check for ipv6 host reachability Other Operations open 2017-04-28 2017-04-27
164042 Racktables: clearly show when hosts are decommissioned Other Operations open 2017-04-28 2017-04-27
164123 tools-k8s-master-01 has two floating IPs Other Operations open 2017-04-29 2017-06-06
164238 move icinga contacts file to public repo Other Operations open 2017-05-03 2017-05-02
164248 HTTP responses from app servers sometimes stall for >1s Other Operations open 2017-05-03 2017-09-12
164290 Set up external DNS record for wikitech-static Other Operations open 2017-06-07 2017-06-06
164341 Decommission old memcached hosts - mc1001->mc1018 Other Operations open 2017-05-04 2017-10-18
164460 Use DNS discovery record for deployment CNAME Other Operations open 2017-05-05 2017-05-09
164490 maintain-meta_p hangs on connecting to wikimedia.org.uk Other Operations open 2017-05-05 2017-05-04
164703 Integrate jessie 8.8 point release Other Operations open 2017-05-09 2017-05-08
164819 reprepro: Support for buildinfo files / dbgsym packages Other Operations open 2017-05-10 2017-06-02
164993 archiva artifact links point to 127.0.0.1 Other Operations open 2017-05-11 2017-05-10
165105 Wiley requests for DOI and some other publishers don't work in production Other Operations open 2017-05-26 2017-08-25
165136 Ferm rules for labstore NFS hosts Other Operations open 2017-05-13 2017-10-06
165170 rack/setup/install ores2001-2009 Other Operations open 2017-05-13 2017-09-06
165173 rack/setup/install dumpsdata100[12] Other Operations open 2017-05-13 2017-07-06
165323 Add Prometheus machine metric to track core dumps Other Operations open 2017-05-16 2017-05-15
165345 decommission indium Other Operations open 2017-05-16 2017-07-20
165511 Change automatic shortlink in blog theme Other Operations open 2017-06-20 2017-06-20
165519 rack and setup mw1307-1348 Other Operations open 2017-05-17 2017-10-20
165618 Audit / document reasons for not enabling HT? Other Operations open 2017-05-18 2017-05-17
165631 move gerrit.wm.org SSH service to private/behind LVS like phab-vcs Other Operations open 2017-05-18 2017-05-17
165779 rack/setup/install labnet100[34] Other Operations open 2017-05-20 2017-08-22
165781 rack/setup/install labcontrol100[34] Other Operations open 2017-05-20 2017-07-12
165784 rack/setup/install labmon1002 Other Operations open 2017-05-20 2017-08-22
165885 Create a cron to clean clientbucket every day or hour Other Operations open 2017-05-21 2017-05-20
166038 Sync internal nutcracker package with Debian package Other Operations open 2017-05-23 2017-05-29
166066 Integrate the puppet compiler in the puppet CI pipeline Other Operations open 2017-05-23 2017-07-06
166081 rack/setup/install conf1004-conf1006 Other Operations open 2017-05-23 2017-09-28
166233 Update redis puppet class to support stretch Other Operations open 2017-05-25 2017-05-24
166291 Exim panics when spamd reaches maxchildren Other Operations open 2017-05-26 2017-09-18
166322 spam from phabricator in labs Other Operations open 2017-05-26 2017-05-26
166368 Wipe of spare/replacement disks Other Operations open 2017-05-27 2017-05-26
166937 Broken /a/refinery-source/guard/run_all_guards.sh script on stat1002 Other Operations open 2017-06-04 2017-10-19
167035 stretch acct monthly cron will spam when /var/log/wtmp.1 doesn't exist Other Operations open 2017-06-06 2017-06-05
167091 Elasticsearch errors about BulkShardRequest Other Operations open 2017-06-07 2017-08-03
167225 Upload hhvm to stretch apt repo in apt.wikimedia.org Other Operations open 2017-06-22 2017-09-21
167239 Redirect status.wikipedia.org to status.wikimedia.org Other Operations open 2017-06-07 2017-06-14
167245 prometheus-node-exporter - invalid group: ‘prometheus:prometheus' Other Operations open 2017-06-08 2017-06-07
167292 Collate jessie-wikimedia/backports into jessie-wikimedia/main Other Operations open 2017-06-08 2017-06-07
167412 host-vmem.erb is doing operations that make no sense Other Operations open 2017-06-09 2017-06-17
167422 Monitoring: add link to graph for Icinga timeseries alarms Other Operations open 2017-06-09 2017-06-08
167549 Create Icinga alert when OSM replication lags on maps Other Operations open 2017-06-10 2017-09-07
167689 Add RIPE atlas data to Prometheus Other Operations open 2017-08-02 2017-08-01
167966 Look into feasibility of disabling sha-1 host keys on our ssh daemons Other Operations open 2017-06-16 2017-07-12
168403 Aggregate prometheus functions yielding different results in grafana vs. prometheus console Other Operations open 2017-06-21 2017-06-20
168407 rack/setup/install labnodepool1002.eqiad.wmnet Other Operations open 2017-06-21 2017-07-12
168445 Reboots of cloud servers Other Operations open 2017-06-21 2017-06-20
168460 Update certificates on productions replicas of corp.wikimedia.org LDAP Other Operations open 2017-06-21 2017-06-20
168490 upgrade planet instances to stretch Other Operations open 2017-06-22 2017-06-21
168562 Reimage gerrit2001 as stretch Other Operations open 2017-06-22 2017-09-28
168619 Degraded RAID on lvs3001 Other Operations open 2017-06-23 2017-09-13
168767 Monitor PostgreSQL connection slots Other Operations open 2017-06-28 2017-06-27
168891 rack/setup/install labtestmetal2001.codfw.wmnet Other Operations open 2017-06-27 2017-07-26
168967 Upload shiny-server .deb to our Jessie apt repository Other Operations open 2017-06-28 2017-09-21
169035 bast3002 sdb broken Other Operations open 2017-06-29 2017-07-03
169246 Stress/capacity test new ores* cluster Other Operations open 2017-06-30 2017-10-06
169249 /usr/local/bin/xenon-generate-svgs and flamegraph.pl cronspam Other Operations open 2017-06-30 2017-09-12
169286 labstore1005 A PCIe link training failure error on boot Other Operations open 2017-07-01 2017-06-30
169287 etcd config depends on puppet certs, but puppet doesn't know Other Operations open 2017-07-01 2017-06-30
169290 New anti-stackclash (4.9.25-1~bpo8+3 ) kernel super bad for NFS Other Operations open 2017-07-01 2017-07-06
169318 Use multiple puppetdbs on puppet masters Other Operations open 2017-07-01 2017-06-30
169322 Research whether it makes sense to have OTRS installation in an HA setup Other Operations open 2017-07-01 2017-06-30
169518 Decommission esams ms-fe / ms-be Other Operations open 2017-07-04 2017-08-29
169548 Prepare for Puppet 4 Other Operations open 2017-07-04 2017-10-03
169564 MD RAID: remove mdadm daily check Other Operations open 2017-07-04 2017-07-03
169570 nfs-manage failover script needs to be tested with real load and fixed Other Operations open 2017-07-04 2017-09-14
169680 NFS on dataset1001 overloaded, high load on the hosts that mount it Other Operations open 2017-07-05 2017-10-12
169849 Architecture and puppetize setup for dumpsdata boxes Other Operations open 2017-07-07 2017-07-12
169884 Jobrunners generate mediawiki exceptions upon calling Closure$RecentChange::save Other Operations open 2017-07-07 2017-07-07
169937 Services Q1 2017/18 goal: Begin migrating job queue processing to multi-DC enabled eventbus infrastructure. Other Operations open 2017-07-07 2017-09-06
169969 Regularly purge old ores graphite metrics Other Operations open 2017-07-08 2017-10-03
170108 Operations Q1 goal: Streamlined Service Delivery Other Operations open 2017-07-11 2017-07-12
170150 Evaluate Grafana's LDAP group options and deprecate grafana-admin if possible Other Operations open 2017-07-11 2017-10-02
170152 mc2023 / mc2025 fail to mount root partition within 90 seconds using Linux 4.9 Other Operations open 2017-07-11 2017-07-10
170298 sshd stretch puppet support Other Operations open 2017-07-12 2017-07-12
170353 Icinga: timeseries checks should have the link to a graph with the data Other Operations open 2017-07-12 2017-10-02
170365 move legal-tm-vio alias to OIT Other Operations open 2017-07-13 2017-08-23
170453 FY2017/18 Program 6: Streamlined Service delivery Other Operations open 2017-07-13 2017-09-06
170456 FY2017/18 Program 6 - Outcome 2 - Objective 3: Integrated, container-based development environment Other Operations open 2017-07-13 2017-07-18
170474 Decommisson and store old row D network gear. Other Operations open 2017-07-13 2017-07-20
170480 FY2017/18 Program 6 - Outcome 2: Developers are able to develop and test their applications through a unified pipeline towards production deployment. Other Operations open 2017-07-13 2017-07-18
170481 FY2017/18 Program 6 - Outcome 2 - Objective 2: Set up a continuous integration and deployment pipeline Other Operations open 2017-07-13 2017-10-02
170628 Jessie rsvg/cairo can't render specific SVG file on Commons Other Operations open 2017-07-19 2017-10-12
170640 reports.frdev.wm.o -- still in use? Other Operations open 2017-07-14 2017-07-18
170740 PuppetDB misbehaving on 2017-07-15 Other Operations open 2017-07-16 2017-07-19
170817 Upgrade Thumbor servers to Stretch Other Operations open 2017-07-18 2017-08-23
171048 Eventbus does not handle gracefully changes in DNS recursors Other Operations open 2017-07-20 2017-10-09
171122 librenms: consider using Distributed Poller with multiple netmon servers Other Operations open 2017-07-20 2017-07-19
171157 Monitor internal CA expirations Other Operations open 2017-07-21 2017-08-21
171188 Move the main WMCS puppetmaster into the Labs realm Other Operations open 2017-07-21 2017-07-24
171191 Should puppet auto-restart slapd? Other Operations open 2017-07-21 2017-07-21
171452 Integrate jessie 8.9 point release Other Operations open 2017-07-25 2017-07-24
171473 labvirt1015 crashes Other Operations open 2017-07-25 2017-10-18
171482 Programmatic generation of grafana dashboards Other Operations open 2017-07-25 2017-09-15
171619 ORES should use a git large file plugin for storing serialized binaries Other Operations open 2017-07-27 2017-10-06
171745 nscd does not cache localhost causing high CPU usage when localhost is often resolved Other Operations open 2017-07-27 2017-10-02
171758 Support git-lfs files in gerrit Other Operations open 2017-07-27 2017-10-16
171851 Reimage ores* hosts with Debian Stretch Other Operations open 2017-07-28 2017-09-06
171923 thorium - failed git clone of geowiki-data-private Other Operations open 2017-07-29 2017-08-04
172409 Copper root (/) 95% full Other Operations open 2017-08-04 2017-09-20
172479 Collect error logs from jobchron/jobrunner services in Logstash Other Operations open 2017-08-05 2017-10-13
172538 rack/setup/install labvirt10(19|20).eqiad.wmnet Other Operations open 2017-08-05 2017-09-14
172628 conf2002 etcdmirror-conftool-eqiad-wmnet died Other Operations open 2017-08-07 2017-08-06
172815 Improve stability and maintainability of our browser-based PDF render service Other Operations open 2017-08-09 2017-10-13
173056 Import Wiki Loves Monuments photos from Flickr to Commons Other Operations open 2017-08-11 2017-10-10
173374 Deleting file on Commons "Error deleting file: An unknown error occurred in storage backend "local-multiwrite"." Other Operations open 2017-08-16 2017-10-19
173436 Delete graphite metrics for old CFs Other Operations open 2017-08-17 2017-10-05
173492 Tune Varnishkafka delivery errors to be more sensitive Other Operations open 2017-08-18 2017-10-02
173710 Job queue is increasing non-stop Other Operations open 2017-08-22 2017-10-22
173721 Track down the source of periodic increases in requests to swift eqiad Other Operations open 2017-08-22 2017-08-23
174172 unused grafana-dashboard indices on elasticsearch / logstash Other Operations open 2017-08-26 2017-08-25
174269 Two cases of local-multiwrite storage backend failure Other Operations open 2017-08-28 2017-08-27
174402 Review and fix file handle management in worker and celery processes Other Operations open 2017-08-29 2017-10-17
174431 Migration of mw* servers to stretch Other Operations open 2017-08-30 2017-10-17
174449 tin has a failing hdd Other Operations open 2017-08-30 2017-09-07
174465 Puppet admin module should support adding system users to managed groups Other Operations open 2017-08-30 2017-09-20
174475 update firmware on scs consoles Other Operations open 2017-08-30 2017-09-11
174720 letsencrypt::cert::integrated and non-http servers Other Operations open 2017-09-01 2017-10-17
174916 electron/pdfrender hangs Other Operations open 2017-09-05 2017-10-13
174959 swift-recon-cron on ms-be203[34]: [Errno 17] File exists: '/var/lock/swift-recon-object-cron' Other Operations open 2017-09-05 2017-09-05
175150 Decommission stat1003.eqiad.wmnet Other Operations open 2017-09-07 2017-09-14
175206 2017/18 Annual Plan Program 8: Multi-datacenter support Other Operations open 2017-09-07 2017-10-19
175210 Select candidate jobs for transferring to the new infrastucture Other Operations open 2017-09-07 2017-09-28
175213 2017/18 Annual Plan Program 8: Multi-datacenter support, Q2 goals Other Operations open 2017-09-07 2017-10-19
175242 all log producers need to use the logstash LVS endpoint Other Operations open 2017-09-08 2017-09-20
175288 setup/install/deploy deploy1001 as deployment server Other Operations open 2017-09-08 2017-09-08
175361 Upgrade mx1001/mx2001 to stretch Other Operations open 2017-09-09 2017-09-20
175362 Split MXes into inbound and outbound Other Operations open 2017-09-09 2017-09-20
175625 scs-c1-eqiad unresponsive Other Operations open 2017-09-12 2017-09-25
175637 End of September milestone: Migrate first production use case Other Operations open 2017-09-12 2017-09-26
175710 Add profiling for Varnish and VCL Other Operations open 2017-09-13 2017-09-12
175798 Port non-deprecated Diamond collectors to Prometheus Other Operations open 2017-09-14 2017-10-16
175799 port elasticsearch diamond collector to prometheus Other Operations open 2017-09-14 2017-09-13
175830 decommission logstash100[1-3] Other Operations open 2017-09-14 2017-10-09
175850 Spike: Enumerate remaining unported stats Other Operations open 2017-09-14 2017-09-15
175876 document all scs connections Other Operations open 2017-09-14 2017-09-18
175885 Toolforge's static webserver broken by Puppet changes and stale nginx packages Other Operations open 2017-09-15 2017-09-14
176153 Create affcom-staff email account Other Operations open 2017-09-26 2017-09-25
176335 logs sent to logstash are lost when the elasticsearch cirrus cluster is unavailable Other Operations open 2017-09-21 2017-10-19
176364 Request access to logstash (nda group) for @framawiki Other Operations open 2017-09-23 2017-10-16
176370 Migrate to PHP 7 in WMF production Other Operations open 2017-09-21 2017-10-17
176437 puppet ca_server confusion Other Operations open 2017-09-22 2017-09-21
176445 Systematically test load speeds of Watchlist and Recent Changes Other Operations open 2017-09-22 2017-10-12
176666 Qualtrics email-LDAP issue Other Operations open 2017-09-26 2017-09-25
176774 Reimage cobalt as stretch Other Operations open 2017-09-27 2017-09-26
176957 Deprecate host copper.eqiad.wmnet Other Operations open 2017-09-29 2017-09-28
177055 Update docker image docker-registry.wikimedia.org/wikimedia-jessie Other Operations open 2017-09-30 2017-10-20
177099 Large number of "A page you created was linked on Wikidata" emails to one recipient in short period of time Other Operations open 2017-09-30 2017-09-29
177195 Reduce technical debt in metrics monitoring Other Operations open 2017-10-03 2017-10-21
177196 Port non-deprecated Diamond collectors to Prometheus Other Operations open 2017-10-03 2017-10-19
177197 Export Prometheus-compatible JVM metrics from JVMs in production Other Operations open 2017-10-03 2017-10-04
177208 Provide dedicated database resources for wikidata Other Operations open 2017-10-03 2017-10-17
177225 Uninstall ganglia from the fleet Other Operations open 2017-10-03 2017-10-21
177253 Upgrade puppetDB to version 3.2 or newer Other Operations open 2017-10-03 2017-10-03
177254 Upgrade to puppet 4 (4.8 or newer) Other Operations open 2017-10-03 2017-10-20
177276 Unify production and CI docker image build process Other Operations open 2017-10-05 2017-10-13
177371 Phase out DSA keys for SSH access (ssh-dss) Other Operati