TechnicalOperations Open Tasks

Help.

107610 Setup separate logical External Store for Flow in production DBA 2015-10-02 2016-12-16
118434 Reduce Flow DB queries on Special:Contributions DBA 2015-11-13 2016-04-19
119154 Move echo tables from local wiki databases onto extension1 cluster for mediawikiwiki, metawiki, and officewiki DBA 2015-11-21 2017-02-09
127792 Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace DBA 2016-02-24 2017-08-10
121161 CentralAuthCreateLocalAccountJob failing on meta due to Echo deadlocks DBA 2016-07-14 2017-01-10
149936 Drop flow_subscription table DBA 2016-11-04 2017-05-25
152385 66 rows from external storage (dewiki) gave duplicate key errors on master failover DBA 2017-01-02 2016-12-05
140609 Add pp_propname/pp_value index to Labs replica DBA 2017-01-02 2017-10-17
150547 Add visitingwatchers to watchlist_count DBA 2017-01-02 2016-11-12
125991 Add wl_timestamp to the watchlist table DBA 2017-01-02 2017-05-11
124214 Allow filtering based on tag on Special:NewFiles DBA 2017-01-02 2017-06-27
136335 Allow self-serve database credential and permissions management for Tool Labs projects DBA 2017-01-02 2017-06-08
134809 Apache <=> mariadb SSL/TLS for cross-datacenter writes DBA 2017-01-02 2017-09-28
131065 API: Contributions: Database query error DBA 2017-01-02 2017-05-08
59176 ApiQueryExtLinksUsage::run query has crazy limit DBA 2017-01-02 2017-04-27
72005 Apply enum changes to (img|oi|fa)_major_mime on production DBA 2017-01-02 2017-04-28
133251 archive/archive_userindex is not filled in eswiki_p DBA 2017-01-02 2016-04-21
109964 Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections. DBA 2017-01-02 2015-09-29
133333 Audit MySQL configurations DBA 2017-01-02 2016-11-10
104459 Automatize the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves DBA 2017-01-02 2017-09-14
56401 Backups systems for userDBs hosted on replicas DBA 2017-01-02 2017-01-19
112473 Better mysql monitoring for number of connections and processlist strange patterns DBA 2017-01-02 2017-08-04
59186 blob_tracking indexes apparently unused DBA 2017-01-02 2016-07-20
149077 Certain ApiQueryRecentChanges::run api query is too slow, slowing down dewiki DBA 2017-01-02 2017-06-16
119601 Certain tools users create multiple long running queries that take all memory from labsdb hosts, slowing it down and potentially crashing (tracking) DBA 2017-01-02 2017-06-20
115148 Consider dropping fr_page_qal_rev/fr_page_rev indexes DBA 2017-01-02 2015-10-09
152427 Create a check/calendar alert for MariaDB TLS certs DBA 2017-01-02 2017-02-21
153440 Create a full backup of all external storage records that would be easy to restore/setup a temporary delayed slave DBA 2017-01-02 2016-12-16
140751 Create a logstash input filter to preprocess mysqld syslog messages DBA 2017-01-02 2016-08-05
119479 Create a script that'll allow easier swatting of queries that're overloading labsdb DBA 2017-01-02 2015-11-24
145072 Create a script to regenerate prometheus mysqld exporter listing that works with puppetdb DBA 2017-01-02 2017-07-10
151999 Create script to monitor db dumps for backups are successful (and if not, old backups are not deleted) DBA 2017-01-02 2017-04-12
48047 Database error on es.wikipedia.org in function "Revision::insertOn": "Duplicate entry for key 'PRIMARY'" DBA 2017-01-02 2015-12-05
133185 Database error while saving an article (1205 Lock wait exceeded in Title::invalidateCache) DBA 2017-01-02 2016-04-25
123557 Database query error (internal_api_error_DBQueryError) while getting list=allrevisions DBA 2017-01-02 2017-02-09
3268 Database replication lag issues (tracking) DBA 2017-01-02 2017-10-09
50930 Database replication problems - production and labs (tracking) DBA 2017-01-02 2017-03-23
18660 Database table cleanup (tracking) DBA 2017-01-02 2017-10-09
54921 Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) DBA 2017-01-02 2017-09-01
71182 Database upgrade MariaDB 10: Metadata access in INFORMATION_SCHEMA causes complete blocks DBA 2017-01-02 2015-09-23
148078 Decommission db1015, db1035, db1044 and db1038 DBA 2017-01-02 2017-09-28
134476 Decommission old coredb machines (<=db1050) DBA 2017-01-02 2017-10-13
150850 Decouple roles from mariadb.pp into their own file DBA 2017-01-02 2017-02-09
131956 Disabling general.confirmeduser from dbreports for using up too much db resources DBA 2017-01-02 2016-12-04
71127 Discrepancies with logging table on different wikis DBA 2017-01-02 2017-08-07
151681 DispatchChanges: Avoid long-lasting connections to the master DB DBA 2017-01-02 2017-07-21
141968 Display lag on grafana (prometheus) and dbtree from pt-heartbeat instead (or in addition) of Seconds_Behind_Master DBA 2017-01-02 2016-08-25
153638 Drop echo tables from local wiki databases DBA 2017-01-02 2017-07-12
144010 Drop eu_touched in production DBA 2017-01-02 2017-10-05
114117 Drop externallinks.el_from_namespace on wmf databases DBA 2017-01-02 2017-04-28
51195 Drop fj_new_sha1 field DBA 2017-01-02 2017-04-26
54932 Drop *_old database tables from Wikimedia wikis DBA 2017-01-02 2016-12-30
118764 Drop old devwikiinternal and rel13testwiki from s3 DBA 2017-01-02 2015-12-05
60674 Drop page.page_restrictions column from Wikimedia wikis DBA 2017-01-02 2017-04-26
86338 Dropping page.page_counter on wmf databases DBA 2017-01-02 2017-04-26
86342 Dropping page.page_no_title_convert on wmf databases DBA 2017-01-02 2017-04-26
67448 Dropping rc_cur_time on wmf databases DBA 2017-01-02 2017-04-26
51191 Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases DBA 2017-01-02 2017-04-28
86339 Dropping site_stats.ss_total_views on wmf databases DBA 2017-01-02 2017-04-26
85757 Dropping user.user_options on wmf databases DBA 2017-01-02 2017-04-26
118859 Drop pre_ tables DBA 2017-01-02 2017-05-31
106854 Duplicate key errors (tracking) DBA 2017-01-02 2016-10-12
2323 edits where the rev_user_text and ar_user_text fields contain underscores, initial lower-case letters or consecutive spaces, which could occur in the Phase I and II software, are inaccessible using Special:Contributions DBA 2017-01-02 2015-12-05
119626 Eliminate SPOF at the main database infrastructure DBA 2017-01-02 2017-05-23
108255 Enable MariaDB/MySQL's Strict Mode DBA 2017-01-02 2017-07-21
119371 Enable MariaDB/MySQL strict mode on CI slaves DBA 2017-01-02 2017-05-08
135414 Entries in revision table where rev_len is NULL DBA 2017-01-02 2017-01-10
121306 Error: 2013 Lost connection to MySQL server during query on IndexPager::buildQueryInfo (LogPager) DBA 2017-01-02 2017-05-08
141765 Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) DBA 2017-01-02 2017-05-08
141095 Explore 'Analyze' statement as substitute for Explain DBA 2017-01-02 2016-12-04
115303 Expunge old AFTv5 log entries on WMF wikis DBA 2017-01-02 2016-08-12
104699 Firewall configurations for database hosts DBA 2017-01-02 2016-11-10
33863 Fix use of DB schema so RenameUser is trivial DBA 2017-01-02 2017-09-12
126251 Get HA db support for labs internal services DBA 2017-01-02 2017-02-14
39591 History of a page was lost after several move actions on zhwp DBA 2017-01-02 2016-12-29
151491 Icinga MariaDB disk space check on silver checks the wrong partition DBA 2017-01-02 2016-11-24
121857 Implement a system to automatically deploy schema changes without needing DBA intervention DBA 2017-01-02 2016-11-10
138562 Improve regular production database backups handling DBA 2017-01-02 2017-07-03
57385 Investigate dropping "edit_page_tracking" database table from Wikimedia wikis after archiving it DBA 2017-01-02 2016-07-07
140832 Investigate moving labsdb (replicas) user credential management to 'Striker' (codename) DBA 2017-01-02 2017-08-22
116793 Investigate slow query logging/digest for Beta Cluster DBA 2017-01-02 2016-08-05
140788 Labs databases rearchitecture (tracking) DBA 2017-01-02 2017-09-05
153058 LabsDB infrastructure pending work DBA 2017-01-02 2017-08-11
129830 large s2 database traffic DBA 2017-01-02 2016-03-18
108434 latin1 encoding in sql databases DBA 2017-01-02 2015-08-14
71222 list=logevents slow for users with last log action long time ago DBA 2017-01-02 2016-05-23
85266 Look into Maria 10 parallel-replication DBA 2017-01-02 2017-03-02
89737 Make several mediawiki table fields unsigned ints on wmf databases DBA 2017-01-02 2017-09-07
59617 Make watchlist table available as curated foo_p.watchlist_count on labsdb DBA 2017-01-02 2017-09-29
127636 MariaDB missing logrotate for error and slow logs DBA 2017-01-02 2016-06-08
133906 memcached on db1011/tendril not puppetised DBA 2017-01-02 2016-04-29
150438 Meta ticket: Deploy InnoDB compression where possible DBA 2017-01-02 2017-07-03
128152 Migrate all old DB rows from windows-1252 to UTF-8 on dawiki DBA 2017-01-02 2017-10-07
128155 Migrate all old DB rows from windows-1252 to UTF-8 on dawiktionary DBA 2017-01-02 2017-10-07
128151 Migrate all old DB rows from windows-1252 to UTF-8 on enwiki DBA 2017-01-02 2017-10-07
128154 Migrate all old DB rows from windows-1252 to UTF-8 on nlwiki DBA 2017-01-02 2017-10-07
128153 Migrate all old DB rows from windows-1252 to UTF-8 on svwiki DBA 2017-01-02 2017-10-07
128156 Migrate all old DB rows from windows-1252 to UTF-8 on svwiktionary DBA 2017-01-02 2017-10-07
142807 Migrate all users to new Wiki Replica cluster and decommission old hardware DBA 2017-01-02 2017-10-18
109179 Migrate MySQLs to use ROW-based replication DBA 2017-01-02 2017-06-20
114904 Migrate wb_items_per_site to using prefixed entity IDs instead of numeric IDs DBA 2017-01-02 2017-10-18
114903 Migrate wb_terms to using prefixed entity IDs instead of numeric IDs DBA 2017-01-02 2017-07-24
132838 Missing / Dropped databases? DBA 2017-01-02 2017-07-06
133705 Multiple concurrent long running queries from s51434 overloading labsdb1003/labsdb1009 DBA 2017-01-02 2017-10-22
112282 Multiple pages with no revisions DBA 2017-01-02 2016-09-23
127363 mysqlbinlog doesn't recognize the ssl-* options in [client] DBA 2017-01-02 2017-01-06
138517 mysqldump is timing out preventing all tables from being included in the dump DBA 2017-01-02 2017-04-20
105728 MySQL field aliases in select() do not use any quoting DBA 2017-01-02 2015-08-14
143896 MySQL metrics monitoring DBA 2017-01-02 2017-10-09
100501 mysql user and group should be a system user/group DBA 2017-01-02 2017-06-22
138915 OTRS database is "too large" DBA 2017-01-02 2017-09-18
98110 Pagelinks table contains a row having pl_from = 0 DBA 2017-01-02 2015-10-13
115081 Pages where page.page_latest = 0 on the English Wikipedia DBA 2017-01-02 2015-12-05
153182 Perform schema change to add externallinks.el_index_60 to all wikis DBA 2017-01-02 2017-06-06
140302 plwiki API request is excessively slow when including a badrevid DBA 2017-01-02 2016-08-24
126252 Populate the wikishared db on all dbstores DBA 2017-01-02 2016-02-09
110115 Possible to run writes (e.g. UPDATE) on Beta Cluster replica DBA 2017-01-02 2016-10-15
135851 Preserve InnoDB table auto_increment on restart DBA 2017-01-02 2017-05-08
108968 prop=revisions with rvdir=newer gives internal_api_error_DBQueryError for page with many version DBA 2017-01-02 2017-02-06
57455 Provide dynamic report of differences between replica databases and production databases DBA 2017-01-02 2016-11-15
114476 Puppetize grants for mysql analytics servers DBA 2017-01-02 2017-02-20
111929 Puppetize grants for mysql backups on dbstore hosts DBA 2017-01-02 2017-04-12
148955 Puppetize tendril web user creation DBA 2017-01-02 2016-10-24
128623 Purge all Schema:Echo data after 90 days DBA 2017-01-02 2016-03-07
71088 Queries of commonswiki_p.filearchive for fa_sha1 are slow DBA 2017-01-02 2017-10-16
131266 Querying the logging table on labs is slow DBA 2017-01-02 2016-12-04
150548 Reduce watchlist_count threshold DBA 2017-01-02 2017-02-22
92739 Remove AFT tables from the analytics slaves DBA 2017-01-02 2016-12-14
114902 Remove numeric entity IDs from database schema DBA 2017-01-02 2016-08-13
127570 Rename be_x_oldwiki database to be_taraskwiki DBA 2017-01-02 2017-10-14
51593 Rename two indexes in the Echo extension DBA 2017-01-02 2017-04-26
119847 Replicate ContentTranslation databases on Labs DBA 2017-01-02 2017-03-03
126218 Replicate wikitech wikis to analytics-store.eqiad.wmnet DBA 2017-01-02 2016-02-11
149643 Review Icinga alarms with disabled notifications DBA 2017-01-02 2016-11-02
101631 rev_len should be available also for deleted revisions in database replicas DBA 2017-01-02 2017-10-16
133523 [RFC] improve parsercache replication and sharding handling DBA 2017-01-02 2017-07-05
83609 script & docs to rename wiki databases DBA 2017-01-02 2017-10-14
29112 Select of revisions for stub history files does not explicitly order revisions DBA 2017-01-02 2016-07-18
141255 Separate host lookup from the sql shell script DBA 2017-01-02 2016-07-27
150949 Set barracuda InnoDB file format as the default configuration everywhere DBA 2017-01-02 2017-05-05
141547 Setup automatic failover for misc database servers DBA 2017-01-02 2016-08-17
108850 Set up auto-purging after 90 days {tick} DBA 2017-01-02 2017-02-01
79922 Set up backup strategy for es clusters DBA 2017-01-02 2017-04-12
50694 Show replication lags in Graphite DBA 2017-01-02 2016-03-25
129093 SHOW SLAVE STATUS as a health check should have a low timeout DBA 2017-01-02 2016-03-08
101502 Slow API list=recentchanges query (DBError: Lost connection to MySQL server during query) DBA 2017-01-02 2016-12-20
143870 Some dump hosts are accessing main traffic servers DBA 2017-01-02 2016-08-31
17441 Some tables lack unique or primary keys, may allow confusing duplicate data DBA 2017-01-02 2017-09-05
108782 Special:Contributions: "contributions page filtered for namespace or RevisionDeleted edits" error DBA 2017-01-02 2016-07-11
146821 Spike: Look into transaction isolation level and other tricks for easing db contention DBA 2017-01-02 2016-10-04
144382 Spiky write pattern on core db masters DBA 2017-01-02 2016-09-08
68025 [Story] Monitor size of some Wikidata database tables DBA 2017-01-02 2016-05-11
97414 [Task] Use an own load section for Wikibase DB queries DBA 2017-01-02 2016-02-05
137654 Tendril DB graphs say 24hr but at 12hr DBA 2017-01-02 2016-06-14
121094 Throttling linkwatcher tool user as it is consuming 100% CPU DBA 2017-01-02 2017-04-19
51190 Truncate SHA-1 indexes DBA 2017-01-02 2017-04-26
107676 Undiscoverable log entries DBA 2017-01-02 2015-11-05
105395 UserMerge using SELECT img_name FROM image WHERE img_user = ... but img_user doesn't have an index DBA 2017-01-02 2015-07-09
151583 Use tls for dump backup generation DBA 2017-01-02 2017-04-12
95799 wfWaitForSlaves in JobRunner can massively slow down run rate if just a single slave is lagged DBA 2017-01-02 2017-09-13
147148 Wikipedia requires a patch to load its data from the dumps with mwdumper DBA 2017-01-02 2017-05-08
106151 write irc bot to report high replag of s{1,2,3}.labsdb on #wikimedia-labsdb DBA 2017-01-02 2016-06-22
154490 Delete prefstats tables DBA 2017-01-04 2017-01-03
155734 Investigation: Add old and new length columns to cu_changes DBA 2017-01-24 2017-03-14
156465 For switchovers: A way to check if slaves are up to date DBA 2017-01-28 2017-01-27
156462 Framework to transfer files over the LAN DBA 2017-01-28 2017-01-27
156461 [META ticket] Automation for our DBs tracking task DBA 2017-01-28 2017-01-27
156844 Prep to decommission old dbstore hosts (db1046, db1047) DBA 2017-02-01 2017-10-19
157359 labsdb1006/1007 (postgresql) maintenance DBA 2017-02-07 2017-06-22
85756 Make (redacted) log_search table available on ToolLabs DBA 2017-02-09 2017-02-08
157668 Automate dataset backup recovery DBA 2017-02-10 2017-04-12
157702 Followup for TLS MariaDB server roll-out DBA 2017-02-10 2017-02-21
48098 Disable miser mode ($wgMiserMode) on small wikis (wikis in small.dblist) DBA 2017-02-14 2017-03-27
158166 Discuss labsdb visibility of rev_text_id and ar_comment DBA 2017-02-16 2017-03-13
68961 Logging needs an index to optimize searching by log_title DBA 2017-02-16 2017-08-07
158724 Increase size of categorylinks.cl_collation column DBA 2017-02-23 2017-04-10
159430 convert dbstore1001 to InnoDB compressed by importing db shards to it DBA 2017-03-03 2017-04-05
159423 Meta ticket: Migrate multi-source database hosts to multi-instance DBA 2017-03-03 2017-07-03
151717 Usage tracking: record which statement group is used DBA 2017-03-08 2017-10-20
160062 Allow setting the watchlist table to read-only on a per-wiki basis DBA 2017-03-10 2017-10-17
146055 Improve privilege separation for phabricator's config files and mysql credentials DBA 2017-03-16 2017-03-28
160731 Decom db1048 (BBU Faulty - slave lagging) DBA 2017-03-18 2017-09-13
160983 Optimize SpecialAllPages::showChunk for large wikis DBA 2017-03-22 2017-10-19
161088 Migrate some s4 hosts to file per table DBA 2017-03-23 2017-07-14
161294 run pt-tablechecksum on s5 DBA 2017-03-25 2017-10-05
89741 Expose ar_content_format and ar_content_model columns of archive table on Labs replicas DBA 2017-03-27 2017-08-23
161754 eqiad: (2) hardware access request for labsdb1004 & 5 refresh DBA 2017-03-30 2017-04-07
161755 eqiad: (2) hardware access request for labsdb1006 & 7 refresh DBA 2017-03-30 2017-04-07
162070 Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases DBA 2017-04-04 2017-04-03
162562 Choose a field length for entity id columns and stick to it DBA 2017-04-10 2017-05-05
162558 Use memcached (or something similar) to keep the latest chd_seen state, only flush to table every once in a while DBA 2017-04-10 2017-05-05
162699 Decomissions old s2 eqiad hosts (db1018, db1021, db1024, db1036) DBA 2017-04-12 2017-09-20
162789 Create less overhead on bacula jobs when dumping production databases DBA 2017-04-13 2017-10-04
162807 Run pt-table-checksum on s1 (enwiki) DBA 2017-04-13 2017-07-17
163190 Run pt-table-checksum on s7 DBA 2017-04-19 2017-10-05
163339 pdu phase inbalances: ps1-a3-codfw, ps1-c6-codfw, & ps1-d6-codfw DBA 2017-04-20 2017-05-08
164167 change_tag table needs redesign DBA 2017-05-01 2017-06-07
161066 Merge pagelinks, templatelinks and imagelinks tables DBA 2017-05-02 2017-05-24
164382 Evaluate the need for FORCE INDEX (ls_field_val) [now IGNORE INDEX (ls_log_id)], delete the index hint if not needed anymore DBA 2017-05-04 2017-05-08
164488 Run pt-table-checksum on s3 DBA 2017-05-05 2017-10-16
117272 Wikidata: Possibility to add comments with more than 255 bytes, potential db issue DBA 2017-05-05 2017-05-04
164504 Cleanup x1 database connection patterns DBA 2017-05-09 2017-05-08
164805 Investigate requirements for MySQL access in RESTBase DBA 2017-05-09 2017-06-15
164834 In some database hosts, performance schema loses digest statistics DBA 2017-05-10 2017-10-04
165358 Set up and package wmfmariadbpy helper scripts so they can easily be deployed to all database server and client hosts DBA 2017-05-16 2017-05-15
165625 Evaluate future of wmf puppet module "mysql" DBA 2017-05-18 2017-09-12
165677 Create a backend check for pybal to monitor the MySQL protocol being up DBA 2017-05-19 2017-05-25
165674 Investigate slow servermon updating queries on db1016 DBA 2017-05-19 2017-06-14
165756 Create summary templates on Wikitech wiki to stop writing the same things everywhere, everytime DBA 2017-05-20 2017-09-12
166108 x1 master db1031: Faulty BBU DBA 2017-05-24 2017-06-27
166344 db1016 m1 master: Possibly faulty BBU DBA 2017-05-26 2017-10-05
166798 Decide whether back-compat views for upcoming major schema changes will be provided in the Labs replicas DBA 2017-06-02 2017-09-07
167973 Move wikitech and labstestwiki to s3 DBA 2017-06-16 2017-07-25
168107 Review new Echo table for user group expiration DBA 2017-06-17 2017-07-20
168349 enwiki_p logging vs logging_userindex returning dramatically different results DBA 2017-06-21 2017-09-12
168414 Purge all old data from EventLogging master DBA 2017-06-21 2017-10-02
169501 Move some masters away from B6 DBA 2017-07-04 2017-10-09
143366 Removal of {{#coordinates:}} leaves DB entries behind DBA 2017-07-13 2017-07-13
170662 Productionize 22 new codfw database servers DBA 2017-07-15 2017-10-17
171071 Perform testing for TLS effect on connection rate DBA 2017-07-20 2017-09-12
170508 The "show ip" action should also provide a distinct list of user-agents for each IP DBA 2017-07-20 2017-09-12
171643 Clean up skin properties DBA 2017-07-26 2017-07-31
171570 Rename database labswiki to wikitech DBA 2017-07-26 2017-10-14
171898 Deleting large watchlist takes > 4 seconds causing rollback due to write time limit DBA 2017-07-29 2017-09-10
172207 flaggedrevs.fr_user is unindexed DBA 2017-08-02 2017-09-25
163681 Use a random timestamp on Special:NewFiles on Commons doesn't loads the page (504 Gateway Timeout) DBA 2017-08-02 2017-08-01
172497 Change pt-heartbeat model to not use super-user, avoid SPOF and switch automatically to the real master without puppet dependency DBA 2017-08-05 2017-08-07
172492 Improve database alerting (tracking) DBA 2017-08-05 2017-10-09
172489 Monitor read_only variable and/or uptime on database masters, make it page DBA 2017-08-05 2017-10-09
172490 Monitor swap/memory usage on databases DBA 2017-08-05 2017-08-07
172498 Switch databases to the future parser DBA 2017-08-05 2017-10-16
173511 Implement technical details and process for "datasets_p" on wikireplica hosts DBA 2017-08-18 2017-10-03
173570 Decommission db1015 DBA 2017-08-19 2017-10-18
173647 Prepare and check storage layer for hif.wiktionary DBA 2017-08-21 2017-09-14
173915 Decommission db1041 DBA 2017-08-24 2017-10-18
174076 Decommission db1033 and db1028 DBA 2017-08-25 2017-10-18
174044 Deploy MCR storage layer DBA 2017-08-25 2017-10-13
174370 Create elections committee private wiki DBA 2017-08-29 2017-10-09
174509 Drop now redundant indexes from pagelinks and templatelinks DBA 2017-08-30 2017-10-17
174569 Schema change for refactored comment storage DBA 2017-08-31 2017-10-19
174648 Create MW Schema Diff maintenance script DBA 2017-09-01 2017-09-12
113842 `pr_index`to be replicated to Labs public databases DBA 2017-09-01 2017-10-06
174763 Decommission db1026 DBA 2017-09-02 2017-10-18
174806 Decommission db1045 DBA 2017-09-02 2017-10-18
174802 Drop education program (ep_*) tables on wikis where it is not enabled DBA 2017-09-02 2017-09-01
174801 Drop flaggedrevs tables on wikis where it is not enabled DBA 2017-09-02 2017-09-01
174804 Truncate `updatelog` on all wikis DBA 2017-09-02 2017-09-01
174902 Decommission db1037 DBA 2017-09-05 2017-10-18
175096 Identify tools hosting databases on labsdb100[13] and notify maintainers DBA 2017-09-06 2017-10-12
175264 Decommission db1049 DBA 2017-09-08 2017-10-18
175679 Decommission db1048 (was Move m3 slave to db1059) DBA 2017-09-13 2017-10-18
175685 Decommission db2010 and move m1 codfw to db2078 DBA 2017-09-13 2017-10-10
175672 Make apache/maintenance hosts TLS connections to mariadb work DBA 2017-09-13 2017-10-17
176043 Prepare and check storage layer for amwikimedia (including dropping s7 version of the wiki) DBA 2017-09-17 2017-10-16
176243 Decommission database hosts < db2030 (tracking) DBA 2017-09-20 2017-09-25
176215 decommission db1018 DBA 2017-09-20 2017-10-16
176277 Provision a separate DB shard for wbc_entity_usage DBA 2017-09-20 2017-09-25
176311 decommission db1036 DBA 2017-09-21 2017-10-04
176532 Gerrit is failing to connect to db on gerrit2001 thus preventing systemd from working DBA 2017-09-24 2017-10-15
176754 Regularly purge expired temporary userrights from DB tables DBA 2017-09-27 2017-10-11
176931 Decommission db1035 DBA 2017-09-29 2017-10-11
177223 Determine schema differences between labsdb1001 and labsdb1009 DBA 2017-10-03 2017-10-18
177274 Provide a new sanitarium host for s8 DBA 2017-10-04 2017-10-12
177601 Deploy dropping wb_entity_per_page table DBA 2017-10-07 2017-10-20
177779 Generate instance list of database hosts to be monitored automatically from exported resources DBA 2017-10-10 2017-10-10
177778 Improve database application performance monitoring visibility DBA 2017-10-10 2017-10-13
177782 Reduce false positives on database pages DBA 2017-10-10 2017-10-13
177911 Decommission db1038 DBA 2017-10-12 2017-10-11
178128 Access to raw database tables on labsdb* for wmcs-admin users DBA 2017-10-13 2017-10-18
178162 Decommission db1050 DBA 2017-10-14 2017-10-18
178290 Check recentchanges table and query errors on wikis other than commonswiki and ruwiki DBA 2017-10-17 2017-10-16
178383 db1101 crashed - memory errors DBA 2017-10-18 2017-10-18
178359 Support multi-instance on core hosts DBA 2017-10-18 2017-10-19
178460 db1082 storage crashed DBA 2017-10-19 2017-10-18
178515 Enable description usage tracking on further wikis DBA 2017-10-19 2017-10-20
178661 Drop wb_entity_per_page views in Wiki Replicas DBA 2017-10-21 2017-10-22
178752 gblrename log_type missing on replicas DBA 2017-10-23 2017-10-22
130883 decom cp3011-22 (12 machines) Hardware Requests 2017-01-02 2017-10-03
95742 Decomission amssq31-62 (32 hosts) Hardware Requests 2017-01-02 2017-08-29
139775 eqiad: add all spare network switches to hardware spares tracking Hardware Requests 2017-01-02 2017-04-27
139752 Find and rack 2 EX4200s in rack c1-eqiad Hardware Requests 2017-01-02 2017-08-29
128821 reclaim and return all cisco servers Hardware Requests 2017-01-02 2016-09-23
159480 Decommission bast3001 Hardware Requests 2017-03-03 2017-08-29
159996 decom fluorine Hardware Requests 2017-03-09 2017-07-20
160986 Decommission ms-fe100[1-4] Hardware Requests 2017-03-22 2017-09-20
166341 SSDs for main Kafka clusters Hardware Requests 2017-05-26 2017-05-29
166489 Decommission ms-be1001 - ms-be1012 Hardware Requests 2017-05-30 2017-09-19
167376 Decommission cp300[3456] Hardware Requests 2017-06-09 2017-08-29
167377 Decommission cp4011, cp4012, cp4019, cp4020 Hardware Requests 2017-06-09 2017-08-01
168559 decom silver (was silver has trouble rebooting) Hardware Requests 2017-06-22 2017-08-11
169020 Decommission cp400[1-4] Hardware Requests 2017-06-28 2017-08-04
170157 decommission rcs100[12] Hardware Requests 2017-07-11 2017-08-03
170441 Decommission mw1196 Hardware Requests 2017-07-13 2017-07-20
172323 Decommission WMF3248 (old R510) Hardware Requests 2017-08-04 2017-09-19
172487 decom iridium Hardware Requests 2017-08-05 2017-08-17
173097 Decommission stat1002.eqiad.wmnet Hardware Requests 2017-08-12 2017-09-28
175089 decommission mw1163 Hardware Requests 2017-09-06 2017-09-05
175093 Decommission osmium.eqiad.wmnet Hardware Requests 2017-09-06 2017-10-20
175595 decommission wdqs100[12] Hardware Requests 2017-09-12 2017-10-04
177958 Decommission ocg1001-3 Hardware Requests 2017-10-12 2017-10-18
178392 Replacement hardware for cumin masters Hardware Requests 2017-10-18 2017-10-18
98006 Anycast (Auth)DNS Network Operations 2017-01-02 2017-07-24
122406 Consider renumbering Labs to separate address spaces Network Operations 2017-01-02 2017-07-17
82038 create a test for multicast relay Network Operations 2017-01-02 2017-06-27
120425 dumps.wikimedia.org seems to have poor throughput towards some destinations Network Operations 2017-01-02 2017-06-27
133387 Enabling IGMP snooping on QFX switches breaks IPv6 (HTCP purges flood across codfw) Network Operations 2017-01-02 2017-08-03
150264 Icinga check for VRRP Network Operations 2017-01-02 2017-06-27
83992 Juniper monitoring Network Operations 2017-01-02 2017-06-27
83119 Netflow Collector Project Network Operations 2017-01-02 2017-06-27
124843 Peer with SFMIX at ULSFO in 200 Paul Network Operations 2017-01-02 2017-06-27
150256 Re-setup lvs1007-lvs1012, replace lvs1001-lvs1006 Network Operations 2017-01-02 2017-09-25
106056 set up a looking glass for WMF ASes Network Operations 2017-01-02 2017-06-27
86541 setup wifi in codfw Network Operations 2017-01-02 2017-06-27
157435 Review ACLs for the Analytics VLAN Network Operations 2017-02-08 2017-10-09
163674 Frequent RST returned by appservers to LVS hosts Network Operations 2017-04-25 2017-07-12
166171 rack/setup/wire/deploy msw2-c1-eqiad Network Operations 2017-05-24 2017-09-12
167306 ospf link-protection Network Operations 2017-06-08 2017-06-27
167299 Upgrade BIOS/RBSU/etc on lvs1007 Network Operations 2017-06-08 2017-09-26
167691 High amount of unexpected ICMP dest unreachable toward esams cache clusters Network Operations 2017-06-13 2017-06-27
167841 Cleanup confed BGP peerings and policies Network Operations 2017-06-14 2017-06-27
167842 Find a new PIM RP IP Network Operations 2017-06-14 2017-10-20
167840 Merge AS14907 with AS43821 Network Operations 2017-06-14 2017-10-13
169644 eqiad: rack frack refresh equipment Network Operations 2017-07-05 2017-09-11
170144 Evaluate NetBox as a Racktables replacement & IPAM Network Operations 2017-07-11 2017-09-17
170131 Recurring varnish-be fetch failures in codfw Network Operations 2017-07-11 2017-07-12
171032 Investigate lvs IP pages during codfw row C switch upgrade Network Operations 2017-07-20 2017-07-20
171962 bonded/redundant network connections for fundraising hosts Network Operations 2017-07-29 2017-07-28
172459 eqiad row D switch upgrade Network Operations 2017-08-04 2017-08-29
173489 pmacct should be upgraded to 1.6.2 on Stretch Network Operations 2017-08-18 2017-10-17
173698 Backfill librenms data in graphite with historical RRDs Network Operations 2017-08-22 2017-08-21
174596 dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour Network Operations 2017-08-31 2017-08-30
174616 set up cr3-esams Network Operations 2017-08-31 2017-08-30
174637 Setup esams atlas anchor Network Operations 2017-09-01 2017-09-04
176175 connect second ethernet interface for fundraising codfw hosts Network Operations 2017-09-19 2017-09-18
176338 eqiad: networking audit for support contract renewal Network Operations 2017-09-21 2017-09-21
176337 esams: networking audit for support contract renewal Network Operations 2017-09-21 2017-09-20
176427 unrack/decom pfw1-codfw and pfw2-codfw Network Operations 2017-09-22 2017-09-22
176816 cr2-esams temperature warning Network Operations 2017-09-27 2017-09-26
176975 connect second interface for each frack to opposite switch for each eqiad host Network Operations 2017-09-29 2017-10-02
107108 Flow notification links on mobile point to desktop Other Operations 2015-10-02 2017-07-25
116750 2FA for SSH access to the production cluster Other Operations 2017-01-02 2015-12-18
140879 503 error raises again while trying to load a Wikidata page Other Operations 2017-01-02 2017-06-23
139971 access_new_install role vs. Labs vs. the future Other Operations 2017-01-02 2016-07-11
140316 Add granularity limiter (g=) to wikimedia.org DKIM record(s) Other Operations 2017-01-02 2016-08-05
141783 Add monitoring for detecting when logstash services are down Other Operations 2017-01-02 2016-08-01
128715 Add other Tools administrators to the Icinga notification group Other Operations 2017-01-02 2016-12-04
110171 Alert when ES indexes are freezed for more than 30 minutes Other Operations 2017-01-02 2016-12-08
118331 Alert when used_memory gets too high for redis queues Other Operations 2017-01-02 2015-11-11
95801 Allow customizing the alert message from graphite Other Operations 2017-01-02 2016-11-18
132256 Analytics hosts showed high temperature alarms Other Operations 2017-01-02 2017-10-06
153816 apache::static_site is not working Other Operations 2017-01-02 2016-12-22
151702 API cluster failure / OOM Other Operations 2017-01-02 2017-05-16
56515 Apply editing rate limits for all users Other Operations 2017-01-02 2017-01-29
114337 Assign 3 more servers to video scaler duty Other Operations 2017-01-02 2017-02-14
136562 Audit/fix hosts with no RAID configured Other Operations 2017-01-02 2016-08-24
94819 Audit racktables Other Operations 2017-01-02 2016-03-17
141756 audit / test / upgrade hp smartarray P840 firmware Other Operations 2017-01-02 2017-06-06
130709 authoritative copy of 'root' files for upload.wikimedia.org is only in swift Other Operations 2017-01-02 2017-03-29
135991 Automated service restarts for common low-level system services Other Operations 2017-01-02 2016-06-21
118829 Automate the provisioning and management of MediaWiki clusters Other Operations 2017-01-02 2015-11-17
89829 bond eth interfaces on ms1001 Other Operations 2017-01-02 2016-01-05
138496 bring swift eqiad to one zone per row Other Operations 2017-01-02 2016-06-23
148968 Build Kubernetes for production use Other Operations 2017-01-02 2017-06-05
126619 cassandra slow streaming during (de)commission Other Operations 2017-01-02 2017-07-11
128590 Cassandra uses default ip address for outbound packets while bootstrapping Other Operations 2017-01-02 2016-12-22
137176 catch-all apache vhost on the cluster should return 404 for non-existing sites Other Operations 2017-01-02 2016-11-24
113792 Change LDAP cn to something more useful (was Rename "Dzahn" to "Daniel Zahn" in Gerrit) Other Operations 2017-01-02 2017-04-14
152073 Check concurrency/retry/timeout limits and syncronize those between services Other Operations 2017-01-02 2016-12-30
119274 Check incoming requests to secure.wm.o Other Operations 2017-01-02 2015-12-08
98984 Check power supply balance settings on cp3030+ Other Operations 2017-01-02 2016-03-17
137217 Clean up apt:pin of python modules used for Nodepool Other Operations 2017-01-02 2016-11-16
144933 Cleanup debconf handling in mailman puppet setup Other Operations 2017-01-02 2016-11-29
111540 Clean up labs graphite datapoints Other Operations 2017-01-02 2016-11-18
142002 Clean up puppet & configs for ORES Other Operations 2017-01-02 2016-09-22
130617 Collect metrics on pool counter usage Other Operations 2017-01-02 2016-11-03
130209 Collect threaddumps from elasticsearch at regular intervals Other Operations 2017-01-02 2016-03-17
133913 Completely port l10nupdate to scap Other Operations 2017-01-02 2017-09-26
124413 confctl should provide tags information after writing data Other Operations 2017-01-02 2016-01-22
150460 Configure maps cluster to send statsd metrics to the statsd endpoint in the same datacenter Other Operations 2017-01-02 2016-11-10
150875 Confirm attribution needs Other Operations 2017-01-02 2017-06-29
129847 conftool-merge should report which node is setting attributes for Other Operations 2017-01-02 2016-03-14
153068 Consider mounting labs NFS labstore1003.eqiad.wmnet:/scratch for server-side uploads Other Operations 2017-01-02 2016-12-13
132104 Consider moving policy.wikimedia.org away from WordPress.com Other Operations 2017-01-02 2016-06-08
134811 Consider REST with SSL (HyperSwitch/Cassandra) for session storage Other Operations 2017-01-02 2017-07-12
94277 Convert snapshot hosts to use HHVM and trusty Other Operations 2017-01-02 2017-10-18
151275 cp4008 and cp4012 running on single PSU Other Operations 2017-01-02 2017-10-18
105780 Create a doc explaining the SLA between services and the monitoring tool Other Operations 2017-01-02 2017-07-11
138799 Create a simple puppet role for setting up a singlenode kubernetes install Other Operations 2017-01-02 2016-06-27
151310 create-dbusers service failing on labstore1004 Other Operations 2017-01-02 2016-11-22
136735 create endowment.wm.org microsite Other Operations 2017-01-02 2017-10-12
134551 Create functional cluster checks for all services (and have them page!) Other Operations 2017-01-02 2017-08-15
146657 create notifications about user accounts that have not been used for a long time Other Operations 2017-01-02 2016-09-26
152439 cronspam from labtestservices2001 /etc/dns-floating-ip-updater.py > /dev/null Other Operations 2017-01-02 2016-12-05
152724 Current state and next steps for RESTBase storage Other Operations 2017-01-02 2017-07-12
87790 decom amslvs1-4 (dc work) Other Operations 2017-01-02 2017-08-29
94215 decommission cp3001 & cp3002 Other Operations 2017-01-02 2017-08-29
145065 Decrease time required to fully restart the Cirrus elasticsearch clusters Other Operations 2017-01-02 2016-11-30
131966 Default gateway unreachable on baham.wikimedia.org after reboot Other Operations 2017-01-02 2017-07-20
67270 Default license for operations/puppet Other Operations 2017-01-02 2017-02-07
150185 Deploy ElectronPdfService Extension to production Other Operations 2017-01-02 2017-10-19
135124 Deploy etcddump (or another etcd dump & load tool) to production Other Operations 2017-01-02 2016-05-12
150486 Deploy federation for Prometheus Other Operations 2017-01-02 2016-11-15
148693 Deploy IDS rendering engine to production Other Operations 2017-01-02 2017-08-26
149057 Designate seems very slow to delete records? Other Operations 2017-01-02 2016-10-24
140270 Determine a core set or a checklist of permissions for deployment purpose Other Operations 2017-01-02 2017-05-16
118154 determine hardware needs for dumps in eqiad and codfw Other Operations 2017-01-02 2017-03-24
141128 determine/process/document bios firmware tracking/updating policies Other Operations 2017-01-02 2016-07-22
125411 Diamond load averages do not contain scaled versions Other Operations 2017-01-02 2017-07-25
110240 [Discussion] Consider validating JSON schemas when running x-ample tests? Other Operations 2017-01-02 2016-10-12
153416 docker-engine pulled into our repositories only keeps the latest version Other Operations 2017-01-02 2017-01-04
127797 document all puppet classes / defined types!? Other Operations 2017-01-02 2016-11-28
115757 document debian packaging guidelines Other Operations 2017-01-02 2016-05-23
133164 Document eqiad/codfw transition plan for OCG Other Operations 2017-01-02 2016-05-23
102575 document graphite failover/backfill procedures Other Operations 2017-01-02 2016-05-23
135318 Document how to handle 'inconsistent state within the internal storage backends' issues Other Operations 2017-01-02 2016-07-31
101585 document redis upgrade/restart procedures Other Operations 2017-01-02 2016-05-23
116805 DomainKeys Identified Mail (DKIM) for phabricator.wikimedia.org Other Operations 2017-01-02 2016-08-31
92471 enable authenticated access to Cassandra JMX Other Operations 2017-01-02 2016-11-29
94951 Enable the usage of `hhvm -m debug --debug-host ::1` from mw1017 so developers can step through code (think gdb) in production to see what is going wrong. Other Operations 2017-01-02 2016-12-01
142991 Enable "upload by url" feature at zhwiki Other Operations 2017-01-02 2017-06-08
136312 Encrypt syslog traffic Other Operations 2017-01-02 2017-10-10
119719 Enforce a minimum refresh period for grafana dashboards hitting graphite Other Operations 2017-01-02 2016-11-18
142827 Enforce reference to Phabricator task for all commits to modules/admin/data/data.yaml Other Operations 2017-01-02 2017-03-10
142815 Enhance account handling (meta bug) Other Operations 2017-01-02 2017-03-28
109089 EPIC: Cultivating the Elasticsearch garden (operational lessons from 1.7.1 upgrade) Other Operations 2017-01-02 2016-10-20
137616 Epic: cultivating the Maps garden Other Operations 2017-01-02 2016-12-15
67394 [EPIC] Performance testing environment Other Operations 2017-01-02 2016-03-11
150871 [EPIC] (Proposal) Replicate core OCG features and sunset OCG service Other Operations 2017-01-02 2017-10-10
133744 Epic: switch Maps to production status Other Operations 2017-01-02 2016-07-19
125442 es2009 degraded RAID Other Operations 2017-01-02 2016-04-22
124185 Evaluate alternative web interfaces to icinga 1 core Other Operations 2017-01-02 2017-04-18
124991 evaluate possibility for nscd use with useldap Other Operations 2017-01-02 2017-07-24
150811 Evaluate ScyllaDB as a near-term replacement to Cassandra Other Operations 2017-01-02 2017-01-25
141524 eventbus should send statsd in batches Other Operations 2017-01-02 2016-11-18
152632 Explore hosting the multimedia commons use case Other Operations 2017-01-02 2016-12-07
100777 expose hosts in maintenance state so we can prevent scap from running on them Other Operations 2017-01-02 2017-02-10
138821 extend existing graphite whisper files retention to five years Other Operations 2017-01-02 2016-11-18
151045 Extending Yubico 2FA for production use (meta bug) Other Operations 2017-01-02 2016-11-18
147923 Extract metrics from logs Other Operations 2017-01-02 2017-09-20
148061 Feasibility of hosting podcast setup on Wikimedia servers Other Operations 2017-01-02 2016-10-13
148986 Firewall sets not being loaded post-reboot due to a @resolve race on jessie Other Operations 2017-01-02 2017-05-31
84163 Fix CirrusSearch monitoring Other Operations 2017-01-02 2017-01-22
151046 Fully puppetise yubikey-val Other Operations 2017-01-02 2016-11-18
128615 Get rid of Tool Labs home page check from shinken Other Operations 2017-01-02 2016-12-04
118746 Goal: Strengthen Incident monitoring infrastructure Other Operations 2017-01-02 2016-08-13
148843 GPU upgrade for stats machine Other Operations 2017-01-02 2017-08-06
130512 Grafana: Job Queue Health: Panel is displayed incorrectly Other Operations 2017-01-02 2016-03-20
86546 graphite-web logs are not rotated Other Operations 2017-01-02 2016-11-18
134237 Graphoid returns a 400 on MW API time-out Other Operations 2017-01-02 2016-10-12
116063 Hardware Automation Workflow - Overall Tracking Other Operations 2017-01-02 2017-02-28
133656 Have a paging check for Nova API accessible Other Operations 2017-01-02 2016-12-21
130590 Have dedicated master nodes for elasticsearch Other Operations 2017-01-02 2016-11-03
149287 Heating alerts for mw servers in eqiad Other Operations 2017-01-02 2017-05-25
133674 HHVM is leaking memory on the API appservers Other Operations 2017-01-02 2016-10-10
133091 Highest SSTables / read thresholds Other Operations 2017-01-02 2017-07-11
146090 High failure rate of account creation should trigger an alarm / page people Other Operations 2017-01-02 2017-05-20
133318 High levels of PoolCounter errors should trigger alerts Other Operations 2017-01-02 2017-10-13
98831 Honor DNT header for access logs & varnish logs Other Operations 2017-01-02 2017-05-16
148614 Icinga check for Tor Other Operations 2017-01-02 2017-06-19
151048 Icinga monitoring for Yubikey components Other Operations 2017-01-02 2017-09-07
150300 icinga notification if elevated writing to badpass.log Other Operations 2017-01-02 2017-09-07
141038 implement icinga paging for non-ops teams Other Operations 2017-01-02 2016-07-21
120165 Implement role based hiera lookups for labs Other Operations 2017-01-02 2017-08-18
124179 Improve access to and control over incident and metrics monitoring infrastructure Other Operations 2017-01-02 2017-03-08
138017 Improve automation around Maps servers Other Operations 2017-01-02 2016-12-15
133844 Improve Elasticsearch icinga alerting Other Operations 2017-01-02 2017-05-04
116627 Include 5xx numbers in fluorine fatalmonitor Other Operations 2017-01-02 2015-10-28
137939 Increase frequency of OSM replication Other Operations 2017-01-02 2017-06-06
136702 Increase time before alert for elasticsearch disk space issues Other Operations 2017-01-02 2016-11-11
150651 Information missing from racktables Other Operations 2017-01-02 2017-07-13
135125 Install a second etcd cluster in codfw Other Operations 2017-01-02 2016-05-12
116288 Install mailman-api for internal use Other Operations 2017-01-02 2015-10-23
140141 Install mscorefonts on scaling servers for SVG rendering Other Operations 2017-01-02 2017-07-25
151047 Integrate Yubikey into data.yaml Other Operations 2017-01-02 2016-11-18
149617 Integrating MediaWiki (and other services) with dynamic configuration Other Operations 2017-01-02 2017-10-07
129621 "internal_api_error_MWException: [dbf916b7] Exception Caught: Could not acquire lock for" for some uploads (during upload with Pywikibot OAuth) Other Operations 2017-01-02 2016-07-20
150822 Internal PKI for secure communication - Barcelona Ops offsite 2016 Other Operations 2017-01-02 2017-02-22
135385 investigate carbon-c-relay stalls/drops towards graphite2002 Other Operations 2017-01-02 2016-11-18
133093 Investigate idle appservers in codfw Other Operations 2017-01-02 2016-04-19
147905 investigate lead hardware issue Other Operations 2017-01-02 2017-04-28
118812 Investigate mysterious_sysctl settings and figure out what to do with them Other Operations 2017-01-02 2015-11-17
123560 investigate rsync between dcs with encryption Other Operations 2017-01-02 2016-05-30
148478 Investigate seemingly random Gerrit slow-downs Other Operations 2017-01-02 2017-08-17
130593 investigate slapd memory leak Other Operations 2017-01-02 2017-06-30
149885 Investigate Swift as a storage backend for maps tiles Other Operations 2017-01-02 2016-12-09
140075 investigate swift used space spikes since June 2016 Other Operations 2017-01-02 2016-07-12
109090 Investigate the need for master only (non data nodes) in our ES cluster Other Operations 2017-01-02 2016-12-08
95053 ircecho should accept input via unix sockets Other Operations 2017-01-02 2016-07-15
152782 Kibana functionality missing after upgrade: histograms Other Operations 2017-01-02 2017-05-20
120377 labmon1001 graphite instance archiver keeps archiving the same instances Other Operations 2017-01-02 2016-11-18
153279 labnet/ labtestnet2001 - disk space - nova-api.log needs rotation Other Operations 2017-01-02 2016-12-15
117673 labs precise and jessie instance not accessible after provisioning Other Operations 2017-01-02 2016-06-21
137791 libcglib3-java replaces libcglib-java in Jessie Other Operations 2017-01-02 2016-06-14
64987 librsvg misinterpret quoted font family names that contain whitespaces Other Operations 2017-01-02 2017-05-21
146664 Limit resources used by ORES Other Operations 2017-01-02 2017-05-01
116767 limit the impact of heavy/large graphite queries Other Operations 2017-01-02 2017-01-27
152078 Load balancing "external" traffic to the Kubernetes cluster in production Other Operations 2017-01-02 2017-06-05
151314 logrotate failing with $FILE.1.gz: File exists Other Operations 2017-01-02 2016-11-22
153940 Logrotate fails for: "$FILE No such file or directory" Other Operations 2017-01-02 2017-03-14
149421 Long running mediawiki web requests impacts service availability, specially databases Other Operations 2017-01-02 2017-04-05
148017 lvs2002 repeated usb connect/disconnect message Other Operations 2017-01-02 2017-04-04
151273 lvs4002 power supply failure Other Operations 2017-01-02 2017-10-18
121105 Mails from MediaWiki seem to get (partially) lost Other Operations 2017-01-02 2016-03-08
146627 Make deployment-prep puppetmaster more similar to Production puppetmaster Other Operations 2017-01-02 2016-10-05
143552 Make elasticsearch configuration more robust to loss of network connectivity Other Operations 2017-01-02 2016-08-30
128716 Make icinga-wm report Tools homepage check at #wikimedia-labs, too Other Operations 2017-01-02 2016-05-27
95052 Make ircecho much better Other Operations 2017-01-02 2016-07-16
76203 Make ircecho run as its own user Other Operations 2017-01-02 2016-07-16
119718 Make it easier to ban misbehaving dashboards from graphite Other Operations 2017-01-02 2016-11-18
120585 Make l10nupdate user a system user Other Operations 2017-01-02 2017-09-26
117508 Make ops-l a list for humans again (no cheating) Other Operations 2017-01-02 2015-11-02
104352 Make scap able to depool/repool servers via the conftool API Other Operations 2017-01-02 2017-09-26
113785 Make the Shinken IRC alert and icinga-wm bots use colors Other Operations 2017-01-02 2015-11-02
138136 MB Lateefi Fonts for Sindhi Wikipedia. Other Operations 2017-01-02 2016-12-10
141520 "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) Other Operations 2017-01-02 2017-05-20
126989 MediaWiki logging & encryption Other Operations 2017-01-02 2017-07-10
116747 Meta task "Revamp user authentication" Other Operations 2017-01-02 2016-08-13
87220 Minimize differences between beta and production (Tracking) Other Operations 2017-01-02 2017-09-01
152767 Missing Labs hiera entry in labs-private repo Other Operations 2017-01-02 2017-08-30
135595 mod_deflate + mod_uwsgi causing mangled apache responses Other Operations 2017-01-02 2016-05-18
123809 Module uwsgi doesn't allow passing multiple config params of same name Other Operations 2017-01-02 2016-01-15
86552 Monitor and alarm on SMART attributes Other Operations 2017-01-02 2017-09-14
108985 Monitor MediaWiki sessions Other Operations 2017-01-02 2015-12-02
116580 monitor postgresql replication status Other Operations 2017-01-02 2015-10-26
110169 Monitor redis memory/disk usage Other Operations 2017-01-02 2016-10-03
106937 Monitor [[Special:ListFiles]] for non 200 HTTP statuses in thumbnails Other Operations 2017-01-02 2017-08-03
136311 Monitor the BMC's event log for hardware errors Other Operations 2017-01-02 2016-05-26
136403 Move cp3030+ from OE14 to OE13 in racktables Other Operations 2017-01-02 2016-06-21
114446 move human users out of UID range for system accounts Other Operations 2017-01-02 2016-04-15
95054 Move ircecho config file to be YAML Other Operations 2017-01-02 2016-07-15
122144 Move most (all?) exim personal aliases to OIT Other Operations 2017-01-02 2017-07-12
152445 Move prometheus entry point off port 80 Other Operations 2017-01-02 2017-07-10
115899 Move scap target configuration to etcd Other Operations 2017-01-02 2016-10-05
144006 Move the MW Beta appservers to Debian Other Operations 2017-01-02 2017-08-29
127549 move travel related aliases to OIT Other Operations 2017-01-02 2016-11-05
141959 Moving network::external to hiera broke much of labs Other Operations 2017-01-02 2016-08-03
129188 mw2212 unresponsive Other Operations 2017-01-02 2016-03-08
69015 m.wikipedia.org incorrectly redirects to en.m.wikipedia.org Other Operations 2017-01-02 2016-12-24
121240 Network isolation for production and semi-production services Other Operations 2017-01-02 2017-07-12
56713 Non-NDA users cannot access graphite.wikimedia.org Other Operations 2017-01-02 2016-06-10
138685 notebook1001 shown as DOWN in icinga, due to firewall rules Other Operations 2017-01-02 2016-06-26
88730 Nutcracker needs to automatically recover from MC failure - rebalancing issues Other Operations 2017-01-02 2017-08-04
111934 Nutcracker stats monitoring should only listen on localhost Other Operations 2017-01-02 2016-12-01
97524 ocg alarm ocg_job_status_queue 'flapping' Other Operations 2017-01-02 2016-10-03
135338 On Trusty and Jessie PHP yields: PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/20-xhprof.ini on line 2 Other Operations 2017-01-02 2017-07-01
114801 operations-apache-config-lint replacement doesn't check syntax Other Operations 2017-01-02 2016-07-28
153703 Option: Consider switching back to leveled compaction (LCS) Other Operations 2017-01-02 2016-12-19
146968 OTRS spam classification methods and systems Other Operations 2017-01-02 2017-10-11
126083 overhaul labstore setup [tracking] Other Operations 2017-01-02 2017-07-23
150396 Phabricator leaving old files in /tmp Other Operations 2017-01-02 2017-01-03
120159 Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module Other Operations 2017-01-02 2016-10-04
127054 pinentry-gtk2 pulls in a lot of unneeded Gnome/GTK libs Other Operations 2017-01-02 2016-02-16
123106 PNG thumbnail preview of SVG misses some text Other Operations 2017-01-02 2017-01-06
148637 Port redis statistics to Prometheus Other Operations 2017-01-02 2017-10-19
129180 Preserve SSH host key when re-imaging hosts Other Operations 2017-01-02 2016-03-07
93138 Procure hardware for Sentry Other Operations 2017-01-02 2016-01-06
151050 Proper documentation for Yubico 2FA for production use Other Operations 2017-01-02 2016-11-18
133476 Proposal: Centralize OTRS login methodology Other Operations 2017-01-02 2017-08-15
140813 Protect sensitive user-related information with a UserData / auth / session service Other Operations 2017-01-02 2017-06-08
122917 Provide a good download service of dumps from Wikimedia Other Operations 2017-01-02 2016-07-19
150672 Provide a /parsoid directory on releases.wikimedia.org Other Operations 2017-01-02 2017-05-20
151009 Provide authenticated access to Prometheus native web interface Other Operations 2017-01-02 2017-09-11
123237 Provide production jessie image with node 4.2; use this for service-runner build command Other Operations 2017-01-02 2016-10-12
150466 publish kartotherian / tilerator metrics by cluster Other Operations 2017-01-02 2016-11-10
150823 Puppet CA rollover Other Operations 2017-01-02 2016-11-29
153246 Puppet failures with "Attempt to assign to a reserved variable name: 'trusted'" Other Operations 2017-01-02 2017-09-11
132632 puppetize turning off reserved space for cassandra /srv Other Operations 2017-01-02 2017-07-18
126574 puppet should try to mount all mountable swift filesystems Other Operations 2017-01-02 2016-02-11
136094 Race condition in setting net.netfilter.nf_conntrack_tcp_timeout_time_wait Other Operations 2017-01-02 2017-08-09
135113 Rationalize our jobqueues redis topology Other Operations 2017-01-02 2017-07-07
146841 Reach out to Google about @yahoo.com emails not reaching gmail inboxes (when sent to mailing lists) Other Operations 2017-01-02 2017-07-18
127825 Re-add intel-microcode Other Operations 2017-01-02 2016-09-25
119846 Redirect revisions from svn.wikimedia.org to https://phabricator.wikimedia.org/rSVN Other Operations 2017-01-02 2015-11-30
133179 Redis monitoring needs to be improved Other Operations 2017-01-02 2017-07-07
135122 Reduce etcd technical debt Other Operations 2017-01-02 2016-05-12
109606 Re-evaluate Limesurvey Other Operations 2017-01-02 2017-06-05
150020 Refactor puppet-postgresql module to use custom types Other Operations 2017-01-02 2016-12-09
148647 refresh swift hardware in codfw/eqiad Other Operations 2017-01-02 2017-03-16
131748 Refresh the appservers puppet code/configs Other Operations 2017-01-02 2016-04-04
126281 [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") Other Operations 2017-01-02 2016-08-08
140442 reinstall rdb100[56] with RAID Other Operations 2017-01-02 2017-04-18
120856 Remove all out of warranty unused cp10xx's from A2 Other Operations 2017-01-02 2017-07-20
150917 Remove deprecated features from book creator UI Other Operations 2017-01-02 2017-10-02
144539 Remove /srv/deployment/wdqs/wdqs/rules.log symlink Other Operations 2017-01-02 2016-10-11
112257 rename cassandra cluster Other Operations 2017-01-02 2016-10-13
147872 Rename rhodium to puppetmaster1003 Other Operations 2017-01-02 2016-10-11
146355 Replace etcd internal auth mechanism with a frontend proxy Other Operations 2017-01-02 2016-09-22
134271 Replace ircd-ratbox with something newer/maintained Other Operations 2017-01-02 2016-06-17
150872 Replace OCG in collection extension with Electron Other Operations 2017-01-02 2017-10-02
116951 Reprepro should bail if it can't read and sign using the root keys Other Operations 2017-01-02 2015-10-28
125015 Requests to (hard) redirect pages return their target's contents but are counted as pageviews to the redirect page Other Operations 2017-01-02 2016-01-28
144431 RESTBase k-r-v as Cassandra anti-pattern Other Operations 2017-01-02 2017-03-17
148567 Restrict outgoing network connections from Electron render service Other Operations 2017-01-02 2017-02-18
142984 Review lists of config/sysctl recommendations by "kernel self-protection project" Other Operations 2017-01-02 2016-08-15
141897 Review new service 'pre-deployment to production' checklist Other Operations 2017-01-02 2017-07-12
149804 Review of ferm services without srange Other Operations 2017-01-02 2016-11-02
137397 revisit swift (sys)logging Other Operations 2017-01-02 2016-07-06
119679 Rewrite http://download.wikimedia.org/mediawiki/ -> https://releases.wikimedia.org/mediawiki in less than 3 redirects Other Operations 2017-01-02 2017-05-16
126158 [RFC] Alert about *when* partitions will run out of space, not a percentage/absolute number Other Operations 2017-01-02 2016-05-02
97204 RFC: Request timeouts and retries Other Operations 2017-01-02 2017-03-31
132532 rsync module doesnt work on trusty Other Operations 2017-01-02 2016-04-12
32716 Run our own Tor client for Tor block Other Operations 2017-01-02 2017-06-19
151049 Run systematic availability tests Other Operations 2017-01-02 2016-11-18
133392 save grafana dashboards in revision control / puppet Other Operations 2017-01-02 2017-07-24
85451 scale graphite deployment (tracking) Other Operations 2017-01-02 2017-05-02
150771 Secondary production Jenkins for CI Other Operations 2017-01-02 2017-10-12
94329 secure Cassandra/RESTBase cluster Other Operations 2017-01-02 2016-10-19
93531 secure.wikimedia.org entries still showing up in Google search results Other Operations 2017-01-02 2015-12-08
122210 Security audit for tftp on Carbon Other Operations 2017-01-02 2016-03-29
40860 security@mediawiki.org : Create a public key and publish it on the public key servers Other Operations 2017-01-02 2016-03-13
122825 Service Ownership and Maintenance Other Operations 2017-01-02 2017-08-08
132216 Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm Other Operations 2017-01-02 2016-05-09
143556 Setting up grafana should also setup Anonymous read-only access for the default org Other Operations 2017-01-02 2016-11-29
55457 setup a DB backed parser cache Other Operations 2017-01-02 2017-05-20
106346 setup an alertable threshold for Cassandra heap dumps Other Operations 2017-01-02 2017-07-11
147366 Setup automated topk wide row reporting Other Operations 2017-01-02 2017-07-18
91404 Setup backups of elasticsearch indices Other Operations 2017-01-02 2016-09-22
119660 Set up LVS for labs dns recursors Other Operations 2017-01-02 2016-08-15
84700 Setup management switch in OE12 Other Operations 2017-01-02 2016-03-17
149543 Setup PAWS internal experimentally on notebook* nodes Other Operations 2017-01-02 2016-10-31
106664 Set up role accounts and feedback loops (FBL) with all providers Other Operations 2017-01-02 2016-10-04
123818 setup YubiHSM and laptop at office Other Operations 2017-01-02 2016-01-25
76306 Set warning thresholds for average cluster utilization Other Operations 2017-01-02 2016-12-16
152100 should we make privatewiki list available to puppet without maintaining two lists? Other Operations 2017-01-02 2016-12-07
151486 Silver anomalies Other Operations 2017-01-02 2016-12-14
151489 silver: /dev/md2 mounted twice Other Operations 2017-01-02 2016-12-14
151493 silver: / partition low on space Other Operations 2017-01-02 2016-11-24
131326 smokeping config puppetization issue? Other Operations 2017-01-02 2016-04-15
112774 solve mtp panel issue for row uplinks Other Operations 2017-01-02 2016-03-17
111838 Some files had disappeared from Commons after renaming Other Operations 2017-01-02 2017-07-25
149845 Something is wrong with installer root disk stuff Other Operations 2017-01-02 2017-01-20
17000 Special:Import error: "Import failed: Could not open import file" Other Operations 2017-01-02 2017-05-25
124101 Specific revisions of multiple files missing from Swift - 404 Not Found returned Other Operations 2017-01-02 2017-07-25
126295 Spike: What do we have to package to run the Programs and Events dashboard on production? Other Operations 2017-01-02 2017-02-28
125085 Split the API MediaWiki appserver pool into two external/internal pools Other Operations 2017-01-02 2016-10-12
151317 stat user crontab on stat hosts for old file removal Other Operations 2017-01-02 2016-11-22
134458 status.wikimedia.org should use some Wikimedia favicon if possible Other Operations 2017-01-02 2016-05-05
141704 Storage backend errors on commons when deleting/restoring pages Other Operations 2017-01-02 2017-07-25
148048 Store Wikimedia unified account name (SUL) in LDAP directory Other Operations 2017-01-02 2017-06-20
46016 SVG fails to render properly due to several issues Other Operations 2017-01-02 2016-07-07
140594 svn.wikimedia.org redirects to Diffusion main page, hence hard to find e.g. "flexbisonparse" Other Operations 2017-01-02 2016-07-18
123918 'swift' user/group IDs should be consistent across the fleet Other Operations 2017-01-02 2017-04-10
146285 Switch mwscript from Zend PHP5 to default php alternative (e.g. HHVM or PHP7) Other Operations 2017-01-02 2017-10-18
142821 Synchronise groups defined in data.yaml to LDAP Other Operations 2017-01-02 2016-08-12
121610 system users with UIDs > 500 Other Operations 2017-01-02 2015-12-16
140536 Thumbnails of some specific images show unwanted black lines Other Operations 2017-01-02 2016-07-19
151304 tmpreaper possible race condition Other Operations 2017-01-02 2016-11-22
116742 Track amount of package updates on systems Other Operations 2017-01-02 2016-08-08
132324 Tracking and Reducing cron-spam from root@ Other Operations 2017-01-02 2017-06-15
140942 Tracking: Monitoring and alerts for "business" metrics Other Operations 2017-01-02 2017-06-21
103886 Translation cache exhaustion caused by changes to PHP code in file scope Other Operations 2017-01-02 2017-09-26
137229 Tune thread for osm2pgsql / postgres max connections for Maps Other Operations 2017-01-02 2016-12-15
135128 Turn on etcd TLS for intra-cluster communications Other Operations 2017-01-02 2016-05-12
147040 Two recently uploaded files have disappeared (404) Other Operations 2017-01-02 2017-03-11
134326 udpmxircecho should write stats of messages processed and we should alert when that drops to zero Other Operations 2017-01-02 2016-05-03
134875 udpmxircecho spam/not working if unable to connect to irc server Other Operations 2017-01-02 2016-07-15
101141 udp rcvbuferrors and inerrors on graphite1001 Other Operations 2017-01-02 2017-07-10
131832 Unable to restore file that has a very large file size Other Operations 2017-01-02 2017-07-25
119401 Untangle labs/production roles from labs/instance roles Other Operations 2017-01-02 2015-11-23
147204 Update confd package Other Operations 2017-01-02 2017-09-14
143931 Update ICU version to 55.1 Other Operations 2017-01-02 2016-08-25
136603 Update limit.sh to support systemd-based cgroup management Other Operations 2017-01-02 2017-10-11
129963 Update memcached package and configuration options Other Operations 2017-01-02 2017-08-16
138866 Update & standardize Platform-specific_documentation for HP servers Other Operations 2017-01-02 2016-06-28
131928 Upgrade jessie systems from Linux 3.19 to 4.4 Other Operations 2017-01-02 2016-08-23
97909 Upgrade jobrunners to redis 2.8 Other Operations 2017-01-02 2016-02-13
150532 Upgrade qemu on ganeti clusters to 2.7 Other Operations 2017-01-02 2017-05-16
133643 Upstream our Diamond PowerDNSRecursorCollector Other Operations 2017-01-02 2016-04-26
142205 use granularity (g=) restrictions for wikimedia.org fundraising DKIM records Other Operations 2017-01-02 2016-08-05
120532 Use user-specific passwords for accessing EventLogging database Other Operations 2017-01-02 2017-02-20
132325 Weak digest algorithm (SHA1) used to sign InRelease on apt.wikimedia.org Other Operations 2017-01-02 2017-04-03
150356 Wikidata Query Service is overly verbose toward logstash Other Operations 2017-01-02 2016-12-16
46791 [[wikitech:Server_admin_log]] should not rely on freenode irc for logmsgbot entries Other Operations 2017-01-02 2017-02-14
132856 Write documentation on how / when to use custom Diamond metrics collectors Other Operations 2017-01-02 2016-04-16
154627 Production error message (when servers are down) points users to donate link which is likely to produce the same error message Other Operations 2017-01-05 2017-04-29
154665 Look into behaviour of /etc/exim4/update-exim4.conf.conf related to updates Other Operations 2017-01-06 2017-01-05
154915 Get rid of "import realm.pp" in manifests/site.pp Other Operations 2017-01-10 2017-07-06
155129 Create prometheus nutcracker exporter Other Operations 2017-01-12 2017-08-11
155209 Increase $wgHTTPImportTimeout to a higher value on WMF wikis Other Operations 2017-01-14 2017-01-13
111595 Do not apply spam headers on email assessed NOT to be spam Other Operations 2017-01-16 2017-01-15
155401 Integrate jessie 8.7 point release Other Operations 2017-01-17 2017-04-05
155761 DNS repo: add Jenkins job to ensure there are no duplicates Other Operations 2017-01-20 2017-01-19
149589 Puppet tab in Horizon unusably slow Other Operations 2017-01-21 2017-02-02
155929 Create /community-beacon alternative entry point Other Operations 2017-01-23 2017-05-17
156143 High CPU usage from swift-proxy on frontend machines Other Operations 2017-01-25 2017-01-24
156136 Increase swift replication factor for accounts Other Operations 2017-01-25 2017-01-24
156140 Lots of hosts with hyperthreading disabled Other Operations 2017-01-25 2017-05-25
156232 confctl SubjectAltNameWarning after python-urllib3 upgrade Other Operations 2017-01-26 2017-02-22
156398 Decommission or repair old asw-c2-eqiad Other Operations 2017-01-27 2017-07-20
156475 Investigate spike in 500s during asw-c2-eqiad replacement Other Operations 2017-01-28 2017-02-09
156544 Create backups of Wikimedia content in diverse geographic places Other Operations 2017-01-29 2017-01-28
156570 Investigate issues with wikitech-static.wikimedia.org Other Operations 2017-01-30 2017-01-29
150912 Class 'Memcached' not found when running mwscript eval.php on debug servers Other Operations 2017-02-01 2017-03-07
156924 Allow integration of data from etcd into the MediaWiki configuration Other Operations 2017-02-02 2017-10-19
156937 Provide cross-dc redundancy (active-active or active-passive) to all important misc services Other Operations 2017-02-02 2017-07-28
156955 Standardizing our partman recipes Other Operations 2017-02-02 2017-05-24
157038 Make it possible to run the mediawiki testsuite against a staging repo of apt.wikimedia.org Other Operations 2017-02-03 2017-06-13
157306 Fix config file handling for /etc/hhvm/php.ini Other Operations 2017-02-07 2017-02-07
157972 Puppet fails only once when restarting ferm is not successful Other Operations 2017-02-14 2017-02-13
158022 make apt.wikimedia.org HA Other Operations 2017-02-15 2017-02-27
158196 Reimage labstore1001 and labstore1002 for DRBD storage setup Other Operations 2017-02-16 2017-07-24
158288 Unclean stop of jobrunner service via puppet Other Operations 2017-02-17 2017-02-16
158434 Phabricator: Make sure phabricator works properly including our puppet roles on jessie Other Operations 2017-02-18 2017-05-20
158429 Switch to predictable network interface names? Other Operations 2017-02-18 2017-06-07
158562 Manage apt sources via puppet? Other Operations 2017-02-21 2017-02-20
158583 Restructure our internal repositories further Other Operations 2017-02-21 2017-07-19
158757 Puppet certificate missing subjectAltName Other Operations 2017-02-23 2017-03-22
158837 Consolidate performance website and related software Other Operations 2017-02-24 2017-10-19
88997 Improve graphite failover Other Operations 2017-02-24 2017-07-20
158915 Make sure replying to emails in gerrit 2.14 works Other Operations 2017-02-24 2017-06-14
159242 Segmentation fault creating thumbnail Other Operations 2017-03-01 2017-02-28
159354 Move coal from graphite machine(s) Other Operations 2017-03-02 2017-09-12
159524 backup space is used unwisely Other Operations 2017-03-04 2017-04-12
159536 Puppet constantly trying to stop the already stopped puppetmaster process on Trusty Other Operations 2017-03-04 2017-03-03
159687 etcd switchover/enhancements Other Operations 2017-03-07 2017-05-15
159661 Improve Terbium (and wasat) userland to process server side uploads Other Operations 2017-03-07 2017-03-06
159830 Sanity check global-multiwrite logs for ConfirmEdit usage Other Operations 2017-03-08 2017-03-07
159750 E-mail for people in different OIT LDAP object unit Other Operations 2017-03-09 2017-09-25
160060 Icinga check for sysctl settings Other Operations 2017-03-10 2017-09-07
160101 Upgrade php5-json .deb to at least 1.3.8 Other Operations 2017-03-10 2017-03-09
160146 jobrunner/jobchron services fail in codfw Other Operations 2017-03-11 2017-03-10
160158 Make disabled accounts visible in the corp mirror LDAP replica Other Operations 2017-03-11 2017-03-28
160229 Back up of Commons files Other Operations 2017-03-12 2017-05-14
160412 Add lock_wait_timeout to maintain_views and maintain-meta_p Other Operations 2017-03-15 2017-03-26
160677 Effects on adjusting Prometheus retention Other Operations 2017-03-17 2017-07-10
161003 Cross-check disabled accounts from corp LDAP against data.yaml Other Operations 2017-03-22 2017-03-21
161004 Remove disabled users from internal mailing lists Other Operations 2017-03-22 2017-09-07
161096 confctl no longer logs a non-changing state change Other Operations 2017-03-23 2017-03-22
161145 Fix the general problem of randomly-bad puppet agent cron timings within redundant clusters Other Operations 2017-03-23 2017-07-21
161296 Upgrade mysqld_exporter to 0.10.0 Other Operations 2017-03-25 2017-05-15
161528 incident 20170323-wikibase did not trigger Icinga paging Other Operations 2017-03-28 2017-03-27
161598 Monitor HHVM bytecode cache depletion on mediawiki app servers Other Operations 2017-03-29 2017-06-15
160529 Sender email spoofing Other Operations 2017-03-29 2017-07-28
161864 404 error while accessing some images files e.g. djvu and jpg Other Operations 2017-03-31 2017-03-30
160071 Add slabinfo prometheus exporter Other Operations 2017-03-31 2017-03-30
161835 Convert labstore cluster configuration to hiera and profiles Other Operations 2017-03-31 2017-07-23
160941 Improve SSH access information in onboarding documentation Other Operations 2017-03-31 2017-03-30
161834 Undo special tools-home and tools-project share definitions for NFS Other Operations 2017-03-31 2017-07-23
161904 decommission backup4001 Other Operations 2017-04-01 2017-07-18
161899 End self-service new Trusty instance creation in Cloud VPS; standardize on Debian base images Other Operations 2017-04-01 2017-10-05
161920 logrotate for ruthenium Other Operations 2017-04-01 2017-05-10
161918 videoscalers (mw1168, mw1169) - high load / overheating Other Operations 2017-04-01 2017-04-01
160644 Eventstreams graphite disk usage Other Operations 2017-04-02 2017-08-29
162013 etcd cluster in codfw has raft consensus issues Other Operations 2017-04-04 2017-09-14
162090 Investigate alternative RAID strategies for labstore1001/2 Other Operations 2017-04-04 2017-06-06
162029 Migrate all jessie hosts to Linux 4.9 Other Operations 2017-04-04 2017-04-13
162037 Use SSL certificates with discovery entry for elasticsearch Other Operations 2017-04-04 2017-10-19
162123 Running swiftrepl is not puppetized Other Operations 2017-04-05 2017-09-28
162122 Swiftrepl was stuck in an infinite loop since days Other Operations 2017-04-05 2017-04-12
162245 Enable GC for HHVM CLI (at least for dump runners) Other Operations 2017-04-06 2017-07-27
36947 Incorrect text positioning in SVG rasterization (scale/transform; font-size; kerning) Other Operations 2017-04-08 2017-08-15
162612 codfw/eqiad hosts occasionally spend > 3 minutes starting networking.service with linux 4.9 Other Operations 2017-04-11 2017-07-20
162850 CPU throttling on DELL PowerEdge R320 Other Operations 2017-04-13 2017-09-13
162955 rebuild tools-grid-master as a large instance Other Operations 2017-04-14 2017-06-08
162857 Some Core availability Catchpoint tests might be more expensive than they need to be Other Operations 2017-04-14 2017-10-11
163068 More missing 'original' files on Commons Other Operations 2017-04-17 2017-07-25
163362 audit all codfw pdu tower draws Other Operations 2017-04-20 2017-04-19
163288 Decide on /var/lib vs /home as locations of homedir for l10nupdate Other Operations 2017-04-20 2017-04-19
163393 Determine appropriate proxy_read_timeout setting for Tools Proxy Other Operations 2017-04-20 2017-04-19
163402 Ensure we can survive a loss of labservices1001 Other Operations 2017-04-20 2017-08-03
163354 Find a way to verify mediawiki-config IPs ahead of datacenter switchovers Other Operations 2017-04-20 2017-04-19
163336 kube-proxy pulls in docker and starts service even when it isnt needed Other Operations 2017-04-20 2017-04-19
163507 Intermittent DB connectivity problem on phabricator, needs investigation Other Operations 2017-04-21 2017-05-24
163698 Add flood protection to the ircecho bot (icinga-wm) Other Operations 2017-04-25 2017-04-24
163667 Fix UIDs for deployment server users Other Operations 2017-04-25 2017-04-24
163673 Some swift disks wrongly mounted on 5 ms-be hosts Other Operations 2017-04-25 2017-08-22
163823 During labservices1001 failover fqdn changed from foo.project.eqiad.wmflabs to foo.eqiad.wmflabs Other Operations 2017-04-26 2017-04-25
163996 Icinga check for ipv6 host reachability Other Operations 2017-04-28 2017-04-27
164042 Racktables: clearly show when hosts are decommissioned Other Operations 2017-04-28 2017-04-27
164123 tools-k8s-master-01 has two floating IPs Other Operations 2017-04-29 2017-06-06
123276 URL parameters do not work with pages that have "?" in their names Other Operations 2017-04-29 2017-09-13
164248 HTTP responses from app servers sometimes stall for >1s Other Operations 2017-05-03 2017-09-12
164238 move icinga contacts file to public repo Other Operations 2017-05-03 2017-05-02
164341 Decommission old memcached hosts - mc1001->mc1018 Other Operations 2017-05-04 2017-10-18
164490 maintain-meta_p hangs on connecting to wikimedia.org.uk Other Operations 2017-05-05 2017-05-04
164460 Use DNS discovery record for deployment CNAME Other Operations 2017-05-05 2017-05-09
164703 Integrate jessie 8.8 point release Other Operations 2017-05-09 2017-05-08
164819 reprepro: Support for buildinfo files / dbgsym packages Other Operations 2017-05-10 2017-06-02
164993 archiva artifact links point to 127.0.0.1 Other Operations 2017-05-11 2017-05-10
165136 Ferm rules for labstore NFS hosts Other Operations 2017-05-13 2017-10-06
165173 rack/setup/install dumpsdata100[12] Other Operations 2017-05-13 2017-07-06
165170 rack/setup/install ores2001-2009 Other Operations 2017-05-13 2017-09-06
165323 Add Prometheus machine metric to track core dumps Other Operations 2017-05-16 2017-05-15
165345 decommission indium Other Operations 2017-05-16 2017-07-20
165519 rack and setup mw1307-1348 Other Operations 2017-05-17 2017-10-20
165618 Audit / document reasons for not enabling HT? Other Operations 2017-05-18 2017-05-17
165631 move gerrit.wm.org SSH service to private/behind LVS like phab-vcs Other Operations 2017-05-18 2017-05-17
165781 rack/setup/install labcontrol100[34] Other Operations 2017-05-20 2017-07-12
165784 rack/setup/install labmon1002 Other Operations 2017-05-20 2017-08-22
165779 rack/setup/install labnet100[34] Other Operations 2017-05-20 2017-08-22
165885 Create a cron to clean clientbucket every day or hour Other Operations 2017-05-21 2017-05-20
166066 Integrate the puppet compiler in the puppet CI pipeline Other Operations 2017-05-23 2017-07-06
166081 rack/setup/install conf1004-conf1006 Other Operations 2017-05-23 2017-09-28
166038 Sync internal nutcracker package with Debian package Other Operations 2017-05-23 2017-05-29
163033 Create grafana dashboard for video scaler job runners Other Operations 2017-05-24 2017-05-23
166233 Update redis puppet class to support stretch Other Operations 2017-05-25 2017-05-24
166291 Exim panics when spamd reaches maxchildren Other Operations 2017-05-26 2017-09-18
166322 spam from phabricator in labs Other Operations 2017-05-26 2017-05-26
165105 Wiley requests for DOI and some other publishers don't work in production Other Operations 2017-05-26 2017-08-25
166368 Wipe of spare/replacement disks Other Operations 2017-05-27 2017-05-26
166937 Broken /a/refinery-source/guard/run_all_guards.sh script on stat1002 Other Operations 2017-06-04 2017-10-19
167035 stretch acct monthly cron will spam when /var/log/wtmp.1 doesn't exist Other Operations 2017-06-06 2017-06-05
167091 Elasticsearch errors about BulkShardRequest Other Operations 2017-06-07 2017-08-03
167239 Redirect status.wikipedia.org to status.wikimedia.org Other Operations 2017-06-07 2017-06-14
164290 Set up external DNS record for wikitech-static Other Operations 2017-06-07 2017-06-06
167292 Collate jessie-wikimedia/backports into jessie-wikimedia/main Other Operations 2017-06-08 2017-06-07
167245 prometheus-node-exporter - invalid group: ‘prometheus:prometheus' Other Operations 2017-06-08 2017-06-07
89808 wikitech instances list is blank Other Operations 2017-06-08 2017-06-07
167412 host-vmem.erb is doing operations that make no sense Other Operations 2017-06-09 2017-06-17
167422 Monitoring: add link to graph for Icinga timeseries alarms Other Operations 2017-06-09 2017-06-08
167549 Create Icinga alert when OSM replication lags on maps Other Operations 2017-06-10 2017-09-07
167966 Look into feasibility of disabling sha-1 host keys on our ssh daemons Other Operations 2017-06-16 2017-07-12
165511 Change automatic shortlink in blog theme Other Operations 2017-06-20 2017-06-20
168403 Aggregate prometheus functions yielding different results in grafana vs. prometheus console Other Operations 2017-06-21 2017-06-20
168407 rack/setup/install labnodepool1002.eqiad.wmnet Other Operations 2017-06-21 2017-07-12
168445 Reboots of cloud servers Other Operations 2017-06-21 2017-06-20
168460 Update certificates on productions replicas of corp.wikimedia.org LDAP Other Operations 2017-06-21 2017-06-20
168562 Reimage gerrit2001 as stretch Other Operations 2017-06-22 2017-09-28
168490 upgrade planet instances to stretch Other Operations 2017-06-22 2017-06-21
167225 Upload hhvm to stretch apt repo in apt.wikimedia.org Other Operations 2017-06-22 2017-09-21
168619 Degraded RAID on lvs3001 Other Operations 2017-06-23 2017-09-13
114849 Log lines on flourine overflow at 8092 bytes. Other Operations 2017-06-24 2017-06-23
113104 Set up a service IP for logstash Other Operations 2017-06-24 2017-06-23
168891 rack/setup/install labtestmetal2001.codfw.wmnet Other Operations 2017-06-27 2017-07-26
168767 Monitor PostgreSQL connection slots Other Operations 2017-06-28 2017-06-27
168967 Upload shiny-server .deb to our Jessie apt repository Other Operations 2017-06-28 2017-09-21
169035 bast3002 sdb broken Other Operations 2017-06-29 2017-07-03
169246 Stress/capacity test new ores* cluster Other Operations 2017-06-30 2017-10-06
169249 /usr/local/bin/xenon-generate-svgs and flamegraph.pl cronspam Other Operations 2017-06-30 2017-09-12
169287 etcd config depends on puppet certs, but puppet doesn't know Other Operations 2017-07-01 2017-06-30
169286 labstore1005 A PCIe link training failure error on boot Other Operations 2017-07-01 2017-06-30
169290 New anti-stackclash (4.9.25-1~bpo8+3 ) kernel super bad for NFS Other Operations 2017-07-01 2017-07-06
169322 Research whether it makes sense to have OTRS installation in an HA setup Other Operations 2017-07-01 2017-06-30
169318 Use multiple puppetdbs on puppet masters Other Operations 2017-07-01 2017-06-30
169518 Decommission esams ms-fe / ms-be Other Operations 2017-07-04 2017-08-29
169564 MD RAID: remove mdadm daily check Other Operations 2017-07-04 2017-07-03
169570 nfs-manage failover script needs to be tested with real load and fixed Other Operations 2017-07-04 2017-09-14
169548 Prepare for Puppet 4 Other Operations 2017-07-04 2017-10-03
169680 NFS on dataset1001 overloaded, high load on the hosts that mount it Other Operations 2017-07-05 2017-10-12
169849 Architecture and puppetize setup for dumpsdata boxes Other Operations 2017-07-07 2017-07-12
169884 Jobrunners generate mediawiki exceptions upon calling Closure$RecentChange::save Other Operations 2017-07-07 2017-07-07
169937 Services Q1 2017/18 goal: Begin migrating job queue processing to multi-DC enabled eventbus infrastructure. Other Operations 2017-07-07 2017-09-06
169969 Regularly purge old ores graphite metrics Other Operations 2017-07-08 2017-10-03
170150 Evaluate Grafana's LDAP group options and deprecate grafana-admin if possible Other Operations 2017-07-11 2017-10-02
170152 mc2023 / mc2025 fail to mount root partition within 90 seconds using Linux 4.9 Other Operations 2017-07-11 2017-07-10
170108 Operations Q1 goal: Streamlined Service Delivery Other Operations 2017-07-11 2017-07-12
89887 Clean up permissions for privatedata files on stat1002 - they should be group readable by statistics-privatedata-users Other Operations 2017-07-12 2017-07-13
170353 Icinga: timeseries checks should have the link to a graph with the data Other Operations 2017-07-12 2017-10-02
104671 Rename 'restricted' group? Other Operations 2017-07-12 2017-07-11
170298 sshd stretch puppet support Other Operations 2017-07-12 2017-07-12
170474 Decommisson and store old row D network gear. Other Operations 2017-07-13 2017-07-20
170480 FY2017/18 Program 6 - Outcome 2: Developers are able to develop and test their applications through a unified pipeline towards production deployment. Other Operations 2017-07-13 2017-07-18
170481 FY2017/18 Program 6 - Outcome 2 - Objective 2: Set up a continuous integration and deployment pipeline Other Operations 2017-07-13 2017-10-02
170456 FY2017/18 Program 6 - Outcome 2 - Objective 3: Integrated, container-based development environment Other Operations 2017-07-13 2017-07-18
170453 FY2017/18 Program 6: Streamlined Service delivery Other Operations 2017-07-13 2017-09-06
170365 move legal-tm-vio alias to OIT Other Operations 2017-07-13 2017-08-23
170640 reports.frdev.wm.o -- still in use? Other Operations 2017-07-14 2017-07-18
170740 PuppetDB misbehaving on 2017-07-15 Other Operations 2017-07-16 2017-07-19
170817 Upgrade Thumbor servers to Stretch Other Operations 2017-07-18 2017-08-23
170628 Jessie rsvg/cairo can't render specific SVG file on Commons Other Operations 2017-07-19 2017-10-12
170995 Setup a mirror for R language dependencies (CRAN) Other Operations 2017-07-19 2017-10-11
171048 Eventbus does not handle gracefully changes in DNS recursors Other Operations 2017-07-20 2017-10-09
171122 librenms: consider using Distributed Poller with multiple netmon servers Other Operations 2017-07-20 2017-07-19
83729 Fix monitoring of poolcounter service Other Operations 2017-07-21 2017-10-13
171157 Monitor internal CA expirations Other Operations 2017-07-21 2017-08-21
171188 Move the main WMCS puppetmaster into the Labs realm Other Operations 2017-07-21 2017-07-24
171191 Should puppet auto-restart slapd? Other Operations 2017-07-21 2017-07-21
171452 Integrate jessie 8.9 point release Other Operations 2017-07-25 2017-07-24
171473 labvirt1015 crashes Other Operations 2017-07-25 2017-10-18
171482 Programmatic generation of grafana dashboards Other Operations 2017-07-25 2017-09-15
154026 On mobile, http://wikipedia.org/wiki/Foo redirects to https://www.m.wikipedia.org/wiki/Foo which does not exist Other Operations 2017-07-26 2017-07-27
171745 nscd does not cache localhost causing high CPU usage when localhost is often resolved Other Operations 2017-07-27 2017-10-02
171619 ORES should use a git large file plugin for storing serialized binaries Other Operations 2017-07-27 2017-10-06
171758 Support git-lfs files in gerrit Other Operations 2017-07-27 2017-10-16
171851 Reimage ores* hosts with Debian Stretch Other Operations 2017-07-28 2017-09-06
171923 thorium - failed git clone of geowiki-data-private Other Operations 2017-07-29 2017-08-04
167689 Add RIPE atlas data to Prometheus Other Operations 2017-08-02 2017-08-01
154619 Export ipsec counters as Prometheus metrics Other Operations 2017-08-02 2017-08-01
172409 Copper root (/) 95% full Other Operations 2017-08-04 2017-09-20
172479 Collect error logs from jobchron/jobrunner services in Logstash Other Operations 2017-08-05 2017-10-13
172538 rack/setup/install labvirt10(19|20).eqiad.wmnet Other Operations 2017-08-05 2017-09-14
50029 crackling at start of OGG renditions of MIDI files (fixed in TiMidity++ 2.14.0) Other Operations 2017-08-06 2017-08-05
172628 conf2002 etcdmirror-conftool-eqiad-wmnet died Other Operations 2017-08-07 2017-08-06
172815 Improve stability and maintainability of our browser-based PDF render service Other Operations 2017-08-09 2017-10-13
173056 Import Wiki Loves Monuments photos from Flickr to Commons Other Operations 2017-08-11 2017-10-10
173374 Deleting file on Commons "Error deleting file: An unknown error occurred in storage backend "local-multiwrite"." Other Operations 2017-08-16 2017-10-19
173436 Delete graphite metrics for old CFs Other Operations 2017-08-17 2017-10-05
173492 Tune Varnishkafka delivery errors to be more sensitive Other Operations 2017-08-18 2017-10-02
173710 Job queue is increasing non-stop Other Operations 2017-08-22 2017-10-22
173721 Track down the source of periodic increases in requests to swift eqiad Other Operations 2017-08-22 2017-08-23
174172 unused grafana-dashboard indices on elasticsearch / logstash Other Operations 2017-08-26 2017-08-25
174269 Two cases of local-multiwrite storage backend failure Other Operations 2017-08-28 2017-08-27
174402 Review and fix file handle management in worker and celery processes Other Operations 2017-08-29 2017-10-17
174431 Migration of mw* servers to stretch Other Operations 2017-08-30 2017-10-17
174465 Puppet admin module should support adding system users to managed groups Other Operations 2017-08-30 2017-09-20
174449 tin has a failing hdd Other Operations 2017-08-30 2017-09-07
174475 update firmware on scs consoles Other Operations 2017-08-30 2017-09-11
41785 Create a labs SMTP smarthost Other Operations 2017-08-31 2017-09-07
174720 letsencrypt::cert::integrated and non-http servers Other Operations 2017-09-01 2017-10-17
174916 electron/pdfrender hangs Other Operations 2017-09-05 2017-10-13
174959 swift-recon-cron on ms-be203[34]: [Errno 17] File exists: '/var/lock/swift-recon-object-cron' Other Operations 2017-09-05 2017-09-05
175206 2017/18 Annual Plan Program 8: Multi-datacenter support Other Operations 2017-09-07 2017-10-19
175213 2017/18 Annual Plan Program 8: Multi-datacenter support, Q2 goals Other Operations 2017-09-07 2017-10-19
175150 Decommission stat1003.eqiad.wmnet Other Operations 2017-09-07 2017-09-14
175210 Select candidate jobs for transferring to the new infrastucture Other Operations 2017-09-07 2017-09-28
175242 all log producers need to use the logstash LVS endpoint Other Operations 2017-09-08 2017-09-20
175288 setup/install/deploy deploy1001 as deployment server Other Operations 2017-09-08 2017-09-08
175362 Split MXes into inbound and outbound Other Operations 2017-09-09 2017-09-20
175361 Upgrade mx1001/mx2001 to stretch Other Operations 2017-09-09 2017-09-20
175637 End of September milestone: Migrate first production use case Other Operations 2017-09-12 2017-09-26
175625 scs-c1-eqiad unresponsive Other Operations 2017-09-12 2017-09-25
175710 Add profiling for Varnish and VCL Other Operations 2017-09-13 2017-09-12
175830 decommission logstash100[1-3] Other Operations 2017-09-14 2017-10-09
175876 document all scs connections Other Operations 2017-09-14 2017-09-18
175799 port elasticsearch diamond collector to prometheus Other Operations 2017-09-14 2017-09-13
175798 Port non-deprecated Diamond collectors to Prometheus Other Operations 2017-09-14 2017-10-16
175850 Spike: Enumerate remaining unported stats Other Operations 2017-09-14 2017-09-15
175885 Toolforge's static webserver broken by Puppet changes and stale nginx packages Other Operations 2017-09-15 2017-09-14
82350 update exim::listserve::private::mailing_lists value in puppet Other Operations 2017-09-19 2017-09-18
176335 logs sent to logstash are lost when the elasticsearch cirrus cluster is unavailable Other Operations 2017-09-21 2017-10-19
176370 Migrate to PHP 7 in WMF production Other Operations 2017-09-21 2017-10-17
176437 puppet ca_server confusion Other Operations 2017-09-22 2017-09-21
176445 Systematically test load speeds of Watchlist and Recent Changes Other Operations 2017-09-22 2017-10-12
176364 Request access to logstash (nda group) for @framawiki Other Operations 2017-09-23 2017-10-16
176153 Create affcom-staff email account Other Operations 2017-09-26 2017-09-25
176666 Qualtrics email-LDAP issue Other Operations 2017-09-26 2017-09-25
176774 Reimage cobalt as stretch Other Operations 2017-09-27 2017-09-26
176957 Deprecate host copper.eqiad.wmnet Other Operations 2017-09-29 2017-09-28
177099 Large number of "A page you created was linked on Wikidata" emails to one recipient in short period of time Other Operations 2017-09-30 2017-09-29
177055 Update docker image docker-registry.wikimedia.org/wikimedia-jessie Other Operations 2017-09-30 2017-10-20
177197 Export Prometheus-compatible JVM metrics from JVMs in production Other Operations 2017-10-03 2017-10-04
177196 Port non-deprecated Diamond collectors to Prometheus Other Operations 2017-10-03 2017-10-19
177208 Provide dedicated database resources for wikidata Other Operations 2017-10-03 2017-10-17
177195 Reduce technical debt in metrics monitoring Other Operations 2017-10-03 2017-10-21
177225 Uninstall ganglia from the fleet Other Operations 2017-10-03 2017-10-21
177253 Upgrade puppetDB to version 3.2 or newer Other Operations 2017-10-03 2017-10-03
177254 Upgrade to puppet 4 (4.8 or newer) Other Operations 2017-10-03 2017-10-20
177397 Create scaffolding of services templates for deployment in production/staging Other Operations 2017-10-05 2017-10-16
177387 Decomission mw1161-69 Other Operations 2017-10-05 2017-10-20
177374 decom wtp1001-wtp1024 Other Operations 2017-10-05 2017-10-18
177396 Design pod-level monitoring and service-level alerting Other Operations 2017-10-05 2017-10-16
177394 Experiment with a TLS proxy/router for pods Other Operations 2017-10-05 2017-10-16
177393 Implement authentication/authorization in Kubernetes clusters Other Operations 2017-10-05 2017-10-16
177395 Improve monitoring of the Kubernetes clusters Other Operations 2017-10-05 2017-10-16
177371 Phase out DSA keys for SSH access (ssh-dss) Other Operations 2017-10-05 2017-10-04
177405 rack and setup db1107 and db1108 Other Operations 2017-10-05 2017-10-19
177276 Unify production and CI docker image build process Other Operations 2017-10-05 2017-10-13
177521 Permissions to upload data to the analytics cluster from a machine at Drexel Other Operations 2017-10-06 2017-10-13
177501 Prometheus cluster attribute for new RESTBase Cassandra cluster Other Operations 2017-10-06 2017-10-05
177498 Provide a forward port of ICU 52 for stretch / Investigate best ICU update strategy Other Operations 2017-10-06 2017-10-06
177624 check cp4007 power supply redundancy Other Operations 2017-10-07 2017-10-06
177625 check cp4008 power supply redundancy Other Operations 2017-10-07 2017-10-06
177623 check lvs4002 power supply redundancy Other Operations 2017-10-07 2017-10-06
177638 check mw2160 power supply redundancy Other Operations 2017-10-07 2017-10-06
177639 check mw2176 power supply redundancy Other Operations 2017-10-07 2017-10-06
177622 Multiple systems in ulsfo 1.22 showing PSU failures Other Operations 2017-10-07 2017-10-18
177739 Integrate stretch 9.2 point release Other Operations 2017-10-10 2017-10-11
177742 Investigate Chrony as a replacement for ISC ntpd Other Operations 2017-10-10 2017-10-09
177821 Allow syslog-tls in analytics towards wezen/lithium Other Operations 2017-10-11 2017-10-10
177875 Degraded RAID on bast3002 Other Operations 2017-10-11 2017-10-10
177881 Degraded RAID on lvs3001 Other Operations 2017-10-11 2017-10-10
177826 Upgrade ci ssh key to ecdsa Other Operations 2017-10-11 2017-10-12
177931 Decommission OCG from production Other Operations 2017-10-12 2017-10-11
178008 ensure that services on labtest machines never create SMS from Icinga (not send sms pages for labtest* things to non-cloud folks) Other Operations 2017-10-12 2017-10-16
177914 Switch labstore servers to default SSH configuration Other Operations 2017-10-12 2017-10-13
177891 Update and use php-wikidiff2 to 1.5 in production Other Operations 2017-10-12 2017-10-16
178325 Operations 2017-18 Q2 Program 6 umbrella task Other Operations 2017-10-17 2017-10-16
178445 flapping monitoring for recommendation_api on scb Other Operations 2017-10-18 2017-10-18
178538 Bump PHP requirement to 5.6 in 1.31 Other Operations 2017-10-19 2017-10-20
178454 Icinga disk space alert when a Docker container is running on an host Other Operations 2017-10-19 2017-10-18
84845 improve cron spam visibility Other Operations 2017-10-19 2017-10-18
178491 Make WDQS throttling more aggressive Other Operations 2017-10-19 2017-10-20
178457 nutcracker fails to start due to lack of /var/run/nutcracker (ex: deployment-videoscaler01 has memcached failures) Other Operations 2017-10-19 2017-10-18
178613 add missing asset tag and correct location in rack for cr1-eqdfw Other Operations 2017-10-20 2017-10-19
178575 Add require_package() variant with repository component to wmflib Other Operations 2017-10-20 2017-10-19
178628 Improve puppet alerting Other Operations 2017-10-20 2017-10-20
178625 puppetdb failures Other Operations 2017-10-20 2017-10-19
178570 [subtask] How should we get Chromium for use in puppeteer? Other Operations 2017-10-20 2017-10-20
178690 Better organization for ops grafana dashboards Other Operations 2017-10-21 2017-10-20
178742 Possibly faulty BBU on analytics1029 Other Operations 2017-10-22 2017-10-21
148494 Add shell scripts CI validations Software Development 2017-01-02 2017-09-14
152950 E901 SyntaxError: invalid syntax is wrongly raised on using python's abc by jenkins python CI linter Software Development 2017-01-02 2017-03-27
144169 Flake8 for python files without extension in puppet repo Software Development 2017-01-02 2017-06-05
145192 icinga-downtime script waiting forever if host already in downtime Software Development 2017-01-02 2016-09-09
150560 More verbose messages from service-checker-swagger Software Development 2017-01-02 2017-05-23
154776 Puppet compiler: order resources for easy comparison between hosts Software Development 2017-01-07 2017-01-06
155705 confctl: log to SAL even if the selection doesn't match any host Software Development 2017-01-20 2017-01-19
157001 Puppet compiler: abort on git rebase conflict Software Development 2017-02-03 2017-02-02
157002 Puppet compiler: re-add the concurrency option NUM_THREADS Software Development 2017-02-03 2017-02-02
157133 Consider adding a --skip-conftool option to puppet-merge Software Development 2017-02-04 2017-10-09
159045 Update Puppet repo code that uses maniphest.update and maniphest.createtask conduit api Software Development 2017-02-26 2017-02-25
161545 Cumin: PuppetDB backend, allow to specify boolean values for resource parameters Software Development 2017-03-28 2017-03-27
163363 Switchdc: improvements Software Development 2017-04-20 2017-05-16
163365 Switchdc RO/RW: add check to test it editing a real wiki Software Development 2017-04-20 2017-04-19
164587 cumin could use randomization/splay options Software Development 2017-05-06 2017-05-05
167504 New tool to track package updates/status for hosts and images (debmonitor) Software Development 2017-06-10 2017-10-07
169304 Cumin masters: simplify usage in case of emergency Software Development 2017-07-01 2017-06-30
177385 Upgrade Cumin masters to stretch Software Development 2017-10-05 2017-10-18
178342 Cumin: add support for external backends Software Development 2017-10-17 2017-10-17
178279 Cumin: add support for role/profile shortcuts in PuppetDB backend Software Development 2017-10-17 2017-10-16
137990 Zero: Investigate removing the limit on carrier tagging to m-dot and zero-dot requests Traffic 2016-06-17 2017-07-17
135762 A/B Testing solid framework Traffic 2017-01-02 2017-09-19
120486 add a https-only option to dynamicproxy Traffic 2017-01-02 2017-10-15
97051 adding new languages to DNS langs.tmpl doesn't work until zone template is edited as well Traffic 2017-01-02 2016-10-04
136703 Add LVS public endpoint checks that bypass caches Traffic 2017-01-02 2016-09-30
86650 Add support for setting weight=0 when depooling Traffic 2017-01-02 2017-07-31
63782 Add varnish logs to logstash Traffic 2017-01-02 2017-02-14
141373 Age header reset to 0 after 24 hours on varnish frontends Traffic 2017-01-02 2016-09-30
97333 Align prod + beta-cluster LVS usage Traffic 2017-01-02 2016-09-30
154015 announce breaking change: http > https for entities in rdf Traffic 2017-01-02 2017-07-25
138546 Backend naming in VCL needs to use fqdn+port Traffic 2017-01-02 2016-09-30
138591 Backport iproute2 4.x from debian testing -> our jessie Traffic 2017-01-02 2017-07-27
144187 Better handling for one-hit-wonder objects Traffic 2017-01-02 2016-09-30
152091 Block hotlinking Traffic 2017-01-02 2016-12-05
120509 Cache education dashboard pages Traffic 2017-01-02 2016-09-30
148830 cache_upload: uncompressed images with Content-Encoding: gzip cause content decoding issues Traffic 2017-01-02 2017-04-10
94125 Central login notice appears on unencrypted API format=*fm pages, where reloading does not affect login status Traffic 2017-01-02 2016-10-04
149873 CentralNotice: Review and update Varnish caching for Special:BannerLoader Traffic 2017-01-02 2016-11-22
36670 Check all wikis for inclusions of http resources on https Traffic 2017-01-02 2016-10-04
146832 Clarify caching to enable direct Wikidata Query Service access by <mapframe/link> Traffic 2017-01-02 2016-12-15
131894 Collect Backend-Timing in Graphite (or Prometheus) Traffic 2017-01-02 2017-09-20
154017 compile number of http uses for http://www.wikidata.org/entity Traffic 2017-01-02 2017-07-25
134323 confctl: give regexen more freedom Traffic 2017-01-02 2016-10-04
134324 confctl select needs a -y flag? Traffic 2017-01-02 2016-10-04
112316 Configure varnish to use "Unconfigured domain" page for 404 Not Served (instead of generic error) Traffic 2017-01-02 2016-09-30
113817 Connect Hadoop records of the same request coming via different channels Traffic 2017-01-02 2016-10-04
26413 Consider DNSSec Traffic 2017-01-02 2016-09-30
153563 Consider switching to HTTPS for Wikidata query service links Traffic 2017-01-02 2017-10-11
133821 Content purges are unreliable Traffic 2017-01-02 2017-10-13
148422 cp3009: memory scrubbing error Traffic 2017-01-02 2017-10-03
133548 Create a secure redirect service for large count of non-canonical / junk domains Traffic 2017-01-02 2017-07-14
119396 Create globally-unique varnish cache cluster port/instancename mappings Traffic 2017-01-02 2016-10-04
91820 Create HTTP verb and sticky cookie DC routing in VCL Traffic 2017-01-02 2017-05-05
132629 Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense Traffic 2017-01-02 2017-07-21
96499 dbtree loads third party resources (from jquery.com and google.com) Traffic 2017-01-02 2016-10-04
133001 Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames Traffic 2017-01-02 2017-08-08
124954 Decrease max object TTL in varnishes Traffic 2017-01-02 2017-07-11
96852 Define 3-host infra cluster for traffic pops Traffic 2017-01-02 2017-03-21
66214 Define an official thumb API Traffic 2017-01-02 2017-08-16
109331 Deleted files sometimes remain visible to non-privileged users if permanently linked Traffic 2017-01-02 2017-07-27
148131 Deploy redundant unified certs Traffic 2017-01-02 2017-06-26
133410 Deploy TemplateStyles to WMF production Traffic 2017-01-02 2017-09-19
111899 Deprecate pybal SSH health checks Traffic 2017-01-02 2016-09-30
128409 Detect tools.wmflabs.org tools which are HTTP-only Traffic 2017-01-02 2017-10-15
146619 DNS domains registered to WMF no longer redirecting Traffic 2017-01-02 2016-09-30
127485 Enable VCL applayer datacenter-switch via confd Traffic 2017-01-02 2016-09-30
127482 Enable VCL source-DC switching via confd Traffic 2017-01-02 2016-09-30
121561 Encrypt Kafka traffic, and restrict access via ACLs Traffic 2017-01-02 2017-08-09
147209 etcd cluster has Raft Internal errors sporadically Traffic 2017-01-02 2016-10-04
96853 Evaluate Apache Traffic Server Traffic 2017-01-02 2017-06-21
122867 Evaluate the feasibility of cache invalidation for the action API Traffic 2017-01-02 2017-07-19
105657 Expires header for load.php should be relative to request time instead of cache time Traffic 2017-01-02 2017-10-17
153468 Ferm/DNS library weirdness on deployment-mediawiki boxes Traffic 2017-01-02 2017-09-08
98165 Figure out an etcd deploy strategy that includes multi DC failure scenarios. Traffic 2017-01-02 2017-07-11
136737 Fix lvs1001-6 storage Traffic 2017-01-02 2017-07-19
102178 Fix RESTBase support for wikitech.wikimedia.org Traffic 2017-01-02 2017-07-11
50133 ForeignAPIRepo wrongly returns non-protocol-relative URLs for original "thumbs" Traffic 2017-01-02 2017-06-27
143562 High number of failed inbound TFO connections in esams Mon-Fri Traffic 2017-01-02 2016-10-04
130904 Host rewrite for /static/ not applied to purges Traffic 2017-01-02 2016-09-30
108580 HTTPS for internal service traffic Traffic 2017-01-02 2016-11-16
104681 HTTPS Plans (tracking / high-level info) Traffic 2017-01-02 2017-07-11
86671 HTTPS RFC5077 session tickets encryption key rollovers Traffic 2017-01-02 2016-09-30
119038 Image cache issue when 'over-writing' an image on commons Traffic 2017-01-02 2016-04-28
94697 implement better failure-scenario geoip mapping in gdnsd Traffic 2017-01-02 2016-09-30
92002 implement Public Key Pinning (HPKP) for Wikimedia domains Traffic 2017-01-02 2017-06-26
120121 Improve Varnish XFF processing for trusted proxies Traffic 2017-01-02 2016-10-04
109321 Inbound TLS for tier-1 varnish backend caches Traffic 2017-01-02 2016-09-30
125170 Internal DNS resolver responds with NXDOMAIN for localhost AAAA Traffic 2017-01-02 2016-09-30
104442 Investigate better DNS cache/lookup solutions Traffic 2017-01-02 2017-07-26
124418 Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan Traffic 2017-01-02 2017-09-14
92298 Investigate our mitigation strategy for HTTPS response length attacks Traffic 2017-01-02 2016-11-15
138093 Investigate query parameter normalization for MW/services Traffic 2017-01-02 2017-07-12
133717 Letsencrypt all the prod things we can - planning Traffic 2017-01-02 2017-01-18
141266 letsencrypt puppetization: add parallel rsa+ecdsa cert support Traffic 2017-01-02 2016-09-30
134447 letsencrypt puppetization: upgrade for scalability Traffic 2017-01-02 2016-09-30
129682 Look into solutions for replaying traffic to testing environment(s) Traffic 2017-01-02 2017-07-12
140365 Lower geodns TTLs from 600 to 300 Traffic 2017-01-02 2016-09-30
83467 LVS testing needs to include internal services testing Traffic 2017-01-02 2016-09-30
128188 Make CI run Varnish VCL tests Traffic 2017-01-02 2016-09-30
152882 Many misc wikis lack mobile domains Traffic 2017-01-02 2017-07-25
102367 Migrate tools.wmflabs.org to https only (and set HSTS) Traffic 2017-01-02 2017-10-15
116132 missing H2 coalesce for upload.wm.o for images ref'd in projects' page outputs Traffic 2017-01-02 2016-11-15
141480 mixed-content issues on planet.wikimedia.org Traffic 2017-01-02 2016-10-04
75944 Monitor Varnish caches on beta cluster have two varnishd process running Traffic 2017-01-02 2016-09-30
84543 more robust certificate chain creation in puppet Traffic 2017-01-02 2016-12-05
133149 Move californium to an internal host? Traffic 2017-01-02 2016-10-04
89838 Move proxy IP lists to META for Varnish XFF decoding Traffic 2017-01-02 2017-02-06
78421 m.{project}.org portal/redirect consistency Traffic 2017-01-02 2017-08-08
86915 nan and minnan subdomain redirects are a mess Traffic 2017-01-02 2017-08-24
81605 No IPv6 addresses on Wikimedia nameservers ns(0-2).wikimedia.org Traffic 2017-01-02 2016-09-30
148134 OCSP Stapling for Intermediates Traffic 2017-01-02 2016-11-01
109325 Outbound HTTPS for varnish backend instances Traffic 2017-01-02 2016-09-30
137747 Parametrization of VCL is inconsistent Traffic 2017-01-02 2016-09-30
112765 Phabricator needs to expose notification daemon (websocket) Traffic 2017-01-02 2016-11-15
125938 PHP fatal errors causing Varnish to return 503 - "Junk after gzip data" Traffic 2017-01-02 2017-05-16
118181 Planning for phasing out non-Forward-Secret TLS ciphers Traffic 2017-01-02 2017-05-08
99216 Please set up a CNAME for videoserver.wikimedia.org to Video Editing Server Traffic 2017-01-02 2016-09-30
118468 point wikilovesmonuments.org ns to wmf Traffic 2017-01-02 2016-09-30
144508 Point wikipedia.in to 205.147.101.160 instead of URL forward Traffic 2017-01-02 2016-10-04
101048 Policy decisions for new (and current) DNS domains registered to the WMF Traffic 2017-01-02 2016-09-30
150479 Prometheus varnish metric churn due to VCL reloads Traffic 2017-01-02 2017-03-16
54253 Protocol-relative URLs are poorly supported or unsupported by a number of HTTP clients Traffic 2017-01-02 2017-03-28
119372 Pybal IdleConnectionMonitor with TCP KeepAlive shows random fails if more than 100 servers are involved. Traffic 2017-01-02 2017-03-30
113597 pybal-related issue on host start can break service IPs... Traffic 2017-01-02 2017-02-10
137252 Redirect phabricator.mediawiki.org to phabricator.wikimedia.org Traffic 2017-01-02 2016-10-07
45250 Redo /beacon/impression system (formerly Special:RecordImpression) to remove extra round trips on all FR impressions (title was: S:RI should pyroperish) Traffic 2017-01-02 2017-03-27
147202 Removing support for AES128-SHA TLS cipher Traffic 2017-01-02 2016-10-03
147199 Removing support for DES-CBC3-SHA TLS cipher (drops IE8-on-XP support) Traffic 2017-01-02 2017-10-19
134807 Replace test hostnames in datecenter-specific subdomains with dashed names Traffic 2017-01-02 2016-10-03
23027 Requests with utf-8 in the URL return a outdated page revision Traffic 2017-01-02 2016-10-04
56783 Respect X-Forwarded-For only from trustworthy sources Traffic 2017-01-02 2016-10-04
133178 RESTBase support for www.wikimedia.org missing Traffic 2017-01-02 2017-07-11
129839 restrict upload cache access for private wikis Traffic 2017-01-02 2016-09-30
111588 RFC: API-driven web front-end Traffic 2017-01-02 2017-09-12
114979 Run IPVS in a separate network namespace Traffic 2017-01-02 2017-09-18
120631 Security: Is it safe to enable Zero spoofing Traffic 2017-01-02 2017-07-20
50835 Separate Cache-Control header for proxy and client Traffic 2017-01-02 2017-05-30
128182 Server certificate is classified as invalid on government computers Traffic 2017-01-02 2016-09-30
131930 Set SPF (... -all) for toolserver.org Traffic 2017-01-02 2017-06-14
123854 Set up action API latency / error rate metrics & alerts Traffic 2017-01-02 2017-07-11
136944 Set up LVS connection sync Traffic 2017-01-02 2016-09-30
101525 Set up LVS for current AuthDNS Traffic 2017-01-02 2017-04-25
128374 Sort out analytics service dependency issues for cp* cache hosts Traffic 2017-01-02 2017-10-02
117435 Spike: CentralNotice: Verify that our Special:HideBanners cookie storm works as efficiently as possible Traffic 2017-01-02 2017-08-16
102848 Split GeoIP into a new component Traffic 2017-01-02 2017-04-04
127387 Split slash decoding from general percent normalization in Varnish VCL Traffic 2017-01-02 2017-10-17
147844 Standardize varnish applayer backend definitions Traffic 2017-01-02 2016-10-11
128559 store.wikimedia.org HTTPS issues Traffic 2017-01-02 2017-07-11
144626 Strong cipher preference ordering for cache terminators Traffic 2017-01-02 2016-11-03
148976 Strongswan Icinga check: do not report issues about depooled hosts Traffic 2017-01-02 2016-10-24
137979 Support brotli compression Traffic 2017-01-02 2017-10-10
78963 Support ESI for ResourceLoader Traffic 2017-01-02 2016-12-05
107236 Switch port 80 to nginx on primary clusters Traffic 2017-01-02 2016-09-30
117826 TEST: redirect small portion of unauthenticated desktop users to mobile web Traffic 2017-01-02 2017-10-02
147967 The WMF-Last-Access Set-Cookie header should follow RFC 2965 syntax rather than the pre-RFC Netscape format Traffic 2017-01-02 2016-10-17
150673 Thumb API: Varnish / CDN questions Traffic 2017-01-02 2017-08-16
150022 thumb_handler.php should not set CC:no-cache on renderer 404 responses? Traffic 2017-01-02 2017-03-07
109776 Tilerator should purge Varnish cache Traffic 2017-01-02 2017-03-08
112521 Track/notify cert expiries better Traffic 2017-01-02 2016-09-30
147648 Unexplained increase in thumbnail 500s Traffic 2017-01-02 2016-11-18
134893 Unhandled pybal error causing services to be depooled in etcd but not in lvs Traffic 2017-01-02 2017-01-26
96844 Update TLS/HTTP documentation on wikitech Traffic 2017-01-02 2016-09-30
128358 Uploading 1.2GB ogv results in 503 Traffic 2017-01-02 2016-12-13
106517 upload.wikimedia.org returns HTTP status code 503 for truncated urls, not 404 Traffic 2017-01-02 2016-09-30
149847 Use content hash based image / thumb URLs Traffic 2017-01-02 2017-10-13
101002 Use Upgrade Insecure Requests on Wikimedia wikis Traffic 2017-01-02 2017-04-12
133895 Varnish configuration for mobile domains should be coherent with Apache configuration Traffic 2017-01-02 2016-09-30
74186 Varnish: Mobile site redirect interferes with OAuth authorization process Traffic 2017-01-02 2016-12-05
134404 Varnish support for active:active backend services Traffic 2017-01-02 2016-10-03
144194 Varnish-triggered CN campaign about browser security Traffic 2017-01-02 2016-10-13
148595 Visting [[c:File:FEZ_trial_gameplay_HD.webm]] in IE11 shows errors in developer console about insecure data:image/png;base64 "URL" Traffic 2017-01-02 2016-10-19
91372 $wgMFAnonymousEditing = true is sometimes not respected: cache? Traffic 2017-01-02 2016-05-04
127573 wikiknihy.cz - transfer to Wikimedia Czech Republic? Traffic 2017-01-02 2016-10-04
152622 Wikipedia.cz and other domains owned by WMCZ have invalid certificate Traffic 2017-01-02 2017-07-14
88861 wikipedia.lol Traffic 2017-01-02 2016-09-30
154702 Fix broken referer categorization for visits from Safari browsers Traffic 2017-01-06 2017-09-01
154801 Investigate varnishd child crashes when multiple nodes get depooled/pooled concurrently Traffic 2017-01-07 2017-02-07
155314 Varnish does not cache Action API responses when logged in Traffic 2017-01-15 2017-07-19
155806 Add CAA records to our domains Traffic 2017-01-21 2017-05-30
156027 Configuration for Asia Cache DC hosts Traffic 2017-01-24 2017-10-19
156026 Enable Service in Asia Cache DC Traffic 2017-01-24 2017-01-23
156032 Server hardware installation for Asia Cache DC Traffic 2017-01-24 2017-01-23
156033 Server hardware purchasing for Asia Cache DC Traffic 2017-01-24 2017-04-20
156031 Turn up network links for Asia Cache DC Traffic 2017-01-24 2017-04-11
156256 Allocate address space for Singapore (APNIC) Traffic 2017-01-26 2017-10-20
156320 $wgServer with initial https:// does not force HTTPS (wgSecureLogin) Traffic 2017-01-27 2017-04-28
82849 lvs servers report 'Memory allocation problem' on bootup Traffic 2017-02-11 2017-10-06
157786 Unhandled error stopping pybal: 'RunCommandMonitoringProtocol' object has no attribute 'checkCall' Traffic 2017-02-11 2017-02-10
119366 Disable caching on the main page for anonymous users Traffic 2017-02-13 2017-03-02
158599 Samsung Internet's desktop mode getting redirected to mobile site Traffic 2017-02-21 2017-07-19
159056 cp2017 froze and stopped serving traffic Traffic 2017-02-26 2017-02-27
159137 certspotter: Error retrieving STH from log Traffic 2017-02-28 2017-02-28
146332 Create short link for outreachdashboard.wmflabs.org Traffic 2017-03-01 2017-02-28
159346 convert mail servers from GS to LE certificates Traffic 2017-03-02 2017-03-02
159429 Allow setting varnish connection timeouts in puppet Traffic 2017-03-03 2017-03-07
159412 Convert all of our site.pp/roles to the role/profile paradigm Traffic 2017-03-03 2017-03-26
159411 Uniform cluster nomenclature across puppet Traffic 2017-03-03 2017-03-26
161148 AuthDNS CM/CI refactor Traffic 2017-03-23 2017-03-24
161256 multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert Traffic 2017-03-24 2017-03-28
161360 404 loading images from Virgin Media Traffic 2017-03-25 2017-03-28
114104 pybal doesn't fully manage LVS table leaving stale services (on IP change) Traffic 2017-03-29 2017-03-30
161517 Allow anonymous users to change interface language on Commons with ULS Traffic 2017-03-30 2017-07-25
79730 Add pybal check to ensure service IP is bound Traffic 2017-03-31 2017-03-30
81305 Make PyBal respect advertised BGP capabilities Traffic 2017-03-31 2017-03-30
162099 lvs2002 random shut down Traffic 2017-04-04 2017-04-07
162684 Network hardware configuration for Asia Cache DC Traffic 2017-04-12 2017-04-11
162683 Network hardware purchasing for Asia Cache DC Traffic 2017-04-12 2017-04-17
162818 icinga alerts on nodejs services when a recdns server is depooled Traffic 2017-04-13 2017-06-27
163141 dbtree: make wasat a working backend and become active-active Traffic 2017-04-18 2017-07-25
163251 Communicate dropping IE8-on-XP support (a security change) to affected editors and other community members Traffic 2017-04-19 2017-10-04
163541 cache hosts should auto-repool iff OCSP files are sane Traffic 2017-04-22 2017-04-21
164259 Add VSL error counters to Varnishkafka stats Traffic 2017-05-03 2017-08-04
164327 replace ulsfo aging servers Traffic 2017-05-03 2017-10-19
164376 [Discuss] Split ORES scores in datacenters based on wiki Traffic 2017-05-04 2017-05-11
164456 Build nginx without image filter support Traffic 2017-05-05 2017-05-09
164609 Merge cache_misc into cache_text functionally Traffic 2017-05-06 2017-06-27
164768 Explicitly limit varnishd transient storage Traffic 2017-05-09 2017-06-20
164868 SSL error for https://wikispecies.org/ Traffic 2017-05-10 2017-06-20
165252 cp1053 possible hardware issues Traffic 2017-05-16 2017-07-20
165560 Artificial spike in offset of unique devices from November to February 6th on wikidata Traffic 2017-05-18 2017-06-26
165651 Collect Google IPs pinging the load balancers Traffic 2017-05-19 2017-05-18
165764 Fully-redundant LVS clusters using Pybal per-service MED feature Traffic 2017-05-20 2017-09-19
165765 Refactor pybal/LVS config for shared failover Traffic 2017-05-20 2017-06-01
166782 wikimediafoundation.org's language selector is confusing to most visitors who don't have accounts there Traffic 2017-06-02 2017-08-24
166965 Degraded RAID on lvs3001 Traffic 2017-06-05 2017-07-03
167513 Redirect lzh.wikipedia to zh-classical.wikipedia Traffic 2017-06-10 2017-06-12
167060 en.wiki domain owned by us, but isn't hosted by us?? Traffic 2017-06-15 2017-06-14
125226 [feature request] Redirect root API path to docs page Traffic 2017-06-15 2017-06-14
167906 Make API usage limits easier to understand, implement, and more adaptive to varying request costs / concurrency limiting Traffic 2017-06-15 2017-10-20
167400 Disable serving unpatrolled new files to Wikipedia Zero users Traffic 2017-06-16 2017-09-02
167972 Respect host header in RESTBase, and redirect /rest_v1 to /rest_v1/ Traffic 2017-06-16 2017-06-20
168539 Unhandled pybal error: OpenSSL.SSL.Error - ssl handshake failure Traffic 2017-06-22 2017-06-22
168529 Upgrade to Varnish 5 Traffic 2017-06-22 2017-10-10
168699 Verify that the codfw lvs is configured correctly for Phabricator Traffic 2017-06-24 2017-08-08
169765 pybal should automatically reconnect to etcd Traffic 2017-07-06 2017-07-18
170605 ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION Traffic 2017-07-14 2017-08-09
170518 Non zero rated LVS IPs Traffic 2017-07-14 2017-08-14
170567 Support TLSv1.3 Traffic 2017-07-14 2017-07-13
170847 Icinga check for pybal HTTP connections to etcd Traffic 2017-07-18 2017-07-18
171168 cp1050 apparently stuck while "Initializing firmware interfaces..." Traffic 2017-07-21 2017-07-25
99531 [Task] move wikiba.se webhosting to wikimedia misc-cluster Traffic 2017-07-22 2017-10-17
171498 Implement machine-local forwarding DNS caches Traffic 2017-07-25 2017-09-13
171470 Monitor DNS delegations Traffic 2017-07-25 2017-07-24
171850 Backport ipvsadm Traffic 2017-07-28 2017-10-11
171967 setup/install cp4022 Traffic 2017-07-29 2017-08-18
171966 setup/install cp402[34] Traffic 2017-07-29 2017-08-18
172123 Determine how to upload Zim files to Swift infrastructure Traffic 2017-08-01 2017-08-12
172148 Determine URL paths for Zim files Traffic 2017-08-01 2017-09-12
172116 Improve OCSP fetching and monitoring strategies Traffic 2017-08-01 2017-08-01
172103 IPVS issues with UDP services, pybal depooling strategy Traffic 2017-08-01 2017-08-01
172124 PyBal Feature: progressive depooling strategy for monitored failures Traffic 2017-08-01 2017-09-06
172198 setup/install cp402[5-8].ulsfo.wmnet Traffic 2017-08-02 2017-08-21
173966 Like nan.wikipedia.org, redirect other nan.*.org to the proper zh-min-nan.*.org domains Traffic 2017-08-24 2017-08-23
174342 Missing IP addresses for Maroc Telecom Traffic 2017-08-29 2017-09-21
174432 Unclear LVS bandwidth graph in "load balancers" dashboard Traffic 2017-08-30 2017-08-30
174640 Invalid "wikimedia" family in unique devices data due to misplaced WMF-Last-Access-Global cookie Traffic 2017-09-01 2017-10-09
174891 cp4024 kernel errors Traffic 2017-09-04 2017-10-17
174960 Varnish does not vary elasticsearch query by request body Traffic 2017-09-05 2017-09-07
175203 Implement stateless TCP balancing in our LVS servers Traffic 2017-09-07 2017-09-07
175319 cp1066 unexplained 503 spikes Traffic 2017-09-09 2017-09-19
175636 prometheus -> grafana stats for per-numa-node meminfo Traffic 2017-09-12 2017-10-02
176366 Decom cp4005-8,13-16 (8 nodes) Traffic 2017-09-21 2017-10-17
176388 pybal: race condition in alerts instrumentation Traffic 2017-09-22 2017-09-28
176875 Allow access to wdqs.svc.eqiad.wmnet on port 8888 Traffic 2017-09-28 2017-10-13
176905 Evaluate requesting a rate limit change from Letsencrypt Traffic 2017-09-28 2017-09-28
177199 Add Prometheus client support for varnish/statsd metrics daemons Traffic 2017-10-03 2017-10-10
177228 Multiple systems in esams OE10 showing PSU failures Traffic 2017-10-03 2017-10-10
177927 Refactor kafka_config.rb and and kafka_cluster_name.rb in puppet to avoid explicit hiera calls Traffic 2017-10-12 2017-10-13
177961 Upgrade LVS servers to stretch Traffic 2017-10-12 2017-10-13
178151 Add UDP monitor for pybal Traffic 2017-10-14 2017-10-13
178173 Renew unified certificates 2017 Traffic 2017-10-14 2017-10-18
178423 rack/setup/install cp40(29|3[012]).ulsfo.wmnet Traffic 2017-10-18 2017-10-19
178436 rack/setup/install lvs400[567].ulsfo.wmnet Traffic 2017-10-18 2017-10-18
178535 decommission lvs400[1-4].ulsfo.wmnet Traffic 2017-10-19 2017-10-18
178592 decommission/replace bast4001.wikimedia.org Traffic 2017-10-20 2017-10-19
178567 Server error (500) while trying to download files from Commons from PAWS Traffic 2017-10-21 2017-10-20
178778 Parsoid, VisualEditor not working with SSL / HTTPS Traffic 2017-10-23 2017-10-22