TechnicalOperations Open Tasks

Help.

150547 Add visitingwatchers to watchlist_count DBA 2017-01-02 2017-12-11
125991 Add wl_timestamp to the watchlist table DBA 2017-01-02 2017-05-11
124214 Allow filtering based on tag on Special:NewFiles DBA 2017-01-02 2017-06-27
136335 Allow self-serve database credential and permissions management for Tool Labs projects DBA 2017-01-02 2018-02-14
134809 Apache <=> mariadb SSL/TLS for cross-datacenter writes DBA 2017-01-02 2018-01-18
131065 API: Contributions: Database query error DBA 2017-01-02 2017-12-11
59176 ApiQueryExtLinksUsage::run query has crazy limit DBA 2017-01-02 2018-05-15
72005 Apply enum changes to (img|oi|fa)_major_mime on production DBA 2017-01-02 2018-03-05
109964 Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections. DBA 2017-01-02 2015-09-29
133333 Audit MySQL configurations DBA 2017-01-02 2016-11-10
104459 Automatize the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves DBA 2017-01-02 2017-09-14
56401 Backups systems for userDBs hosted on replicas DBA 2017-01-02 2018-01-29
112473 Better mysql monitoring for number of connections and processlist strange patterns DBA 2017-01-02 2017-08-04
59186 blob_tracking indexes apparently unused DBA 2017-01-02 2016-07-20
121161 CentralAuthCreateLocalAccountJob failing on meta due to Echo deadlocks DBA 2017-01-02 2017-12-11
149077 Certain ApiQueryRecentChanges::run api query is too slow, slowing down dewiki DBA 2017-01-02 2017-12-11
119601 Certain tools users create multiple long running queries that take all memory and/or CPU from labsdb hosts, slowing it down and potentially crashing (tracking) DBA 2017-01-02 2018-05-11
132838 Certain wiki databases missing from replicas? DBA 2017-01-02 2017-12-11
152427 Create a check/calendar alert for MariaDB TLS certs DBA 2017-01-02 2017-02-21
140751 Create a logstash input filter to preprocess mysqld syslog messages DBA 2017-01-02 2017-12-11
119479 Create a script that'll allow easier swatting of queries that're overloading labsdb DBA 2017-01-02 2015-11-24
145072 Create a script to regenerate prometheus mysqld exporter listing that works with puppetdb DBA 2017-01-02 2017-12-11
151999 Create script to monitor db dumps for backups are successful DBA 2017-01-02 2018-03-09
133185 Database error while saving an article (1205 Lock wait exceeded in Title::invalidateCache) DBA 2017-01-02 2016-04-25
3268 Database replication lag issues (tracking) DBA 2017-01-02 2017-10-09
50930 Database replication problems - production and labs (tracking) DBA 2017-01-02 2017-03-23
54921 Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) DBA 2017-01-02 2018-05-16
71182 Database upgrade MariaDB 10: Metadata access in INFORMATION_SCHEMA causes complete blocks DBA 2017-01-02 2015-09-23
134476 Decommission old coredb machines (<=db1050) DBA 2017-01-02 2018-05-03
71127 Discrepancies with logging table on different wikis DBA 2017-01-02 2017-08-07
151681 DispatchChanges: Avoid long-lasting connections to the master DB DBA 2017-01-02 2017-12-11
141968 Display lag on grafana (prometheus) and dbtree from pt-heartbeat instead (or in addition) of Seconds_Behind_Master DBA 2017-01-02 2016-08-25
153638 Drop echo tables from local wiki databases DBA 2017-01-02 2017-07-12
144010 Drop eu_touched in production DBA 2017-01-02 2017-10-05
114117 Drop externallinks.el_from_namespace on wmf databases DBA 2017-01-02 2018-03-05
51195 Drop fj_new_sha1 field DBA 2017-01-02 2017-12-08
86338 Dropping page.page_counter on wmf databases DBA 2017-01-02 2017-04-26
86342 Dropping page.page_no_title_convert on wmf databases DBA 2017-01-02 2017-04-26
67448 Dropping rc_cur_time on wmf databases DBA 2017-01-02 2017-04-26
51191 Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases DBA 2017-01-02 2017-04-28
86339 Dropping site_stats.ss_total_views on wmf databases DBA 2017-01-02 2017-04-26
85757 Dropping user.user_options on wmf databases DBA 2017-01-02 2017-04-26
106854 Duplicate key errors (tracking) DBA 2017-01-02 2018-03-26
2323 edits where the rev_user_text and ar_user_text fields contain underscores, initial lower-case letters or consecutive spaces, which could occur in the Phase I and II software, are inaccessible using Special:Contributions DBA 2017-01-02 2017-12-08
119626 Eliminate SPOF at the main database infrastructure DBA 2017-01-02 2017-11-30
127792 Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace DBA 2017-01-02 2017-08-10
108255 Enable MariaDB/MySQL's Strict Mode DBA 2017-01-02 2018-04-27
119371 Enable MariaDB/MySQL strict mode on CI db hosts DBA 2017-01-02 2018-05-07
135414 Entries in revision table where rev_len is NULL DBA 2017-01-02 2017-12-11
141765 Error: 2013 Lost connection to MySQL server during query on NewFilesPager (again) DBA 2017-01-02 2017-05-08
141095 Explore 'Analyze' statement as substitute for Explain DBA 2017-01-02 2017-12-11
115303 Expunge old AFTv5 log entries on WMF wikis DBA 2017-01-02 2016-08-12
104699 Firewall configurations for database hosts DBA 2017-01-02 2018-04-04
33863 Fix use of DB schema so RenameUser is trivial DBA 2017-01-02 2017-09-12
126251 Get HA db support for labs internal services DBA 2017-01-02 2017-02-14
39591 History of a page was lost after several move actions on zhwp DBA 2017-01-02 2017-12-11
121857 Implement a system to automatically deploy schema changes without needing DBA intervention DBA 2017-01-02 2016-11-10
138562 Improve regular production database backups handling DBA 2017-01-02 2018-04-17
140832 Investigate moving labsdb (replicas) user credential management to 'Striker' (codename) DBA 2017-01-02 2017-08-22
116793 Investigate slow query logging/digest for Beta Cluster DBA 2017-01-02 2016-08-05
129830 large s2 database traffic DBA 2017-01-02 2016-03-18
108434 latin1 encoding in sql databases DBA 2017-01-02 2015-08-14
71222 list=logevents slow for users with last log action long time ago DBA 2017-01-02 2016-05-23
85266 Look into Maria 10 parallel-replication DBA 2017-01-02 2017-03-02
89737 Make several mediawiki table fields unsigned ints on wmf databases DBA 2017-01-02 2018-03-01
59617 Make watchlist table available as curated foo_p.watchlist_count on labsdb DBA 2017-01-02 2018-05-04
127636 MariaDB missing logrotate for error and slow logs DBA 2017-01-02 2017-12-11
142807 Migrate all users to new Wiki Replica cluster and decommission old hardware DBA 2017-01-02 2018-03-26
109179 Migrate MySQLs to use ROW-based replication DBA 2017-01-02 2018-02-07
114904 Migrate wb_items_per_site to using prefixed entity IDs instead of numeric IDs DBA 2017-01-02 2017-11-14
114903 Migrate wb_terms to using prefixed entity IDs instead of numeric IDs DBA 2017-01-02 2018-03-06
119154 Move echo tables from local wiki databases onto extension1 cluster for mediawikiwiki, metawiki, and officewiki DBA 2017-01-02 2017-02-09
112282 Multiple pages with no revisions DBA 2017-01-02 2016-09-23
127363 mysqlbinlog doesn't recognize the ssl-* options in [client] DBA 2017-01-02 2017-01-06
138517 mysqldump is timing out preventing all tables from being included in the dump DBA 2017-01-02 2017-04-20
143896 MySQL metrics monitoring DBA 2017-01-02 2017-10-09
100501 mysql user and group should be a system user/group DBA 2017-01-02 2017-06-22
138915 OTRS database is "too large" DBA 2017-01-02 2018-03-12
115081 Pages where page.page_latest = 0 on the English Wikipedia DBA 2017-01-02 2015-12-05
140302 plwiki API request is excessively slow when including a badrevid DBA 2017-01-02 2017-12-11
126252 Populate the wikishared db on all dbstores DBA 2017-01-02 2016-02-09
110115 Possible to run writes (e.g. UPDATE) on Beta Cluster replica DBA 2017-01-02 2018-05-01
135851 Preserve InnoDB table auto_increment on restart DBA 2017-01-02 2017-05-08
108968 prop=revisions with rvdir=newer gives internal_api_error_DBQueryError for page with many version DBA 2017-01-02 2017-02-06
57455 Provide dynamic report of differences between replica databases and production databases DBA 2017-01-02 2016-11-15
114476 Puppetize grants for mysql analytics servers DBA 2017-01-02 2017-02-20
111929 Puppetize grants for mysql hosts that are the source of recovery (dbstore, passive misc) DBA 2017-01-02 2017-04-12
128623 Purge all Schema:Echo data after 90 days DBA 2017-01-02 2017-12-11
71088 Queries of commonswiki_p.filearchive for fa_sha1 are slow DBA 2017-01-02 2017-10-26
131266 Querying the logging table on labs is slow DBA 2017-01-02 2016-12-04
128821 reclaim and return all cisco servers DBA 2017-01-02 2018-05-03
150548 Reduce watchlist_count threshold DBA 2017-01-02 2017-02-22
92739 Remove AFT tables from the analytics slaves DBA 2017-01-02 2016-12-14
114902 Remove numeric entity IDs from database schema DBA 2017-01-02 2017-11-17
127570 Rename be_x_oldwiki database to be_taraskwiki DBA 2017-01-02 2017-10-14
51593 Rename two indexes in the Echo extension DBA 2017-01-02 2017-04-26
126218 Replicate wikitech wikis to analytics-store.eqiad.wmnet DBA 2017-01-02 2016-02-11
101631 rev_len should be available also for deleted revisions in database replicas DBA 2017-01-02 2017-10-26
133523 [RFC] improve parsercache replication and sharding handling DBA 2017-01-02 2018-02-12
83609 script & docs to rename wiki databases DBA 2017-01-02 2017-10-14
29112 Select of revisions for stub history files does not explicitly order revisions DBA 2017-01-02 2018-05-08
141255 Separate host lookup from the sql shell script DBA 2017-01-02 2017-12-11
141547 Setup automatic failover for misc database servers DBA 2017-01-02 2016-08-17
79922 Set up backup strategy for es clusters DBA 2017-01-02 2017-04-12
107610 Setup separate logical External Store for Flow in production DBA 2017-01-02 2018-04-06
50694 Show replication lags in Graphite DBA 2017-01-02 2016-03-25
129093 SHOW SLAVE STATUS as a health check should have a low timeout DBA 2017-01-02 2016-03-08
101502 Slow API list=recentchanges query (DBError: Lost connection to MySQL server during query) DBA 2017-01-02 2016-12-20
143870 Some dump hosts are accessing main traffic servers DBA 2017-01-02 2018-04-25
17441 Some tables lack unique or primary keys, may allow confusing duplicate data DBA 2017-01-02 2018-04-03
108782 Special:Contributions: "contributions page filtered for namespace or RevisionDeleted edits" error DBA 2017-01-02 2016-07-11
68025 [Story] Monitor size of some Wikidata database tables DBA 2017-01-02 2016-05-11
97414 [Task] Use an own load section for Wikibase DB queries DBA 2017-01-02 2016-02-05
137654 Tendril DB graphs say 24hr but at 12hr DBA 2017-01-02 2016-06-14
121094 Throttling linkwatcher tool user as it is consuming 100% CPU DBA 2017-01-02 2017-04-19
51190 Truncate SHA-1 indexes DBA 2017-01-02 2017-04-26
107676 Undiscoverable log entries DBA 2017-01-02 2015-11-05
105395 UserMerge using SELECT img_name FROM image WHERE img_user = ... but img_user doesn't have an index DBA 2017-01-02 2015-07-09
151583 Use tls for dump backup generation DBA 2017-01-02 2017-12-11
95799 wfWaitForSlaves in JobRunner can massively slow down run rate if just a single slave is lagged DBA 2017-01-02 2018-05-01
147148 Wikipedia requires a patch to load its data from the dumps with mwdumper DBA 2017-01-02 2017-05-08
106151 write irc bot to report high replag of s{1,2,3}.labsdb on #wikimedia-labsdb DBA 2017-01-02 2016-06-22
155734 Investigation: Add old and new length columns to cu_changes DBA 2017-01-24 2018-02-23
156465 For switchovers: A way to check if slaves are up to date DBA 2017-01-28 2017-01-27
156461 [META ticket] Automation for our DBs tracking task DBA 2017-01-28 2017-01-27
157359 labsdb1006/1007 (postgresql) maintenance DBA 2017-02-07 2018-02-17
85756 Make (redacted) log_search table available on ToolLabs DBA 2017-02-09 2018-01-10
157702 Followup for TLS MariaDB server roll-out DBA 2017-02-10 2018-03-22
48098 Disable miser mode ($wgMiserMode) on small wikis (wikis in small.dblist) DBA 2017-02-14 2017-03-27
158166 Discuss labsdb visibility of rev_text_id and ar_comment DBA 2017-02-16 2017-03-13
68961 Logging needs an index to optimize searching by log_title DBA 2017-02-16 2017-11-20
158724 Increase size of categorylinks.cl_collation column DBA 2017-02-23 2018-03-20
159423 Meta ticket: Migrate multi-source database hosts to multi-instance DBA 2017-03-03 2018-05-15
146055 Improve privilege separation for phabricator's config files and mysql credentials DBA 2017-03-16 2018-04-23
160731 Decom db1048 (BBU Faulty - slave lagging) DBA 2017-03-18 2017-09-13
160983 Optimize SpecialAllPages::showChunk for large wikis DBA 2017-03-22 2017-12-11
89741 Expose ar_content_format and ar_content_model columns of archive table on Labs replicas DBA 2017-03-27 2018-02-09
161754 eqiad: (2) hardware access request for labsdb1004 & 5 refresh DBA 2017-03-30 2017-04-07
161755 eqiad: (2) hardware access request for labsdb1006 & 7 refresh DBA 2017-03-30 2017-04-07
162070 Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases DBA 2017-04-04 2018-02-07
162562 Choose a field length for entity id columns and stick to it DBA 2017-04-10 2017-05-05
162558 Use memcached (or something similar) to keep the latest chd_seen state, only flush to table every once in a while DBA 2017-04-10 2017-05-05
163339 pdu phase inbalances: ps1-a3-codfw, ps1-c6-codfw, & ps1-d6-codfw DBA 2017-04-20 2017-05-08
164167 change_tag table needs redesign DBA 2017-05-01 2018-01-29
164382 Evaluate the need for FORCE INDEX (ls_field_val) [now IGNORE INDEX (ls_log_id)], delete the index hint if not needed anymore DBA 2017-05-04 2017-12-11
117272 Wikidata: Possibility to add comments with more than 255 bytes, potential db issue DBA 2017-05-05 2017-05-04
164504 Tracking: Cleanup x1 database connection patterns DBA 2017-05-09 2018-02-21
164834 In some database hosts, performance schema loses digest statistics DBA 2017-05-10 2017-10-04
165358 Set up and package wmfmariadbpy helper scripts so they can easily be deployed to all database server and client hosts DBA 2017-05-16 2018-02-07
164810 Switch phabricator production to codfw DBA 2017-05-17 2018-04-01
165625 Evaluate future of wmf puppet module "mysql" DBA 2017-05-18 2018-05-14
165677 Create a backend check for pybal to monitor the MySQL protocol being up DBA 2017-05-19 2017-05-25
165674 Investigate slow servermon updating queries on db1016 DBA 2017-05-19 2018-02-14
165756 Create summary templates on Wikitech wiki to stop writing the same things everywhere, everytime DBA 2017-05-20 2017-09-12
161066 Merge pagelinks, templatelinks and imagelinks tables DBA 2017-05-20 2017-05-24
166108 x1 master db1031: Faulty BBU DBA 2017-05-24 2018-01-19
166798 Decide whether back-compat views for upcoming major schema changes will be provided in the Labs replicas DBA 2017-06-02 2017-09-07
167973 Move wikitech and labstestwiki to s5 DBA 2017-06-16 2018-03-13
168349 enwiki_p logging vs logging_userindex returning dramatically different results DBA 2017-06-21 2017-09-12
143366 Removal of {{#coordinates:}} leaves DB entries behind DBA 2017-07-13 2018-03-02
171071 Perform testing for TLS effect on connection rate DBA 2017-07-20 2018-04-06
170508 The "show ip" action should also provide a distinct list of user-agents for each IP DBA 2017-07-20 2017-11-22
171643 Clean up skin properties DBA 2017-07-26 2018-02-01
171570 Rename database labswiki to wikitech DBA 2017-07-26 2017-10-14
171898 Deleting large watchlist takes > 4 seconds causing rollback due to write time limit DBA 2017-07-29 2017-09-10
163681 Use a random timestamp on Special:NewFiles on Commons doesn't loads the page (504 Gateway Timeout) DBA 2017-08-02 2017-12-11
172497 Change pt-heartbeat model to not use super-user, avoid SPOF and switch automatically to the real master without puppet dependency DBA 2017-08-05 2017-08-07
172492 Improve database alerting (tracking) DBA 2017-08-05 2017-10-09
172489 Monitor read_only variable and/or uptime on database masters, make it page DBA 2017-08-05 2017-10-09
172490 Monitor swap/memory usage on databases DBA 2017-08-05 2017-08-07
173511 Implement technical details and process for "datasets_p" on wikireplica hosts DBA 2017-08-23 2017-11-15
174648 Create MW Schema Diff maintenance script DBA 2017-09-01 2017-09-12
113842 `pr_index` to be replicated to Labs public databases DBA 2017-09-01 2018-04-26
174802 Drop education program (ep_*) tables on wikis where it is not enabled DBA 2017-09-02 2017-09-01
175672 Make apache/maintenance hosts TLS connections to mariadb work DBA 2017-09-13 2018-05-08
176277 Provision a separate DB shard for wbc_entity_usage DBA 2017-09-20 2017-09-25
176532 Gerrit is failing to connect to db on gerrit2001 thus preventing systemd from working DBA 2017-09-24 2018-03-22
174044 Deploy MCR storage layer DBA 2017-09-26 2018-05-15
157762 Drop ptrl_comment in production DBA 2017-10-02 2018-03-05
177779 Generate instance list of database hosts to be monitored automatically from exported resources DBA 2017-10-10 2017-10-10
177778 Improve database application performance monitoring visibility DBA 2017-10-10 2018-01-09
177782 Reduce false positives on database pages DBA 2017-10-10 2017-11-16
178290 Check recentchanges table and query errors on wikis other than commonswiki and ruwiki DBA 2017-10-17 2018-03-16
178877 operations/software repo: flake8 check DBA 2017-10-25 2017-10-25
179010 Re-enable Wikidata Recent Changes integration on Commons DBA 2017-10-26 2018-05-09
180158 Investigate using macbre/index-digest over WMF MW sql query workload DBA 2017-11-10 2017-11-10
180636 Make Dispenser's principle_links table accessible in new Wiki replica cluster DBA 2017-11-16 2017-12-11
182563 Identify and delete non-useful files on database hosts taking space unnecesarily DBA 2017-12-12 2017-12-11
180979 List of Newsletters should have a column showing the number of issues DBA 2017-12-21 2017-12-21
183983 Re-institute query killer for the analytics WikiReplica DBA 2018-01-03 2018-03-22
184054 Decommission db1029 and db1031 DBA 2018-01-04 2018-05-07
184540 Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running DBA 2018-01-10 2018-01-18
184615 Once MCR is deployed, drop the rev_text_id, rev_content_model, and rev_content_format fields to be dropped from revision DBA 2018-01-11 2018-05-15
184805 Move some wikis to s5 DBA 2018-01-13 2018-02-21
185084 Allow use of EtcdConfig to configure slave databases DBA 2018-01-18 2018-02-07
185673 Add base36 functions to ToolForge database DBA 2018-01-26 2018-02-07
173943 Display count of remaining content space errors DBA 2018-02-01 2018-02-07
186188 Failover DB masters in row D DBA 2018-02-02 2018-04-25
186320 Decommission db1051-db1060 (DBA tracking) DBA 2018-02-03 2018-05-21
187089 Fix WMF schemas to not break when comment store goes WRITE_NEW DBA 2018-02-13 2018-04-17
187295 Apply AbuseFilter patch-fix-index DBA 2018-02-15 2018-05-21
187579 Create ip_logging table to query for logged actions by IP ranges DBA 2018-02-17 2018-02-16
187584 Investigation: Using log_search to query for logged actions against IPs in a given range DBA 2018-02-17 2018-03-06
187521 Optimize recentchanges and wbc_entity_usage table across wikis DBA 2018-02-17 2018-04-17
147894 Create index for cu_agents in cu_changes table DBA 2018-02-25 2018-02-25
188299 Schema change for refactored actor storage DBA 2018-02-27 2018-04-18
189026 IndexPager::buildQueryInfo (contributions page unfiltered) query needs tuning DBA 2018-03-07 2018-04-28
188992 Make wb_terms table fancy DBA 2018-03-07 2018-05-09
189107 DB meta task for next DC failover issues DBA 2018-03-08 2018-03-20
190148 Change DEFAULT 0 for rev_text_id on production DBs DBA 2018-03-21 2018-05-07
190704 Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy DBA 2018-03-27 2018-05-18
191316 Schema change to make archive.ar_rev_id NOT NULL DBA 2018-04-04 2018-04-24
191391 Apply schema changes to an isolated database and examine the results DBA 2018-04-05 2018-05-22
191519 Schema change for rc_namespace_title_timestamp index DBA 2018-04-06 2018-04-18
191581 Nullify ct_rc_id and ts_rc_id when recent changes entries are being deleted DBA 2018-04-07 2018-04-06
192358 Setup database logical backups on eqiad DBA 2018-04-18 2018-05-04
173917 Cannot add a previously used change tag to an abuse filter DBA 2018-04-23 2018-05-01
192875 Debmonitor: request for misc DB allocation DBA 2018-04-25 2018-05-03
192926 Schema change to drop archive.ar_text and archive.ar_flags DBA 2018-04-25 2018-05-15
192979 Productionize 8 eqiad hosts DBA 2018-04-26 2018-05-16
193187 Prepare and check storage layer for idwikimedia DBA 2018-04-27 2018-05-21
193224 Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished DBA 2018-04-28 2018-04-27
193226 Test MySQL 8.0 with production data and evaluate its fit for WMF databases DBA 2018-04-28 2018-04-27
179604 Move AbuseFilter slow filters data from Logstash to per-filter profiling DBA 2018-04-29 2018-04-28
193736 Decommission db1056 DBA 2018-05-04 2018-05-08
193732 Decommission db1060 DBA 2018-05-04 2018-05-15
194118 Decommission db1055 DBA 2018-05-09 2018-05-15
194273 Clean up indexes of wb_terms table DBA 2018-05-10 2018-05-22
194270 Drop 'tmp1' index from wb_terms table in production DBA 2018-05-10 2018-05-22
194343 quentinv57-tools/tools/globalcontribs.php generates slow/complex SQL queries which impact server performance DBA 2018-05-11 2018-05-18
194634 Decommission db1053 DBA 2018-05-15 2018-05-16
194780 rack/setup/install db112[45].eqiad.wmnet (sanitarium expansion) DBA 2018-05-16 2018-05-18
194781 rack/setup/install db209[45].codfw.wmnet (sanitarium expansion) DBA 2018-05-16 2018-05-18
194870 Failover s2 primary master DBA 2018-05-18 2018-05-21
195008 Prepare and check storage layer for pmswikisource DBA 2018-05-19 2018-05-21
195228 db2064 crashed and totally broken - decommission it DBA 2018-05-22 2018-05-22
139775 eqiad: add all spare network switches to hardware spares tracking Hardware Requests 2017-01-02 2017-04-27
178392 Replacement hardware for cumin masters Hardware Requests 2017-10-18 2018-05-03
181264 Refresh or replace oxygen Hardware Requests 2017-11-24 2018-04-05
181936 Give misc dump crons their own host Hardware Requests 2017-12-05 2018-05-10
98006 Anycast (Auth)DNS Network Operations 2017-01-02 2017-07-24
122406 Consider renumbering Labs to separate address spaces Network Operations 2017-01-02 2018-05-16
82038 create a test for multicast relay Network Operations 2017-01-02 2017-06-27
120425 dumps.wikimedia.org seems to have poor throughput towards some destinations Network Operations 2017-01-02 2018-04-21
133387 Enabling IGMP snooping on QFX switches breaks IPv6 (HTCP purges flood across codfw) Network Operations 2017-01-02 2017-08-03
150264 Icinga check for VRRP Network Operations 2017-01-02 2017-06-27
136671 Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR Network Operations 2017-01-02 2018-02-04
83992 Juniper monitoring Network Operations 2017-01-02 2017-06-27
83119 Netflow Collector Project Network Operations 2017-01-02 2017-06-27
124843 Peer with SFMIX at ULSFO in 200 Paul Network Operations 2017-01-02 2017-06-27
122210 Security audit for tftp on install1001 Network Operations 2017-01-02 2018-03-22
106056 set up a looking glass for WMF ASes Network Operations 2017-01-02 2018-02-06
86541 setup wifi in codfw Network Operations 2017-01-02 2018-02-06
163674 Frequent RST returned by appservers to LVS hosts Network Operations 2017-04-25 2017-11-09
167306 ospf link-protection Network Operations 2017-06-08 2017-06-27
167691 High amount of unexpected ICMP dest unreachable toward esams cache clusters Network Operations 2017-06-13 2017-11-09
167841 Cleanup confed BGP peerings and policies Network Operations 2017-06-14 2017-06-27
167842 Find a new PIM RP IP Network Operations 2017-06-14 2017-10-20
170144 Evaluate NetBox as a Racktables replacement & IPAM Network Operations 2017-07-11 2018-03-20
171032 Investigate lvs IP pages during codfw row C switch upgrade Network Operations 2017-07-20 2017-07-20
172459 eqiad row D switch upgrade Network Operations 2017-08-04 2018-02-07
173698 Backfill librenms data in graphite with historical RRDs Network Operations 2017-08-22 2017-08-21
174596 dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour Network Operations 2017-08-31 2017-08-30
174616 set up cr3-esams Network Operations 2017-08-31 2018-01-03
174637 Setup esams atlas anchor Network Operations 2017-09-01 2018-01-03
180179 Evaluate the possibility to add Juniper images to Openstack Network Operations 2017-11-10 2018-01-12
181036 Pull netflow data in realtime from Kafka via Tranquillity/Spark Network Operations 2017-11-22 2018-04-19
183390 unrack/decom pfw1-eqiad and pfw2-eqiad Network Operations 2017-12-21 2018-05-03
184067 Complete router migration from cr1-esams to cr3-esams Network Operations 2018-01-04 2018-01-03
185151 replace msw1-esams Network Operations 2018-01-18 2018-02-02
185171 replace mr1-eqiad Network Operations 2018-01-19 2018-04-24
185337 rack spare switches in c1-eqiad Network Operations 2018-01-20 2018-03-14
185926 review and fix scs config Network Operations 2018-01-30 2018-01-29
186021 reconfigure esams switch port for new bastion Network Operations 2018-01-31 2018-01-30
186550 Anycast recdns Network Operations 2018-02-06 2018-02-13
187820 Rack/cable/configure mr1-eqiad Network Operations 2018-02-21 2018-02-20
187929 Cloud IPv6 subnets Network Operations 2018-02-22 2018-02-21
187960 Rack/cable/configure asw2-a-eqiad switch stack Network Operations 2018-02-22 2018-04-24
187962 Rack/cable/configure asw2-c-eqiad switch stack Network Operations 2018-02-22 2018-05-18
189519 Audit switch ports/descriptions/enable Network Operations 2018-03-13 2018-04-24
189522 Detect IP address collisions Network Operations 2018-03-13 2018-03-13
189552 Rack/cable/configure ulsfo MX204 Network Operations 2018-03-14 2018-05-02
189689 Connection timeout from 195.77.175.64/29 to text-lb.esams.wikimedia.org Network Operations 2018-03-15 2018-03-14
190090 Offload pings to dedicated server Network Operations 2018-03-20 2018-04-03
190184 Netbox: setup backups Network Operations 2018-03-21 2018-03-20
190364 eqiad 10G ports needs Network Operations 2018-03-22 2018-03-26
190424 modify labs-hosts1-vlans for http load of installer kernel Network Operations 2018-03-23 2018-03-22
191371 Enabling graceful-switchover causes core dumps on cr1-codfw Network Operations 2018-04-05 2018-04-06
191667 Juniper HA audit Network Operations 2018-04-07 2018-04-11
193496 Allocate public v4 IPs for Neutron setup in eqiad Network Operations 2018-05-02 2018-05-16
194515 eqdfw: Patch GTT cross-connect Network Operations 2018-05-12 2018-05-11
116750 2FA for SSH access to the production cluster Other Operations 2017-01-02 2015-12-18
140879 503 error raises again while trying to load a Wikidata page Other Operations 2017-01-02 2017-06-23
139971 access_new_install role vs. Labs vs. the future Other Operations 2017-01-02 2016-07-11
140316 Add granularity limiter (g=) to wikimedia.org DKIM record(s) Other Operations 2017-01-02 2016-08-05
141783 Add monitoring for detecting when logstash services are down Other Operations 2017-01-02 2016-08-01
128715 Add other Tools administrators to the Icinga notification group Other Operations 2017-01-02 2016-12-04
84279 Admin module should allow group management of system users Other Operations 2017-01-02 2015-01-30
110171 Alert when ES indexes are freezed for more than 30 minutes Other Operations 2017-01-02 2016-12-08
118331 Alert when used_memory gets too high for redis queues Other Operations 2017-01-02 2015-11-11
95801 Allow customizing the alert message from graphite Other Operations 2017-01-02 2016-11-18
153816 apache::static_site is not working Other Operations 2017-01-02 2016-12-22
56515 Apply editing rate limits for all users Other Operations 2017-01-02 2018-04-12
114337 Assign 3 more servers to video scaler duty Other Operations 2017-01-02 2018-02-23
136562 Audit/fix hosts with no RAID configured Other Operations 2017-01-02 2018-05-07
94819 Audit racktables Other Operations 2017-01-02 2016-03-17
141756 audit / test / upgrade hp smartarray P840 firmware Other Operations 2017-01-02 2018-04-23
130709 authoritative copy of 'root' files for upload.wikimedia.org is only in swift Other Operations 2017-01-02 2017-03-29
135991 Automated service restarts for common low-level system services Other Operations 2017-01-02 2016-06-21
118829 Automate the provisioning and management of MediaWiki clusters Other Operations 2017-01-02 2015-11-17
138496 bring swift eqiad to one zone per row Other Operations 2017-01-02 2016-06-23
128590 Cassandra uses default ip address for outbound packets while bootstrapping Other Operations 2017-01-02 2016-12-22
137176 catch-all apache vhost on the cluster should return 404 for non-existing sites Other Operations 2017-01-02 2016-11-24
113792 Change LDAP cn to something more useful (was Rename "Dzahn" to "Daniel Zahn" in Gerrit) Other Operations 2017-01-02 2017-04-14
119274 Check incoming requests to secure.wm.o Other Operations 2017-01-02 2015-12-08
98984 Check power supply balance settings on cp3030+ Other Operations 2017-01-02 2018-01-03
144933 Cleanup debconf handling in mailman puppet setup Other Operations 2017-01-02 2016-11-29
111540 Clean up labs graphite datapoints Other Operations 2017-01-02 2016-11-18
142002 Clean up puppet & configs for ORES Other Operations 2017-01-02 2016-09-22
130617 Collect metrics on pool counter usage Other Operations 2017-01-02 2016-11-03
133913 Completely port l10nupdate to scap Other Operations 2017-01-02 2017-09-26
124413 confctl should provide tags information after writing data Other Operations 2017-01-02 2016-01-22
150460 Configure maps cluster to send statsd metrics to the statsd endpoint in the same datacenter Other Operations 2017-01-02 2016-11-10
150875 Confirm attribution needs Other Operations 2017-01-02 2017-06-29
129847 conftool-merge should report which node is setting attributes for Other Operations 2017-01-02 2016-03-14
153068 Consider mounting labs NFS labstore1003.eqiad.wmnet:/scratch for server-side uploads Other Operations 2017-01-02 2016-12-13
132104 Consider moving policy.wikimedia.org away from WordPress.com Other Operations 2017-01-02 2016-06-08
134811 Consider REST with SSL (HyperSwitch/Cassandra) for session storage Other Operations 2017-01-02 2017-07-12
105780 Create a doc explaining the SLA between services and the monitoring tool Other Operations 2017-01-02 2017-07-11
138799 Create a simple puppet role for setting up a singlenode kubernetes install Other Operations 2017-01-02 2016-06-27
134551 Create functional cluster checks for all services (and have them page!) Other Operations 2017-01-02 2018-04-13
146657 create notifications about user accounts that have not been used for a long time Other Operations 2017-01-02 2016-09-26
152439 cronspam from labtestservices2001 /etc/dns-floating-ip-updater.py > /dev/null Other Operations 2017-01-02 2016-12-05
152724 Current state and next steps for RESTBase storage Other Operations 2017-01-02 2017-07-12
87790 decom amslvs1-4 (dc work) Other Operations 2017-01-02 2018-01-30
130883 decom cp3011-22 (12 machines) Other Operations 2017-01-02 2018-05-03
95742 Decomission amssq31-62 (32 hosts) Other Operations 2017-01-02 2018-05-03
94215 decommission cp3001 & cp3002 Other Operations 2017-01-02 2018-01-03
145065 Decrease time required to fully restart the Cirrus elasticsearch clusters Other Operations 2017-01-02 2017-10-24
131966 Default gateway unreachable on baham.wikimedia.org after reboot Other Operations 2017-01-02 2017-07-20
67270 Default license for operations/puppet Other Operations 2017-01-02 2018-02-19
135124 Deploy etcddump (or another etcd dump & load tool) to production Other Operations 2017-01-02 2016-05-12
150486 Deploy federation for Prometheus Other Operations 2017-01-02 2016-11-15
148693 Deploy IDS rendering engine to production Other Operations 2017-01-02 2017-08-26
149057 Designate seems very slow to delete records? Other Operations 2017-01-02 2016-10-24
140270 Determine a core set or a checklist of permissions for deployment purpose Other Operations 2017-01-02 2017-05-16
141128 determine/process/document bios firmware tracking/updating policies Other Operations 2017-01-02 2018-01-28
125411 Diamond load averages do not contain scaled versions Other Operations 2017-01-02 2017-07-25
110240 [Discussion] Consider validating JSON schemas when running x-ample tests? Other Operations 2017-01-02 2016-10-12
153416 docker-engine pulled into our repositories only keeps the latest version Other Operations 2017-01-02 2017-01-04
127797 document all puppet classes / defined types!? Other Operations 2017-01-02 2016-11-28
115757 document debian packaging guidelines Other Operations 2017-01-02 2016-05-23
133164 Document eqiad/codfw transition plan for OCG Other Operations 2017-01-02 2016-05-23
102575 document graphite failover/backfill procedures Other Operations 2017-01-02 2016-05-23
135318 Document how to handle 'inconsistent state within the internal storage backends' issues Other Operations 2017-01-02 2016-07-31
101585 document redis upgrade/restart procedures Other Operations 2017-01-02 2016-05-23
116805 DomainKeys Identified Mail (DKIM) for phabricator.wikimedia.org Other Operations 2017-01-02 2016-08-31
92471 enable authenticated access to Cassandra JMX Other Operations 2017-01-02 2018-03-02
94951 Enable the usage of `hhvm -m debug --debug-host ::1` from mw1017 so developers can step through code (think gdb) in production to see what is going wrong. Other Operations 2017-01-02 2016-12-01
142991 Enable "upload by url" feature at zhwiki Other Operations 2017-01-02 2017-06-08
136312 Encrypt syslog traffic Other Operations 2017-01-02 2018-05-04
119719 Enforce a minimum refresh period for grafana dashboards hitting graphite Other Operations 2017-01-02 2016-11-18
142827 Enforce reference to Phabricator task for all commits to modules/admin/data/data.yaml Other Operations 2017-01-02 2017-03-10
142815 Enhance account handling (meta bug) Other Operations 2017-01-02 2017-03-28
109089 EPIC: Cultivating the Elasticsearch garden (operational lessons from 1.7.1 upgrade) Other Operations 2017-01-02 2016-10-20
137616 Epic: cultivating the Maps garden Other Operations 2017-01-02 2016-12-15
67394 [EPIC] Performance testing environment Other Operations 2017-01-02 2016-03-11
150871 [EPIC] (Proposal) Replicate core OCG features and sunset OCG service Other Operations 2017-01-02 2018-02-23
133744 Epic: switch Maps to production status Other Operations 2017-01-02 2016-07-19
125442 es2009 degraded RAID Other Operations 2017-01-02 2016-04-22
124185 Evaluate alternative web interfaces to icinga 1 core Other Operations 2017-01-02 2017-04-18
124991 evaluate possibility for nscd use with useldap Other Operations 2017-01-02 2017-07-24
141524 eventbus should send statsd in batches Other Operations 2017-01-02 2016-11-18
152632 Explore hosting the multimedia commons use case Other Operations 2017-01-02 2017-12-18
100777 expose hosts in maintenance state so we can prevent scap from running on them Other Operations 2017-01-02 2018-02-01
138821 extend existing graphite whisper files retention to five years Other Operations 2017-01-02 2016-11-18
151045 Extending Yubico 2FA for production use (meta bug) Other Operations 2017-01-02 2016-11-18
147923 Extract metrics from logs Other Operations 2017-01-02 2018-03-28
148061 Feasibility of hosting podcast setup on Wikimedia servers Other Operations 2017-01-02 2016-10-13
148986 Firewall sets not being loaded post-reboot due to a @resolve race on jessie Other Operations 2017-01-02 2017-05-31
84163 Fix CirrusSearch monitoring Other Operations 2017-01-02 2017-01-22
107108 Flow notification links on mobile point to desktop Other Operations 2017-01-02 2017-07-25
151046 Fully puppetise yubikey-val Other Operations 2017-01-02 2016-11-18
145885 Gerrit shows HTTP 500 error when pasting extended unicode characters Other Operations 2017-01-02 2018-04-24
128615 Get rid of Tool Labs home page check from shinken Other Operations 2017-01-02 2016-12-04
118746 Goal: Strengthen Incident monitoring infrastructure Other Operations 2017-01-02 2016-08-13
148843 GPU upgrade for stats machine Other Operations 2017-01-02 2018-05-21
86546 graphite-web logs are not rotated Other Operations 2017-01-02 2016-11-18
134237 Graphoid returns a 400 on MW API time-out Other Operations 2017-01-02 2016-10-12
116063 Hardware Automation Workflow - Overall Tracking Other Operations 2017-01-02 2018-01-08
133656 Have a paging check for Nova API accessible Other Operations 2017-01-02 2016-12-21
130590 Have dedicated master nodes for elasticsearch Other Operations 2017-01-02 2016-11-03
149287 Heating alerts for mw servers in eqiad Other Operations 2017-01-02 2018-05-16
133674 HHVM is leaking memory on the API appservers Other Operations 2017-01-02 2016-10-10
133091 Highest SSTables / read thresholds Other Operations 2017-01-02 2017-07-11
146090 High failure rate of account creation should trigger an alarm / page people Other Operations 2017-01-02 2017-05-20
133318 High levels of PoolCounter errors should trigger alerts Other Operations 2017-01-02 2018-01-09
98831 Honor DNT header for access logs & varnish logs Other Operations 2017-01-02 2018-02-14
148614 Icinga check for Tor Other Operations 2017-01-02 2018-05-15
151048 Icinga monitoring for Yubikey components Other Operations 2017-01-02 2017-09-07
150300 icinga notification if elevated writing to badpass.log Other Operations 2017-01-02 2018-05-03
141038 implement icinga paging for non-ops teams Other Operations 2017-01-02 2016-07-21
124179 Improve access to and control over incident and metrics monitoring infrastructure Other Operations 2017-01-02 2017-03-08
138017 Improve automation around Maps servers Other Operations 2017-01-02 2016-12-15
133844 Improve Elasticsearch icinga alerting Other Operations 2017-01-02 2017-12-02
116627 Include 5xx numbers in fluorine fatalmonitor Other Operations 2017-01-02 2015-10-28
137939 Increase frequency of OSM replication Other Operations 2017-01-02 2017-06-06
135125 Install a second etcd cluster in codfw Other Operations 2017-01-02 2016-05-12
140141 Install mscorefonts on scaling servers for SVG rendering Other Operations 2017-01-02 2017-07-25
151047 Integrate Yubikey into data.yaml Other Operations 2017-01-02 2016-11-18
150822 Internal PKI for secure communication - Barcelona Ops offsite 2016 Other Operations 2017-01-02 2017-02-22
135385 investigate carbon-c-relay stalls/drops towards graphite2002 Other Operations 2017-01-02 2016-11-18
133093 Investigate idle appservers in codfw Other Operations 2017-01-02 2016-04-19
118812 Investigate mysterious_sysctl settings and figure out what to do with them Other Operations 2017-01-02 2015-11-17
123560 investigate rsync between dcs with encryption Other Operations 2017-01-02 2016-05-30
130593 investigate slapd memory leak Other Operations 2017-01-02 2017-06-30
149885 Investigate Swift as a storage backend for maps tiles Other Operations 2017-01-02 2016-12-09
140075 investigate swift used space spikes since June 2016 Other Operations 2017-01-02 2016-07-12
109090 Investigate the need for master only (non data nodes) in our ES cluster Other Operations 2017-01-02 2016-12-08
95053 ircecho should accept input via unix sockets Other Operations 2017-01-02 2016-07-15
152782 Kibana functionality missing after upgrade: histograms Other Operations 2017-01-02 2017-05-20
120377 labmon1001 graphite instance archiver keeps archiving the same instances Other Operations 2017-01-02 2016-11-18
153279 labnet/ labtestnet2001 - disk space - nova-api.log needs rotation Other Operations 2017-01-02 2016-12-15
117673 labs precise and jessie instance not accessible after provisioning Other Operations 2017-01-02 2016-06-21
64987 librsvg misinterpret quoted font family names that contain whitespaces Other Operations 2017-01-02 2017-10-25
116767 limit the impact of heavy/large graphite queries Other Operations 2017-01-02 2017-01-27
151314 logrotate failing with $FILE.1.gz: File exists Other Operations 2017-01-02 2016-11-22
153940 Logrotate fails for: "$FILE No such file or directory" Other Operations 2017-01-02 2017-03-14
149421 Long running mediawiki web requests impacts service availability, specially databases Other Operations 2017-01-02 2018-03-23
148017 lvs2002 repeated usb connect/disconnect message Other Operations 2017-01-02 2017-04-04
146627 Make deployment-prep puppetmaster more similar to Production puppetmaster Other Operations 2017-01-02 2016-10-05
143552 Make elasticsearch configuration more robust to loss of network connectivity Other Operations 2017-01-02 2016-08-30
128716 Make icinga-wm report Tools homepage check at #wikimedia-labs, too Other Operations 2017-01-02 2016-05-27
95052 Make ircecho much better Other Operations 2017-01-02 2016-07-16
76203 Make ircecho run as its own user Other Operations 2017-01-02 2016-07-16
119718 Make it easier to ban misbehaving dashboards from graphite Other Operations 2017-01-02 2016-11-18
120585 Make l10nupdate user a system user Other Operations 2017-01-02 2017-09-26
117508 Make ops-l a list for humans again (no cheating) Other Operations 2017-01-02 2015-11-02
104352 Make scap able to depool/repool servers via the conftool API Other Operations 2017-01-02 2017-09-26
113785 Make the Shinken IRC alert and icinga-wm bots use colors Other Operations 2017-01-02 2015-11-02
141520 "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) Other Operations 2017-01-02 2017-05-20
126989 MediaWiki logging & encryption Other Operations 2017-01-02 2017-07-10
116747 Meta task "Revamp user authentication" Other Operations 2017-01-02 2016-08-13
87220 Minimize differences between beta and production (Tracking) Other Operations 2017-01-02 2017-09-01
152767 Missing Labs hiera entry in labs-private repo Other Operations 2017-01-02 2017-08-30
135595 mod_deflate + mod_uwsgi causing mangled apache responses Other Operations 2017-01-02 2016-05-18
123809 Module uwsgi doesn't allow passing multiple config params of same name Other Operations 2017-01-02 2016-01-15
86552 Monitor and alarm on SMART attributes Other Operations 2017-01-02 2018-02-12
108985 Monitor MediaWiki sessions Other Operations 2017-01-02 2015-12-02
116580 monitor postgresql replication status Other Operations 2017-01-02 2015-10-26
110169 Monitor redis memory/disk usage Other Operations 2017-01-02 2016-10-03
106937 Monitor [[Special:ListFiles]] for non 200 HTTP statuses in thumbnails Other Operations 2017-01-02 2017-08-03
136311 Monitor the BMC's event log for hardware errors Other Operations 2017-01-02 2016-05-26
136403 Move cp3030+ from OE14 to OE13 in racktables Other Operations 2017-01-02 2016-06-21
114446 move human users out of UID range for system accounts Other Operations 2017-01-02 2017-12-06
95054 Move ircecho config file to be YAML Other Operations 2017-01-02 2016-07-15
152445 Move prometheus entry point off port 80 Other Operations 2017-01-02 2017-07-10
115899 Move scap target configuration to etcd Other Operations 2017-01-02 2016-10-05
141959 Moving network::external to hiera broke much of labs Other Operations 2017-01-02 2016-08-03
129188 mw2212 unresponsive Other Operations 2017-01-02 2016-03-08
118677 Nastaleeq font for Western Punjabi Other Operations 2017-01-02 2018-02-06
121240 Network isolation for production and semi-production services Other Operations 2017-01-02 2017-07-12
56713 Non-NDA users cannot access graphite.wikimedia.org Other Operations 2017-01-02 2016-06-10
138685 notebook1001 shown as DOWN in icinga, due to firewall rules Other Operations 2017-01-02 2016-06-26
88730 Nutcracker needs to automatically recover from MC failure - rebalancing issues Other Operations 2017-01-02 2018-04-16
111934 Nutcracker stats monitoring should only listen on localhost Other Operations 2017-01-02 2016-12-01
97524 ocg alarm ocg_job_status_queue 'flapping' Other Operations 2017-01-02 2016-10-03
135338 On Trusty and Jessie PHP yields: PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/20-xhprof.ini on line 2 Other Operations 2017-01-02 2017-07-01
114801 operations-apache-config-lint replacement doesn't check syntax Other Operations 2017-01-02 2017-12-07
146968 OTRS spam classification methods and systems Other Operations 2017-01-02 2017-10-11
126083 overhaul labstore setup [tracking] Other Operations 2017-01-02 2017-07-23
150396 Phabricator leaving old files in /tmp Other Operations 2017-01-02 2017-01-03
127054 pinentry-gtk2 pulls in a lot of unneeded Gnome/GTK libs Other Operations 2017-01-02 2016-02-16
129180 Preserve SSH host key when re-imaging hosts Other Operations 2017-01-02 2016-03-07
93138 Procure hardware for Sentry Other Operations 2017-01-02 2016-01-06
151050 Proper documentation for Yubico 2FA for production use Other Operations 2017-01-02 2016-11-18
133476 Proposal: Centralize OTRS login methodology Other Operations 2017-01-02 2017-08-15
140813 Protect sensitive user-related information with a UserData / auth / session service Other Operations 2017-01-02 2017-12-20
122917 Provide a good download service of dumps from Wikimedia Other Operations 2017-01-02 2016-07-19
151009 Provide authenticated access to Prometheus native web interface Other Operations 2017-01-02 2018-05-11
123237 Provide production jessie image with node 4.2; use this for service-runner build command Other Operations 2017-01-02 2016-10-12
150466 publish kartotherian / tilerator metrics by cluster Other Operations 2017-01-02 2016-11-10
150823 Puppet CA rollover Other Operations 2017-01-02 2016-11-29
153246 Puppet failures with "Attempt to assign to a reserved variable name: 'trusted'" Other Operations 2017-01-02 2017-09-11
126574 puppet should try to mount all mountable swift filesystems Other Operations 2017-01-02 2016-02-11
136094 Race condition in setting net.netfilter.nf_conntrack_tcp_timeout_time_wait Other Operations 2017-01-02 2017-08-09
135113 Rationalize our jobqueues redis topology Other Operations 2017-01-02 2017-07-07
146841 Reach out to Google about @yahoo.com emails not reaching gmail inboxes (when sent to mailing lists) Other Operations 2017-01-02 2018-05-11
127825 Re-add intel-microcode Other Operations 2017-01-02 2018-01-05
119846 Redirect revisions from svn.wikimedia.org to https://phabricator.wikimedia.org/rSVN Other Operations 2017-01-02 2015-11-30
133179 Redis monitoring needs to be improved Other Operations 2017-01-02 2017-07-07
135122 Reduce etcd technical debt Other Operations 2017-01-02 2016-05-12
109606 Re-evaluate Limesurvey Other Operations 2017-01-02 2017-06-05
150020 Refactor puppet-postgresql module to use custom types Other Operations 2017-01-02 2016-12-09
148647 refresh swift hardware in codfw/eqiad Other Operations 2017-01-02 2017-03-16
131748 Refresh the appservers puppet code/configs Other Operations 2017-01-02 2016-04-04
140442 reinstall rdb100[56] with RAID Other Operations 2017-01-02 2018-05-07
120856 Remove all out of warranty unused cp10xx's from A2 Other Operations 2017-01-02 2017-07-20
150917 Remove deprecated features from book creator UI Other Operations 2017-01-02 2018-02-15
144539 Remove /srv/deployment/wdqs/wdqs/rules.log symlink Other Operations 2017-01-02 2018-03-06
147872 Rename rhodium to puppetmaster1003 Other Operations 2017-01-02 2016-10-11
146355 Replace etcd internal auth mechanism with a frontend proxy Other Operations 2017-01-02 2016-09-22
134271 Replace ircd-ratbox with something newer/maintained Other Operations 2017-01-02 2018-01-19
150872 Replace OCG in collection extension with Electron Other Operations 2017-01-02 2017-10-02
116951 Reprepro should bail if it can't read and sign using the root keys Other Operations 2017-01-02 2015-10-28
125015 Requests to (hard) redirect pages return their target's contents but are counted as pageviews to the redirect page Other Operations 2017-01-02 2016-01-28
144431 RESTBase k-r-v as Cassandra anti-pattern Other Operations 2017-01-02 2018-04-13
149643 Review Icinga alarms with disabled notifications Other Operations 2017-01-02 2017-11-15
142984 Review lists of config/sysctl recommendations by "kernel self-protection project" Other Operations 2017-01-02 2016-08-15
141897 Review new service 'pre-deployment to production' checklist Other Operations 2017-01-02 2017-07-12
149804 Review of ferm services without srange Other Operations 2017-01-02 2016-11-02
119679 Rewrite http://download.wikimedia.org/mediawiki/ -> https://releases.wikimedia.org/mediawiki in less than 3 redirects Other Operations 2017-01-02 2017-05-16
126158 [RFC] Alert about *when* partitions will run out of space, not a percentage/absolute number Other Operations 2017-01-02 2016-05-02
132532 rsync module doesnt work on trusty Other Operations 2017-01-02 2016-04-12
32716 Run our own Tor client for Tor block Other Operations 2017-01-02 2017-06-19
151049 Run systematic availability tests Other Operations 2017-01-02 2016-11-18
85451 scale graphite deployment (tracking) Other Operations 2017-01-02 2017-12-18
150771 Secondary production Jenkins for CI Other Operations 2017-01-02 2017-10-12
94329 secure Cassandra/RESTBase cluster Other Operations 2017-01-02 2016-10-19
93531 secure.wikimedia.org entries still showing up in Google search results Other Operations 2017-01-02 2015-12-08
40860 security@mediawiki.org : Create a public key and publish it on the public key servers Other Operations 2017-01-02 2017-12-11
122825 Service Ownership and Maintenance Other Operations 2017-01-02 2018-03-07
132216 Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm Other Operations 2017-01-02 2016-05-09
143556 Setting up grafana should also setup Anonymous read-only access for the default org Other Operations 2017-01-02 2016-11-29
55457 setup a DB backed parser cache Other Operations 2017-01-02 2017-05-20
106346 setup an alertable threshold for Cassandra heap dumps Other Operations 2017-01-02 2017-07-11
91404 Setup backups of elasticsearch indices Other Operations 2017-01-02 2016-09-22
119660 Set up LVS for labs dns recursors Other Operations 2017-01-02 2016-08-15
84700 Setup management switch in OE12 Other Operations 2017-01-02 2016-03-17
149543 Setup PAWS internal experimentally on notebook* nodes Other Operations 2017-01-02 2016-10-31
106664 Set up role accounts and feedback loops (FBL) with all providers Other Operations 2017-01-02 2016-10-04
76306 Set warning thresholds for average cluster utilization Other Operations 2017-01-02 2016-12-16
152100 should we make privatewiki list available to puppet without maintaining two lists? Other Operations 2017-01-02 2016-12-07
131326 smokeping config puppetization issue? Other Operations 2017-01-02 2016-04-15
112774 solve mtp panel issue for row uplinks Other Operations 2017-01-02 2016-03-17
111838 Some files had disappeared from Commons after renaming Other Operations 2017-01-02 2017-07-25
17000 Special:Import error: "Import failed: Could not open import file" Other Operations 2017-01-02 2018-01-18
124101 Specific revisions of multiple files missing from Swift - 404 Not Found returned Other Operations 2017-01-02 2018-04-16
126295 Spike: What do we have to package to run the Programs and Events dashboard on production? Other Operations 2017-01-02 2017-02-28
125085 Split the API MediaWiki appserver pool into two external/internal pools Other Operations 2017-01-02 2016-10-12
151317 stat user crontab on stat hosts for old file removal Other Operations 2017-01-02 2016-11-22
134458 status.wikimedia.org should use some Wikimedia favicon if possible Other Operations 2017-01-02 2016-05-05
141704 Storage backend errors on commons when deleting/restoring pages Other Operations 2017-01-02 2018-04-10
148048 Store Wikimedia unified account name (SUL) in LDAP directory Other Operations 2017-01-02 2018-03-12
46016 SVG fails to render properly due to several issues Other Operations 2017-01-02 2016-07-07
140594 svn.wikimedia.org redirects to Diffusion main page, hence hard to find e.g. "flexbisonparse" Other Operations 2017-01-02 2016-07-18
123918 'swift' user/group IDs should be consistent across the fleet Other Operations 2017-01-02 2017-04-10
142821 Synchronise groups defined in data.yaml to LDAP Other Operations 2017-01-02 2016-08-12
121610 system users with UIDs > 500 Other Operations 2017-01-02 2015-12-16
151304 tmpreaper possible race condition Other Operations 2017-01-02 2016-11-22
116742 Track amount of package updates on systems Other Operations 2017-01-02 2016-08-08
132324 Tracking and Reducing cron-spam to root@ Other Operations 2017-01-02 2018-01-18
140942 Tracking: Monitoring and alerts for "business" metrics Other Operations 2017-01-02 2017-06-21
122127 Translation of namespaces for Gilaki Other Operations 2017-01-02 2018-02-06
137229 Tune thread for osm2pgsql / postgres max connections for Maps Other Operations 2017-01-02 2016-12-15
135128 Turn on etcd TLS for intra-cluster communications Other Operations 2017-01-02 2016-05-12
147040 Two recently uploaded files have disappeared (404) Other Operations 2017-01-02 2017-03-11
134326 udpmxircecho should write stats of messages processed and we should alert when that drops to zero Other Operations 2017-01-02 2016-05-03
134875 udpmxircecho spam/not working if unable to connect to irc server Other Operations 2017-01-02 2016-07-15
101141 udp rcvbuferrors and inerrors on graphite1001 Other Operations 2017-01-02 2017-07-10
131832 Unable to restore file that has a very large file size Other Operations 2017-01-02 2017-07-25
119401 Untangle labs/production roles from labs/instance roles Other Operations 2017-01-02 2015-11-23
147204 Update confd package Other Operations 2017-01-02 2017-11-14
136603 Update limit.sh to support systemd-based cgroup management Other Operations 2017-01-02 2018-05-01
129963 Update memcached package and configuration options Other Operations 2017-01-02 2018-04-16
138866 Update & standardize Platform-specific_documentation for HP servers Other Operations 2017-01-02 2016-06-28
97909 Upgrade jobrunners to redis 2.8 Other Operations 2017-01-02 2016-02-13
142205 use granularity (g=) restrictions for wikimedia.org fundraising DKIM records Other Operations 2017-01-02 2016-08-05
120532 Use user-specific passwords for accessing EventLogging database Other Operations 2017-01-02 2017-02-20
132325 Weak digest algorithm (SHA1) used to sign InRelease on apt.wikimedia.org Other Operations 2017-01-02 2017-04-03
150356 Wikidata Query Service is overly verbose toward logstash Other Operations 2017-01-02 2016-12-16
46791 [[wikitech:Server_admin_log]] should not rely on freenode irc for logmsgbot entries Other Operations 2017-01-02 2017-02-14
154627 Production error message (when servers are down) points users to donate link which is likely to produce the same error message Other Operations 2017-01-05 2017-04-29
154665 Look into behaviour of /etc/exim4/update-exim4.conf.conf related to updates Other Operations 2017-01-06 2017-01-05
154915 Get rid of "import realm.pp" in manifests/site.pp Other Operations 2017-01-10 2018-05-02
155209 Increase $wgHTTPImportTimeout to a higher value on WMF wikis Other Operations 2017-01-14 2017-01-13
111595 Do not apply spam headers on email assessed NOT to be spam Other Operations 2017-01-16 2017-01-15
155761 DNS repo: add Jenkins job to ensure there are no duplicates Other Operations 2017-01-20 2017-01-19
149589 Puppet tab in Horizon unusably slow Other Operations 2017-01-21 2017-02-02
155929 Create /community-beacon alternative entry point Other Operations 2017-01-23 2017-05-17
156143 High CPU usage from swift-proxy on frontend machines Other Operations 2017-01-25 2017-01-24
156136 Increase swift replication factor for accounts Other Operations 2017-01-25 2017-01-24
156140 Lots of hosts with hyperthreading disabled Other Operations 2017-01-25 2017-05-25
156232 confctl SubjectAltNameWarning after python-urllib3 upgrade Other Operations 2017-01-26 2017-02-22
156398 Decommission or repair old asw-c2-eqiad Other Operations 2017-01-27 2017-07-20
156475 Investigate spike in 500s during asw-c2-eqiad replacement Other Operations 2017-01-28 2018-05-14
156544 Create backups of Wikimedia content in diverse geographic places Other Operations 2017-01-29 2017-12-28
156570 Investigate issues with wikitech-static.wikimedia.org Other Operations 2017-01-30 2017-01-29
156937 Provide cross-dc redundancy (active-active or active-passive) to all important misc services Other Operations 2017-02-02 2018-03-23
156955 Standardizing our partman recipes Other Operations 2017-02-02 2018-03-29
157030 cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 Other Operations 2017-02-03 2018-04-06
157038 Make it possible to run the mediawiki testsuite against a staging repo of apt.wikimedia.org Other Operations 2017-02-03 2017-06-13
157306 Fix config file handling for /etc/hhvm/php.ini Other Operations 2017-02-07 2017-02-07
157972 Puppet fails only once when restarting ferm is not successful Other Operations 2017-02-14 2017-02-13
158022 make apt.wikimedia.org HA Other Operations 2017-02-15 2017-02-27
158288 Unclean stop of jobrunner service via puppet Other Operations 2017-02-17 2017-02-16
158434 Phabricator: Make sure phabricator works properly including our puppet roles on jessie Other Operations 2017-02-18 2017-05-20
158429 Switch to predictable network interface names? Other Operations 2017-02-18 2017-06-07
158562 Manage apt sources via puppet? Other Operations 2017-02-21 2017-02-20
158757 Puppet certificate missing subjectAltName Other Operations 2017-02-23 2017-03-22
158837 Consolidate performance website and related software Other Operations 2017-02-24 2018-05-22
88997 Improve graphite failover Other Operations 2017-02-24 2017-12-14
158915 Setup reply emails for gerrit Other Operations 2017-02-24 2017-06-14
159242 Segmentation fault creating thumbnail Other Operations 2017-03-01 2017-11-24
159480 Decommission bast3001 Other Operations 2017-03-03 2018-05-03
159524 backup space is used unwisely Other Operations 2017-03-04 2017-04-12
159536 Puppet constantly trying to stop the already stopped puppetmaster process on Trusty Other Operations 2017-03-04 2017-03-03
159687 etcd switchover/enhancements Other Operations 2017-03-07 2017-05-15
159661 Improve Terbium (and wasat) userland to process server side uploads Other Operations 2017-03-07 2017-03-06
159830 Sanity check global-multiwrite logs for ConfirmEdit usage Other Operations 2017-03-08 2017-03-07
159750 E-mail for people in different OIT LDAP object unit Other Operations 2017-03-09 2017-09-25
160060 Icinga check for sysctl settings Other Operations 2017-03-10 2017-09-07
160101 Upgrade php5-json .deb to at least 1.3.8 Other Operations 2017-03-10 2017-03-09
160146 jobrunner/jobchron services fail in codfw Other Operations 2017-03-11 2017-03-10
160158 Make disabled accounts visible in the corp mirror LDAP replica Other Operations 2017-03-11 2017-03-28
160229 Back up of Commons files Other Operations 2017-03-12 2017-05-14
160412 Add lock_wait_timeout to maintain_views and maintain-meta_p Other Operations 2017-03-15 2017-03-26
161003 Cross-check disabled accounts from corp LDAP against data.yaml Other Operations 2017-03-22 2017-03-21
161004 Remove disabled users from internal mailing lists Other Operations 2017-03-22 2018-03-26
161096 confctl no longer logs a non-changing state change Other Operations 2017-03-23 2017-03-22
161145 Fix the general problem of randomly-bad puppet agent cron timings within redundant clusters Other Operations 2017-03-23 2017-07-21
161296 Upgrade mysqld_exporter to 0.10.0 Other Operations 2017-03-25 2017-05-15
161528 incident 20170323-wikibase did not trigger Icinga paging Other Operations 2017-03-28 2017-03-27
161598 Monitor HHVM bytecode cache depletion on mediawiki app servers Other Operations 2017-03-29 2018-02-20
160529 Sender email spoofing Other Operations 2017-03-29 2017-07-28
161864 404 error while accessing some images files e.g. djvu and jpg Other Operations 2017-03-31 2017-03-30
160071 Add slabinfo prometheus exporter Other Operations 2017-03-31 2017-03-30
161835 Convert labstore cluster configuration to hiera and profiles Other Operations 2017-03-31 2017-07-23
160941 Improve SSH access information in onboarding documentation Other Operations 2017-03-31 2017-03-30
161834 Undo special tools-home and tools-project share definitions for NFS Other Operations 2017-03-31 2017-07-23
161904 decommission backup4001 Other Operations 2017-04-01 2017-07-18
161920 logrotate for ruthenium Other Operations 2017-04-01 2017-05-10
161918 videoscalers (mw1168, mw1169) - high load / overheating Other Operations 2017-04-01 2017-04-01
162029 Migrate all jessie hosts to Linux 4.9 Other Operations 2017-04-04 2017-04-13
162037 Use SSL certificates with discovery entry for elasticsearch Other Operations 2017-04-04 2017-10-19
162123 Running swiftrepl is not puppetized Other Operations 2017-04-05 2017-09-28
162122 Swiftrepl was stuck in an infinite loop since days Other Operations 2017-04-05 2017-04-12
36947 Incorrect text positioning in SVG rasterization (scale/transform; font-size; kerning) Other Operations 2017-04-08 2018-03-26
162612 codfw/eqiad hosts occasionally spend > 3 minutes starting networking.service with linux 4.9 Other Operations 2017-04-11 2017-07-20
162850 CPU throttling on DELL PowerEdge R320 Other Operations 2017-04-13 2017-09-13
162955 rebuild tools-grid-master as a large instance Other Operations 2017-04-14 2017-06-08
163068 More missing 'original' files on Commons Other Operations 2017-04-17 2017-07-25
163362 audit all codfw pdu tower draws Other Operations 2017-04-20 2017-04-19
163288 Decide on /var/lib vs /home as locations of homedir for l10nupdate Other Operations 2017-04-20 2018-02-01
163393 Determine appropriate proxy_read_timeout setting for Tools Proxy Other Operations 2017-04-20 2017-04-19
163402 Ensure we can survive a loss of labservices1001 Other Operations 2017-04-20 2017-08-03
163354 Find a way to verify mediawiki-config IPs ahead of datacenter switchovers Other Operations 2017-04-20 2017-04-19
163336 kube-proxy pulls in docker and starts service even when it isnt needed Other Operations 2017-04-20 2017-04-19
163507 Intermittent DB connectivity problem on phabricator, needs investigation Other Operations 2017-04-21 2017-05-24
163698 Add flood protection to the ircecho bot (icinga-wm) Other Operations 2017-04-25 2017-04-24
163667 Fix UIDs for deployment server users Other Operations 2017-04-25 2017-04-24
163673 Some swift disks wrongly mounted on 5 ms-be hosts Other Operations 2017-04-25 2017-08-22
163823 During labservices1001 failover fqdn changed from foo.project.eqiad.wmflabs to foo.eqiad.wmflabs Other Operations 2017-04-26 2017-04-25
163996 Icinga check for ipv6 host reachability Other Operations 2017-04-28 2018-04-30
164042 Racktables: clearly show when hosts are decommissioned Other Operations 2017-04-28 2017-04-27
164123 tools-k8s-master-01 has two floating IPs Other Operations 2017-04-29 2017-06-06
123276 URL parameters do not work with pages that have "?" in their names Other Operations 2017-04-29 2017-09-13
164238 move icinga contacts file to public repo Other Operations 2017-05-03 2018-02-01
164341 Decommission old memcached hosts - mc1001->mc1018 Other Operations 2017-05-04 2018-04-19
164490 maintain-meta_p hangs on connecting to wikimedia.org.uk Other Operations 2017-05-05 2017-05-04
164819 reprepro: Support for buildinfo files / dbgsym packages Other Operations 2017-05-10 2017-06-02
164993 archiva artifact links point to 127.0.0.1 Other Operations 2017-05-11 2017-05-10
165105 Some requests for DOIs are failing or very slow; if we have a DOI and the request is taking too long, just use CrossRef data instead. Other Operations 2017-05-12 2018-05-03
165136 Ferm rules for labstore NFS hosts Other Operations 2017-05-13 2017-10-06
165323 Add Prometheus machine metric to track core dumps Other Operations 2017-05-16 2017-05-15
165618 Audit / document reasons for not enabling HT? Other Operations 2017-05-18 2017-05-17
165631 move gerrit.wm.org SSH service to private/behind LVS like phab-vcs Other Operations 2017-05-18 2018-01-25
165885 Create a cron to clean clientbucket every day or hour Other Operations 2017-05-21 2017-05-20
166066 Integrate the puppet compiler in the puppet CI pipeline Other Operations 2017-05-23 2018-04-08
166081 rack/setup/install conf1004-conf1006 Other Operations 2017-05-23 2018-05-14
166038 Sync internal nutcracker package with Debian package Other Operations 2017-05-23 2017-05-29
163033 Create grafana dashboard for video scaler job runners Other Operations 2017-05-24 2017-05-23
166233 Update redis puppet class to support stretch Other Operations 2017-05-25 2017-05-24
166291 Exim panics when spamd reaches maxchildren Other Operations 2017-05-26 2017-09-18
166368 Wipe of spare/replacement disks Other Operations 2017-05-27 2017-05-26
166937 Broken /a/refinery-source/guard/run_all_guards.sh script on stat1002 Other Operations 2017-06-04 2018-01-08
167035 stretch acct monthly cron will spam when /var/log/wtmp.1 doesn't exist Other Operations 2017-06-06 2017-06-05
167091 Elasticsearch errors about BulkShardRequest Other Operations 2017-06-07 2017-10-24
164290 Set up external DNS record for wikitech-static Other Operations 2017-06-07 2017-06-06
167292 Collate jessie-wikimedia/backports into jessie-wikimedia/main Other Operations 2017-06-08 2017-06-07
167245 prometheus-node-exporter - invalid group: ‘prometheus:prometheus' Other Operations 2017-06-08 2017-06-07
89808 wikitech instances list is blank Other Operations 2017-06-08 2017-06-07
167376 Decommission cp300[3456] Other Operations 2017-06-09 2018-05-03
167377 Decommission cp4011, cp4012, cp4019, cp4020 Other Operations 2017-06-09 2018-05-03
167412 host-vmem.erb is doing operations that make no sense Other Operations 2017-06-09 2017-06-17
167422 Monitoring: add link to graph for Icinga timeseries alarms Other Operations 2017-06-09 2017-06-08
167549 Create Icinga alert when OSM replication lags on maps Other Operations 2017-06-10 2018-02-26
167966 Look into feasibility of disabling sha-1 host keys on our ssh daemons Other Operations 2017-06-16 2017-07-12
165511 Change automatic shortlink in blog theme Other Operations 2017-06-20 2017-06-20
168403 Aggregate prometheus functions yielding different results in grafana vs. prometheus console Other Operations 2017-06-21 2017-06-20
168407 rack/setup/install labnodepool1002.eqiad.wmnet Other Operations 2017-06-21 2018-01-10
168460 Update certificates on productions replicas of corp.wikimedia.org LDAP Other Operations 2017-06-21 2017-06-20
168490 upgrade planet instances to stretch Other Operations 2017-06-22 2017-11-22
114849 Log lines on flourine overflow at 8092 bytes. Other Operations 2017-06-24 2017-06-23
113104 Set up a service IP for logstash Other Operations 2017-06-24 2017-06-23
169020 Decommission cp400[1-4] Other Operations 2017-06-28 2018-05-03
168767 Monitor PostgreSQL connection slots Other Operations 2017-06-28 2017-06-27
168967 Upload shiny-server .deb to our Jessie apt repository Other Operations 2017-06-28 2018-04-26
169035 bast3002 sdb broken Other Operations 2017-06-29 2018-04-20
169287 etcd config depends on puppet certs, but puppet doesn't know Other Operations 2017-07-01 2017-06-30
169286 labstore1005 A PCIe link training failure error on boot Other Operations 2017-07-01 2017-06-30
169290 New anti-stackclash (4.9.25-1~bpo8+3 ) kernel super bad for NFS Other Operations 2017-07-01 2018-03-06
169322 Research whether it makes sense to have OTRS installation in an HA setup Other Operations 2017-07-01 2017-06-30
169318 Use multiple puppetdbs on puppet masters Other Operations 2017-07-01 2017-06-30
169518 Decommission esams ms-fe / ms-be Other Operations 2017-07-04 2018-05-03
169564 MD RAID: remove mdadm daily check Other Operations 2017-07-04 2017-07-03
169570 nfs-manage failover script needs to be tested with real load and fixed Other Operations 2017-07-04 2017-09-14
169680 NFS on dataset1001 overloaded, high load on the hosts that mount it Other Operations 2017-07-05 2018-04-03
169884 Jobrunners generate mediawiki exceptions upon calling Closure$RecentChange::save Other Operations 2017-07-07 2017-07-07
103886 Translation cache exhaustion caused by changes to PHP code in file scope Other Operations 2017-07-09 2018-03-19
170150 Evaluate Grafana's LDAP group options and deprecate grafana-admin if possible Other Operations 2017-07-11 2018-04-16
170152 mc2023 / mc2025 fail to mount root partition within 90 seconds using Linux 4.9 Other Operations 2017-07-11 2017-07-10
170108 Operations Q1 goal: Streamlined Service Delivery Other Operations 2017-07-11 2017-07-12
104671 Rename 'restricted' group? Other Operations 2017-07-12 2017-07-11
170298 sshd stretch puppet support Other Operations 2017-07-12 2017-07-12
170474 Decommisson and store old row D network gear. Other Operations 2017-07-13 2017-07-20
170480 FY2017/18 Program 6 - Outcome 2: Developers are able to develop and test their applications through a unified pipeline towards production deployment. Other Operations 2017-07-13 2017-07-18
170481 FY2017/18 Program 6 - Outcome 2 - Objective 2: Set up a continuous integration and deployment pipeline Other Operations 2017-07-13 2018-05-04
170456 FY2017/18 Program 6 - Outcome 2 - Objective 3: Integrated, container-based development environment Other Operations 2017-07-13 2017-12-28
170453 FY2017/18 Program 6: Streamlined Service delivery Other Operations 2017-07-13 2017-09-06
170640 reports.frdev.wm.o -- still in use? Other Operations 2017-07-14 2017-07-18
170740 PuppetDB misbehaving on 2017-07-15 Other Operations 2017-07-16 2017-07-19
170817 Upgrade Thumbor servers to Stretch Other Operations 2017-07-18 2018-04-03
170628 Jessie rsvg/cairo can't render specific SVG file on Commons Other Operations 2017-07-19 2018-03-14
132632 puppetize turning off reserved space for cassandra /srv Other Operations 2017-07-19 2017-07-18
170995 Setup a mirror for R language dependencies (CRAN) Other Operations 2017-07-19 2018-04-23
147366 Setup automated topk wide row reporting Other Operations 2017-07-19 2017-07-18
171048 Eventbus does not handle gracefully changes in DNS recursors Other Operations 2017-07-20 2018-04-02
171122 librenms: consider using Distributed Poller with multiple netmon servers Other Operations 2017-07-20 2017-07-19
83729 Fix monitoring of poolcounter service Other Operations 2017-07-21 2018-01-09
171157 Monitor internal CA expirations Other Operations 2017-07-21 2018-01-08
171188 Move the main WMCS puppetmaster into the Labs realm Other Operations 2017-07-21 2017-07-24
171191 Should puppet auto-restart slapd? Other Operations 2017-07-21 2017-07-21
171482 Programmatic generation of grafana dashboards Other Operations 2017-07-25 2018-05-08
171619 ORES should use a git large file plugin for storing serialized binaries Other Operations 2017-07-28 2017-11-29
167689 Add RIPE atlas data to Prometheus Other Operations 2017-08-02 2017-08-01
154619 Export ipsec counters as Prometheus metrics Other Operations 2017-08-02 2017-08-01
172479 Collect error logs from jobchron/jobrunner services in Logstash Other Operations 2017-08-05 2018-02-13
172584 Securing external binaries run by MediaWiki Other Operations 2017-08-06 2017-12-13
172815 Improve stability and maintainability of our browser-based PDF render service Other Operations 2017-08-09 2017-11-22
173097 Decommission stat1002.eqiad.wmnet Other Operations 2017-08-12 2018-05-03
173492 Tune Varnishkafka delivery errors to be more sensitive Other Operations 2017-08-18 2018-03-29
173721 Track down the source of periodic increases in requests to swift eqiad Other Operations 2017-08-22 2017-08-23
174172 unused grafana-dashboard indices on elasticsearch / logstash Other Operations 2017-08-26 2017-08-25
174269 Two cases of local-multiwrite storage backend failure Other Operations 2017-08-28 2017-08-27
160644 Eventstreams graphite disk usage Other Operations 2017-08-30 2017-08-29
174465 Puppet admin module should support adding system users to managed groups Other Operations 2017-08-30 2018-05-14
174449 tin has a failing hdd Other Operations 2017-08-30 2017-09-07
174475 update firmware on scs consoles Other Operations 2017-08-30 2017-09-11
174431 Upgrade mw* servers to Debian Stretch (using HHVM) Other Operations 2017-08-30 2018-05-10
41785 Create a labs SMTP smarthost Other Operations 2017-08-31 2018-01-06
174720 letsencrypt::cert::integrated and non-http servers Other Operations 2017-09-01 2017-10-17
174916 electron/pdfrender hangs Other Operations 2017-09-05 2017-10-13
174959 swift-recon-cron on ms-be203[34]: [Errno 17] File exists: '/var/lock/swift-recon-object-cron' Other Operations 2017-09-05 2017-09-05
175206 2017/18 Annual Plan Program 8: Multi-datacenter support Other Operations 2017-09-07 2017-10-19
175213 2017/18 Annual Plan Program 8: Multi-datacenter support, Q2 goals Other Operations 2017-09-07 2017-10-19
175210 Select candidate jobs for transferring to the new infrastucture Other Operations 2017-09-07 2018-04-16
175288 setup/install/deploy deploy1001 as deployment server Other Operations 2017-09-08 2018-05-22
175362 Split MXes into inbound and outbound Other Operations 2017-09-09 2017-09-20
175361 Upgrade mx1001/mx2001 to stretch Other Operations 2017-09-09 2018-04-19
175625 scs-c1-eqiad unresponsive Other Operations 2017-09-12 2017-11-28
175710 Add profiling for Varnish and VCL Other Operations 2017-09-13 2018-02-02
175738 Long term storage for frack prometheus data Other Operations 2017-09-13 2017-11-14
175876 document all scs connections Other Operations 2017-09-14 2018-01-29
175885 Toolforge's static webserver broken by Puppet changes and stale nginx packages Other Operations 2017-09-15 2017-11-20
82350 update exim::listserve::private::mailing_lists value in puppet Other Operations 2017-09-19 2017-09-18
176335 logs sent to logstash are lost when the elasticsearch cirrus cluster is unavailable Other Operations 2017-09-21 2017-10-24
176370 Migrate to PHP 7 in WMF production Other Operations 2017-09-21 2018-05-06
176437 puppet ca_server confusion Other Operations 2017-09-22 2017-09-21
176445 Systematically test load speeds of Watchlist and Recent Changes Other Operations 2017-09-23 2017-10-12
176153 Create affcom-staff email account Other Operations 2017-09-26 2017-09-25
176666 Qualtrics cannot send email to wikimedia.org addresses Other Operations 2017-09-26 2018-04-19
176774 Reimage cobalt as stretch Other Operations 2017-09-27 2018-02-06
176957 Decommission host copper.eqiad.wmnet Other Operations 2017-09-29 2018-05-03
177099 Large number of "A page you created was linked on Wikidata" emails to one recipient in short period of time Other Operations 2017-09-30 2017-09-29
177197 Export Prometheus-compatible JVM metrics from JVMs in production Other Operations 2017-10-03 2018-04-16
177196 Port non-deprecated Diamond collectors to Prometheus Other Operations 2017-10-03 2017-12-21
177195 Reduce technical debt in metrics monitoring Other Operations 2017-10-03 2018-02-01
177371 Phase out DSA keys for SSH access (ssh-dss) Other Operations 2017-10-05 2018-04-26
177826 Upgrade ci ssh key to ecdsa Other Operations 2017-10-11 2017-10-23
177958 Decommission ocg1001-3 Other Operations 2017-10-12 2018-05-03
177914 Switch labstore servers to default SSH configuration Other Operations 2017-10-12 2017-10-13
177891 Update and use php-wikidiff2 1.5.1 & MovedParagraphDetectionCutoff in production Other Operations 2017-10-12 2017-11-28
178445 flapping monitoring for recommendation_api on scb Other Operations 2017-10-18 2017-11-02
84845 improve cron spam visibility Other Operations 2017-10-19 2017-10-18
178575 Add require_package() variant with repository component to wmflib Other Operations 2017-10-20 2017-10-19
178628 Improve puppet alerting Other Operations 2017-10-20 2017-10-20
178690 Better organization for ops grafana dashboards Other Operations 2017-10-21 2018-04-16
178663 Switch CI Docker Storage Driver to its own partition and to use devicemapper Other Operations 2017-10-21 2018-03-21
178839 New upstream jvm-tools Other Operations 2017-10-24 2018-01-02
178810 Wikibase: Increase batch size for HTMLCacheUpdateJobs triggered by repo changes. Other Operations 2017-10-24 2017-10-27
179022 Backport firejail 0.9.52 for use on Wikimedia appservers Other Operations 2017-10-26 2018-02-04
179078 mpt raid controller not detected as fact on maps-test2* Other Operations 2017-10-27 2017-10-26
179099 puppetmaster hostcert and hostprivkey point to nonexistent files Other Operations 2017-10-27 2018-04-23
179192 Check analytics1037 power supply status Other Operations 2017-10-28 2018-04-04
179181 Puppet4: hiera() can only be called using the 4.x function API. Other Operations 2017-10-28 2017-11-29
179230 Puppet wmf-style-guide: array of classes not detected properly Other Operations 2017-10-29 2017-10-28
179353 Scap: Standardize git version Other Operations 2017-10-31 2018-02-01
179395 Cluster puppet variable and ganglia decommission Other Operations 2017-11-01 2017-11-14
179371 Move deployment-prep redis instances to stretch Other Operations 2017-11-01 2018-04-06
179354 wikimedia-jessie & wikimedia-stretch docker images don't have deb-src set for apt.wikimedia.org Other Operations 2017-11-01 2017-11-02
179463 Create a single application to provision and manage developer (LDAP) accounts Other Operations 2017-11-02 2018-03-22
179501 Use external dsh group to list pooled ORES nodes Other Operations 2017-11-02 2017-11-01
179562 Create jenkins job for creating deployment artifacts for `docker-pkg-deploy` Other Operations 2017-11-03 2017-12-07
179696 Homepage for https://docker-registry.wikimedia.org Other Operations 2017-11-04 2017-11-20
179856 Improve documentation for mirrors.wikimedia.org Other Operations 2017-11-07 2017-11-06
180023 [DRAFT][RfC] Deployment of python applications in production Other Operations 2017-11-09 2017-12-04
180051 Reduce the number of fields declared in elasticsearch by logstash Other Operations 2017-11-09 2017-12-12
180330 Add CI to all operations/* repositories and archive obsolete ones Other Operations 2017-11-14 2017-11-20
180498 planet.wikimedia.org: replace planet-venus software with rawdog Other Operations 2017-11-15 2017-11-14
180628 Install git-lfs client (at least on scap targets & masters) Other Operations 2017-11-16 2018-03-19
180641 reinstall RT server with private IP and stretch Other Operations 2017-11-16 2017-11-15
180853 Bring discourse.mediawiki.org to production Other Operations 2017-11-18 2018-05-04
180944 Passenger spews Exception NoMethodError in Rack application object Other Operations 2017-11-21 2017-11-27
181205 let quarry use the mariadb module Other Operations 2017-11-24 2017-11-23
181559 Investigate redis-cluster or other techniques for making Redis not a single point of failure. Other Operations 2017-11-29 2017-12-05
181546 Let the ORES application set log severity, not uWSGI Other Operations 2017-11-29 2017-11-28
181632 Celery manager implodes horribly if Redis goes down Other Operations 2017-11-30 2017-12-05
181634 Investigate overload condition, seems that we lose nodes Other Operations 2017-11-30 2017-12-05
181678 Plan migration of ORES repos to git-lfs Other Operations 2017-11-30 2018-05-01
181630 Send celery and wsgi service logs to logstash Other Operations 2017-11-30 2017-12-05
181621 What is causing ORES celery workers to suddenly require more CPU? Other Operations 2017-11-30 2017-12-05
181750 decommission mobile 1004 and mobile1005 Other Operations 2017-12-01 2018-04-30
181763 Decommission niobium Other Operations 2017-12-01 2017-12-12
181855 scap support for git-lfs Other Operations 2017-12-02 2018-04-26
181200 Use "Charter" as preferred typeface on Electron Other Operations 2017-12-02 2018-03-18
182034 Decommission osm-cp100[1-4] Other Operations 2017-12-05 2018-02-15
182033 Decommission osm-web100[1-4] Other Operations 2017-12-05 2018-02-15
182016 Decommission server zinc Other Operations 2017-12-05 2017-12-12
182015 Decommission Vanadium Other Operations 2017-12-05 2017-12-12
181971 Disable hiera autolookups Other Operations 2017-12-05 2017-12-05
181988 Investigate and improve memory allocation rates of WDQS Other Operations 2017-12-05 2018-05-18
181967 Update puppet code to conform to puppet 4.x and later standards Other Operations 2017-12-05 2017-12-04
182228 run-no-puppet leave puppet disabled on kill/crash Other Operations 2017-12-07 2017-12-06
182203 Tuning profile::ores::celery parameters should cause a Celery service restart Other Operations 2017-12-07 2017-12-06
182274 Create custom per-job metric reporters capability Other Operations 2017-12-08 2018-04-16
182331 [Epic] Deploy ORES in kubernetes cluster Other Operations 2017-12-08 2017-12-07
182497 Update log config for scb* boxes, to deal with ORES verbose logging Other Operations 2017-12-10 2017-12-11
182249 Diagnose and fix 4.5k req/min ceiling for ores* requests Other Operations 2017-12-12 2017-12-18
182597 Use EtcdConfig in production to allow automation of a datacenter switch Other Operations 2017-12-12 2018-03-06
182702 Debian Jessie reimage/install ends up in kernel panic with 8.10 netboot image Other Operations 2017-12-13 2017-12-15
182699 Use firmware-enriched Debian installation images Other Operations 2017-12-13 2017-12-12
182759 Add Prometheus exporter to Jenkins instances Other Operations 2017-12-14 2018-04-16
182819 custom fact interface_primary breaks under newer versions of facter Other Operations 2017-12-14 2017-12-13
182812 Forward security@tools.wmflabs.org to security@wikimedia.org Other Operations 2017-12-14 2017-12-15
182822 Generate a list of files that are supposed to exist but 404s Other Operations 2017-12-14 2018-03-29
94457 Install nodejs, nginx and other dependencies on francium Other Operations 2017-12-19 2017-12-18
183146 Monitor resource usage on a per-cgroup basis Other Operations 2017-12-19 2018-03-16
183236 After reimage Puppet order: sudo command failed Other Operations 2017-12-20 2017-12-19
183454 Deprovision Diamond collectors no longer in use Other Operations 2017-12-22 2018-04-26
183565 Fix regex.yaml single-regex issue Other Operations 2017-12-23 2017-12-22
183585 Rack/cable/configure asw2-b-eqiad switch stack Other Operations 2017-12-23 2018-04-04
111653 Encrypt all the things Other Operations 2017-12-27 2017-12-26
82937 re-create script for manual paging Other Operations 2017-12-29 2018-05-14
183814 Degraded RAID on bast3002 Other Operations 2017-12-31 2018-01-03
183920 2018-01-02: labstore Tools and Misc share very full Other Operations 2018-01-03 2018-01-10
183954 templatetiger is using 827G of 8T available tools nfs storage Other Operations 2018-01-03 2018-03-13
184086 Add prometheus exporter to Gerrit Other Operations 2018-01-04 2018-02-26
182832 Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state Other Operations 2018-01-04 2018-04-16
184064 Prepare racks OE14, OE15 and OE16 with new infrastructure Other Operations 2018-01-04 2018-01-03
184066 Procure and install new PDUs Other Operations 2018-01-04 2018-01-03
184063 Remove all decommissioned hardware Other Operations 2018-01-04 2018-01-03
184065 Setup new access switches Other Operations 2018-01-04 2018-01-03
184061 SRE 2017-18 Q3 goal Cleanup esams and refresh servers and infrastructure (tracking) Other Operations 2018-01-04 2018-02-02
184186 Fix unknown variables warning that occur with puppet 4.x Other Operations 2018-01-05 2018-01-04
184245 Create some mechanism for instances in projects to modify the project Designate records Other Operations 2018-01-06 2018-01-19
184230 Disavow emails from wikipedia.com Other Operations 2018-01-06 2018-02-09
184236 Puppet broken on deployment-ms-be0[34] with evaluation error in swift module Other Operations 2018-01-07 2018-03-31
184461 Discourse migration from wmflabs to production Other Operations 2018-01-09 2018-05-04
184462 Serve one production service via Kubernetes Other Operations 2018-01-09 2018-02-03
184561 Modernize Puppet Configuration Management (2017-18 Q3 Goal) Other Operations 2018-01-10 2018-04-03
184564 Plan Puppet 5 upgrade Other Operations 2018-01-10 2018-04-04
184522 To purchase for next esams visit Other Operations 2018-01-10 2018-01-09
184634 Netbox: postgres cannot be restarted w/ current config Other Operations 2018-01-11 2018-01-22
184655 logstash group1 dashboard incorrectly shows testwikidatawiki Other Operations 2018-01-12 2018-02-28
184714 Puppet fail to properly refresh Icinga Other Operations 2018-01-12 2018-01-11
184832 Decommission labsdb1001 and labsdb1003 Other Operations 2018-01-13 2018-05-03
184936 install/designate other machine as esams bastion Other Operations 2018-01-16 2018-01-31
184924 Utilize the deployment pipeline (stretch) Other Operations 2018-01-16 2018-03-27
185004 Decommission mw1201-mw1220 Other Operations 2018-01-17 2018-05-03
185024 Readd complete URL parsing fix from 3.18.7 release Other Operations 2018-01-17 2018-01-23
185055 Stack overflow when Redis is down Other Operations 2018-01-18 2018-01-18
185236 Password Vault for Security Team Other Operations 2018-01-19 2018-01-18
185215 Puppet compiler failure to lookup some keys Other Operations 2018-01-19 2018-03-01
185275 replace tin (new hardware) Other Operations 2018-01-19 2018-04-11
185189 scap sudo violation on first puppet run Other Operations 2018-01-19 2018-01-18
185195 tmpreaper doesn't play along with PrivateTmp systemd units Other Operations 2018-01-19 2018-04-16
185306 ms-be2023 unresponsive while rebuilding one disk Other Operations 2018-01-20 2018-01-19
185298 xfs_db blocked / timeout on ms-be2023 Other Operations 2018-01-20 2018-01-19
185504 Netbox: add Icinga check for PostgreSQL Other Operations 2018-01-23 2018-04-10
185644 Switch phabricator from using apache to nginx Other Operations 2018-01-25 2018-01-31
185814 /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 Other Operations 2018-01-28 2018-01-27
185815 The Rack Puppet master server is deprecated and will be removed in a future release. Please use Puppet Server instead. Other Operations 2018-01-28 2018-01-27
186069 Icinga: page in case all MediaWiki are throwing 5xx Other Operations 2018-01-31 2018-05-02
186073 Rack/setup frmon1001 Other Operations 2018-01-31 2018-03-01
186153 outdated DjVu file page thumbnail in cache Other Operations 2018-02-01 2018-02-04
78135 Provide a pxe-bootable rescue image Other Operations 2018-02-03 2018-03-29
186288 replace all Ubuntu (trusty) hosts in production with Debian Other Operations 2018-02-03 2018-03-13
186416 Allow selecting which images to build Other Operations 2018-02-04 2018-03-20
186625 apply hostname labels to bast1002/WMF4749 Other Operations 2018-02-07 2018-02-06
186734 Clean up redundant ORES celery_workers defaults Other Operations 2018-02-08 2018-02-07
186748 New service request: chromium-render/deploy Other Operations 2018-02-08 2018-05-22
104774 Publishing translations for central notice banners fails Other Operations 2018-02-08 2018-02-21
186918 prometheus: ganglia-gen and outdated Ganglia:cluster resource name Other Operations 2018-02-10 2018-02-09
187076 Deploy error: insufficient permission for adding an object to repository database .git/objects Other Operations 2018-02-13 2018-02-12
187078 Re-consider ` >/dev/null 2>&1` as output of many cron'd MW maintenance scripts Other Operations 2018-02-13 2018-02-12
187101 Setup some alert mechanism when some 'critical' cron jobs fail Other Operations 2018-02-13 2018-02-12
187190 Decommission graphite1002 Other Operations 2018-02-14 2018-05-09
187257 puppetdb4: systemd config review Other Operations 2018-02-14 2018-02-13
187194 zotero translation server: code stewardship request Other Operations 2018-02-14 2018-03-28
187364 rack frdata1001 Other Operations 2018-02-15 2018-03-01
187365 rack frpig1001 Other Operations 2018-02-15 2018-03-01
187456 Decommission labstore100[12] and their disk shelves Other Operations 2018-02-16 2018-03-10
187467 Decommission mw2017 and mw2099 Other Operations 2018-02-16 2018-05-03
187474 Decommission old and unused/spare servers in codfw Other Operations 2018-02-16 2018-05-03
187473 Decommission old and unused/spare servers in eqiad Other Operations 2018-02-16 2018-05-03
187445 Decommission osm-db200[12] and osm-web200[1234] Other Operations 2018-02-16 2018-03-15
187434 Include apache_exporter in puppet module apache Other Operations 2018-02-16 2018-02-15
187651 Setting packages on 'hold' breaks puppet runs Other Operations 2018-02-19 2018-04-19
187658 Setup cron for foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php on Beta Other Operations 2018-02-19 2018-02-20
187673 Build and deploy hhvm-luasandbox 3.0.1 to Wikimedia wikis Other Operations 2018-02-20 2018-03-07
187754 Figure out why HHVM isn't using error_document404 setting Other Operations 2018-02-20 2018-02-19
187736 Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) Other Operations 2018-02-20 2018-04-04
187991 Have swift metrics available in Prometheus Other Operations 2018-02-23 2018-02-22
187994 netfilter software at WMF: iptables vs nftables Other Operations 2018-02-23 2018-03-05
187984 Update OTRS to the latest stable version (6.x.x) Other Operations 2018-02-23 2018-04-26
187987 Upgrade to Prometheus 2.x Other Operations 2018-02-23 2018-04-20
188098 Add Prometheus collector for Tor Other Operations 2018-02-24 2018-02-23
45952 Incorrect "non-identical file already exists" error when undeleting file on Commons Other Operations 2018-02-26 2018-02-25
188317 Detect high server load earlier – prometheus alert? Other Operations 2018-02-27 2018-02-27
188453 Google Search Console access for Search Platform team Other Operations 2018-02-28 2018-04-17
188377 Import some Analytics git puppet submodules to operations/puppet Other Operations 2018-02-28 2018-04-05
188602 Decrease the amount of IRC spam in case of widespread puppet failures Other Operations 2018-03-02 2018-03-01
188601 Gain visibility into httpd mod_proxy actions Other Operations 2018-03-02 2018-03-01
188947 Create an LVS endpoint for jobrunners on videoscalers Other Operations 2018-03-06 2018-05-22
188913 "Obama" page on Beta Cluster often responds with 503 Other Operations 2018-03-06 2018-04-16
188985 https://meta.wikimedia.org/wiki/Special:Contact/Stewards is being abused by spammers Other Operations 2018-03-07 2018-03-06
189137 Migrate CirrusSearch jobs to Kafka queue Other Operations 2018-03-08 2018-04-25
189065 Outbound mail from Greenhouse is broken Other Operations 2018-03-08 2018-04-10
189435 Integrate stretch 9.4 point update Other Operations 2018-03-13 2018-03-12
189566 Decommission eventlog1001 Other Operations 2018-03-14 2018-05-03
189629 rename role::xenon Other Operations 2018-03-14 2018-03-13
189741 Build .deb package of python3-aiokafka Other Operations 2018-03-15 2018-03-15
189729 Build .deb package of python3-typing for jessie Other Operations 2018-03-15 2018-03-15
189801 setup backup1001.eqiad.wmnet Other Operations 2018-03-16 2018-04-04
189890 add ssh key comparison to cross-validate-accounts.py Other Operations 2018-03-17 2018-03-20
189921 decom californium Other Operations 2018-03-18 2018-05-03
190086 Decommission old server wmf4077 Other Operations 2018-03-20 2018-05-03
190085 Reclaim/Decommission Silver.wikimedia.org Other Operations 2018-03-20 2018-05-03
190111 VirtualHost for mod_status breaks debugging Apache/MediaWiki from localhost Other Operations 2018-03-20 2018-04-06
190225 Decommission unused host wmf3565 Other Operations 2018-03-21 2018-04-30
190318 remove puppet_major_version and puppetdb_major_version variables. clean up puppet master/db hieradata Other Operations 2018-03-22 2018-04-20
190455 Logstash no longer captures DB queries in debug mode Other Operations 2018-03-23 2018-04-24
190568 Reimage both phab1001 and phab2001 to stretch Other Operations 2018-03-24 2018-04-24
190693 Extend dpkg Icinga check to also check for inconsistent apt state Other Operations 2018-03-27 2018-03-26
190717 Update wikidiff2 library on the WMF production cluster Other Operations 2018-03-27 2018-05-18
190766 add ci test for admin module indentation Other Operations 2018-03-28 2018-03-27
191018 Provide an option menu when booting via PXE Other Operations 2018-03-30 2018-03-29
191153 decom bast1001 Other Operations 2018-04-01 2018-05-03
191199 Page allocation stalls on scb1001, scb1002 Other Operations 2018-04-03 2018-04-02
191191 pdfrender logs to /var/log/syslog as well as to /srv/log/pdfrender Other Operations 2018-04-03 2018-04-02
191315 Cassandra Graphite metrics space usage audit and cleanup Other Operations 2018-04-04 2018-04-05
191348 decommission uranium/WMF3128 Other Operations 2018-04-04 2018-05-03
191355 decom niobium/WMF3428 Other Operations 2018-04-04 2018-05-03
191362 decom promethium/WMF3571 Other Operations 2018-04-04 2018-05-03
191357 decom silver/WMF3434 Other Operations 2018-04-04 2018-05-03
191360 decom spare server lawrencium/WMF3542 Other Operations 2018-04-04 2018-05-03
191364 decom spare server osmium/wmf4546 Other Operations 2018-04-04 2018-05-03
191351 decom vanadium/WMF3291 Other Operations 2018-04-04 2018-05-03
191352 decom zinc/WMF3298 Other Operations 2018-04-04 2018-05-03
191388 Puppet: tracking catalogs that changes at every run Other Operations 2018-04-05 2018-04-04
191438 Upgrade Puppet compilers to Stretch Other Operations 2018-04-05 2018-04-04
191491 Adjust bandwidth/connection limits, memory settings on labstore1006,7 as appropriate Other Operations 2018-04-06 2018-04-05
191659 Configure a threshold for earlier notification of /srv/cassandra/instance-data Other Operations 2018-04-07 2018-04-16
191625 Create RC feed for login.wikimedia Other Operations 2018-04-07 2018-04-06
191627 Remove Cassandra 2.2.6 packages from jessie-wikimedia/thirdparty apt repo Other Operations 2018-04-07 2018-04-06
191648 uwsgi::app sorts config keys, but the .ini file behavior depends on order Other Operations 2018-04-07 2018-04-13
191842 Deployment git server can't supply ORES hosts in parallel Other Operations 2018-04-10 2018-04-09
191921 mwscript rebuildLocalisationCache.php takes 40 minutes Other Operations 2018-04-11 2018-05-14
191956 Document how to fix IPMI issues on Wikitech Other Operations 2018-04-12 2018-04-11
192103 Decommission notebook1001 Other Operations 2018-04-13 2018-05-03
192102 deprecate and remove --autoload in uwsgi puppet class Other Operations 2018-04-13 2018-04-13
192092 setup replacement for terbium (maintenance_server) on stretch Other Operations 2018-04-13 2018-05-22
192370 Deploy mcrouter to production as a wancache backend Other Operations 2018-04-18 2018-05-08
192457 Reallocate former image scalers Other Operations 2018-04-19 2018-05-03
192551 atop on stretch overloading a host Other Operations 2018-04-20 2018-04-25
192532 Figure out a way to enable volunteers to use the puppet compiler Other Operations 2018-04-20 2018-05-02
192547 Improve remote IPMI monitoring Other Operations 2018-04-20 2018-04-19
192610 prometheus on bast3002 misbehaving Other Operations 2018-04-20 2018-04-24
192561 Upgrade deployment-prep deployment servers to stretch Other Operations 2018-04-20 2018-04-19
192771 mcrouter production architecture Other Operations 2018-04-24 2018-05-19
192948 Upgrade prometheus-jmx-exporter on all services using it Other Operations 2018-04-25 2018-04-26
193025 Decommision poolcounter1002 Other Operations 2018-04-26 2018-05-03
193072 TTS server deployment strategy Other Operations 2018-04-26 2018-05-19
190716 Deploying FileExporter and FileImporter Other Operations 2018-04-27 2018-05-11
193155 IPMI Audit 2018-04 Other Operations 2018-04-27 2018-04-26
193112 Jobs writing to the Elasticsearch cluster in codfw are timing out, causing all type of issues Other Operations 2018-04-27 2018-05-02
193196 labnet1003 and labnet1004 moving and enabling 10G NICs Other Operations 2018-04-27 2018-05-17
193160 Monitor the BIOS boot order and parameters Other Operations 2018-04-27 2018-04-26
193121 Upgrade ganeti hosts to stretch Other Operations 2018-04-27 2018-04-30
193272 Prometheus vs. CPU usage vs. hyperthreading Other Operations 2018-04-28 2018-05-01
193420 Decommission hafnium Other Operations 2018-05-01 2018-05-03
193394 Degraded RAID on wasat Other Operations 2018-05-01 2018-05-04
193414 Servers using tidy-html5 are rendering pages differently, especially with <bdi> Other Operations 2018-05-01 2018-05-09
193408 SPF record for canonical domains Other Operations 2018-05-01 2018-04-30
193473 Add HTTPS support to wdqs-internal service Other Operations 2018-05-02 2018-05-02
193605 Alert when elasticsearch writes are frozen for too long Other Operations 2018-05-03 2018-05-08
193664 Knock down puppet 4 deprecation warnings Other Operations 2018-05-03 2018-05-03
193649 migrate elasticsearch to stretch (from jessie) Other Operations 2018-05-03 2018-05-03
193655 rack/setup/install labstore1008 & labstore1009 Other Operations 2018-05-03 2018-05-17
125976 Run mediawiki::maintenance scripts in Beta Cluster Other Operations 2018-05-03 2018-05-02
193628 tungsten disk 1 and 8 SMART failure Other Operations 2018-05-03 2018-05-02
193738 dbstore1002 disk 5 not healthy Other Operations 2018-05-04 2018-05-03
193733 Move dispatching of wikidata to a dedicated node Other Operations 2018-05-04 2018-05-03
193766 Ship host syslogs to ELK Other Operations 2018-05-04 2018-05-08
193910 Build and upload jenkins-debian-glue_0.18.4-wmf3 for jessie Other Operations 2018-05-05 2018-05-04
193916 rename naos to deploy2001 and reinstall with stretch Other Operations 2018-05-05 2018-05-17
193915 rename wasat to mwmaint2001 and reinstall it with stretch Other Operations 2018-05-05 2018-05-04
194036 mw1230 sdb "Raw_Read_Error_Rate" SMART Other Operations 2018-05-08 2018-05-07
194172 mw2213 correctable memory errors Other Operations 2018-05-09 2018-05-08
194186 rack/setup/install cloudelastic100[1-4].eqiad.wmnet systems Other Operations 2018-05-09 2018-05-15
194184 rack/setup/install wdqs10[09|10].eqiad.wmnet Other Operations 2018-05-09 2018-05-17
194171 rdb2002 correctable memory errors Other Operations 2018-05-09 2018-05-08
194174 wtp2013 memory correctable errors Other Operations 2018-05-09 2018-05-08
194176 wtp2020 correctable memory errors Other Operations 2018-05-09 2018-05-08
194234 anaytics1032's BBU is not working correctly Other Operations 2018-05-10 2018-05-15
194249 kafka1023 correctable memory errors Other Operations 2018-05-10 2018-05-15
194558 Enable CAPTCHA on mailman instances Other Operations 2018-05-13 2018-05-12
155683 Rename (create anew) the TC team mailing list Other Operations 2018-05-13 2018-05-12
183832 Rename project mailing list for Africa Wikimedia Developers project Other Operations 2018-05-13 2018-05-12
186311 wikitech-l is mangling my PGP/MIME emails, causing signature validation to fail Other Operations 2018-05-13 2018-05-15
194653 Ban clients of WDQS which don't follow throttling directives for some time Other Operations 2018-05-15 2018-05-18
194669 Provide a mean to mass discard/reject subscription requests on Wikimedia mailing lists Other Operations 2018-05-15 2018-05-14
192751 Please upload large file to Wikimedia Commons Other Operations 2018-05-16 2018-05-15
194855 Degraded RAID on labvirt1020 Other Operations 2018-05-17 2018-05-18
194835 mw2182 crash Other Operations 2018-05-17 2018-05-16
194907 Degraded RAID on labvirt1019 Other Operations 2018-05-18 2018-05-18
194964 Connect or troubleshoot eth1 on labvirt1019 and labvirt1020 Other Operations 2018-05-19 2018-05-18
194998 Create custom deployment-prep role that allows editing of Designate records only Other Operations 2018-05-19 2018-05-20
194966 disk usage increase on maps servers Other Operations 2018-05-19 2018-05-22
194997 Track more detailed disk usage on maps servers Other Operations 2018-05-19 2018-05-22
195059 Cannot add or update records under DNS zones in Horizon Other Operations 2018-05-20 2018-05-19
195252 Update prometheus-varnish-exporter on debian to 1.4 Other Operations 2018-05-22 2018-05-21
195293 503 error attempting to open multiple projects (Wikipedia and meta wiki are loading very slowly) Other Operations 2018-05-23 2018-05-22
195289 Add Addshore & possibly other WMDE devs/deployers to the wikidata icinga contact list Other Operations 2018-05-23 2018-05-22
195306 Degraded RAID on elastic2020 Other Operations 2018-05-23 2018-05-22
148494 Add shell scripts CI validations Software Development 2017-01-02 2017-09-14
152950 E901 SyntaxError: invalid syntax is wrongly raised on using python's abc by jenkins python CI linter Software Development 2017-01-02 2017-03-27
144169 Flake8 for python files without extension in puppet repo Software Development 2017-01-02 2018-01-08
150560 More verbose messages from service-checker-swagger Software Development 2017-01-02 2017-05-23
154776 Puppet compiler: order resources for easy comparison between hosts Software Development 2017-01-07 2017-01-06
155705 confctl: log to SAL even if the selection doesn't match any host Software Development 2017-01-20 2017-01-19
157001 Puppet compiler: abort on git rebase conflict Software Development 2017-02-03 2017-02-02
157002 Puppet compiler: re-add the concurrency option NUM_THREADS Software Development 2017-02-03 2017-02-02
157133 Consider adding a --skip-conftool option to puppet-merge Software Development 2017-02-04 2018-02-23
159045 Update Puppet repo code that uses maniphest.update and maniphest.createtask conduit api Software Development 2017-02-26 2017-02-25
161545 Cumin: PuppetDB backend, allow to specify boolean values for resource parameters Software Development 2017-03-28 2017-03-27
163363 Switchdc: improvements Software Development 2017-04-20 2017-05-16
163365 Switchdc RO/RW: add check to test it editing a real wiki Software Development 2017-04-20 2017-04-19
164587 cumin could use randomization/splay options Software Development 2017-05-06 2017-05-05
167504 New tool to track package updates/status for hosts and images (debmonitor) Software Development 2017-06-10 2018-04-04
169304 Cumin masters: simplify usage in case of emergency Software Development 2017-07-01 2017-06-30
177385 Upgrade Cumin masters to stretch Software Development 2017-10-05 2017-10-18
179816 Cumin: create external backend for WMCS Puppet API Software Development 2017-11-07 2017-11-06
182028 DNS repo: add CI checks for obvious configuration errors Software Development 2017-12-05 2017-12-04
184435 Puppet tox: properly lint both Py2 and Py3 files Software Development 2018-01-09 2018-01-28
186818 Cumin: add --limit to randomly select N hosts Software Development 2018-02-09 2018-02-13
187709 Cumin feature idea: Prometheus backend Software Development 2018-02-20 2018-02-19
187751 wmf-auto-reimage: migrate script to Python3 Software Development 2018-02-20 2018-02-21
187773 Cumin: upgrade it to 3.0.1 in production Software Development 2018-02-21 2018-02-21
188112 cumin 3.0.1-1 is broken on labs master Software Development 2018-02-24 2018-04-07
191300 Debmonitor: deploy the agent across the fleet Software Development 2018-04-04 2018-04-03
191299 Debmonitor: deploy the service in production Software Development 2018-04-04 2018-05-03
191298 Release and deploy Debmonitor (patch management software) [Technology Goal 2017-18_Q4] Software Development 2018-04-04 2018-04-24
120486 add a https-only option to dynamicproxy Traffic 2017-01-02 2017-12-29
97051 adding new languages to DNS langs.tmpl doesn't work until zone template is edited as well Traffic 2017-01-02 2016-10-04
136703 Add LVS public endpoint checks that bypass caches Traffic 2017-01-02 2016-09-30
86650 Add support for setting weight=0 when depooling Traffic 2017-01-02 2017-07-31
97333 Align prod + beta-cluster LVS usage Traffic 2017-01-02 2016-09-30
154015 announce breaking change: http > https for entities in rdf Traffic 2017-01-02 2018-02-22
138546 Backend naming in VCL needs to use fqdn+port Traffic 2017-01-02 2016-09-30
138591 Backport iproute2 4.x from debian testing -> our jessie Traffic 2017-01-02 2017-07-27
144187 Better handling for one-hit-wonder objects Traffic 2017-01-02 2017-11-06
152091 Block hotlinking Traffic 2017-01-02 2016-12-05
120509 Cache education dashboard pages Traffic 2017-01-02 2016-09-30
94125 Central login notice appears on unencrypted API format=*fm pages, where reloading does not affect login status Traffic 2017-01-02 2016-10-04
149873 CentralNotice: Review and update Varnish caching for Special:BannerLoader Traffic 2017-01-02 2016-11-22
36670 Check all wikis for inclusions of http resources on https Traffic 2017-01-02 2016-10-04
146832 Clarify caching to enable direct Wikidata Query Service access by <mapframe/link> Traffic 2017-01-02 2016-12-15
131894 Collect Backend-Timing in Graphite (or Prometheus) Traffic 2017-01-02 2018-04-19
154017 compile number of http uses for http://www.wikidata.org/entity Traffic 2017-01-02 2018-02-22
134323 confctl: give regexen more freedom Traffic 2017-01-02 2016-10-04
134324 confctl select needs a -y flag? Traffic 2017-01-02 2016-10-04
112316 Configure varnish to use "Unconfigured domain" page for 404 Not Served (instead of generic error) Traffic 2017-01-02 2016-09-30
113817 Connect Hadoop records of the same request coming via different channels Traffic 2017-01-02 2016-10-04
26413 Consider DNSSec Traffic 2017-01-02 2017-10-23
153563 Consider switching to HTTPS for Wikidata query service links Traffic 2017-01-02 2017-11-12
133821 Content purges are unreliable Traffic 2017-01-02 2018-01-15
148422 cp3009: memory scrubbing error Traffic 2017-01-02 2018-01-03
133548 Create a secure redirect service for large count of non-canonical / junk domains Traffic 2017-01-02 2018-05-18
91820 Create HTTP verb and sticky cookie DC routing in VCL Traffic 2017-01-02 2018-05-20
132629 Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense Traffic 2017-01-02 2017-07-21
96499 dbtree loads third party resources (from jquery.com and google.com) Traffic 2017-01-02 2016-10-04
133001 Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames Traffic 2017-01-02 2017-11-08
96852 Define 3-host infra cluster for traffic pops Traffic 2017-01-02 2017-03-21
66214 Define an official thumb API Traffic 2017-01-02 2018-04-25
109331 Deleted files sometimes remain visible to non-privileged users if permanently linked Traffic 2017-01-02 2017-07-27
133410 Deploy TemplateStyles to WMF production Traffic 2017-01-02 2018-05-03
111899 Deprecate pybal SSH health checks Traffic 2017-01-02 2016-09-30
128409 Detect tools.wmflabs.org tools which are HTTP-only Traffic 2017-01-02 2017-10-15
146619 DNS domains registered to WMF no longer redirecting Traffic 2017-01-02 2016-09-30
127482 Enable VCL source-DC switching via confd Traffic 2017-01-02 2016-09-30
147209 etcd cluster has Raft Internal errors sporadically Traffic 2017-01-02 2016-10-04
96853 Evaluate Apache Traffic Server Traffic 2017-01-02 2017-06-21
122867 Evaluate the feasibility of cache invalidation for the action API Traffic 2017-01-02 2017-07-19
105657 Expires header for load.php should be relative to request time instead of cache time Traffic 2017-01-02 2018-02-07
153468 Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances Traffic 2017-01-02 2018-03-31
98165 Figure out an etcd deploy strategy that includes multi DC failure scenarios. Traffic 2017-01-02 2017-07-11
102178 Fix RESTBase support for wikitech.wikimedia.org Traffic 2017-01-02 2017-07-11
50133 ForeignAPIRepo wrongly returns non-protocol-relative URLs for original "thumbs" Traffic 2017-01-02 2017-06-27
143562 High number of failed inbound TFO connections in esams Mon-Fri Traffic 2017-01-02 2016-10-04
130904 Host rewrite for /static/ not applied to purges Traffic 2017-01-02 2016-09-30
108580 HTTPS for internal service traffic Traffic 2017-01-02 2016-11-16
104681 HTTPS Plans (tracking / high-level info) Traffic 2017-01-02 2017-07-11
86671 HTTPS RFC5077 session tickets encryption key rollovers Traffic 2017-01-02 2017-10-23
119038 Image cache issue when 'over-writing' an image on commons Traffic 2017-01-02 2016-04-28
94697 implement better failure-scenario geoip mapping in gdnsd Traffic 2017-01-02 2016-09-30
120121 Improve Varnish XFF processing for trusted proxies Traffic 2017-01-02 2016-10-04
109321 Inbound TLS for tier-1 varnish backend caches Traffic 2017-01-02 2016-09-30
125170 Internal DNS resolver responds with NXDOMAIN for localhost AAAA Traffic 2017-01-02 2016-09-30
104442 Investigate better DNS cache/lookup solutions Traffic 2017-01-02 2017-07-26
124418 Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan Traffic 2017-01-02 2018-03-19
92298 Investigate our mitigation strategy for HTTPS response length attacks Traffic 2017-01-02 2016-11-15
138093 Investigate query parameter normalization for MW/services Traffic 2017-01-02 2017-07-12
133717 Letsencrypt all the prod things we can - planning Traffic 2017-01-02 2017-01-18
141266 letsencrypt puppetization: add parallel rsa+ecdsa cert support Traffic 2017-01-02 2016-09-30
134447 letsencrypt puppetization: upgrade for scalability Traffic 2017-01-02 2016-09-30
129682 Look into solutions for replaying traffic to testing environment(s) Traffic 2017-01-02 2017-07-12
140365 Lower geodns TTLs from 600 (10min) to 300 (5min) Traffic 2017-01-02 2016-09-30
83467 LVS testing needs to include internal services testing Traffic 2017-01-02 2016-09-30
128188 Make CI run Varnish VCL tests Traffic 2017-01-02 2016-09-30
152882 Many misc wikis lack mobile domains Traffic 2017-01-02 2017-07-25
102367 Migrate tools.wmflabs.org to https only (and set HSTS) Traffic 2017-01-02 2018-05-14
116132 missing H2 coalesce for upload.wm.o for images ref'd in projects' page outputs Traffic 2017-01-02 2018-03-19
141480 mixed-content issues on planet.wikimedia.org Traffic 2017-01-02 2016-10-04
75944 Monitor Varnish caches on beta cluster have two varnishd process running Traffic 2017-01-02 2016-09-30
89838 Move proxy IP lists to META for Varnish XFF decoding Traffic 2017-01-02 2017-02-06
78421 m.{project}.org portal/redirect consistency Traffic 2017-01-02 2017-08-08
86915 nan and minnan subdomain redirects are a mess Traffic 2017-01-02 2017-08-24
81605 No IPv6 addresses on Wikimedia nameservers ns(0-2).wikimedia.org Traffic 2017-01-02 2016-09-30
148134 OCSP Stapling for Intermediates Traffic 2017-01-02 2016-11-01
109325 Outbound HTTPS for varnish backend instances Traffic 2017-01-02 2016-09-30
137747 Parametrization of VCL is inconsistent Traffic 2017-01-02 2016-09-30
118181 Planning for phasing out non-Forward-Secret TLS ciphers Traffic 2017-01-02 2017-05-08
99216 Please set up a CNAME for videoserver.wikimedia.org to Video Editing Server Traffic 2017-01-02 2016-09-30
118468 point wikilovesmonuments.org ns to wmf Traffic 2017-01-02 2016-09-30
144508 Point wikipedia.in to 205.147.101.160 instead of URL forward Traffic 2017-01-02 2016-10-04
101048 Policy decisions for new (and current) DNS domains registered to the WMF Traffic 2017-01-02 2018-03-26
150479 Prometheus varnish metric churn due to VCL reloads Traffic 2017-01-02 2017-03-16
54253 Protocol-relative URLs are poorly supported or unsupported by a number of HTTP clients Traffic 2017-01-02 2017-03-28
119372 Pybal IdleConnectionMonitor with TCP KeepAlive shows random fails if more than 100 servers are involved. Traffic 2017-01-02 2017-03-30
113597 pybal-related issue on host start can break service IPs... Traffic 2017-01-02 2017-02-10
137252 Redirect phabricator.mediawiki.org to phabricator.wikimedia.org Traffic 2017-01-02 2016-10-07
45250 Redo /beacon/impression system (formerly Special:RecordImpression) to remove extra round trips on all FR impressions (title was: S:RI should pyroperish) Traffic 2017-01-02 2017-03-27
147202 Removing support for AES128-SHA TLS cipher Traffic 2017-01-02 2016-10-03
134807 Replace test hostnames in datecenter-specific subdomains with dashed names Traffic 2017-01-02 2016-10-03
23027 Requests with utf-8 in the URL return a outdated page revision Traffic 2017-01-02 2016-10-04
56783 Respect X-Forwarded-For only from trustworthy sources Traffic 2017-01-02 2016-10-04
129839 restrict upload cache access for private wikis Traffic 2017-01-02 2016-09-30
111588 RFC: API-driven web front-end Traffic 2017-01-02 2018-05-15
149847 RFC: Use content hash based image / thumb URLs Traffic 2017-01-02 2018-04-12
114979 Run IPVS in a separate network namespace Traffic 2017-01-02 2017-09-18
120631 Security: Is it safe to enable Zero spoofing Traffic 2017-01-02 2017-07-20
50835 Separate Cache-Control header for proxy and client Traffic 2017-01-02 2017-05-30
131930 Set SPF (... -all) for toolserver.org Traffic 2017-01-02 2017-06-14
123854 Set up action API latency / error rate metrics & alerts Traffic 2017-01-02 2017-07-11
136944 Set up LVS connection sync Traffic 2017-01-02 2016-09-30
101525 Set up LVS for current AuthDNS Traffic 2017-01-02 2017-04-25
128374 Sort out analytics service dependency issues for cp* cache hosts Traffic 2017-01-02 2018-04-02
117435 Spike: CentralNotice: Verify that our Special:HideBanners cookie storm works as efficiently as possible Traffic 2017-01-02 2017-08-16
102848 Split GeoIP into a new component Traffic 2017-01-02 2017-11-13
127387 Split slash decoding from general percent normalization in Varnish VCL Traffic 2017-01-02 2017-11-14
128559 store.wikimedia.org HTTPS issues Traffic 2017-01-02 2018-04-26
148976 Strongswan Icinga check: do not report issues about depooled hosts Traffic 2017-01-02 2016-10-24
137979 Support brotli compression Traffic 2017-01-02 2017-10-10
78963 Support ESI for ResourceLoader Traffic 2017-01-02 2016-12-05
107236 Switch port 80 to nginx on primary clusters Traffic 2017-01-02 2016-09-30
117826 TEST: redirect small portion of unauthenticated desktop users to mobile web Traffic 2017-01-02 2017-10-02
147967 The WMF-Last-Access Set-Cookie header should follow RFC 2965 syntax rather than the pre-RFC Netscape format Traffic 2017-01-02 2016-10-17
150673 Thumb API: Varnish / CDN questions Traffic 2017-01-02 2017-08-16
150022 thumb.php should not set CC:no-cache on renderer 404 responses? Traffic 2017-01-02 2018-05-17
109776 Tilerator should purge Varnish cache Traffic 2017-01-02 2017-03-08
147648 Unexplained increase in thumbnail 500s Traffic 2017-01-02 2016-11-18
134893 Unhandled pybal error causing services to be depooled in etcd but not in lvs Traffic 2017-01-02 2017-01-26
96844 Update TLS/HTTP documentation on wikitech Traffic 2017-01-02 2016-09-30
128358 Uploading 1.2GB ogv results in 503 Traffic 2017-01-02 2016-12-13
106517 upload.wikimedia.org returns HTTP status code 503 for truncated urls, not 404 Traffic 2017-01-02 2017-11-30
101002 Use Upgrade Insecure Requests on Wikimedia wikis Traffic 2017-01-02 2017-10-23
133895 Varnish configuration for mobile domains should be coherent with Apache configuration Traffic 2017-01-02 2016-09-30
74186 Varnish: Mobile site redirect interferes with OAuth authorization process Traffic 2017-01-02 2017-11-04
148595 Visting [[c:File:FEZ_trial_gameplay_HD.webm]] in IE11 shows errors in developer console about insecure data:image/png;base64 "URL" Traffic 2017-01-02 2016-10-19
91372 $wgMFAnonymousEditing = true is sometimes not respected: cache? Traffic 2017-01-02 2016-05-04
127573 wikiknihy.cz - transfer to Wikimedia Czech Republic? Traffic 2017-01-02 2016-10-04
152622 Wikipedia.cz and other domains owned by WMCZ have invalid certificate Traffic 2017-01-02 2017-07-14
88861 wikipedia.lol Traffic 2017-01-02 2016-09-30
154702 Fix broken referer categorization for visits from Safari browsers Traffic 2017-01-06 2017-09-01
154801 Investigate varnishd child crashes when multiple nodes get depooled/pooled concurrently Traffic 2017-01-07 2017-02-07
155314 Varnish does not cache Action API responses when logged in Traffic 2017-01-15 2017-07-19
156462 Framework to transfer files over the LAN Traffic 2017-01-28 2018-05-17
82849 lvs servers report 'Memory allocation problem' on bootup Traffic 2017-02-11 2017-10-06
157786 Unhandled error stopping pybal: 'RunCommandMonitoringProtocol' object has no attribute 'checkCall' Traffic 2017-02-11 2017-02-10
119366 Disable caching on the main page for anonymous users Traffic 2017-02-13 2018-03-29
158599 Samsung Internet's desktop mode getting redirected to mobile site Traffic 2017-02-21 2017-07-19
146332 Create short link for outreachdashboard.wmflabs.org Traffic 2017-02-28 2017-02-28
159412 Convert all of our site.pp/roles to the role/profile paradigm Traffic 2017-03-03 2017-12-01
159411 Uniform cluster nomenclature across puppet Traffic 2017-03-03 2017-03-26
161148 AuthDNS CM/CI refactor Traffic 2017-03-23 2017-03-24
161256 multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert Traffic 2017-03-24 2017-03-28
161360 404 loading images from Virgin Media Traffic 2017-03-25 2017-03-28
114104 pybal doesn't fully manage LVS table leaving stale services (on IP change) Traffic 2017-03-29 2017-03-30
161517 Allow anonymous users to change interface language on Commons with ULS Traffic 2017-03-30 2017-07-25
79730 Add pybal check to ensure service IP is bound Traffic 2017-03-31 2017-03-30
81305 Make PyBal respect advertised BGP capabilities Traffic 2017-03-31 2017-03-30
162818 icinga alerts on nodejs services when a recdns server is depooled Traffic 2017-04-13 2017-06-27
163141 dbtree: make wasat a working backend and become active-active Traffic 2017-04-18 2017-12-11
163541 cache hosts should auto-repool iff OCSP files are sane Traffic 2017-04-22 2017-04-21
164259 Add VSL error counters to Varnishkafka stats Traffic 2017-05-03 2018-04-05
164327 replace ulsfo aging servers Traffic 2017-05-03 2017-12-18
164456 Migrate to nginx-light Traffic 2017-05-05 2017-05-09
164460 Use DNS discovery record for deployment CNAME Traffic 2017-05-05 2017-05-09
164609 Merge cache_misc into cache_text functionally Traffic 2017-05-06 2017-06-27
164768 Explicitly limit varnishd transient storage Traffic 2017-05-09 2017-06-20
165252 cp1053 possible hardware issues Traffic 2017-05-16 2017-10-23
165560 Artificial spike in offset of unique devices from November to February 6th on wikidata Traffic 2017-05-18 2017-06-26
165764 Fully-redundant LVS clusters using Pybal per-service MED feature Traffic 2017-05-20 2017-11-23
165765 Refactor pybal/LVS config for shared failover Traffic 2017-05-20 2017-06-01
133178 RESTBase support for www.wikimedia.org missing Traffic 2017-05-31 2018-01-31
166782 wikimediafoundation.org's language selector is confusing to most visitors who don't have accounts there Traffic 2017-06-02 2017-08-24
167239 Redirect status.wikipedia.org to status.wikimedia.org Traffic 2017-06-07 2017-06-14
167513 Redirect lzh.wikipedia to zh-classical.wikipedia Traffic 2017-06-10 2017-06-12
125226 [feature request] Redirect root API path to docs page Traffic 2017-06-15 2017-06-14
167906 Make API usage limits easier to understand, implement, and more adaptive to varying request costs / concurrency limiting Traffic 2017-06-15 2017-12-22
167400 Disable serving unpatrolled new files to Wikipedia Zero users Traffic 2017-06-16 2018-02-06
167972 Respect host header in RESTBase, and redirect /rest_v1 to /rest_v1/ Traffic 2017-06-16 2017-06-20
168539 Unhandled pybal error: OpenSSL.SSL.Error - ssl handshake failure Traffic 2017-06-22 2017-06-22
169765 pybal should automatically reconnect to etcd Traffic 2017-07-06 2018-02-16
170567 Support TLSv1.3 Traffic 2017-07-14 2017-10-23
170605 Unable to render file from upload.wikimedia.org "Error 349 ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION" Traffic 2017-07-14 2017-08-09
171168 cp1050 apparently stuck while "Initializing firmware interfaces..." Traffic 2017-07-21 2017-11-09
99531 [Task] move wikiba.se webhosting to wikimedia misc-cluster Traffic 2017-07-22 2018-05-07
171498 Implement machine-local forwarding DNS caches Traffic 2017-07-25 2017-09-13
171470 Monitor DNS delegations Traffic 2017-07-25 2017-07-24
171850 Backport ipvsadm Traffic 2017-07-28 2017-10-11
172123 Determine how to upload Zim files to Swift infrastructure Traffic 2017-08-01 2018-04-06
172148 Determine URL paths for Zim files Traffic 2017-08-01 2018-03-12
172103 IPVS issues with UDP services, pybal depooling strategy Traffic 2017-08-01 2017-08-01
172124 PyBal Feature: progressive depooling strategy for monitored failures Traffic 2017-08-01 2017-09-06
173966 Like nan.wikipedia.org, redirect other nan.*.org to the proper zh-min-nan.*.org domains Traffic 2017-08-24 2017-08-23
174342 Missing IP addresses for Maroc Telecom Traffic 2017-08-29 2017-11-16
174432 Unclear LVS bandwidth graph in "load balancers" dashboard Traffic 2017-08-30 2017-08-30
174932 Recurrent 'mailbox lag' critical alerts and 500s Traffic 2017-09-05 2018-03-29
174960 Varnish does not vary elasticsearch query by request body Traffic 2017-09-05 2017-09-07
175203 Implement stateless TCP balancing in our LVS servers Traffic 2017-09-07 2017-09-07
175319 cp1066 unexplained 503 spikes Traffic 2017-09-09 2017-09-19
175636 prometheus -> grafana stats for per-numa-node meminfo Traffic 2017-09-12 2017-10-02
176366 Decom cp4005-8,13-16 (8 nodes) Traffic 2017-09-21 2018-05-03
176388 pybal: race condition in alerts instrumentation Traffic 2017-09-22 2017-09-28
176875 Allow access to wdqs.svc.eqiad.wmnet on port 8888 Traffic 2017-09-28 2018-03-05
177742 Investigate Chrony as a replacement for ISC ntpd Traffic 2017-10-10 2018-03-23
177927 Refactor kafka_config.rb and and kafka_cluster_name.rb in puppet to avoid explicit hiera calls Traffic 2017-10-12 2018-04-16
177961 Upgrade LVS servers to stretch Traffic 2017-10-12 2018-03-12
178173 Renew unified certificates 2017 Traffic 2017-10-14 2017-11-16
178535 decommission lvs400[1-4].ulsfo.wmnet Traffic 2017-10-19 2017-11-27
178592 decommission/replace bast4001.wikimedia.org Traffic 2017-10-20 2017-12-12
178815 decom cp40(09|1[078]) Traffic 2017-10-24 2017-11-17
158604 Investigate usefulness of SameSite cookies for logged-in accounts Traffic 2017-10-25 2017-10-25
179025 LVS hosts should have static-mapped IPv6 on all virtual interfaces Traffic 2017-10-26 2017-10-25
179026 LVS IPv6 IPs should all be recorded in DNS Traffic 2017-10-26 2017-10-25
179027 Puppetize LVS interface IP sets per-DC for easy use in ferm rules Traffic 2017-10-26 2017-11-20
179050 setup bast4002/WMF7218 Traffic 2017-10-27 2017-11-09
179197 Investigate what caused the the unattended varnish upgrade in Beta Cluster Traffic 2017-10-28 2017-11-07
179953 cp3043 disk failure Traffic 2017-11-08 2018-01-03
180257 Puppet / LVS: confusion in service vs IP name Traffic 2017-11-11 2017-11-14
180269 Wikimedia's recent upgrade to nginx v. 1.13.6 breaks older Android HTTP libraries Traffic 2017-11-11 2017-11-14
180329 Add CI to all operations/software/varnish/* repositories and archive obsolete ones Traffic 2017-11-14 2017-11-20
180407 Change "CP" cookie from subdomain to project level Traffic 2017-11-14 2017-11-14
180434 Uncacheable content handling: hfp vs hfm Traffic 2017-11-15 2017-11-16
180655 Phabricator and Gerrit: Improve the way that maintenance downtime is communicated to users. Traffic 2017-11-17 2017-12-07
180712 VCL: handling of uncacheable responses in wikimedia-common Traffic 2017-11-17 2018-03-23
180921 Referrer policy for browsers which only support the old spec Traffic 2017-11-20 2018-05-20
181315 Varnish HTTP response from app servers taking 160s (only 0.031s inside Apache) Traffic 2017-11-25 2018-05-05
181368 Log source port for anonymous users and expose it for sysops/checkusers Traffic 2017-11-27 2017-11-27
181569 rack/setup scs-eqsin.mgmt.eqsin.wmnet Traffic 2017-11-29 2018-02-13
182993 TLS security review of the Kafka stack Traffic 2017-12-16 2018-04-17
183177 memory errors not showing in icinga Traffic 2017-12-19 2018-05-14
183554 Unified certs bloat reduction? Traffic 2017-12-23 2017-12-22
184293 rack/setup/install lvs101[3-6] Traffic 2018-01-06 2018-04-30
184534 Cached page previews not shown when refreshed Traffic 2018-01-10 2018-03-30
184942 Deprecate python varnish cachestats Traffic 2018-01-16 2018-04-30
185239 Puppet hosts with signed certificate present on agent but not master Traffic 2018-01-19 2018-01-22
185350 Vet reliability of the response_size field for data analysis purposes Traffic 2018-01-20 2018-02-05
186732 Decide on Cache-Control headers for map tiles Traffic 2018-02-08 2018-05-08
187157 cp5006 unresponsive Traffic 2018-02-14 2018-04-13
188087 Some etcd connections not established at startup Traffic 2018-02-24 2018-02-23
188776 Move Foundation Wiki to new URL when new Wikimedia Foundation website launches Traffic 2018-03-03 2018-04-30
188807 Update documentation for "https" field in X-Analytics Traffic 2018-03-04 2018-03-12
188804 Investigate and fix odd uri_host values Traffic 2018-03-06 2018-03-07
189250 WP Zero workarounds for eqsin Traffic 2018-03-09 2018-03-21
189305 cp3034: Uncorrectable Memory Error Traffic 2018-03-10 2018-03-19
189290 Tune systemd journal rate limiting for PyBal Traffic 2018-03-10 2018-04-16
190244 en-wp.org certificate error Traffic 2018-03-22 2018-04-18
188561 SSL cert for links.email.wikimedia.org Traffic 2018-03-23 2018-04-17
190607 cp3048 hardware issues Traffic 2018-03-25 2018-03-26
190843 Spammy events coming our way for sites such us https://ru.wikipedia.kim Traffic 2018-03-28 2018-03-29
190992 prometheus: slow dashboards due to suboptimal query_range performance Traffic 2018-03-30 2018-03-29
191017 Unwanted service startups and their triggers Traffic 2018-03-30 2018-03-29
190993 Upgrade pybal-test instances to stretch Traffic 2018-03-30 2018-04-12
147162 upload.wikimedia.org returns HTTP 501 instead of 416 for non-satisfiable byte ranges Traffic 2018-03-30 2018-03-29
191393 Puppet: tlsproxy localssl default_server make a Notify at each run Traffic 2018-04-05 2018-04-18
191940 Investigate 2018-04-10 global traffic drop Traffic 2018-04-11 2018-04-16
192082 lvs2006 Embedded Flash/SD-CARD iLO errors Traffic 2018-04-13 2018-04-12
192206 Remove wildcard vhost for *.wikimedia.org Traffic 2018-04-15 2018-05-07
192280 sda failure in hydrogen.wikimedia.org Traffic 2018-04-17 2018-04-17
192368 Unconditional return(deliver) in vcl_hit Traffic 2018-04-18 2018-04-27
192437 Pybal support of configuration from the kubernetes API Traffic 2018-04-19 2018-05-21
192555 Begin execution of non-forward-secret ciphers deprecation Traffic 2018-04-20 2018-05-14
192559 Establish timeline and methodology for upcoming deprecation of non-forward-secret ciphers and TLSv1.0 Traffic 2018-04-20 2018-04-19
193521 Consider adding expect-CT: header to enforce certificate transparency Traffic 2018-05-02 2018-05-07
193445 Update Media dashboard in Grafana to use Prometheus metrics Traffic 2018-05-03 2018-05-07
193865 Enable numa_networking on all caches Traffic 2018-05-05 2018-05-07
194031 Setup a new PKI software as an alternative to the puppet CA for managing services certificates Traffic 2018-05-08 2018-05-21
194380 Identify bots using AES128-SHA maintainers running on toolforge Traffic 2018-05-11 2018-05-17
194757 cp1068 memory correctable errors Traffic 2018-05-16 2018-05-16
194724 Deprecate `base::service_unit` in puppet Traffic 2018-05-16 2018-05-16
194814 Remove unnecessary response headers Traffic 2018-05-17 2018-05-17
194962 Create and deploy a centralized letsencrypt service Traffic 2018-05-19 2018-05-20
194965 gdnsd plugin support for ACME DNS challenges Traffic 2018-05-19 2018-05-22
195327 Normalise the Accept-Language header for REST API requests Traffic 2018-05-23 2018-05-22