TechnicalOperations Status Report

Project Operations from 2018-01-01 to 2018-01-19

Help

Network Operations 183390 unrack/decom pfw1-eqiad and pfw2-eqiad Unknown None
Network Operations 185228 replace ulsfo access switches Screep None
Network Operations 185171 replace mr1-eqiad Screep None
Network Operations 185151 replace msw1-esams Screep None
Network Operations 184067 Complete router migration from cr1-esams to cr3-esams Screep None
Network Operations 176338 eqiad: networking audit for support contract renewal In-Scope Done None
Network Operations 174637 Setup esams atlas anchor In-Scope Open None
Network Operations 98006 Anycast (Auth)DNS In-Scope Open None
Network Operations 176427 unrack/decom pfw1-codfw and pfw2-codfw In-Scope Open None
Network Operations 174616 set up cr3-esams In-Scope Open None
Network Operations 167299 Upgrade BIOS/RBSU/etc on lvs1007 In-Scope Open None
Network Operations 181036 Pull netflow data in realtime from Kafka via Tranquillity/Spark In-Scope Open None
Network Operations 163674 Frequent RST returned by appservers to LVS hosts In-Scope Open None
Network Operations 82038 create a test for multicast relay In-Scope Open None
Network Operations 179042 Setup eqsin RIPE Atlas anchor In-Scope Open None
Network Operations 167306 ospf link-protection In-Scope Open None
Network Operations 173698 Backfill librenms data in graphite with historical RRDs In-Scope Open None
Network Operations 150256 Re-setup lvs1007-lvs1012, replace lvs1001-lvs1006 In-Scope Open None
Network Operations 157435 Review ACLs for the Analytics VLAN In-Scope Open None
Network Operations 150264 Icinga check for VRRP In-Scope Open None
Network Operations 172459 eqiad row D switch upgrade In-Scope Open None
Network Operations 170144 Evaluate NetBox as a Racktables replacement & IPAM In-Scope Open None
Network Operations 176337 esams: networking audit for support contract renewal In-Scope Open None
Network Operations 180179 Evaluate the possibility to add Juniper images to Openstack In-Scope Open None
Network Operations 86541 setup wifi in codfw In-Scope Open None
Network Operations 120425 dumps.wikimedia.org seems to have poor throughput towards some destinations In-Scope Open None
Network Operations 133387 Enabling IGMP snooping on QFX switches breaks IPv6 (HTCP purges flood across codfw) In-Scope Open None
Network Operations 171032 Investigate lvs IP pages during codfw row C switch upgrade In-Scope Open None
Network Operations 122406 Consider renumbering Labs to separate address spaces In-Scope Open None
Network Operations 83992 Juniper monitoring In-Scope Open None
Network Operations 167840 Merge AS14907 with AS43821 In-Scope Open None
Network Operations 167842 Find a new PIM RP IP In-Scope Open None
Traffic 183902 Swift invalid range requests causing 501s Screep None
Traffic 184255 Lower varnish caching length on doc.wikimedia.org Screep None
Traffic 182993 TLS security review of the Kafka stack Unknown 13.0
Traffic 184245 Create some mechanism for instances in projects to modify the project Designate records Screep None
Traffic 185239 Puppet hosts with signed certificate present on agent but not master Screep None
Traffic 184293 rack/setup/install lvs101[3-6] Screep None
Traffic 184448 Upgrade cache_text to Varnish 5 Elaborated None
Traffic 184942 Deprecate python varnish cachestats Elaborated None
Traffic 184715 pybal's "can-depool" logic only takes downServers into account Screep Done None
Traffic 183926 Limit http methods reported by varnishmtail Elaborated Done None
Traffic 184721 Alert instrumentation returning 500 errors Screep Done None
Traffic 177199 Add Prometheus client support for varnish/statsd metrics daemons In-Scope Done None
Traffic 166965 Degraded RAID on lvs3001 In-Scope Done None
Traffic 177228 Multiple systems in esams OE10 showing PSU failures In-Scope Done None
Traffic 184833 Inconsistent behavior when fetching redirected pages with Cache-Control header Screep Done None
Traffic 118468 point wikilovesmonuments.org ns to wmf In-Scope Open None
Traffic 164259 Add VSL error counters to Varnishkafka stats In-Scope Open None
Traffic 181315 load.php response taking 160s (of which only 0.031s in Apache) In-Scope Open None
Traffic 133717 Letsencrypt all the prod things we can - planning In-Scope Open None
Traffic 165764 Fully-redundant LVS clusters using Pybal per-service MED feature In-Scope Open None
Traffic 172148 Determine URL paths for Zim files In-Scope Open None
Traffic 54253 Protocol-relative URLs are poorly supported or unsupported by a number of HTTP clients In-Scope Open None
Traffic 131894 Collect Backend-Timing in Graphite (or Prometheus) In-Scope Open None
Traffic 117826 TEST: redirect small portion of unauthenticated desktop users to mobile web In-Scope Open None
Traffic 109331 Deleted files sometimes remain visible to non-privileged users if permanently linked In-Scope Open None
Traffic 78963 Support ESI for ResourceLoader In-Scope Open None
Traffic 128358 Uploading 1.2GB ogv results in 503 In-Scope Open None
Traffic 167906 Make API usage limits easier to understand, implement, and more adaptive to varying request costs / concurrency limiting In-Scope Open None
Traffic 136737 Fix lvs1001-6 storage In-Scope Open None
Traffic 81305 Make PyBal respect advertised BGP capabilities In-Scope Open None
Traffic 163141 dbtree: make wasat a working backend and become active-active In-Scope Open None
Traffic 152622 Wikipedia.cz and other domains owned by WMCZ have invalid certificate In-Scope Open None
Traffic 122867 Evaluate the feasibility of cache invalidation for the action API In-Scope Open None
Traffic 127387 Split slash decoding from general percent normalization in Varnish VCL In-Scope Open None
Traffic 179050 setup bast4002/WMF7218 In-Scope Open None
Traffic 75944 Monitor Varnish caches on beta cluster have two varnishd process running In-Scope Open None
Traffic 134447 letsencrypt puppetization: upgrade for scalability In-Scope Open None
Traffic 97051 adding new languages to DNS langs.tmpl doesn't work until zone template is edited as well In-Scope Open None
Traffic 107236 Switch port 80 to nginx on primary clusters In-Scope Open None
Traffic 167060 en.wiki domain owned by us, but isn't hosted by us?? In-Scope Open None
Traffic 91372 $wgMFAnonymousEditing = true is sometimes not respected: cache? In-Scope Open None
Traffic 179025 LVS hosts should have static-mapped IPv6 on all virtual interfaces In-Scope Open None
Traffic 180407 Change "CP" cookie from subdomain to project level In-Scope Open None
Traffic 102178 Fix RESTBase support for wikitech.wikimedia.org In-Scope Open None
Traffic 123854 Set up action API latency / error rate metrics & alerts In-Scope Open None
Traffic 86915 nan and minnan subdomain redirects are a mess In-Scope Open None
Traffic 171470 Monitor DNS delegations In-Scope Open None
Traffic 149847 Use content hash based image / thumb URLs In-Scope Open None
Traffic 154017 compile number of http uses for http://www.wikidata.org/entity In-Scope Open None
Traffic 164609 Merge cache_misc into cache_text functionally In-Scope Open None
Traffic 159411 Uniform cluster nomenclature across puppet In-Scope Open None
Traffic 146619 DNS domains registered to WMF no longer redirecting In-Scope Open None
Traffic 152091 Block hotlinking In-Scope Open None
Traffic 148422 cp3009: memory scrubbing error In-Scope Open None
Traffic 180433 Upgrade cache_upload to Varnish 5 In-Scope Open None
Traffic 144187 Better handling for one-hit-wonder objects In-Scope Open None
Traffic 133548 Create a secure redirect service for large count of non-canonical / junk domains In-Scope Open None
Traffic 156027 Configuration for Asia Cache DC hosts In-Scope Open None
Traffic 105657 Expires header for load.php should be relative to request time instead of cache time In-Scope Open None
Traffic 161256 multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert In-Scope Open None
Traffic 146332 Create short link for outreachdashboard.wmflabs.org In-Scope Open None
Traffic 153563 Consider switching to HTTPS for Wikidata query service links In-Scope Open None
Traffic 96499 dbtree loads third party resources (from jquery.com and google.com) In-Scope Open None
Traffic 164768 Explicitly limit varnishd transient storage In-Scope Open None
Traffic 172123 Determine how to upload Zim files to Swift infrastructure In-Scope Open None
Traffic 174342 Missing IP addresses for Maroc Telecom In-Scope Open None
Traffic 179197 Investigate what caused the the unattended varnish upgrade in Beta Cluster In-Scope Open None
Traffic 119372 Pybal IdleConnectionMonitor with TCP KeepAlive shows random fails if more than 100 servers are involved. In-Scope Open None
Traffic 148134 OCSP Stapling for Intermediates In-Scope Open None
Traffic 156320 $wgServer with initial https:// does not force HTTPS (wgSecureLogin) In-Scope Open None
Traffic 129682 Look into solutions for replaying traffic to testing environment(s) In-Scope Open None
Traffic 180069 Pybal should be able to advertise to multiple routers In-Scope Open None
Traffic 180257 Puppet / LVS: confusion in service vs IP name In-Scope Open None
Traffic 143562 High number of failed inbound TFO connections in esams Mon-Fri In-Scope Open None
Traffic 159412 Convert all of our site.pp/roles to the role/profile paradigm In-Scope Open None
Traffic 178567 Server error (500) while trying to download files from Commons from PAWS In-Scope Open None
Traffic 144508 Point wikipedia.in to 205.147.101.160 instead of URL forward In-Scope Open None
Traffic 99531 [Task] move wikiba.se webhosting to wikimedia misc-cluster In-Scope Open None
Traffic 133821 Content purges are unreliable In-Scope Open None
Traffic 180269 Wikimedia's recent upgrade to nginx v. 1.13.6 breaks older Android HTTP libraries In-Scope Open None
Traffic 167400 Disable serving unpatrolled new files to Wikipedia Zero users In-Scope Open None
Traffic 146832 Clarify caching to enable direct Wikidata Query Service access by <mapframe/link> In-Scope Open None
Traffic 98165 Figure out an etcd deploy strategy that includes multi DC failure scenarios. In-Scope Open None
Traffic 179156 503 spikes and resulting API slowness starting 18:45 October 26 In-Scope Open None
Traffic 181801 "Error: 404, Requested domainname does not exist" when accessing Commons categories/images; works on mobile page In-Scope Open None
Traffic 109776 Tilerator should purge Varnish cache In-Scope Open None
Traffic 161360 404 loading images from Virgin Media In-Scope Open None
Traffic 23027 Requests with utf-8 in the URL return a outdated page revision In-Scope Open None
Traffic 178173 Renew unified certificates 2017 In-Scope Open None
Traffic 162683 Network hardware purchasing for Asia Cache DC In-Scope Open None
Traffic 99216 Please set up a CNAME for videoserver.wikimedia.org to Video Editing Server In-Scope Open None
Traffic 128374 Sort out analytics service dependency issues for cp* cache hosts In-Scope Open None
Traffic 154801 Investigate varnishd child crashes when multiple nodes get depooled/pooled concurrently In-Scope Open None
Traffic 119366 Disable caching on the main page for anonymous users In-Scope Open None
Traffic 159346 convert mail servers from GS to LE certificates In-Scope Open None
Traffic 148976 Strongswan Icinga check: do not report issues about depooled hosts In-Scope Open None
Traffic 127485 Enable VCL applayer datacenter-switch via confd In-Scope Open None
Traffic 104681 HTTPS Plans (tracking / high-level info) In-Scope Open None
Traffic 120486 add a https-only option to dynamicproxy In-Scope Open None
Traffic 170567 Support TLSv1.3 In-Scope Open None
Traffic 133001 Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames In-Scope Open 0.0
Traffic 106517 upload.wikimedia.org returns HTTP status code 503 for truncated urls, not 404 In-Scope Open None
Traffic 102848 Split GeoIP into a new component In-Scope Open None
Traffic 128409 Detect tools.wmflabs.org tools which are HTTP-only In-Scope Open None
Traffic 79730 Add pybal check to ensure service IP is bound In-Scope Open None
Traffic 119038 Image cache issue when 'over-writing' an image on commons In-Scope Open None
Traffic 137990 Zero: Investigate removing the limit on carrier tagging to m-dot and zero-dot requests In-Scope Open None
Traffic 101525 Set up LVS for current AuthDNS In-Scope Open None
Traffic 133410 Deploy TemplateStyles to WMF production In-Scope Open None
Traffic 96844 Update TLS/HTTP documentation on wikitech In-Scope Open None
Traffic 102367 Migrate tools.wmflabs.org to https only (and set HSTS) In-Scope Open None
Traffic 82849 lvs servers report 'Memory allocation problem' on bootup In-Scope Open None
Traffic 177742 Investigate Chrony as a replacement for ISC ntpd In-Scope Open None
Traffic 172124 PyBal Feature: progressive depooling strategy for monitored failures In-Scope Open None
Traffic 130904 Host rewrite for /static/ not applied to purges In-Scope Open None
Traffic 124418 Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan In-Scope Open None
Traffic 125170 Internal DNS resolver responds with NXDOMAIN for localhost AAAA In-Scope Open None
Traffic 176875 Allow access to wdqs.svc.eqiad.wmnet on port 8888 In-Scope Open None
Traffic 167513 Redirect lzh.wikipedia to zh-classical.wikipedia In-Scope Open None
Traffic 173966 Like nan.wikipedia.org, redirect other nan.*.org to the proper zh-min-nan.*.org domains In-Scope Open None
Traffic 155314 Varnish does not cache Action API responses when logged in In-Scope Open None
Traffic 120121 Improve Varnish XFF processing for trusted proxies In-Scope Open None
Traffic 109325 Outbound HTTPS for varnish backend instances In-Scope Open None
Traffic 168529 Upgrade to Varnish 5 In-Scope Open None
Traffic 175636 prometheus -> grafana stats for per-numa-node meminfo In-Scope Open None
Traffic 36670 Check all wikis for inclusions of http resources on https In-Scope Open None
Traffic 138546 Backend naming in VCL needs to use fqdn+port In-Scope Open None
Traffic 158599 Samsung Internet's desktop mode getting redirected to mobile site In-Scope Open None
Traffic 127482 Enable VCL source-DC switching via confd In-Scope Open None
Traffic 176366 Decom cp4005-8,13-16 (8 nodes) In-Scope Open None
Traffic 150673 Thumb API: Varnish / CDN questions In-Scope Open None
Traffic 137252 Redirect phabricator.mediawiki.org to phabricator.wikimedia.org In-Scope Open None
Traffic 171168 cp1050 apparently stuck while "Initializing firmware interfaces..." In-Scope Open None
Traffic 128188 Make CI run Varnish VCL tests In-Scope Open None
Traffic 133149 Move californium to an internal host? In-Scope Open None
Traffic 74186 Varnish: Mobile site redirect interferes with OAuth authorization process In-Scope Open None
Traffic 78421 m.{project}.org portal/redirect consistency In-Scope Open None
Traffic 170605 ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION In-Scope Open None
Traffic 162818 icinga alerts on nodejs services when a recdns server is depooled In-Scope Open None
Traffic 83467 LVS testing needs to include internal services testing In-Scope Open None
Traffic 104442 Investigate better DNS cache/lookup solutions In-Scope Open None
Traffic 120509 Cache education dashboard pages In-Scope Open None
Traffic 154702 Fix broken referer categorization for visits from Safari browsers In-Scope Open None
Traffic 129839 restrict upload cache access for private wikis In-Scope Open None
Traffic 119396 Create globally-unique varnish cache cluster port/instancename mappings In-Scope Open None
Traffic 128559 store.wikimedia.org HTTPS issues In-Scope Open None
Traffic 177927 Refactor kafka_config.rb and and kafka_cluster_name.rb in puppet to avoid explicit hiera calls In-Scope Open None
Traffic 127573 wikiknihy.cz - transfer to Wikimedia Czech Republic? In-Scope Open None
Traffic 134893 Unhandled pybal error causing services to be depooled in etcd but not in lvs In-Scope Open None
Traffic 56783 Respect X-Forwarded-For only from trustworthy sources In-Scope Open None
Traffic 179204 setup/deploy dns400[12]/wmf721[56] In-Scope Open None
Traffic 125938 PHP fatal errors causing Varnish to return 503 - "Junk after gzip data" In-Scope Open None
Traffic 180712 VCL: handling of uncacheable responses in wikimedia-common In-Scope Open None
Traffic 66214 Define an official thumb API In-Scope Open None
Traffic 111588 RFC: API-driven web front-end In-Scope Open None
Traffic 120631 Security: Is it safe to enable Zero spoofing In-Scope Open None
Traffic 117435 Spike: CentralNotice: Verify that our Special:HideBanners cookie storm works as efficiently as possible In-Scope Open 2.0
Traffic 114104 pybal doesn't fully manage LVS table leaving stale services (on IP change) In-Scope Open None
Traffic 137979 Support brotli compression In-Scope Open None
Traffic 177961 Upgrade LVS servers to stretch In-Scope Open None
Traffic 141266 letsencrypt puppetization: add parallel rsa+ecdsa cert support In-Scope Open None
Traffic 166782 wikimediafoundation.org's language selector is confusing to most visitors who don't have accounts there In-Scope Open None
Traffic 165560 Artificial spike in offset of unique devices from November to February 6th on wikidata In-Scope Open None
Traffic 183916 Create subdomain for Research landing page Screep Open None
Traffic 175203 Implement stateless TCP balancing in our LVS servers In-Scope Open None
Traffic 150022 thumb_handler.php should not set CC:no-cache on renderer 404 responses? In-Scope Open None
Traffic 165765 Refactor pybal/LVS config for shared failover In-Scope Open None
Traffic 108580 HTTPS for internal service traffic In-Scope Open None
Traffic 156032 Server hardware installation for Asia Cache DC In-Scope Open None
Traffic 164327 replace ulsfo aging servers In-Scope Open None
Traffic 171498 Implement machine-local forwarding DNS caches In-Scope Open None
Traffic 112316 Configure varnish to use "Unconfigured domain" page for 404 Not Served (instead of generic error) In-Scope Open None
Traffic 180329 Add CI to all operations/software/varnish/* repositories and archive obsolete ones In-Scope Open None
Traffic 94125 Central login notice appears on unencrypted API format=*fm pages, where reloading does not affect login status In-Scope Open None
Traffic 91820 Create HTTP verb and sticky cookie DC routing in VCL In-Scope Open None
Traffic 132629 Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense In-Scope Open None
Traffic 89838 Move proxy IP lists to META for Varnish XFF decoding In-Scope Open None
Traffic 88861 wikipedia.lol In-Scope Open None
Traffic 180434 Uncacheable content handling: hfp vs hfm In-Scope Open None
Traffic 172103 IPVS issues with UDP services, pybal depooling strategy In-Scope Open None
Traffic 170847 Icinga check for pybal HTTP connections to etcd In-Scope Open None
Traffic 163541 cache hosts should auto-repool iff OCSP files are sane In-Scope Open None
Traffic 121561 Encrypt Kafka traffic, and restrict access via ACLs In-Scope Open 0.0
Traffic 178535 decommission lvs400[1-4].ulsfo.wmnet In-Scope Open None
Traffic 45250 Redo /beacon/impression system (formerly Special:RecordImpression) to remove extra round trips on all FR impressions (title was: S:RI should pyroperish) In-Scope Open None
Traffic 180921 Referrer policy for browsers which only support the old spec In-Scope Open None
Traffic 149873 CentralNotice: Review and update Varnish caching for Special:BannerLoader In-Scope Open 2.0
Traffic 138093 Investigate query parameter normalization for MW/services In-Scope Open None
Traffic 170518 Non zero rated LVS IPs In-Scope Open None
Traffic 178815 decom cp40(09|1[078]) In-Scope Open None
Traffic 161517 Allow anonymous users to change interface language on Commons with ULS In-Scope Open None
Traffic 118181 Planning for phasing out non-Forward-Secret TLS ciphers In-Scope Open None
Traffic 180978 Switch on http/2 in apache for gerrit In-Scope Open None
Traffic 178592 decommission/replace bast4001.wikimedia.org In-Scope Open None
Traffic 178151 Add UDP monitor for pybal In-Scope Open None
Traffic 180792 Remove 3DES patch from OpenSSL builds In-Scope Open None
Traffic 153468 Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances In-Scope Open None
Traffic 133895 Varnish configuration for mobile domains should be coherent with Apache configuration In-Scope Open None
Traffic 164456 Migrate to nginx-light In-Scope Open None
Traffic 147202 Removing support for AES128-SHA TLS cipher In-Scope Open None
Traffic 141480 mixed-content issues on planet.wikimedia.org In-Scope Open None
Traffic 150479 Prometheus varnish metric churn due to VCL reloads In-Scope Open None
Traffic 152882 Many misc wikis lack mobile domains In-Scope Open None
Traffic 131930 Set SPF (... -all) for toolserver.org In-Scope Open None
Traffic 171850 Backport ipvsadm In-Scope Open None
DBA 185084 Allow use of EtcdConfig to configure slave databases Screep None
DBA 184805 Move some wikis to s5 Screep None
DBA 183469 Setup newer machines and replace all old misc (m*) and x1 eqiad machines Unknown None
DBA 182556 Decommission db1034 Unknown None
DBA 184797 Move mariadb_maintenance away from terbium/wasat (mediawiki_maintenance) Screep None
DBA 184262 Decommission db1039 Elaborated None
DBA 183470 Setup newer machines and replace all old misc (m*) and x1 codfw machines Unknown None
DBA 184699 Generate consistent logical database backups in CODFW Screep None
DBA 174763 Decommission db1026 In-Scope Done None
DBA 183708 Degraded RAID on db1001 Unknown Done None
DBA 175264 Decommission db1049 In-Scope Done None
DBA 184836 Degraded RAID on db2036 Screep Done None
DBA 183750 pc2005 crashed: CPU2 internal error Unknown Done None
DBA 175679 Decommission db1048 (was Move m3 slave to db1059) In-Scope Done None
DBA 173570 Decommission db1015 In-Scope Done None
DBA 178162 Decommission db1050 In-Scope Done None
DBA 181696 Decommission db1044 In-Scope Done None
DBA 182896 Rack and setup db1113 and db1114 Unknown Done None
DBA 183887 db2054: Disk with predictive failure Screep Done None
DBA 184464 Degraded RAID on db2060 Screep Done None
DBA 181378 Decommission db1021 In-Scope Done None
DBA 169501 Move some masters away from B6 In-Scope Done None
DBA 174806 Decommission db1045 In-Scope Done None
DBA 180788 Rack and setup db1111 and db1112 In-Scope Done None
DBA 184285 Degraded RAID on db2055 Screep Done None
DBA 176532 Gerrit is failing to connect to db on gerrit2001 thus preventing systemd from working In-Scope Open None
DBA 83609 script & docs to rename wiki databases In-Scope Open None
DBA 143896 MySQL metrics monitoring In-Scope Open None
DBA 133523 [RFC] improve parsercache replication and sharding handling In-Scope Open None
DBA 54932 Drop *_old database tables from Wikimedia wikis In-Scope Open None
DBA 152427 Create a check/calendar alert for MariaDB TLS certs In-Scope Open None
DBA 100501 mysql user and group should be a system user/group In-Scope Open None
DBA 163339 pdu phase inbalances: ps1-a3-codfw, ps1-c6-codfw, & ps1-d6-codfw In-Scope Open None
DBA 135851 Preserve InnoDB table auto_increment on restart In-Scope Open None
DBA 153440 Create a full backup of all external storage records that would be easy to restore/setup a temporary delayed slave In-Scope Open None
DBA 134809 Apache <=> mariadb SSL/TLS for cross-datacenter writes In-Scope Open None
DBA 178877 operations/software repo: flake8 check In-Scope Open None
DBA 141547 Setup automatic failover for misc database servers In-Scope Open None
DBA 119154 Move echo tables from local wiki databases onto extension1 cluster for mediawikiwiki, metawiki, and officewiki In-Scope Open None
DBA 134476 Decommission old coredb machines (<=db1050) In-Scope Open None
DBA 109179 Migrate MySQLs to use ROW-based replication In-Scope Open None
DBA 183249 Reimage and upgrade to stretch all dbproxies Unknown Open None
DBA 128821 reclaim and return all cisco servers In-Scope Open None
DBA 141255 Separate host lookup from the sql shell script In-Scope Open None
DBA 119626 Eliminate SPOF at the main database infrastructure In-Scope Open None
DBA 50930 Database replication problems - production and labs (tracking) In-Scope Open None
DBA 104699 Firewall configurations for database hosts In-Scope Open None
DBA 157702 Followup for TLS MariaDB server roll-out In-Scope Open None
DBA 160731 Decom db1048 (BBU Faulty - slave lagging) In-Scope Open None
DBA 176754 Regularly purge expired temporary userrights from DB tables In-Scope Open 3.0
DBA 141968 Display lag on grafana (prometheus) and dbtree from pt-heartbeat instead (or in addition) of Seconds_Behind_Master In-Scope Open None
DBA 164834 In some database hosts, performance schema loses digest statistics In-Scope Open None
DBA 161754 eqiad: (2) hardware access request for labsdb1004 & 5 refresh In-Scope Open None
DBA 162070 Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases In-Scope Open None
DBA 148955 Puppetize tendril web user creation In-Scope Open None
DBA 107610 Setup separate logical External Store for Flow in production In-Scope Open None
DBA 171071 Perform testing for TLS effect on connection rate In-Scope Open None
DBA 166108 x1 master db1031: Faulty BBU In-Scope Open None
DBA 112282 Multiple pages with no revisions In-Scope Open None
DBA 161755 eqiad: (2) hardware access request for labsdb1006 & 7 refresh In-Scope Open None
DBA 127570 Rename be_x_oldwiki database to be_taraskwiki In-Scope Open None
DBA 145072 Create a script to regenerate prometheus mysqld exporter listing that works with puppetdb In-Scope Open None
DBA 126252 Populate the wikishared db on all dbstores In-Scope Open None
DBA 175672 Make apache/maintenance hosts TLS connections to mariadb work In-Scope Open None
DBA 151491 Icinga MariaDB disk space check on silver checks the wrong partition In-Scope Open None
DBA 112473 Better mysql monitoring for number of connections and processlist strange patterns In-Scope Open None
DBA 157359 labsdb1006/1007 (postgresql) maintenance In-Scope Open None
DBA 165625 Evaluate future of wmf puppet module "mysql" In-Scope Open None
DBA 162789 Create less overhead on bacula jobs when dumping production databases In-Scope Open None
DBA 165677 Create a backend check for pybal to monitor the MySQL protocol being up In-Scope Open None
DBA 165674 Investigate slow servermon updating queries on db1016 In-Scope Open None
DBA 151999 Create script to monitor db dumps for backups are successful (and if not, old backups are not deleted) In-Scope Open None
DBA 177779 Generate instance list of database hosts to be monitored automatically from exported resources In-Scope Open None
Software Development 157002 Puppet compiler: re-add the concurrency option NUM_THREADS In-Scope Open None
Software Development 155705 confctl: log to SAL even if the selection doesn't match any host In-Scope Open None
Software Development 148494 Add shell scripts CI validations In-Scope Open None
Software Development 164587 cumin could use randomization/splay options In-Scope Open None
Software Development 154776 Puppet compiler: order resources for easy comparison between hosts In-Scope Open None
Software Development 159045 Update Puppet repo code that uses maniphest.update and maniphest.createtask conduit api In-Scope Open None
Software Development 167504 New tool to track package updates/status for hosts and images (debmonitor) In-Scope Open None
Software Development 177385 Upgrade Cumin masters to stretch In-Scope Open None
Software Development 157133 Consider adding a --skip-conftool option to puppet-merge In-Scope Open None
Software Development 182028 DNS repo: add CI checks for obvious configuration errors In-Scope Open None
Software Development 144169 Flake8 for python files without extension in puppet repo In-Scope Open None
Software Development 152950 E901 SyntaxError: invalid syntax is wrongly raised on using python's abc by jenkins python CI linter In-Scope Open None
Software Development 150560 More verbose messages from service-checker-swagger In-Scope Open None
Software Development 157001 Puppet compiler: abort on git rebase conflict In-Scope Open None
Hardware Requests 184481 hardware request for tin replacement Screep None
Hardware Requests 184551 EQIAD: (1) hardware request for eventlog1001 replacement - eventlog1002. Screep None
Hardware Requests 185004 Decommission mw1201-mw1220 Elaborated None
Hardware Requests 185226 Decommission host erbium Screep None
Hardware Requests 184832 Decommission labsdb1001 and labsdb1003 Elaborated None
Hardware Requests 184480 hardware request for bast1001 replacement Screep None
Hardware Requests 184068 Procure and install LVS and miscellaneous servers Screep None
Hardware Requests 183209 decom uranium Unknown None
Hardware Requests 181784 Decommission db104[67] In-Scope Done None
Hardware Requests 159996 decom fluorine In-Scope Open None
Hardware Requests 166341 SSDs for main Kafka clusters In-Scope Open None
Hardware Requests 175830 decommission logstash100[1-3] In-Scope Open None
Hardware Requests 167377 Decommission cp4011, cp4012, cp4019, cp4020 In-Scope Open None
Hardware Requests 176957 Decommission host copper.eqiad.wmnet In-Scope Open None
Hardware Requests 173097 Decommission stat1002.eqiad.wmnet In-Scope Open None
Hardware Requests 167376 Decommission cp300[3456] In-Scope Open None
Hardware Requests 175089 decommission mw1163 In-Scope Open None
Hardware Requests 175595 decommission wdqs100[12] In-Scope Open None
Hardware Requests 168559 decom silver (was silver has trouble rebooting) In-Scope Open None
Hardware Requests 181264 Refresh or replace oxygen In-Scope Open None
Hardware Requests 130883 decom cp3011-22 (12 machines) In-Scope Open None
Hardware Requests 95742 Decomission amssq31-62 (32 hosts) In-Scope Open None
Hardware Requests 170157 decommission rcs100[12] In-Scope Open 3.0
Hardware Requests 177958 Decommission ocg1001-3 In-Scope Open None
Hardware Requests 169020 Decommission cp400[1-4] In-Scope Open None
Hardware Requests 178392 Replacement hardware for cumin masters In-Scope Open None
Hardware Requests 170441 Decommission mw1196 In-Scope Open None
Hardware Requests 175093 Decommission osmium.eqiad.wmnet In-Scope Open None
Hardware Requests 159480 Decommission bast3001 In-Scope Open None
Hardware Requests 172487 decom iridium In-Scope Open None
Other Operations 182819 custom fact interface_primary breaks under newer versions of facter Unknown None
Other Operations 184186 Fix unknown variables warning that occur with puppet 4.x Screep None
Other Operations 184714 Puppet fail to properly refresh Icinga Screep None
Other Operations 185101 Labstore1006/7 profile for meltdown kernel Screep None
Other Operations 184236 Puppet broken on deployment-ms-be0[34] with evaluation error in swift module Screep None
Other Operations 183177 memory errors not showing in icinga Unknown None
Other Operations 185236 Password Vault for Security Team Screep None
Other Operations 183815 Degraded RAID on lvs3001 Unknown None
Other Operations 182759 Add Prometheus exporter to Jenkins instances Unknown None
Other Operations 183412 replace bast1001 (new hardware) Unknown None
Other Operations 183454 Deprovision Diamond collectors no longer in use Unknown None
Other Operations 184946 Degraded RAID on db2036 Screep None
Other Operations 184514 Degraded RAID on ms-be1033 Screep None
Other Operations 184789 Degraded RAID on ms-be2023 Screep None
Other Operations 184086 Add prometheus exporter to Gerrit Elaborated None
Other Operations 184462 Serve one production service via Kubernetes Screep None
Other Operations 184061 SRE 2017-18 Q3 goal Cleanup esams and refresh servers and infrastructure (tracking) Screep None
Other Operations 185195 Sporadic logrotate issue for stretch mediawiki appservers Elaborated None
Other Operations 183236 After reimage Puppet order: sudo command failed Unknown None
Other Operations 182822 Generate a list of files that are supposed to exist but 404s Unknown None
Other Operations 183971 dumps project is using 2T of 5T available NFS misc storage Screep None
Other Operations 182614 Investigate why ORES logs are being written to syslog despite explicit logging config. Fix. Unknown None
Other Operations 183888 php-luasandbox in Wikimedia's Stretch apt repo depends on php5 Screep None
Other Operations 183814 Degraded RAID on bast3002 Unknown None
Other Operations 185024 HHVM 3.18.5+dfsg-1+wmf3 changes parse_url causing unit tests to fail Screep None
Other Operations 184230 Disavow emails from wikipedia.com Screep None
Other Operations 184616 Replace snapshot1001 with a proper testbed host (new hardware) Screep None
Other Operations 185153 attach furud's new arrays (furud-array[3-7]) Screep None
Other Operations 182228 run-no-puppet leave puppet disabled on kill/crash Unknown None
Other Operations 184065 Setup new access switches Screep None
Other Operations 123147 Wikipedia.com warns about bad certificate Unknown None
Other Operations 184562 Upgrade Puppet Master Infrastructure to Debian Stretch Screep None
Other Operations 183935 rack/setup/install notebook[34] Screep None
Other Operations 184924 Utilize the deployment pipeline (stretch) Screep None
Other Operations 184655 logstash group1 dashboard incorrectly shows testwikidatawiki Screep None
Other Operations 184730 Reboot non-labvirt cloud provider hardware for meltdown Screep None
Other Operations 182955 Decommission kafka1018 Unknown None
Other Operations 183920 2018-01-02: labstore Tools and Misc share very full Screep None
Other Operations 182331 [Epic] Deploy ORES in kubernetes cluster Unknown None
Other Operations 184722 Hardware check on mw1271 Screep None
Other Operations 182597 Use EtcdConfig in production to allow automation of a datacenter switch Unknown None
Other Operations 185189 scap sudo violation on first puppet run Screep None
Other Operations 182832 Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state Screep None
Other Operations 184066 Procure and install new PDUs Screep None
Other Operations 183146 Monitor resource usage on a per-cgroup basis Unknown None
Other Operations 185215 Puppet compiler failure to lookup some keys Screep None
Other Operations 184435 Puppet tox: properly lint both Py2 and Py3 files Screep None
Other Operations 182203 Tuning profile::ores::celery parameters should cause a Celery service restart Unknown None
Other Operations 184909 labvirt1021-1022 spam the dhcp server with requests Screep None
Other Operations 184634 Netbox: postgres cannot be restarted w/ current config Screep None
Other Operations 184923 Validate whether the (implemented) standardized application environment works as expected Screep None
Other Operations 183937 rack/setup/install labvirt102[12] Screep None
Other Operations 182585 Puppet: Setting configtimeout is deprecated Unknown None
Other Operations 184564 Plan Puppet 5 upgrade Screep None
Other Operations 183565 Fix regex.yaml single-regex issue Unknown None
Other Operations 182497 Update log config for scb* boxes, to deal with ORES verbose logging Unknown None
Other Operations 184444 Puppet hosts with their cert revoked can still run puppet Screep None
Other Operations 183507 Create 'releng' LDAP group Unknown None
Other Operations 182699 Use firmware-enriched Debian installation images Unknown None
Other Operations 184242 Puppet broken on deployment-netbox, looks like it thinks its a prod box Elaborated None
Other Operations 184270 rebuild php-wikidiff2 and php-luasandbox for php7 and stretch Screep None
Other Operations 184224 Package/upload service-checker for Debian stretch Screep None
Other Operations 185080 Prepare conftool for safely editing mediawiki-config values Screep None
Other Operations 184063 Remove all decommissioned hardware Screep None
Other Operations 185275 replace tin (new hardware) Screep None
Other Operations 184919 Serve at least 50% of Mathoid via kubernetes Screep None
Other Operations 184528 Degraded RAID on lvs3001 Screep None
Other Operations 184064 Prepare racks OE14, OE15 and OE16 with new infrastructure Screep None
Other Operations 182656 Integrate jessie 8.10 point release Unknown None
Other Operations 185078 Test EtcdConfig in different failure scenarios Screep None
Other Operations 184561 Modernize Puppet Configuration Management (2017-18 Q3 Goal) Screep None
Other Operations 184530 Degraded RAID on lvs3001 Screep None
Other Operations 184592 Setup reply via email in discourse-mediawiki.wmflabs.org Elaborated None
Other Operations 182702 Debian Jessie reimage/install ends up in kernel panic with 8.10 netboot image Unknown None
Other Operations 184533 Degraded RAID on lvs3001 Screep None
Other Operations 185219 check americium eth1 cabling and link Elaborated None
Other Operations 182304 Cleanup multiple definitions of logstash endpoint in puppet / hiera Unknown None
Other Operations 184936 install/designate other machine as esams bastion Elaborated None
Other Operations 182274 Create custom per-job metric reporters capability Unknown None
Other Operations 184563 Investigate landscape of PuppetDB Frontends and Provision One Screep None
Other Operations 185055 Stack overflow when Redis is down Screep None
Other Operations 184048 HHVM hangs on the API cluster Screep None
Other Operations 184443 Reboot snapshot*, dumpsdata*, dataset1001, ms1001, francium Screep None
Other Operations 182655 Integrate stretch 9.3 point update Unknown None
Other Operations 182812 Forward security@tools.wmflabs.org to security@wikimedia.org Unknown None
Other Operations 178778 Parsoid, VisualEditor not working with SSL / HTTPS In-Scope Cut 0.0
Other Operations 184338 Update people.wikimedia.org with the 2017 Wikimedia hackathon group photo Screep Done None
Other Operations 183385 Package Poolcounter for Debian Stretch Unknown Done None
Other Operations 183970 wikidumpparse is using 1.2TB of 5T available NFS misc storage Screep Done None
Other Operations 182894 Trusty puppet 4 approach Unknown Done None
Other Operations 179721 puppet4: The following unknown setting(s) are being ignored: parser In-Scope Done None
Other Operations 180924 Deploy Noto fonts or their derivatives for Chinese (and J&K?) In-Scope Done None
Other Operations 183144 labcontrol1002 Error: unable to connect to node rabbit@labcontrol1002: nodedown Unknown Done None
Other Operations 165519 rack and setup mw1307-1348 In-Scope Done None
Other Operations 181263 mw2251 failed memory dimm In-Scope Done None
Other Operations 183046 scap 3.7.4-2 is broken Unknown Done None
Other Operations 181906 Emails send by subscribers don't arrive on the mailing list Moderators-nl In-Scope Done None
Other Operations 183569 npm 1.4.21 can't use a http proxy Unknown Done None
Other Operations 177521 Permissions to upload data to the analytics cluster from a machine at Drexel In-Scope Done None
Other Operations 184196 cp1066's DRAC not responding to SSH Screep Done None
Other Operations 184617 Give access to S4 (procurement tasks) to Erika Bjune Screep Done None
Other Operations 183972 Allow "releasers-mediawiki" sudo rights to manage Jenkins Screep Done None
Other Operations 184582 Request access to analytics cluster for bawolff Screep Done None
Other Operations 184103 ircecho doesn't reconnect on failure Screep Done None
Other Operations 184555 All IP addresses used for sending emails by Wikimedia's services Screep Done None
Other Operations 184189 Cloud: Labvirt and instance reboots for Meltdown Screep Done None
Other Operations 183953 tools.iabot is using 1.3T of 8T available tools nfs storage Screep Done None
Other Operations 177254 Upgrade to puppet 4 (4.8 or newer) In-Scope Done None
Other Operations 184190 Requesting access to Production SSH, statistics-privatedata-users, analytics-privatedata-users, perf-team for imarlier Screep Done None
Other Operations 155129 Create prometheus nutcracker exporter In-Scope Done None
Other Operations 177208 Provide dedicated database resources for wikidata In-Scope Done None
Other Operations 177276 Unify production and CI docker image build process In-Scope Done None
Other Operations 183895 Decommission mw1180-1200 Elaborated Done None
Other Operations 184085 Requesting extended access to stat1005 for jdcc Screep Done None
Other Operations 170353 Icinga: timeseries checks should have the link to a graph with the data In-Scope Done None
Other Operations 178325 Operations 2017-18 Q2 Program 6 umbrella task In-Scope Done None
Other Operations 184774 SCAP: Upload debian package version 3.7.5-1 Screep Done None
Other Operations 183954 templatetiger is using 827G of 8T available tools nfs storage Screep Done None
Other Operations 184620 Request to be added to the ldap/wmde group Screep Done None
Other Operations 184473 Requesting access to Production Shell for cy534 Screep Done None
Other Operations 168619 Degraded RAID on lvs3001 In-Scope Done None
Other Operations 184206 Requesting access to researchers and analytics-privatedata-users for Ed Sanders Screep Done None
Other Operations 182583 maps-test2001 is low on disk space Unknown Done None
Other Operations 184053 Degraded RAID on ms-be1013 Screep Done None
Other Operations 183917 remove cloud VPS project 'ganglia' Screep Done None
Other Operations 184179 Missing references to s8 on maintenance and cloud scripts (and potentially others) Elaborated Done None
Other Operations 169548 Prepare for Puppet 4 In-Scope Done None
Other Operations 184110 Set up RESTBase on Cassandra 3 nodes Screep Done None
Other Operations 182215 install_server: switch to stretch as default install image Unknown Done None
Other Operations 183896 Degraded RAID on ms-be1033 Screep Done None
Other Operations 184787 Degraded RAID on ms-be2023 Screep Done None
Other Operations 184796 Configure puppetdb to export metrics via Prometheus JMX Agent Screep Done None
Other Operations 169969 Regularly purge old ores graphite metrics In-Scope Done None
Other Operations 151310 create-dbusers service failing on labstore1004 In-Scope Done None
Other Operations 184390 Degraded RAID on ms-be2037 Screep Done None
Other Operations 184785 ms-be2023 fails to (re)boot Screep Done None
Other Operations 162245 Enable GC for HHVM CLI (at least for dump runners) In-Scope Done None
Other Operations 158196 Reimage labstore1001 and labstore1002 for DRBD storage setup In-Scope Done None
Other Operations 184664 Install Noto fonts on scaling servers for SVG rendering Screep Done None
Other Operations 184434 prometheus-blazegraph-exporter failing to start after reboot Screep Done None
Other Operations 180167 Update puppetmaster1001 puppet certificate In-Scope Done None
Other Operations 184017 Puppet - Error: /Stage[main]/Apparmor/Service[apparmor]: Provider init is not functional on this host Screep Done None
Other Operations 185102 labstore2003 reboots into mode missing /srv disks Screep Done None
Other Operations 180854 Create discourse-mediawiki.wmflabs.org (pilot instance) In-Scope Done None
Other Operations 181989 Icinga check for WDQS should do an actual query In-Scope Done None
Other Operations 179640 mw1191 ipmi-sel cpu errors In-Scope Done None
Other Operations 181952 Requesting access to EventLogging data for Vinitha In-Scope Done None
Other Operations 119718 Make it easier to ban misbehaving dashboards from graphite In-Scope Open None
Other Operations 120532 Use user-specific passwords for accessing EventLogging database In-Scope Open None
Other Operations 170108 Operations Q1 goal: Streamlined Service Delivery In-Scope Open None
Other Operations 167245 prometheus-node-exporter - invalid group: ‘prometheus:prometheus' In-Scope Open None
Other Operations 163336 kube-proxy pulls in docker and starts service even when it isnt needed In-Scope Open None
Other Operations 179230 Puppet wmf-style-guide: array of classes not detected properly In-Scope Open None
Other Operations 159536 Puppet constantly trying to stop the already stopped puppetmaster process on Trusty In-Scope Open None
Other Operations 133179 Redis monitoring needs to be improved In-Scope Open None
Other Operations 167422 Monitoring: add link to graph for Icinga timeseries alarms In-Scope Open None
Other Operations 151486 Silver anomalies In-Scope Open None
Other Operations 181559 Investigate redis-cluster or other techniques for making Redis not a single point of failure. In-Scope Open None
Other Operations 118746 Goal: Strengthen Incident monitoring infrastructure In-Scope Open None
Other Operations 154619 Export ipsec counters as Prometheus metrics In-Scope Open None
Other Operations 136562 Audit/fix hosts with no RAID configured In-Scope Open None
Other Operations 122825 Service Ownership and Maintenance In-Scope Open None
Other Operations 116580 monitor postgresql replication status In-Scope Open None
Other Operations 158562 Manage apt sources via puppet? In-Scope Open None
Other Operations 160677 Effects on adjusting Prometheus retention In-Scope Open None
Other Operations 178839 New upstream jvm-tools In-Scope Open None
Other Operations 170453 FY2017/18 Program 6: Streamlined Service delivery In-Scope Open None
Other Operations 86552 Monitor and alarm on SMART attributes In-Scope Open None
Other Operations 133392 save grafana dashboards in revision control / puppet In-Scope Open None
Other Operations 168407 rack/setup/install labnodepool1002.eqiad.wmnet In-Scope Open None
Other Operations 89808 wikitech instances list is blank In-Scope Open None
Other Operations 176437 puppet ca_server confusion In-Scope Open None
Other Operations 158915 Make sure replying to emails in gerrit 2.14 works In-Scope Open None
Other Operations 177498 Provide a forward port of ICU 52 for stretch / Investigate best ICU update strategy In-Scope Open None
Other Operations 160529 Sender email spoofing In-Scope Open None
Other Operations 92471 enable authenticated access to Cassandra JMX In-Scope Open None
Other Operations 146664 Limit resources used by ORES In-Scope Open None
Other Operations 151009 Provide authenticated access to Prometheus native web interface In-Scope Open None
Other Operations 124101 Specific revisions of multiple files missing from Swift - 404 Not Found returned In-Scope Open None
Other Operations 143552 Make elasticsearch configuration more robust to loss of network connectivity In-Scope Open None
Other Operations 173721 Track down the source of periodic increases in requests to swift eqiad In-Scope Open None
Other Operations 167412 host-vmem.erb is doing operations that make no sense In-Scope Open None
Other Operations 146285 Switch mwscript from Zend PHP5 to default php alternative (e.g. HHVM or PHP7) In-Scope Open None
Other Operations 106937 Monitor [[Special:ListFiles]] for non 200 HTTP statuses in thumbnails In-Scope Open None
Other Operations 158434 Phabricator: Make sure phabricator works properly including our puppet roles on jessie In-Scope Open None
Other Operations 101585 document redis upgrade/restart procedures In-Scope Open None
Other Operations 177197 Export Prometheus-compatible JVM metrics from JVMs in production In-Scope Open None
Other Operations 89887 Clean up permissions for privatedata files on stat1002 - they should be group readable by statistics-privatedata-users In-Scope Open None
Other Operations 184461 Discourse migration from wmflabs to production Elaborated Open None
Other Operations 165885 Create a cron to clean clientbucket every day or hour In-Scope Open None
Other Operations 152078 Load balancing "external" traffic to the Kubernetes cluster in production In-Scope Open None
Other Operations 181750 decommission mobile 1004 and mobile1005 In-Scope Open None
Other Operations 114801 operations-apache-config-lint replacement doesn't check syntax In-Scope Open None
Other Operations 150822 Internal PKI for secure communication - Barcelona Ops offsite 2016 In-Scope Open None
Other Operations 180051 Reduce the number of fields declared in elasticsearch by logstash In-Scope Open None
Other Operations 175206 2017/18 Annual Plan Program 8: Multi-datacenter support In-Scope Open None
Other Operations 160060 Icinga check for sysctl settings In-Scope Open None
Other Operations 130617 Collect metrics on pool counter usage In-Scope Open None
Other Operations 161528 incident 20170323-wikibase did not trigger Icinga paging In-Scope Open None
Other Operations 141897 Review new service 'pre-deployment to production' checklist In-Scope Open None
Other Operations 122144 Move most (all?) exim personal aliases to OIT In-Scope Open None
Other Operations 147872 Rename rhodium to puppetmaster1003 In-Scope Open None
Other Operations 163673 Some swift disks wrongly mounted on 5 ms-be hosts In-Scope Open None
Other Operations 136311 Monitor the BMC's event log for hardware errors In-Scope Open None
Other Operations 142205 use granularity (g=) restrictions for wikimedia.org fundraising DKIM records In-Scope Open None
Other Operations 150912 Class 'Memcached' not found when running mwscript eval.php on debug servers In-Scope Open None
Other Operations 136403 Move cp3030+ from OE14 to OE13 in racktables In-Scope Open None
Other Operations 116063 Hardware Automation Workflow - Overall Tracking In-Scope Open None
Other Operations 132104 Consider moving policy.wikimedia.org away from WordPress.com In-Scope Open None
Other Operations 130593 investigate slapd memory leak In-Scope Open None
Other Operations 161918 videoscalers (mw1168, mw1169) - high load / overheating In-Scope Open None
Other Operations 154915 Get rid of "import realm.pp" in manifests/site.pp In-Scope Open None
Other Operations 116805 DomainKeys Identified Mail (DKIM) for phabricator.wikimedia.org In-Scope Open None
Other Operations 150871 [EPIC] (Proposal) Replicate core OCG features and sunset OCG service In-Scope Open None
Other Operations 159830 Sanity check global-multiwrite logs for ConfirmEdit usage In-Scope Open None
Other Operations 140879 503 error raises again while trying to load a Wikidata page In-Scope Open None
Other Operations 161598 Monitor HHVM bytecode cache depletion on mediawiki app servers In-Scope Open None
Other Operations 169884 Jobrunners generate mediawiki exceptions upon calling Closure$RecentChange::save In-Scope Open None
Other Operations 163402 Ensure we can survive a loss of labservices1001 In-Scope Open None
Other Operations 150875 Confirm attribution needs In-Scope Open None
Other Operations 176445 Systematically test load speeds of Watchlist and Recent Changes In-Scope Open None
Other Operations 167966 Look into feasibility of disabling sha-1 host keys on our ssh daemons In-Scope Open None
Other Operations 150356 Wikidata Query Service is overly verbose toward logstash In-Scope Open None
Other Operations 181988 Investigate and improve memory allocation rates of WDQS In-Scope Open None
Other Operations 95801 Allow customizing the alert message from graphite In-Scope Open None
Other Operations 168460 Update certificates on productions replicas of corp.wikimedia.org LDAP In-Scope Open None
Other Operations 182033 Decommission osm-web100[1-4] In-Scope Open None
Other Operations 169287 etcd config depends on puppet certs, but puppet doesn't know In-Scope Open None
Other Operations 172538 rack/setup/install labvirt10(19|20).eqiad.wmnet In-Scope Open None
Other Operations 113785 Make the Shinken IRC alert and icinga-wm bots use colors In-Scope Open None
Other Operations 175213 2017/18 Annual Plan Program 8: Multi-datacenter support, Q2 goals In-Scope Open None
Other Operations 180641 reinstall RT server with private IP and stretch In-Scope Open None
Other Operations 177371 Phase out DSA keys for SSH access (ssh-dss) In-Scope Open None
Other Operations 182249 Diagnose and fix 4.5k req/min ceiling for ores* requests In-Scope Open None
Other Operations 148614 Icinga check for Tor In-Scope Open None
Other Operations 174431 Upgrade mw* servers to Debian Stretch (using HHVM) In-Scope Open None
Other Operations 126295 Spike: What do we have to package to run the Programs and Events dashboard on production? In-Scope Open None
Other Operations 161003 Cross-check disabled accounts from corp LDAP against data.yaml In-Scope Open None
Other Operations 142815 Enhance account handling (meta bug) In-Scope Open None
Other Operations 179078 mpt raid controller not detected as fact on maps-test2* In-Scope Open None
Other Operations 128716 Make icinga-wm report Tools homepage check at #wikimedia-labs, too In-Scope Open None
Other Operations 169286 labstore1005 A PCIe link training failure error on boot In-Scope Open None
Other Operations 142821 Synchronise groups defined in data.yaml to LDAP In-Scope Open None
Other Operations 158837 Consolidate performance website and related software In-Scope Open None
Other Operations 110240 [Discussion] Consider validating JSON schemas when running x-ample tests? In-Scope Open None
Other Operations 171157 Monitor internal CA expirations In-Scope Open None
Other Operations 162029 Migrate all jessie hosts to Linux 4.9 In-Scope Open None
Other Operations 86546 graphite-web logs are not rotated In-Scope Open None
Other Operations 135318 Document how to handle 'inconsistent state within the internal storage backends' issues In-Scope Open None
Other Operations 124991 evaluate possibility for nscd use with useldap In-Scope Open None
Other Operations 171851 Reimage ores* hosts with Debian Stretch In-Scope Open None
Other Operations 166937 Broken /a/refinery-source/guard/run_all_guards.sh script on stat1002 In-Scope Open None
Other Operations 135124 Deploy etcddump (or another etcd dump & load tool) to production In-Scope Open None
Other Operations 150823 Puppet CA rollover In-Scope Open None
Other Operations 141520 "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) In-Scope Open None
Other Operations 133844 Improve Elasticsearch icinga alerting In-Scope Open None
Other Operations 129847 conftool-merge should report which node is setting attributes for In-Scope Open None
Other Operations 180498 planet.wikimedia.org: replace planet-venus software with rawdog In-Scope Open None
Other Operations 179022 Backport firejail 0.9.52 for use on Wikimedia appservers In-Scope Open None
Other Operations 169290 New anti-stackclash (4.9.25-1~bpo8+3 ) kernel super bad for NFS In-Scope Open None
Other Operations 181519 tls expiry check for mx vs acme-setup renewal period In-Scope Open None
Other Operations 151702 API cluster failure / OOM In-Scope Open None
Other Operations 177826 Upgrade ci ssh key to ecdsa In-Scope Open None
Other Operations 137397 revisit swift (sys)logging In-Scope Open None
Other Operations 181556 rack/setup/install dns500[12] In-Scope Open None
Other Operations 176666 Qualtrics cannot send email to wikimedia.org addresses In-Scope Open None
Other Operations 160229 Back up of Commons files In-Scope Open None
Other Operations 152767 Missing Labs hiera entry in labs-private repo In-Scope Open None
Other Operations 161145 Fix the general problem of randomly-bad puppet agent cron timings within redundant clusters In-Scope Open None
Other Operations 140594 svn.wikimedia.org redirects to Diffusion main page, hence hard to find e.g. "flexbisonparse" In-Scope Open None
Other Operations 149617 Integrating MediaWiki (and other services) with dynamic configuration In-Scope Open None
Other Operations 127797 document all puppet classes / defined types!? In-Scope Open None
Other Operations 67394 [EPIC] Performance testing environment In-Scope Open None
Other Operations 127825 Re-add intel-microcode In-Scope Open None
Other Operations 150020 Refactor puppet-postgresql module to use custom types In-Scope Open None
Other Operations 107108 Flow notification links on mobile point to desktop In-Scope Open None
Other Operations 135128 Turn on etcd TLS for intra-cluster communications In-Scope Open None
Other Operations 179192 Check analytics1037 power supply status In-Scope Open None
Other Operations 164341 Decommission old memcached hosts - mc1001->mc1018 In-Scope Open None
Other Operations 169249 /usr/local/bin/xenon-generate-svgs and flamegraph.pl cronspam In-Scope Open None
Other Operations 162955 rebuild tools-grid-master as a large instance In-Scope Open None
Other Operations 157306 Fix config file handling for /etc/hhvm/php.ini In-Scope Open None
Other Operations 175361 Upgrade mx1001/mx2001 to stretch In-Scope Open None
Other Operations 165781 rack/setup/install labcontrol100[34] In-Scope Open None
Other Operations 169318 Use multiple puppetdbs on puppet masters In-Scope Open None
Other Operations 119274 Check incoming requests to secure.wm.o In-Scope Open None
Other Operations 170481 FY2017/18 Program 6 - Outcome 2 - Objective 2: Set up a continuous integration and deployment pipeline In-Scope Open None
Other Operations 127549 move travel related aliases to OIT In-Scope Open None
Other Operations 151314 logrotate failing with $FILE.1.gz: File exists In-Scope Open None
Other Operations 119846 Redirect revisions from svn.wikimedia.org to https://phabricator.wikimedia.org/rSVN In-Scope Open None
Other Operations 123237 Provide production jessie image with node 4.2; use this for service-runner build command In-Scope Open None
Other Operations 150771 Secondary production Jenkins for CI In-Scope Open None
Other Operations 156136 Increase swift replication factor for accounts In-Scope Open None
Other Operations 152073 Check concurrency/retry/timeout limits and syncronize those between services In-Scope Open None
Other Operations 120856 Remove all out of warranty unused cp10xx's from A2 In-Scope Open None
Other Operations 155761 DNS repo: add Jenkins job to ensure there are no duplicates In-Scope Open None
Other Operations 136094 Race condition in setting net.netfilter.nf_conntrack_tcp_timeout_time_wait In-Scope Open None
Other Operations 138136 MB Lateefi Fonts for Sindhi Wikipedia. In-Scope Open None
Other Operations 175710 Add profiling for Varnish and VCL In-Scope Open None
Other Operations 170995 Setup a mirror for R language dependencies (CRAN) In-Scope Open 10.0
Other Operations 148647 refresh swift hardware in codfw/eqiad In-Scope Open None
Other Operations 165784 rack/setup/install labmon1002 In-Scope Open None
Other Operations 165511 Change automatic shortlink in blog theme In-Scope Open None
Other Operations 174172 unused grafana-dashboard indices on elasticsearch / logstash In-Scope Open None
Other Operations 84163 Fix CirrusSearch monitoring In-Scope Open None
Other Operations 119660 Set up LVS for labs dns recursors In-Scope Open None
Other Operations 163068 More missing 'original' files on Commons In-Scope Open None
Other Operations 180944 Passenger spews Exception NoMethodError in Rack application object In-Scope Open None
Other Operations 179501 Use external dsh group to list pooled ORES nodes In-Scope Open None
Other Operations 84700 Setup management switch in OE12 In-Scope Open None
Other Operations 123918 'swift' user/group IDs should be consistent across the fleet In-Scope Open None
Other Operations 164819 reprepro: Support for buildinfo files / dbgsym packages In-Scope Open None
Other Operations 146657 create notifications about user accounts that have not been used for a long time In-Scope Open None
Other Operations 166233 Update redis puppet class to support stretch In-Scope Open None
Other Operations 145065 Decrease time required to fully restart the Cirrus elasticsearch clusters In-Scope Open None
Other Operations 118154 determine hardware needs for dumps in eqiad and codfw In-Scope Open None
Other Operations 162612 codfw/eqiad hosts occasionally spend > 3 minutes starting networking.service with linux 4.9 In-Scope Open None
Other Operations 164042 Racktables: clearly show when hosts are decommissioned In-Scope Open None
Other Operations 152632 Explore hosting the multimedia commons use case In-Scope Open None
Other Operations 138799 Create a simple puppet role for setting up a singlenode kubernetes install In-Scope Open None
Other Operations 141524 eventbus should send statsd in batches In-Scope Open None
Other Operations 177196 Port non-deprecated Diamond collectors to Prometheus In-Scope Open None
Other Operations 180330 Add CI to all operations/* repositories and archive obsolete ones In-Scope Open None
Other Operations 76306 Set warning thresholds for average cluster utilization In-Scope Open None
Other Operations 117673 labs precise and jessie instance not accessible after provisioning In-Scope Open None
Other Operations 158288 Unclean stop of jobrunner service via puppet In-Scope Open None
Other Operations 148048 Store Wikimedia unified account name (SUL) in LDAP directory In-Scope Open None
Other Operations 98831 Honor DNT header for access logs & varnish logs In-Scope Open None
Other Operations 132324 Tracking and Reducing cron-spam from root@ In-Scope Open None
Other Operations 165170 rack/setup/install ores2001-2009 In-Scope Open None
Other Operations 101141 udp rcvbuferrors and inerrors on graphite1001 In-Scope Open None
Other Operations 124413 confctl should provide tags information after writing data In-Scope Open None
Other Operations 95052 Make ircecho much better In-Scope Open None
Other Operations 151489 silver: /dev/md2 mounted twice In-Scope Open None
Other Operations 95053 ircecho should accept input via unix sockets In-Scope Open None
Other Operations 158022 make apt.wikimedia.org HA In-Scope Open None
Other Operations 167225 Upload hhvm to stretch apt repo in apt.wikimedia.org In-Scope Open None
Other Operations 159687 etcd switchover/enhancements In-Scope Open None
Other Operations 142991 Enable "upload by url" feature at zhwiki In-Scope Open None
Other Operations 138017 Improve automation around Maps servers In-Scope Open None
Other Operations 156143 High CPU usage from swift-proxy on frontend machines In-Scope Open None
Other Operations 138866 Update & standardize Platform-specific_documentation for HP servers In-Scope Open None
Other Operations 151050 Proper documentation for Yubico 2FA for production use In-Scope Open None
Other Operations 181205 let quarry use the mariadb module In-Scope Open None
Other Operations 171188 Move the main WMCS puppetmaster into the Labs realm In-Scope Open None
Other Operations 164290 Set up external DNS record for wikitech-static In-Scope Open None
Other Operations 181621 What is causing ORES celery workers to suddenly require more CPU? In-Scope Open None
Other Operations 179353 Scap: Standardize git version In-Scope Open None
Other Operations 181557 rack/setup/install cp50(0[1-9]|1[0-2]) In-Scope Open None
Other Operations 181200 Use "Charter" as preferred typeface on Electron In-Scope Open None
Other Operations 87220 Minimize differences between beta and production (Tracking) In-Scope Open None
Other Operations 166291 Exim panics when spamd reaches maxchildren In-Scope Open None
Other Operations 98984 Check power supply balance settings on cp3030+ In-Scope Open None
Other Operations 172815 Improve stability and maintainability of our browser-based PDF render service In-Scope Open None
Other Operations 171191 Should puppet auto-restart slapd? In-Scope Open None
Other Operations 131748 Refresh the appservers puppet code/configs In-Scope Open None
Other Operations 132216 Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm In-Scope Open None
Other Operations 111934 Nutcracker stats monitoring should only listen on localhost In-Scope Open None
Other Operations 151493 silver: / partition low on space In-Scope Open None
Other Operations 115899 Move scap target configuration to etcd In-Scope Open None
Other Operations 166066 Integrate the puppet compiler in the puppet CI pipeline In-Scope Open None
Other Operations 162122 Swiftrepl was stuck in an infinite loop since days In-Scope Open None
Other Operations 161834 Undo special tools-home and tools-project share definitions for NFS In-Scope Open None
Other Operations 165105 Wiley requests for DOI and some other publishers don't work in production In-Scope Open None
Other Operations 151045 Extending Yubico 2FA for production use (meta bug) In-Scope Open None
Other Operations 149421 Long running mediawiki web requests impacts service availability, specially databases In-Scope Open None
Other Operations 177914 Switch labstore servers to default SSH configuration In-Scope Open None
Other Operations 171745 nscd does not cache localhost causing high CPU usage when localhost is often resolved In-Scope Open None
Other Operations 152724 Current state and next steps for RESTBase storage In-Scope Open None
Other Operations 177253 Upgrade puppetDB to version 3.2 or newer In-Scope Open None
Other Operations 156232 confctl SubjectAltNameWarning after python-urllib3 upgrade In-Scope Open None
Other Operations 146841 Reach out to Google about @yahoo.com emails not reaching gmail inboxes (when sent to mailing lists) In-Scope Open None
Other Operations 150486 Deploy federation for Prometheus In-Scope Open None
Other Operations 123818 setup YubiHSM and laptop at office In-Scope Open None
Other Operations 179461 Use the term "developer account" for Wikimedia LDAP accounts In-Scope Open None
Other Operations 134271 Replace ircd-ratbox with something newer/maintained In-Scope Open None
Other Operations 88997 Improve graphite failover In-Scope Open None
Other Operations 137616 Epic: cultivating the Maps garden In-Scope Open None
Other Operations 178628 Improve puppet alerting In-Scope Open None
Other Operations 170628 Jessie rsvg/cairo can't render specific SVG file on Commons In-Scope Open None
Other Operations 171482 Programmatic generation of grafana dashboards In-Scope Open None
Other Operations 167035 stretch acct monthly cron will spam when /var/log/wtmp.1 doesn't exist In-Scope Open None
Other Operations 165136 Ferm rules for labstore NFS hosts In-Scope Open None
Other Operations 181546 Let the ORES application set log severity, not uWSGI In-Scope Open None
Other Operations 142827 Enforce reference to Phabricator task for all commits to modules/admin/data/data.yaml In-Scope Open None
Other Operations 106346 setup an alertable threshold for Cassandra heap dumps In-Scope Open None
Other Operations 150396 Phabricator leaving old files in /tmp In-Scope Open None
Other Operations 166179 singapore caching center: eqiad staging tracking task In-Scope Open None
Other Operations 169570 nfs-manage failover script needs to be tested with real load and fixed In-Scope Open None
Other Operations 174720 letsencrypt::cert::integrated and non-http servers In-Scope Open None
Other Operations 128715 Add other Tools administrators to the Icinga notification group In-Scope Open None
Other Operations 179371 Move deployment-prep redis instances to stretch In-Scope Open None
Other Operations 177195 Reduce technical debt in metrics monitoring In-Scope Open None
Other Operations 152100 should we make privatewiki list available to puppet without maintaining two lists? In-Scope Open None
Other Operations 179271 Disk space checks complaining on docker build hosts when building containers In-Scope Open None
Other Operations 162123 Running swiftrepl is not puppetized In-Scope Open None
Other Operations 163823 During labservices1001 failover fqdn changed from foo.project.eqiad.wmflabs to foo.eqiad.wmflabs In-Scope Open None
Other Operations 178810 Wikibase: Increase batch size for HTMLCacheUpdateJobs triggered by repo changes. In-Scope Open None
Other Operations 146968 OTRS spam classification methods and systems In-Scope Open None
Other Operations 135385 investigate carbon-c-relay stalls/drops towards graphite2002 In-Scope Open None
Other Operations 147040 Two recently uploaded files have disappeared (404) In-Scope Open None
Other Operations 128615 Get rid of Tool Labs home page check from shinken In-Scope Open None
Other Operations 156398 Decommission or repair old asw-c2-eqiad In-Scope Open None
Other Operations 181121 Hardware errors on ganeti1005- ganeti1008 In-Scope Open None
Other Operations 137791 libcglib3-java replaces libcglib-java in Jessie In-Scope Open None
Other Operations 175885 Toolforge's static webserver broken by Puppet changes and stale nginx packages In-Scope Open None
Other Operations 150532 Upgrade qemu on ganeti clusters to 2.7 In-Scope Open None
Other Operations 177622 Multiple systems in ulsfo 1.22 showing PSU failures In-Scope Open None
Other Operations 84845 improve cron spam visibility In-Scope Open None
Other Operations 181971 Disable hiera autolookups In-Scope Open None
Other Operations 115757 document debian packaging guidelines In-Scope Open None
Other Operations 132325 Weak digest algorithm (SHA1) used to sign InRelease on apt.wikimedia.org In-Scope Open None
Other Operations 161920 logrotate for ruthenium In-Scope Open None
Other Operations 151049 Run systematic availability tests In-Scope Open None
Other Operations 112774 solve mtp panel issue for row uplinks In-Scope Open None
Other Operations 159661 Improve Terbium (and wasat) userland to process server side uploads In-Scope Open None
Other Operations 171619 ORES should use a git large file plugin for storing serialized binaries In-Scope Open None
Other Operations 141783 Add monitoring for detecting when logstash services are down In-Scope Open None
Other Operations 160071 Add slabinfo prometheus exporter In-Scope Open None
Other Operations 159354 Move coal from graphite machine(s) In-Scope Open None
Other Operations 162037 Use SSL certificates with discovery entry for elasticsearch In-Scope Open None
Other Operations 94215 decommission cp3001 & cp3002 In-Scope Open None
Other Operations 160644 Eventstreams graphite disk usage In-Scope Open None
Other Operations 124179 Improve access to and control over incident and metrics monitoring infrastructure In-Scope Open None
Other Operations 149057 Designate seems very slow to delete records? In-Scope Open None
Other Operations 125442 es2009 degraded RAID In-Scope Open None
Other Operations 126574 puppet should try to mount all mountable swift filesystems In-Scope Open None
Other Operations 36947 Incorrect text positioning in SVG rasterization (scale/transform; font-size; kerning) In-Scope Open 0.0
Other Operations 122210 Security audit for tftp on Carbon In-Scope Open None
Other Operations 153416 docker-engine pulled into our repositories only keeps the latest version In-Scope Open None
Other Operations 163507 Intermittent DB connectivity problem on phabricator, needs investigation In-Scope Open None
Other Operations 148693 Deploy IDS rendering engine to production In-Scope Open None
Other Operations 163362 audit all codfw pdu tower draws In-Scope Open None
Other Operations 160412 Add lock_wait_timeout to maintain_views and maintain-meta_p In-Scope Open None
Other Operations 137939 Increase frequency of OSM replication In-Scope Open None
Other Operations 168445 Reboots of cloud servers In-Scope Open None
Other Operations 111540 Clean up labs graphite datapoints In-Scope Open None
Other Operations 144006 Move the MW Beta appservers to Debian In-Scope Open None
Other Operations 116742 Track amount of package updates on systems In-Scope Open None
Other Operations 160101 Upgrade php5-json .deb to at least 1.3.8 In-Scope Open None
Other Operations 109089 EPIC: Cultivating the Elasticsearch garden (operational lessons from 1.7.1 upgrade) In-Scope Open None
Other Operations 161864 404 error while accessing some images files e.g. djvu and jpg In-Scope Open None
Other Operations 166081 rack/setup/install conf1004-conf1006 In-Scope Open None
Other Operations 144539 Remove /srv/deployment/wdqs/wdqs/rules.log symlink In-Scope Open None
Other Operations 164993 archiva artifact links point to 127.0.0.1 In-Scope Open None
Other Operations 170298 sshd stretch puppet support In-Scope Open None
Other Operations 153940 Logrotate fails for: "$FILE No such file or directory" In-Scope Open None
Other Operations 120159 Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module In-Scope Open None
Other Operations 133674 HHVM is leaking memory on the API appservers In-Scope Open None
Other Operations 134811 Consider REST with SSL (HyperSwitch/Cassandra) for session storage In-Scope Open None
Other Operations 138821 extend existing graphite whisper files retention to five years In-Scope Open None
Other Operations 102575 document graphite failover/backfill procedures In-Scope Open None
Other Operations 135338 On Trusty and Jessie PHP yields: PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/20-xhprof.ini on line 2 In-Scope Open None
Other Operations 164460 Use DNS discovery record for deployment CNAME In-Scope Open None
Other Operations 164238 move icinga contacts file to public repo In-Scope Open None
Other Operations 181920 etcd-mirror failure In-Scope Open None
Other Operations 159750 E-mail for people in different OIT LDAP object unit In-Scope Open None
Other Operations 110169 Monitor redis memory/disk usage In-Scope Open None
Other Operations 170474 Decommisson and store old row D network gear. In-Scope Open None
Other Operations 177891 Update and use php-wikidiff2 1.5.1 & MovedParagraphDetectionCutoff in production In-Scope Open None
Other Operations 143556 Setting up grafana should also setup Anonymous read-only access for the default org In-Scope Open None
Other Operations 138496 bring swift eqiad to one zone per row In-Scope Open None
Other Operations 175362 Split MXes into inbound and outbound In-Scope Open None
Other Operations 112257 rename cassandra cluster In-Scope Open None
Other Operations 182085 Requesting access to swift for Phabricator's git-lfs storage In-Scope Open None
Other Operations 165323 Add Prometheus machine metric to track core dumps In-Scope Open None
Other Operations 169680 NFS on dataset1001 overloaded, high load on the hosts that mount it In-Scope Open None
Other Operations 118812 Investigate mysterious_sysctl settings and figure out what to do with them In-Scope Open None
Other Operations 135113 Rationalize our jobqueues redis topology In-Scope Open None
Other Operations 104352 Make scap able to depool/repool servers via the conftool API In-Scope Open None
Other Operations 120585 Make l10nupdate user a system user In-Scope Open None
Other Operations 131832 Unable to restore file that has a very large file size In-Scope Open None
Other Operations 132632 puppetize turning off reserved space for cassandra /srv In-Scope Open None
Other Operations 129180 Preserve SSH host key when re-imaging hosts In-Scope Open None
Other Operations 161904 decommission backup4001 In-Scope Open None
Other Operations 153279 labnet/ labtestnet2001 - disk space - nova-api.log needs rotation In-Scope Open None
Other Operations 133913 Completely port l10nupdate to scap In-Scope Open None
Other Operations 137229 Tune thread for osm2pgsql / postgres max connections for Maps In-Scope Open None
Other Operations 133093 Investigate idle appservers in codfw In-Scope Open None
Other Operations 148478 Investigate seemingly random Gerrit slow-downs In-Scope Open None
Other Operations 140942 Tracking: Monitoring and alerts for "business" metrics In-Scope Open None
Other Operations 181728 Stop using jmx_exporter deployed via scap in favour of Debian package In-Scope Open None
Other Operations 114446 move human users out of UID range for system accounts In-Scope Open None
Other Operations 104671 Rename 'restricted' group? In-Scope Open None
Other Operations 139971 access_new_install role vs. Labs vs. the future In-Scope Open None
Other Operations 167689 Add RIPE atlas data to Prometheus In-Scope Open None
Other Operations 141756 audit / test / upgrade hp smartarray P840 firmware In-Scope Open None
Other Operations 175210 Select candidate jobs for transferring to the new infrastucture In-Scope Open None
Other Operations 179562 Create jenkins job for creating deployment artifacts for `docker-pkg-deploy` In-Scope Open None
Other Operations 146355 Replace etcd internal auth mechanism with a frontend proxy In-Scope Open None
Other Operations 40860 security@mediawiki.org : Create a public key and publish it on the public key servers In-Scope Open None
Other Operations 153068 Consider mounting labs NFS labstore1003.eqiad.wmnet:/scratch for server-side uploads In-Scope Open None
Other Operations 176335 logs sent to logstash are lost when the elasticsearch cirrus cluster is unavailable In-Scope Open None
Other Operations 169322 Research whether it makes sense to have OTRS installation in an HA setup In-Scope Open None
Other Operations 181678 Plan migration of ORES repos to git-lfs In-Scope Open None
Other Operations 152439 cronspam from labtestservices2001 /etc/dns-floating-ip-updater.py > /dev/null In-Scope Open None
Other Operations 93531 secure.wikimedia.org entries still showing up in Google search results In-Scope Open None
Other Operations 133476 Proposal: Centralize OTRS login methodology In-Scope Open None
Other Operations 46016 SVG fails to render properly due to several issues In-Scope Open None
Other Operations 164703 Integrate jessie 8.8 point release In-Scope Open None
Other Operations 175625 scs-c1-eqiad unresponsive In-Scope Open None
Other Operations 32716 Run our own Tor client for Tor block In-Scope Open None
Other Operations 149287 Heating alerts for mw servers in eqiad In-Scope Open None
Other Operations 147366 Setup automated topk wide row reporting In-Scope Open None
Other Operations 93138 Procure hardware for Sentry In-Scope Open None
Other Operations 155929 Create /community-beacon alternative entry point In-Scope Open None
Other Operations 109090 Investigate the need for master only (non data nodes) in our ES cluster In-Scope Open None
Other Operations 169518 Decommission esams ms-fe / ms-be In-Scope Open None
Other Operations 147204 Update confd package In-Scope Open None
Other Operations 162850 CPU throttling on DELL PowerEdge R320 In-Scope Open None
Other Operations 179395 Cluster puppet variable and ganglia decommission In-Scope Open None
Other Operations 174449 tin has a failing hdd In-Scope Open None
Other Operations 67270 Default license for operations/puppet In-Scope Open None
Other Operations 109606 Re-evaluate Limesurvey In-Scope Open None
Other Operations 173710 Job queue is increasing non-stop In-Scope Open None
Other Operations 181627 Port elasticsearch metrics to Prometheus In-Scope Open None
Other Operations 135125 Install a second etcd cluster in codfw In-Scope Open None
Other Operations 177739 Integrate stretch 9.2 point release In-Scope Open None
Other Operations 106664 Set up role accounts and feedback loops (FBL) with all providers In-Scope Open None
Other Operations 46791 [[wikitech:Server_admin_log]] should not rely on freenode irc for logmsgbot entries In-Scope Open None
Other Operations 178575 Add require_package() variant with repository component to wmflib In-Scope Open None
Other Operations 114849 Log lines on flourine overflow at 8092 bytes. In-Scope Open None
Other Operations 17000 Special:Import error: "Import failed: Could not open import file" In-Scope Open None
Other Operations 121610 system users with UIDs > 500 In-Scope Open None
Other Operations 116767 limit the impact of heavy/large graphite queries In-Scope Open None
Other Operations 167549 Create Icinga alert when OSM replication lags on maps In-Scope Open None
Other Operations 119401 Untangle labs/production roles from labs/instance roles In-Scope Open None
Other Operations 147905 investigate lead hardware issue In-Scope Open None
Other Operations 119679 Rewrite http://download.wikimedia.org/mediawiki/ -> https://releases.wikimedia.org/mediawiki in less than 3 redirects In-Scope Open None
Other Operations 134875 udpmxircecho spam/not working if unable to connect to irc server In-Scope Open None
Other Operations 149845 Something is wrong with installer root disk stuff In-Scope Open None
Other Operations 76203 Make ircecho run as its own user In-Scope Open None
Other Operations 154665 Look into behaviour of /etc/exim4/update-exim4.conf.conf related to updates In-Scope Open None
Other Operations 120165 Implement role based hiera lookups for labs In-Scope Open None
Other Operations 163354 Find a way to verify mediawiki-config IPs ahead of datacenter switchovers In-Scope Open None
Other Operations 181071 Cache ORES virtualenv within versioned source In-Scope Open None
Other Operations 181855 scap support for git-lfs In-Scope Open None
Other Operations 170152 mc2023 / mc2025 fail to mount root partition within 90 seconds using Linux 4.9 In-Scope Open None
Other Operations 140270 Determine a core set or a checklist of permissions for deployment purpose In-Scope Open None
Other Operations 118331 Alert when used_memory gets too high for redis queues In-Scope Open None
Other Operations 130590 Have dedicated master nodes for elasticsearch In-Scope Open None
Other Operations 159524 backup space is used unwisely In-Scope Open None
Other Operations 145885 Gerrit shows HTTP 500 error when pasting extended unicode characters In-Scope Open None
Other Operations 150460 Configure maps cluster to send statsd metrics to the statsd endpoint in the same datacenter In-Scope Open None
Other Operations 125085 Split the API MediaWiki appserver pool into two external/internal pools In-Scope Open None
Other Operations 180023 [DRAFT][RfC] Deployment of python applications in production In-Scope Open None
Other Operations 56713 Non-NDA users cannot access graphite.wikimedia.org In-Scope Open None
Other Operations 129188 mw2212 unresponsive In-Scope Open None
Other Operations 135122 Reduce etcd technical debt In-Scope Open None
Other Operations 179856 Improve documentation for mirrors.wikimedia.org In-Scope Open None
Other Operations 175876 document all scs connections In-Scope Open None
Other Operations 97524 ocg alarm ocg_job_status_queue 'flapping' In-Scope Open None
Other Operations 166368 Wipe of spare/replacement disks In-Scope Open None
Other Operations 133318 High levels of PoolCounter errors should trigger alerts In-Scope Open None
Other Operations 157038 Make it possible to run the mediawiki testsuite against a staging repo of apt.wikimedia.org In-Scope Open None
Other Operations 156937 Provide cross-dc redundancy (active-active or active-passive) to all important misc services In-Scope Open None
Other Operations 180628 Install git-lfs client (at least on scap targets & masters) In-Scope Open None
Other Operations 167239 Redirect status.wikipedia.org to status.wikimedia.org In-Scope Open None
Other Operations 181763 Decommission niobium In-Scope Open None
Other Operations 131928 Upgrade jessie systems from Linux 3.19 to 4.4 In-Scope Open None
Other Operations 161296 Upgrade mysqld_exporter to 0.10.0 In-Scope Open None
Other Operations 82350 update exim::listserve::private::mailing_lists value in puppet In-Scope Open None
Other Operations 181825 decommission rcs1001/1002 In-Scope Open None
Other Operations 144431 RESTBase k-r-v as Cassandra anti-pattern In-Scope Open None
Other Operations 156475 Investigate spike in 500s during asw-c2-eqiad replacement In-Scope Open None
Other Operations 175150 Decommission stat1003.eqiad.wmnet In-Scope Open None
Other Operations 87790 decom amslvs1-4 (dc work) In-Scope Open None
Other Operations 134326 udpmxircecho should write stats of messages processed and we should alert when that drops to zero In-Scope Open None
Other Operations 124185 Evaluate alternative web interfaces to icinga 1 core In-Scope Open None
Other Operations 144933 Cleanup debconf handling in mailman puppet setup In-Scope Open None
Other Operations 88730 Nutcracker needs to automatically recover from MC failure - rebalancing issues In-Scope Open None
Other Operations 136603 Update limit.sh to support systemd-based cgroup management In-Scope Open None
Other Operations 171048 Eventbus does not handle gracefully changes in DNS recursors In-Scope Open None
Other Operations 114337 Assign 3 more servers to video scaler duty In-Scope Open None
Other Operations 156544 Create backups of Wikimedia content in diverse geographic places In-Scope Open None
Other Operations 97909 Upgrade jobrunners to redis 2.8 In-Scope Open None
Other Operations 181554 rack/setup/install bast5001 In-Scope Open None
Other Operations 121240 Network isolation for production and semi-production services In-Scope Open None
Other Operations 113104 Set up a service IP for logstash In-Scope Open None
Other Operations 160146 jobrunner/jobchron services fail in codfw In-Scope Open None
Other Operations 123809 Module uwsgi doesn't allow passing multiple config params of same name In-Scope Open None
Other Operations 149643 Review Icinga alarms with disabled notifications In-Scope Open None
Other Operations 148968 Build Kubernetes for production use In-Scope Open None
Other Operations 133744 Epic: switch Maps to production status In-Scope Open None
Other Operations 164490 maintain-meta_p hangs on connecting to wikimedia.org.uk In-Scope Open None
Other Operations 176774 Reimage cobalt as stretch In-Scope Open None
Other Operations 168891 rack/setup/install labtestmetal2001.codfw.wmnet In-Scope Open None
Other Operations 126083 overhaul labstore setup [tracking] In-Scope Open None
Other Operations 130709 authoritative copy of 'root' files for upload.wikimedia.org is only in swift In-Scope Open None
Other Operations 163996 Icinga check for ipv6 host reachability In-Scope Open None
Other Operations 120377 labmon1001 graphite instance archiver keeps archiving the same instances In-Scope Open None
Other Operations 156140 Lots of hosts with hyperthreading disabled In-Scope Open None
Other Operations 113792 Change LDAP cn to something more useful (was Rename "Dzahn" to "Daniel Zahn" in Gerrit) In-Scope Open None
Other Operations 146090 High failure rate of account creation should trigger an alarm / page people In-Scope Open None
Other Operations 177374 decom wtp1001-wtp1024 In-Scope Open None
Other Operations 161096 confctl no longer logs a non-changing state change In-Scope Open None
Other Operations 175288 setup/install/deploy deploy1001 as deployment server In-Scope Open None
Other Operations 164123 tools-k8s-master-01 has two floating IPs In-Scope Open None
Other Operations 170365 move legal-tm-vio alias to OIT In-Scope Open None
Other Operations 150300 icinga notification if elevated writing to badpass.log In-Scope Open None
Other Operations 132532 rsync module doesnt work on trusty In-Scope Open None
Other Operations 151304 tmpreaper possible race condition In-Scope Open None
Other Operations 141038 implement icinga paging for non-ops teams In-Scope Open None
Other Operations 94819 Audit racktables In-Scope Open None
Other Operations 140075 investigate swift used space spikes since June 2016 In-Scope Open None
Other Operations 181967 Update puppet code to conform to puppet 4.x and later standards In-Scope Open None
Other Operations 171122 librenms: consider using Distributed Poller with multiple netmon servers In-Scope Open None
Other Operations 111838 Some files had disappeared from Commons after renaming In-Scope Open None
Other Operations 56515 Apply editing rate limits for all users In-Scope Open None
Other Operations 174465 Puppet admin module should support adding system users to managed groups In-Scope Open None
Other Operations 64987 librsvg misinterpret quoted font family names that contain whitespaces In-Scope Open None
Other Operations 154627 Production error message (when servers are down) points users to donate link which is likely to produce the same error message In-Scope Open None
Other Operations 174959 swift-recon-cron on ms-be203[34]: [Errno 17] File exists: '/var/lock/swift-recon-object-cron' In-Scope Open None
Other Operations 116627 Include 5xx numbers in fluorine fatalmonitor In-Scope Open None
Other Operations 173492 Tune Varnishkafka delivery errors to be more sensitive In-Scope Open None
Other Operations 94329 secure Cassandra/RESTBase cluster In-Scope Open None
Other Operations 111653 Encrypt all the things In-Scope Open None
Other Operations 135595 mod_deflate + mod_uwsgi causing mangled apache responses In-Scope Open None
Other Operations 181630 Send celery and wsgi service logs to logstash In-Scope Open None
Other Operations 168967 Upload shiny-server .deb to our Jessie apt repository In-Scope Open None
Other Operations 153246 Puppet failures with "Attempt to assign to a reserved variable name: 'trusted'" In-Scope Open None
Other Operations 149804 Review of ferm services without srange In-Scope Open None
Other Operations 83729 Fix monitoring of poolcounter service In-Scope Open None
Other Operations 179354 wikimedia-jessie & wikimedia-stretch docker images don't have deb-src set for apt.wikimedia.org In-Scope Open None
Other Operations 182034 Decommission osm-cp100[1-4] In-Scope Open None
Other Operations 169564 MD RAID: remove mdadm daily check In-Scope Open None
Other Operations 178445 flapping monitoring for recommendation_api on scb In-Scope Open None
Other Operations 179787 upload.wikimedia.org reports wrong mimetype for svg In-Scope Open None
Other Operations 141959 Moving network::external to hiera broke much of labs In-Scope Open None
Other Operations 140316 Add granularity limiter (g=) to wikimedia.org DKIM record(s) In-Scope Open None
Other Operations 155209 Increase $wgHTTPImportTimeout to a higher value on WMF wikis In-Scope Open None
Other Operations 168490 upgrade planet instances to stretch In-Scope Open None
Other Operations 140813 Protect sensitive user-related information with a UserData / auth / session service In-Scope Open None
Other Operations 110171 Alert when ES indexes are freezed for more than 30 minutes In-Scope Open None
Other Operations 162090 Investigate alternative RAID strategies for labstore1001/2 In-Scope Open None
Other Operations 166038 Sync internal nutcracker package with Debian package In-Scope Open None
Other Operations 134551 Create functional cluster checks for all services (and have them page!) In-Scope Open None
Other Operations 178663 Switch CI Docker Storage Driver to devicemapper In-Scope Open None
Other Operations 170640 reports.frdev.wm.o -- still in use? In-Scope Open None
Other Operations 163033 Create grafana dashboard for video scaler job runners In-Scope Open None
Other Operations 91404 Setup backups of elasticsearch indices In-Scope Open None
Other Operations 147923 Extract metrics from logs In-Scope Open None
Other Operations 174269 Two cases of local-multiwrite storage backend failure In-Scope Open None
Other Operations 163698 Add flood protection to the ircecho bot (icinga-wm) In-Scope Open None
Other Operations 118829 Automate the provisioning and management of MediaWiki clusters In-Scope Open None
Other Operations 165345 decommission indium In-Scope Open None
Other Operations 150672 Provide a /parsoid directory on releases.wikimedia.org In-Scope Open None
Other Operations 148017 lvs2002 repeated usb connect/disconnect message In-Scope Open None
Other Operations 150917 Remove deprecated features from book creator UI In-Scope Open None
Other Operations 177099 Large number of "A page you created was linked on Wikidata" emails to one recipient in short period of time In-Scope Open None
Other Operations 151048 Icinga monitoring for Yubikey components In-Scope Open None
Other Operations 142002 Clean up puppet & configs for ORES In-Scope Open None
Other Operations 117508 Make ops-l a list for humans again (no cheating) In-Scope Open None
Other Operations 105780 Create a doc explaining the SLA between services and the monitoring tool In-Scope Open None
Other Operations 125411 Diamond load averages do not contain scaled versions In-Scope Open None
Other Operations 148986 Firewall sets not being loaded post-reboot due to a @resolve race on jessie In-Scope Open None
Other Operations 170150 Evaluate Grafana's LDAP group options and deprecate grafana-admin if possible In-Scope Open None
Other Operations 176370 Migrate to PHP 7 in WMF production In-Scope Open None
Other Operations 116951 Reprepro should bail if it can't read and sign using the root keys In-Scope Open None
Other Operations 103886 Translation cache exhaustion caused by changes to PHP code in file scope In-Scope Open None
Other Operations 94457 Install nodejs, nginx and other dependencies on francium In-Scope Open None
Other Operations 136312 Encrypt syslog traffic In-Scope Open None
Other Operations 164248 HTTP responses from app servers sometimes stall for >1s In-Scope Open None
Other Operations 179696 Homepage for https://docker-registry.wikimedia.org In-Scope Open None
Other Operations 175738 Long term storage for frack prometheus data In-Scope Open None
Other Operations 176153 Create affcom-staff email account In-Scope Open None
Other Operations 156570 Investigate issues with wikitech-static.wikimedia.org In-Scope Open None
Other Operations 174916 electron/pdfrender hangs In-Scope Open None
Other Operations 111595 Do not apply spam headers on email assessed NOT to be spam In-Scope Open None
Other Operations 41785 Create a labs SMTP smarthost In-Scope Open None
Other Operations 150466 publish kartotherian / tilerator metrics by cluster In-Scope Open None
Other Operations 123560 investigate rsync between dcs with encryption In-Scope Open None
Other Operations 152445 Move prometheus entry point off port 80 In-Scope Open None
Other Operations 131326 smokeping config puppetization issue? In-Scope Open None
Other Operations 170817 Upgrade Thumbor servers to Stretch In-Scope Open None
Other Operations 129963 Update memcached package and configuration options In-Scope Open None
Other Operations 123276 URL parameters do not work with pages that have "?" in their names In-Scope Open None
Other Operations 168403 Aggregate prometheus functions yielding different results in grafana vs. prometheus console In-Scope Open None
Other Operations 160941 Improve SSH access information in onboarding documentation In-Scope Open None
Other Operations 119719 Enforce a minimum refresh period for grafana dashboards hitting graphite In-Scope Open None
Other Operations 125015 Requests to (hard) redirect pages return their target's contents but are counted as pageviews to the redirect page In-Scope Open None
Other Operations 149589 Puppet tab in Horizon unusably slow In-Scope Open None
Other Operations 170740 PuppetDB misbehaving on 2017-07-15 In-Scope Open None
Other Operations 167091 Elasticsearch errors about BulkShardRequest In-Scope Open None
Other Operations 163288 Decide on /var/lib vs /home as locations of homedir for l10nupdate In-Scope Open None
Other Operations 94951 Enable the usage of `hhvm -m debug --debug-host ::1` from mw1017 so developers can step through code (think gdb) in production to see what is going wrong. In-Scope Open None
Other Operations 172479 Collect error logs from jobchron/jobrunner services in Logstash In-Scope Open None
Other Operations 138685 notebook1001 shown as DOWN in icinga, due to firewall rules In-Scope Open None
Other Operations 163667 Fix UIDs for deployment server users In-Scope Open None
Other Operations 181634 Investigate overload condition, seems that we lose nodes In-Scope Open None
Other Operations 134458 status.wikimedia.org should use some Wikimedia favicon if possible In-Scope Open None
Other Operations 140442 reinstall rdb100[56] with RAID In-Scope Open None
Other Operations 159242 Segmentation fault creating thumbnail In-Scope Open None
Other Operations 150872 Replace OCG in collection extension with Electron In-Scope Open None
Other Operations 146627 Make deployment-prep puppetmaster more similar to Production puppetmaster In-Scope Open None
Other Operations 108985 Monitor MediaWiki sessions In-Scope Open None
Other Operations 165631 move gerrit.wm.org SSH service to private/behind LVS like phab-vcs In-Scope Open None
Other Operations 135991 Automated service restarts for common low-level system services In-Scope Open None
Other Operations 116750 2FA for SSH access to the production cluster In-Scope Open None
Other Operations 156924 Allow integration of data from etcd into the MediaWiki configuration In-Scope Open None
Other Operations 151317 stat user crontab on stat hosts for old file removal In-Scope Open None
Other Operations 157972 Puppet fails only once when restarting ferm is not successful In-Scope Open None
Other Operations 181409 Check for long running screen/tmux should mention usernames In-Scope Open None
Other Operations 179181 Puppet4: hiera() can only be called using the 4.x function API. In-Scope Open None
Other Operations 133164 Document eqiad/codfw transition plan for OCG In-Scope Open None
Other Operations 116288 Install mailman-api for internal use In-Scope Open None
Other Operations 182015 Decommission Vanadium In-Scope Open None
Other Operations 161835 Convert labstore cluster configuration to hiera and profiles In-Scope Open None
Other Operations 149885 Investigate Swift as a storage backend for maps tiles In-Scope Open None
Other Operations 156955 Standardizing our partman recipes In-Scope Open None
Other Operations 141128 determine/process/document bios firmware tracking/updating policies In-Scope Open None
Other Operations 179463 Create a single application to provision and manage developer (LDAP) accounts In-Scope Open None
Other Operations 160158 Make disabled accounts visible in the corp mirror LDAP replica In-Scope Open None
Other Operations 162857 Some Core availability Catchpoint tests might be more expensive than they need to be In-Scope Open None
Other Operations 151046 Fully puppetise yubikey-val In-Scope Open None
Other Operations 131966 Default gateway unreachable on baham.wikimedia.org after reboot In-Scope Open None
Other Operations 149543 Setup PAWS internal experimentally on notebook* nodes In-Scope Open None
Other Operations 180853 Bring discourse.mediawiki.org to production In-Scope Open None
Other Operations 174475 update firmware on scs consoles In-Scope Open None
Other Operations 134237 Graphoid returns a 400 on MW API time-out In-Scope Open None
Other Operations 55457 setup a DB backed parser cache In-Scope Open None
Other Operations 179099 puppetmaster hostcert and hostprivkey point to nonexistent files In-Scope Open None
Other Operations 161004 Remove disabled users from internal mailing lists In-Scope Open None
Other Operations 95054 Move ircecho config file to be YAML In-Scope Open None
Other Operations 163393 Determine appropriate proxy_read_timeout setting for Tools Proxy In-Scope Open None
Other Operations 165618 Audit / document reasons for not enabling HT? In-Scope Open None
Other Operations 143931 Update ICU version to 55.1 In-Scope Open None
Other Operations 148843 GPU upgrade for stats machine In-Scope Open None
Other Operations 168767 Monitor PostgreSQL connection slots In-Scope Open None
Other Operations 82937 re-create script for manual paging In-Scope Open None
Other Operations 140141 Install mscorefonts on scaling servers for SVG rendering In-Scope Open None
Other Operations 178690 Better organization for ops grafana dashboards In-Scope Open None
Other Operations 133091 Highest SSTables / read thresholds In-Scope Open None
Other Operations 165779 rack/setup/install labnet100[34] In-Scope Open None
Other Operations 133656 Have a paging check for Nova API accessible In-Scope Open None
Other Operations 108027 Collect per-cgroup cpu/mem and other system level metrics In-Scope Open None
Other Operations 158757 Puppet certificate missing subjectAltName In-Scope Open None
Other Operations 148061 Feasibility of hosting podcast setup on Wikimedia servers In-Scope Open None
Other Operations 172628 conf2002 etcdmirror-conftool-eqiad-wmnet died In-Scope Open None
Other Operations 150185 Deploy ElectronPdfService Extension to production In-Scope Open None
Other Operations 142984 Review lists of config/sysctl recommendations by "kernel self-protection project" In-Scope Open None
Other Operations 151047 Integrate Yubikey into data.yaml In-Scope Open None
Other Operations 141704 Storage backend errors on commons when deleting/restoring pages In-Scope Open None
Other Operations 158429 Switch to predictable network interface names? In-Scope Open None
Other Operations 127054 pinentry-gtk2 pulls in a lot of unneeded Gnome/GTK libs In-Scope Open None
Other Operations 169035 bast3002 sdb broken In-Scope Open None
Other Operations 126158 [RFC] Alert about *when* partitions will run out of space, not a percentage/absolute number In-Scope Open None
Other Operations 182016 Decommission server zinc In-Scope Open None
Other Operations 122917 Provide a good download service of dumps from Wikimedia In-Scope Open None
Other Operations 100777 expose hosts in maintenance state so we can prevent scap from running on them In-Scope Open None
Other Operations 153816 apache::static_site is not working In-Scope Open None
Other Operations 137176 catch-all apache vhost on the cluster should return 404 for non-existing sites In-Scope Open None
Other Operations 172584 Securing external binaries run by MediaWiki In-Scope Open None
Other Operations 167292 Collate jessie-wikimedia/backports into jessie-wikimedia/main In-Scope Open None
Other Operations 181632 Celery manager implodes horribly if Redis goes down In-Scope Open None
Other Operations 152782 Kibana functionality missing after upgrade: histograms In-Scope Open None
Other Operations 170456 FY2017/18 Program 6 - Outcome 2 - Objective 3: Integrated, container-based development environment In-Scope Open None
Other Operations 170480 FY2017/18 Program 6 - Outcome 2: Developers are able to develop and test their applications through a unified pipeline towards production deployment. In-Scope Open None
Other Operations 85451 scale graphite deployment (tracking) In-Scope Open None
Other Operations 128590 Cassandra uses default ip address for outbound packets while bootstrapping In-Scope Open None
Other Operations 126989 MediaWiki logging & encryption In-Scope Open None
Other Operations 123106 PNG thumbnail preview of SVG misses some text In-Scope Open None
Other Operations 182023 Migrate htmlCacheUpdate job to Kafka In-Scope Open None
Other Operations 116747 Meta task "Revamp user authentication" In-Scope Open None