Meeting Notes on Incident Management and Technical Discussions

Meeting Overview

This document summarizes a discussion held among team members regarding various technical issues, incident management, and project updates. Key participants included Michael, Michelle, Waseem, Eric, and Mario, among others. This meeting predominantly focused on incident handling, technical issues, and planning for systems upgrades.

Introductions and Participant Updates

  • The meeting began with casual greetings among team members. Individuals were identified, including Michelle and Eric who joined the call.

  • Acknowledgment of participants who were unable to attend, including Waseem.

Incident Management Discussion

Provo Site Incident

  • Incident Number: INC1605593

  • Issue: Ongoing concerns related to the Provo site's conditions. The power cable was reportedly checked and confirmed as properly connected by an internal engineer. The issue appears intermittent and requires monitoring.

  • RMA Process: The team discussed whether they should initiate a Return Merchandise Authorization (RMA) for the device. Noted that further verification is required regarding whether issues persist post-checkup.

Tinton Falls Site Incident

  • Incident Number: INC1604596

  • Issue: Latency issues were identified, correlated with CPU spikes on the IONS. The team had already rebooted the device and attempted a version upgrade but issues resurfaced.

  • Palo Alto Upgrade Recommendations: It was mentioned that Palo Alto does not recommend the current version upgrade, and insights were exchanged regarding whether they should proceed with the upgrade.

  • Conclusion: Since the upgrade seems necessary to address ongoing latency issues, especially due to a known memory leak, the recommendation is to keep the upgrade on the schedule for today at 8 PM Eastern Time.

Incident Breakdown

  • Current Incidents: 27 open tickets.

  • Incidents on Hold: 1 ticket.

  • Pending Carrier: Issues awaiting feedback from the carrier. The status of incidents was clarified, indicating that while many are on hold, they may not all be new incidents.

Upgrade Discussions

  • Concerns were raised regarding the potential new upgrades and fixes. A specific mention was given to how incidents could arise from system maintenance rather than operational failures.

  • Michael recommended being cautious with upgrades in the Tinton Falls site due to CPU spike issues.

  • Memory Leak Reference: A significant memory leak issue causing CPU spikes was discussed, detailing its effects and the necessity for system restarts to clear temp files.

Team Collaboration

  • Team members were encouraged to continue communication and collaboration on incident management and technical challenges. Specific references were made to engage with the carrier (Zayo) concerning the performance estimates being compromised.

  • The importance of tracking down incidents was discussed, especially in relation to underlying system monitoring that can auto-generate tickets based on system alerts.

Action Items and Final Thoughts

  • Next Steps: A follow-up on the Tinton Falls upgrade planned for tonight at 8 PM, pending confirmation from the team lead on whether the upgrade should continue amidst concerns raised about the version.

  • Discussion on further clarifying incident qualifications and ensuring that maintenance alerts do not clutter the incident reporting system.

  • The meeting concluded with acknowledgments and reinforced the importance of continued collaboration on these issues until resolved.