Debugging HPC Applications Notes

Debugging HPC Applications

Tools

  • Debugging code is complex, aided by open-source and proprietary tools.

  • Tools allow stepping through code, placing breakpoints, and viewing memory.

  • Common open-source debugging tools are serial but adaptable for parallel debugging.

  • Key tools:

    • GNU Debugger (GDB): Breakpoints, watchpoints, catchpoints, backtrace, variable setting, thread debugging, cheat sheet.

    • Valgrind: Memory analysis tool.

    • Commercial Parallel Debuggers: Enhanced features for multi-threaded and distributed applications.

GNU Debugger (GDB)

  • Enables stepping through code, setting breakpoints, viewing/changing variables, and tracking variables.

  • Command-line debugger: requires commands for setting breakpoints, continuing execution, etc.

  • Compile code with debugging information: use the -g flag.

  • Inspects program execution, sets breakpoints, analyzes memory, and controls execution flow.

  • Breakpoints: Stop execution at specific points using break command (e.g., break main.c:10).

  • Watchpoints: Stop execution when a variable's value changes (e.g., watch variable_name).

  • Catchpoints: Halt execution on specific events like system calls or exceptions (e.g., catch throw).

  • Backtrace: Displays the call stack (backtrace or bt command).

  • Setting Variables: Modifies variable values during debugging using set variable var_name = new_value (e.g., set variable x = 10).

  • Threads: Debugs multi-threaded apps; info threads lists threads; switch with thread <thread-id>.

  • GDB Cheat Sheet Commands

    • run: Starts the program.

    • break <function/line>: Sets a breakpoint.

    • continue: Resumes execution.

    • next: Executes the next line.

    • step: Steps into a function.

    • print <variable>: Prints a variable's value.

Valgrind

  • Memory analysis tool: detects memory leaks, buffer overflows, and uninitialized memory reads.

  • Primarily used with memcheck.

  • Command: valgrind --leak-check=full ./program

  • Improves memory efficiency, assists in debugging memory-related errors.

Commercial Parallel Debuggers

  • Enhanced debugging for multi-threaded/distributed applications.

  • Examples:

    • TotalView: Supports MPI and OpenMP debugging.

    • DDT (Arm Forge): Advanced debugging for parallel HPC applications.

    • Intel Inspector: Debugs memory/threading issues on Intel architectures.

  • Features: graphical interfaces, automated bug detection, scalability.

GDB Sample Execution

  • Compile: gcc–g–oexampleexample.c</code></p></li><li><p>StartGDB:<code>gdbexample</code></p></li><li><p>Setbreakpoint:<code>(gdb)breakmain</code></p></li><li><p>Run:<code>(gdb)run</code></p></li><li><p>Step:<code>(gdb)next</code>or<code>(gdb)step</code></p></li><li><p>Printvariable:<code>(gdb)printa</code>,<code>(gdb)printsum</code></p></li><li><p>Continue:<code>(gdb)continue</code></p></li><li><p>Quit:<code>(gdb)quit</code></p></li></ul><h4id="a654dcae−128e−448d−8759−fe6bbe4a1669"data−toc−id="a654dcae−128e−448d−8759−fe6bbe4a1669"collapsed="false"seolevelmigrated="true">DebuggingonHPC</h4><ul><li><p>Requiresdetailedviewofsupercomputersoftwareandhardware.</p></li><li><p>Invokedviacommand:<code>gdb<executablename></code>.</p></li><li><p><code>−g</code>flagduringcompilationisimportant.</p></li></ul><h4id="4d6e69c1−d67c−4421−a817−aceb83d54763"data−toc−id="4d6e69c1−d67c−4421−a817−aceb83d54763"collapsed="false"seolevelmigrated="true">BreakPoints(GDB)</h4><ul><li><p>Interruptscodeexecutiontoexamineprogramstate.</p></li><li><p>Setusingfunctionname,linenumber,filename,orcondition.</p></li><li><p>Examples(usingcodefromFig.14.2):</p><ul><li><p><code>breakprintf</code></p></li><li><p><code>break17</code></p></li><li><p><code>breakdotprodserial.c:17</code></p></li><li><p><code>breakdotprodserial.c:16ifi==4</code></p></li></ul></li><li><p><code>infobreakpoints</code>:Queriesbreakpointinformation.</p></li><li><p>Disable:<code>disable<breakpointid></code>;enable:<code>enable<breakpointid></code>;delete:<code>delete<breakpointid></code>;temporarybreakpoint:<code>tbreak<location></code>.</p></li></ul><h4id="8df3b644−7a20−48ed−9af6−77b1f947dc92"data−toc−id="8df3b644−7a20−48ed−9af6−77b1f947dc92"collapsed="false"seolevelmigrated="true">WatchPointsandCatchPoints</h4><ul><li><p>Conditionalbreakpointsbasedonvariablewritesorevents.</p></li><li><p><code>watch<expression></code>:Watchesforchangestoavariable.</p><ul><li><p>Example:TowatchforchangestothevalueofthesumvariableinFig.14.2,thecommandwatchsumwouldbeissuedtothegdbcommandlineoncethevariablesumwasinthecurrentcontextatline20</p></li></ul></li><li><p><code>infowatchpoints</code>:Commandtogettheinformationonwatchpoints.</p></li></ul><h4id="2df8a813−0efb−4fc1−9fd7−adcfa55c75cf"data−toc−id="2df8a813−0efb−4fc1−9fd7−adcfa55c75cf"collapsed="false"seolevelmigrated="true">BackTrace</h4><ul><li><p>Showsthecallstackwhenexecutionpauses.</p></li><li><p>Use<code>backtrace</code>(or<code>bt</code>)command.</p></li><li><p>Use<code>up</code>and<code>down</code>totraversethecallstack.</p></li></ul><h4id="7f41ac8a−a1bd−41b3−85d0−de34a94f83af"data−toc−id="7f41ac8a−a1bd−41b3−85d0−de34a94f83af"collapsed="false"seolevelmigrated="true">SettingaVariable(GDB)</h4><ul><li><p>Modifyvariablevaluesduringexecutiontotestscenarios.</p></li><li><p>Usethe<code>set</code>command:<code>setvariablevarname=newvalue</code>(e.g.,<code>setvariablex=10</code>).</p></li></ul><h4id="bf9b7a69−cbe9−4bda−a199−e469ad044f82"data−toc−id="bf9b7a69−cbe9−4bda−a199−e469ad044f82"collapsed="false"seolevelmigrated="true">Threads(GDB)</h4><ul><li><p>Formultithreadedapplications(e.g.,OpenMP).</p></li><li><p><code>infothreads</code>:ListsthreadsandIDs.</p></li><li><p><code>thread<thread−id></code>:Switchesbetweenthreads.</p></li></ul><h4id="2581b4ce−5073−432b−8cbf−61323eef8a2c"data−toc−id="2581b4ce−5073−432b−8cbf−61323eef8a2c"collapsed="false"seolevelmigrated="true">Valgrind</h4><ul><li><p>Toolsuitefordebugging,especiallymemoryerrorsandthreaddataraces.</p></li><li><p>Compilewith<code>−g</code>flag.</p></li><li><p>Usage:<code>valgrind–tool=helgrind<programexecutable></code>(fordataracedetection).</p></li><li><p>Ifnotoolspecified,runsMemcheck(memoryerroridentification).</p></li><li><p>Seetable14.2forGDBcommandsummaries.</p></li></ul><h4id="fa1b29d5−7d64−450f−b9a9−e8a8da23900c"data−toc−id="fa1b29d5−7d64−450f−b9a9−e8a8da23900c"collapsed="false"seolevelmigrated="true">DebuggingParallelBugs</h4><ul><li><p>Parallelprogramscanhave"parallel−only"bugs.</p><ul><li><p>RaceConditions</p><ul><li><p>Arearesultfrommisuseofsharedvariables</p></li></ul></li><li><p>DeadLocks</p><ul><li><p>Whentasksareperpetuallywaitingforamessageorsignalthatneverarrives.</p></li></ul></li></ul></li></ul><h4id="fecc9641−92f4−4105−be96−176b4508856a"data−toc−id="fecc9641−92f4−4105−be96−176b4508856a"collapsed="false"seolevelmigrated="true">RaceConditions</h4><ul><li><p>Occurduetomisuseofsharedvariableswithoutproperprotection.</p></li><li><p>Mitigation:</p><ul><li><p>Use<code>default(none)</code>inOpenMPpragmas.</p></li><li><p>Testcodewithout<code>nowait</code>clauses.</p></li></ul></li><li><p>Manifestaswrongandvariablecoderesults.</p></li><li><p>Debugging:focusonsharedvariables.</p></li></ul><h4id="64f7e6b7−674a−4218−95be−edaef10d4c34"data−toc−id="64f7e6b7−674a−4218−95be−edaef10d4c34"collapsed="false"seolevelmigrated="true">Deadlocks</h4><ul><li><p>Threadslockupwaitingonresourcesthatneverbecomeavailable.</p></li><li><p>Sign:programhangs.</p></li><li><p>Prevention:</p><ul><li><p>BecarefulwiththreadIDconditionalclauses.</p></li><li><p>Use<code>flush</code>pragmaforthreadcommunication.</p></li><li><p>Unsetlocksaftersettingthem.</p></li></ul></li></ul><h4id="583db1b3−2993−405d−a9a6−432324d8844c"data−toc−id="583db1b3−2993−405d−a9a6−432324d8844c"collapsed="false"seolevelmigrated="true">ToolsforParallelDebugging</h4><ul><li><p>Useproperdebuggingandprofilingtools.</p></li><li><p>SHARCNETtools:</p><ul><li><p>DDT:commercialdebugger(MPI,OpenMP,CUDA).</p></li><li><p>VALGRIND:open−sourcememorydebugger.</p></li></ul></li></ul><h4id="0b13df0b−28d1−43a7−ad67−e38adb62813d"data−toc−id="0b13df0b−28d1−43a7−ad67−e38adb62813d"collapsed="false"seolevelmigrated="true">DDT(DistributedDebuggingTool)</h4><ul><li><p>CommercialdebuggerwithGUIforparallelprograms(MPI,OpenMP,CUDA).</p></li><li><p>SupportsC,C++,Fortran.</p></li></ul><h4id="f92a6ff6−9f3a−4d61−a918−46e9ab7289c1"data−toc−id="f92a6ff6−9f3a−4d61−a918−46e9ab7289c1"collapsed="false"seolevelmigrated="true">DebuggingOpenMPExample:UnprotectedSharedVariable</h4><ul><li><p>Commonerror:accessingsharedvariablewithoutprotection.</p></li><li><p>UseValgrindtoidentifydataraces.</p></li><li><p>Example:</p><ul><li><p>Preventingraceconditionsisdonethroughsynchronizationmechanismssuchas:</p><ul><li><p>CriticalSection</p></li><li><p>AtomicOperations</p></li><li><p>ReductionClause</p></li><li><p>Locks</p></li></ul></li></ul></li></ul><h4id="3d2658e2−618a−4d4e−88c5−9b9668192bab"data−toc−id="3d2658e2−618a−4d4e−88c5−9b9668192bab"collapsed="false"seolevelmigrated="true">DebuggingMPIExample:Deadlock</h4><ul><li><p>Commonerror:competingrequestsblockprogress.</p></li><li><p>DebugusingGDBforeachprocess.</p></li><li><p>Twoapproaches:</p><ul><li><p>Launchxtermwindowforeachprocess(maynotworkonallclusters).</p></li><li><p>ModifycodetoprintPIDsandpauseexecution.</p></li></ul></li></ul><h4id="068570d5−96ac−4ec4−93c8−dc8ce1da8a3b"data−toc−id="068570d5−96ac−4ec4−93c8−dc8ce1da8a3b"collapsed="false"seolevelmigrated="true">CompilerFlagsforDebugging</h4><ul><li><p>Compilerwarningsassistindebugging.</p></li><li><p>Usecommand−lineoptionstocheckformistakes.</p></li><li><p>Seetable14.5forGNU,Intel,LLVM,andPGIoptions.</p></li></ul><h4id="1dcd3772−399f−42db−bad6−c18c7e8d2e6f"data−toc−id="1dcd3772−399f−42db−bad6−c18c7e8d2e6f"collapsed="false"seolevelmigrated="true">SystemMonitorstoAidDebugging</h4><ul><li><p>Clustersusemonitoringsoftwarefornodehardwarestatusandworkloadinformation.</p></li><li><p>MonitorsCPUutilization,memoryusage,I/O,networktraffic,diskspace.</p></li><li><p>Lightweightdaemonscollectinformationregularly.</p></li><li><p>Commonmonitors:Nagios,Ganglia.</p></li></ul><h4id="dcfe6276−42d4−4d9d−a085−e133a24a5d1e"data−toc−id="dcfe6276−42d4−4d9d−a085−e133a24a5d1e"collapsed="false"seolevelmigrated="true">TypesofErrorsandDebuggingTools</h4><ul><li><p>SyntaxErrors:</p><ul><li><p>CompilerErrorMessages</p></li><li><p>StaticAnalysisTools</p></li></ul></li><li><p>RuntimeErrors:</p><ul><li><p>GDB</p></li><li><p>Valgrind</p></li><li><p>AddressSanitizer</p></li><li><p>CoreDumps</p></li></ul></li><li><p>LogicalErrors:</p><ul><li><p>PrintfDebugging</p></li><li><p>GDB/LLDB</p></li><li><p>UnitTestingFrameworks</p></li></ul></li><li><p>MemoryErrors:</p><ul><li><p>Valgrind</p></li><li><p>AddressSanitizer</p></li><li><p>Heaptrack</p></li></ul></li><li><p>ConcurrencyErrors:</p><ul><li><p>ThreadSanitizer</p></li><li><p>IntelInspector</p></li><li><p>Helgrind</p></li></ul></li><li><p>DistributedSystemErrors</p><ul><li><p>TotalView</p></li><li><p>MPICHECK</p></li><li><p>DTrace/strace</p></li></ul></li><li><p>PerformanceErrors:</p><ul><li><p>Perf</p></li><li><p>Gprof</p></li><li><p>VtuneProfiler</p></li></ul></li></ul><h4id="a0ffbd08−7e82−4700−8a9a−42946f1efe16"data−toc−id="a0ffbd08−7e82−4700−8a9a−42946f1efe16"collapsed="false"seolevelmigrated="true">ProfilingTools</h4><ul><li><p>Linux:Perf,PAPI,Perfctr,Gprof</p></li><li><p>Windows:IntelVTuneProfiler</p></li><li><p>MeasureCPUusage,memoryallocation,functioncallfrequencies,andmore.</p></li><li><p>Identifybottlenecksandoptimizecode.</p></li><li><p>Findhotspots,measurecacheefficiency,analyzebranchmispredictions,andtrackI/Obottlenecks.</p></li></ul><h4id="0c15d4d6−fc2c−4baf−b2dc−2cfe9e607870"data−toc−id="0c15d4d6−fc2c−4baf−b2dc−2cfe9e607870"collapsed="false"seolevelmigrated="true">PerfTool</h4><ul><li><p>ProfilertoolforLinux.</p></li><li><p>Identifieshotspots,measurescacheefficiencyandanalyzesbranchmispredictions</p></li><li><p>TypicaloutputMetrics:thenumberofCPUcycles,thenumberofinstructionsexecuted,cachehitsandmisses</p></li><li><p>BasicCommands:</p><ul><li><p><code>perfstat</code>:Providesasummaryofperformancecounterswhileacommandruns.</p></li><li><p><code>perfrecord</code>:Capturesdetailedperformancedataforlateranalysis.</p></li></ul></li></ul><h4id="80e56254−ff58−4dec−903c−acc866351c0a"data−toc−id="80e56254−ff58−4dec−903c−acc866351c0a"collapsed="false"seolevelmigrated="true">Gnuplot</h4><ul><li><p>Graphingutility,commandlinedriven.</p></li><li><p>Cross−platformfunctionality</p></li><li><p>Generateawiderangeofplots</p></li><li><p>ControlledviaScripts</p></li><li><p>BasicCommandSequence:</p><ul><li><p><code>gcc –g –o example example.c</code></p></li><li><p>Start GDB: <code>gdb example</code></p></li><li><p>Set breakpoint: <code>(gdb) break main</code></p></li><li><p>Run: <code>(gdb) run</code></p></li><li><p>Step: <code>(gdb) next</code> or <code>(gdb) step</code></p></li><li><p>Print variable: <code>(gdb) print a</code>, <code>(gdb) print sum</code></p></li><li><p>Continue: <code>(gdb) continue</code></p></li><li><p>Quit: <code>(gdb) quit</code></p></li></ul><h4 id="a654dcae-128e-448d-8759-fe6bbe4a1669" data-toc-id="a654dcae-128e-448d-8759-fe6bbe4a1669" collapsed="false" seolevelmigrated="true">Debugging on HPC</h4><ul><li><p>Requires detailed view of supercomputer software and hardware.</p></li><li><p>Invoked via command: <code>gdb <executable name></code>.</p></li><li><p><code>-g</code> flag during compilation is important.</p></li></ul><h4 id="4d6e69c1-d67c-4421-a817-aceb83d54763" data-toc-id="4d6e69c1-d67c-4421-a817-aceb83d54763" collapsed="false" seolevelmigrated="true">Break Points (GDB)</h4><ul><li><p>Interrupts code execution to examine program state.</p></li><li><p>Set using function name, line number, file name, or condition.</p></li><li><p>Examples (using code from Fig. 14.2):</p><ul><li><p><code>break printf</code></p></li><li><p><code>break 17</code></p></li><li><p><code>break dotprod_serial.c:17</code></p></li><li><p><code>break dotprod_serial.c:16 if i==4</code></p></li></ul></li><li><p><code>info breakpoints</code>: Queries break point information.</p></li><li><p>Disable: <code>disable <breakpoint_id></code>; enable: <code>enable <breakpoint_id></code>; delete: <code>delete <breakpoint_id></code>; temporary breakpoint: <code>tbreak <location></code>.</p></li></ul><h4 id="8df3b644-7a20-48ed-9af6-77b1f947dc92" data-toc-id="8df3b644-7a20-48ed-9af6-77b1f947dc92" collapsed="false" seolevelmigrated="true">Watch Points and Catch Points</h4><ul><li><p>Conditional breakpoints based on variable writes or events.</p></li><li><p><code>watch <expression></code>: Watches for changes to a variable.</p><ul><li><p>Example: To watch for changes to the value of the sum variable in Fig. 14.2, the command watch sum would be issued to the gdb command line once the variable sum was in the current context at line 20</p></li></ul></li><li><p><code>info watchpoints</code>: Command to get the information on watch points.</p></li></ul><h4 id="2df8a813-0efb-4fc1-9fd7-adcfa55c75cf" data-toc-id="2df8a813-0efb-4fc1-9fd7-adcfa55c75cf" collapsed="false" seolevelmigrated="true">Back Trace</h4><ul><li><p>Shows the call stack when execution pauses.</p></li><li><p>Use <code>backtrace</code> (or <code>bt</code>) command.</p></li><li><p>Use <code>up</code> and <code>down</code> to traverse the call stack.</p></li></ul><h4 id="7f41ac8a-a1bd-41b3-85d0-de34a94f83af" data-toc-id="7f41ac8a-a1bd-41b3-85d0-de34a94f83af" collapsed="false" seolevelmigrated="true">Setting a Variable (GDB)</h4><ul><li><p>Modify variable values during execution to test scenarios.</p></li><li><p>Use the <code>set</code> command: <code>set variable var_name = new_value</code> (e.g., <code>set variable x = 10</code>).</p></li></ul><h4 id="bf9b7a69-cbe9-4bda-a199-e469ad044f82" data-toc-id="bf9b7a69-cbe9-4bda-a199-e469ad044f82" collapsed="false" seolevelmigrated="true">Threads (GDB)</h4><ul><li><p>For multithreaded applications (e.g., OpenMP).</p></li><li><p><code>info threads</code>: Lists threads and IDs.</p></li><li><p><code>thread <thread-id></code>: Switches between threads.</p></li></ul><h4 id="2581b4ce-5073-432b-8cbf-61323eef8a2c" data-toc-id="2581b4ce-5073-432b-8cbf-61323eef8a2c" collapsed="false" seolevelmigrated="true">Valgrind</h4><ul><li><p>Tool suite for debugging, especially memory errors and thread data races.</p></li><li><p>Compile with <code>-g</code> flag.</p></li><li><p>Usage: <code>valgrind –tool=helgrind <program executable></code> (for data race detection).</p></li><li><p>If no tool specified, runs Memcheck (memory error identification).</p></li><li><p>See table 14.2 for GDB command summaries.</p></li></ul><h4 id="fa1b29d5-7d64-450f-b9a9-e8a8da23900c" data-toc-id="fa1b29d5-7d64-450f-b9a9-e8a8da23900c" collapsed="false" seolevelmigrated="true">Debugging Parallel Bugs</h4><ul><li><p>Parallel programs can have "parallel-only" bugs.</p><ul><li><p>Race Conditions</p><ul><li><p>Are a result from misuse of shared variables</p></li></ul></li><li><p>Dead Locks</p><ul><li><p>When tasks are perpetually waiting for a message or signal that never arrives.</p></li></ul></li></ul></li></ul><h4 id="fecc9641-92f4-4105-be96-176b4508856a" data-toc-id="fecc9641-92f4-4105-be96-176b4508856a" collapsed="false" seolevelmigrated="true">Race Conditions</h4><ul><li><p>Occur due to misuse of shared variables without proper protection.</p></li><li><p>Mitigation:</p><ul><li><p>Use <code>default(none)</code> in OpenMP pragmas.</p></li><li><p>Test code without <code>nowait</code> clauses.</p></li></ul></li><li><p>Manifest as wrong and variable code results.</p></li><li><p>Debugging: focus on shared variables.</p></li></ul><h4 id="64f7e6b7-674a-4218-95be-edaef10d4c34" data-toc-id="64f7e6b7-674a-4218-95be-edaef10d4c34" collapsed="false" seolevelmigrated="true">Deadlocks</h4><ul><li><p>Threads lock up waiting on resources that never become available.</p></li><li><p>Sign: program hangs.</p></li><li><p>Prevention:</p><ul><li><p>Be careful with threadID conditional clauses.</p></li><li><p>Use <code>flush</code> pragma for thread communication.</p></li><li><p>Unset locks after setting them.</p></li></ul></li></ul><h4 id="583db1b3-2993-405d-a9a6-432324d8844c" data-toc-id="583db1b3-2993-405d-a9a6-432324d8844c" collapsed="false" seolevelmigrated="true">Tools for Parallel Debugging</h4><ul><li><p>Use proper debugging and profiling tools.</p></li><li><p>SHARCNET tools:</p><ul><li><p>DDT: commercial debugger (MPI, OpenMP, CUDA).</p></li><li><p>VALGRIND: open-source memory debugger.</p></li></ul></li></ul><h4 id="0b13df0b-28d1-43a7-ad67-e38adb62813d" data-toc-id="0b13df0b-28d1-43a7-ad67-e38adb62813d" collapsed="false" seolevelmigrated="true">DDT (Distributed Debugging Tool)</h4><ul><li><p>Commercial debugger with GUI for parallel programs (MPI, OpenMP, CUDA).</p></li><li><p>Supports C, C++, Fortran.</p></li></ul><h4 id="f92a6ff6-9f3a-4d61-a918-46e9ab7289c1" data-toc-id="f92a6ff6-9f3a-4d61-a918-46e9ab7289c1" collapsed="false" seolevelmigrated="true">Debugging OpenMP Example: Unprotected Shared Variable</h4><ul><li><p>Common error: accessing shared variable without protection.</p></li><li><p>Use Valgrind to identify data races.</p></li><li><p>Example:</p><ul><li><p>Preventing race conditions is done through synchronization mechanisms such as:</p><ul><li><p>Critical Section</p></li><li><p>Atomic Operations</p></li><li><p>Reduction Clause</p></li><li><p>Locks</p></li></ul></li></ul></li></ul><h4 id="3d2658e2-618a-4d4e-88c5-9b9668192bab" data-toc-id="3d2658e2-618a-4d4e-88c5-9b9668192bab" collapsed="false" seolevelmigrated="true">Debugging MPI Example: Deadlock</h4><ul><li><p>Common error: competing requests block progress.</p></li><li><p>Debug using GDB for each process.</p></li><li><p>Two approaches:</p><ul><li><p>Launch xterm window for each process (may not work on all clusters).</p></li><li><p>Modify code to print PIDs and pause execution.</p></li></ul></li></ul><h4 id="068570d5-96ac-4ec4-93c8-dc8ce1da8a3b" data-toc-id="068570d5-96ac-4ec4-93c8-dc8ce1da8a3b" collapsed="false" seolevelmigrated="true">Compiler Flags for Debugging</h4><ul><li><p>Compiler warnings assist in debugging.</p></li><li><p>Use command-line options to check for mistakes.</p></li><li><p>See table 14.5 for GNU, Intel, LLVM, and PGI options.</p></li></ul><h4 id="1dcd3772-399f-42db-bad6-c18c7e8d2e6f" data-toc-id="1dcd3772-399f-42db-bad6-c18c7e8d2e6f" collapsed="false" seolevelmigrated="true">System Monitors to Aid Debugging</h4><ul><li><p>Clusters use monitoring software for node hardware status and workload information.</p></li><li><p>Monitors CPU utilization, memory usage, I/O, network traffic, disk space.</p></li><li><p>Lightweight daemons collect information regularly.</p></li><li><p>Common monitors: Nagios, Ganglia.</p></li></ul><h4 id="dcfe6276-42d4-4d9d-a085-e133a24a5d1e" data-toc-id="dcfe6276-42d4-4d9d-a085-e133a24a5d1e" collapsed="false" seolevelmigrated="true">Types of Errors and Debugging Tools</h4><ul><li><p>Syntax Errors:</p><ul><li><p>Compiler Error Messages</p></li><li><p>Static Analysis Tools</p></li></ul></li><li><p>Runtime Errors:</p><ul><li><p>GDB</p></li><li><p>Valgrind</p></li><li><p>AddressSanitizer</p></li><li><p>Core Dumps</p></li></ul></li><li><p>Logical Errors:</p><ul><li><p>Printf Debugging</p></li><li><p>GDB/LLDB</p></li><li><p>Unit Testing Frameworks</p></li></ul></li><li><p>Memory Errors:</p><ul><li><p>Valgrind</p></li><li><p>AddressSanitizer</p></li><li><p>Heaptrack</p></li></ul></li><li><p>Concurrency Errors:</p><ul><li><p>ThreadSanitizer</p></li><li><p>Intel Inspector</p></li><li><p>Helgrind</p></li></ul></li><li><p>Distributed System Errors</p><ul><li><p>TotalView</p></li><li><p>MPICHECK</p></li><li><p>DTrace/strace</p></li></ul></li><li><p>Performance Errors:</p><ul><li><p>Perf</p></li><li><p>Gprof</p></li><li><p>Vtune Profiler</p></li></ul></li></ul><h4 id="a0ffbd08-7e82-4700-8a9a-42946f1efe16" data-toc-id="a0ffbd08-7e82-4700-8a9a-42946f1efe16" collapsed="false" seolevelmigrated="true">Profiling Tools</h4><ul><li><p>Linux: Perf, PAPI, Perfctr, Gprof</p></li><li><p>Windows: Intel VTune Profiler</p></li><li><p>Measure CPU usage, memory allocation, function call frequencies, and more.</p></li><li><p>Identify bottlenecks and optimize code.</p></li><li><p>Find hotspots, measure cache efficiency, analyze branch mispredictions, and track I/O bottlenecks.</p></li></ul><h4 id="0c15d4d6-fc2c-4baf-b2dc-2cfe9e607870" data-toc-id="0c15d4d6-fc2c-4baf-b2dc-2cfe9e607870" collapsed="false" seolevelmigrated="true">Perf Tool</h4><ul><li><p>Profiler tool for Linux.</p></li><li><p>Identifies hotspots, measures cache efficiency and analyzes branch mispredictions</p></li><li><p>Typical output Metrics: the number of CPU cycles, the number of instructions executed, cache hits and misses</p></li><li><p>Basic Commands:</p><ul><li><p><code>perf stat</code>: Provides a summary of performance counters while a command runs.</p></li><li><p><code>perfrecord</code>: Captures detailed performance data for later analysis.</p></li></ul></li></ul><h4 id="80e56254-ff58-4dec-903c-acc866351c0a" data-toc-id="80e56254-ff58-4dec-903c-acc866351c0a" collapsed="false" seolevelmigrated="true">Gnuplot</h4><ul><li><p>Graphing utility, command line driven.</p></li><li><p>Cross-platform functionality</p></li><li><p>Generate a wide range of plots</p></li><li><p>Controlled via Scripts</p></li><li><p>Basic Command Sequence:</p><ul><li><p><code>gnuplot <gnuplot> plot sin(x)