gem5 and garnet
gem5 is a simulation platform for computer-system architecture research. It came as a merger of the m5 simulator from the University of Michigan Ann Arbor, and the GEMS simulator from the University of Wisconsin Madison.
gem5’s on-chip network implementation is called Garnet.
A subset of instructions are given here for installing and running gem5/garnet at Georgia Tech.
The gem5 wiki describes the full details of getting started, running workloads, and the implemented features.
gem5 needs the mercurial version control system.
Software Environment
We strongly recommend using the following ECE Machines :
- ece-linlabsrv01.ece.gatech.edu
ssh <GTid>@ece-linlabsrv01.ece.gatech.edu
If you are a CS student, you have been given ECE accounts for this class.
Remember that you need to be on the GT network/VPN in order to connect to the ECE machines (https://faq.oit.gatech.edu/content/how-do-i-install-campus-vpn)
CS Machines (shuttle3.cc.gatech.edu): (Not supported)
Personal Machine: (Not recommended)
- g++ version 4.7 or newer or clang version 3.1 or newer.
- Python, version 2.5 – 2.7 (we don’t support Python 3.X). gem5 links in the Python interpreter, so you need the Python header files and shared library (e.g., /usr/lib/libpython2.5.so) in addition to the interpreter executable. These may or may not be installed by default. For example, on Debian/Ubuntu, you need the “python-dev” package in addition to the “python” package. If you need a newer or different Python installation but can’t or don’t want to upgrade the default Python on your system, see our page on using a non-default Python installation.
- SCons, version 0.98.1 or newer. SCons is a powerful replacement for make. See here to download SCons. If you don’t have administrator privileges on your machine, you can use the “scons-local” package to install scons in your m5 directory, or install SCons in your home directory using the ‘–prefix=’ option.
- SWIG, version 2.0.4 or newer.
- zlib, any recent version. For Debian/Ubuntu, you will need the “zlib-dev” or “zlib1g-dev” package to get the zlib.h header file as well as the library itself.
- m4, the macro processor.
The following is optional, but highly recommended:
- protobuf, version 2.1 or newer for trace capture and playback support.
- pydot, Pythons interface to graphviz, is needed for generation of graphical representations of the simulated system topology.
Example – Commands on Ubuntu:
$sudo apt-get install g++
$sudo apt-get install python
$sudo apt-get install python-dev
$sudo apt-get install swig
$sudo apt-get install zlib
$sudo apt-get install m4
For scons, you need to download it from here.
After extract the compressed file, go to the directory and run the following command.
$sudo python setup.py install.
Downloading gem5
hg clone /nethome/tkrishna3/teaching/simulators/gem5/repo/gem5
Useful hg commands:
hg statusshows what files have been modified in your repositoryhg diffshows a diff of the modified files.
Network-on-Chip (Garnet) Simulations
gem5 can be run in a special Garnet_standalone mode, where Garnet2.0, the network-on-chip simulator within gem5, runs with synthetic traffic injectors, without real CPUs or caches.
The following set of instructions are for running gem5 in this mode.
Setting the environment
(run this everytime you log in, OR add it into your .my-cshrc file)
source <path_to_gem5>/my_scripts/set_env.cshrc
Building/compiling gem5 for Garnet_standalone simulations (ECE Red Hat Enterprise Linux Machines)
(this command only needs to be run once when you start. After that you need to run it only if you change any code).
cd gem5
./my_scripts/build_Garnet_standalone.sh
Running gem5 in Garnet_standalone mode
(run everytime you run a new simulation).
./build/Garnet_standalone/gem5.debug configs/example/garnet_synth_traffic.py \
--network=garnet2.0 \
--num-cpus=16 \
--num-dirs=16 \
--topology=Mesh_XY \
--mesh-rows=4 \
--sim-cycles=1000 \
--injectionrate=0.01 \
--synthetic=uniform_random
This command simulates a 4×4 Mesh for 1000 cycles, and injects uniform random traffic at an injection rate of 0.01.
Note: The \ is added to add a line break. All the command line options can be written in one line without the \
Command Line Parameters
The list of command line parameters are: [Note there are two dashes in front of each option]
| System Configuration | Description |
|---|---|
| –num-cpus | Number of cpus. This is the number of source (injection) nodes in the network. |
| –num-dirs | Number of directories. This is the number of destination (ejection) nodes in the network. |
| –network | Network model: simple or garnet2.0. Use garnet2.0 for running synthetic traffic. |
| –topology | Topology for connecting the cpus and dirs to the network routers/switches. More detail about different topologies can be found here. |
| –mesh-rows | The number of rows in the mesh. Only valid when –topology is Mesh_* or MeshDirCorners_*. |
| –garnet-deadlock-threshold | The number of cycles of no activity for any packet when a possible deadlock alert is issued. |
| Network Configuration | Description |
|---|---|
| –router-latency | Default number of pipeline stages in the garnet router. Has to be >= 1. Can be over-ridden on a per router basis in the topology file. |
| –link-latency | Default latency of each link in the network. Has to be >= 1. Can be over-ridden on a per link basis in the topology file. |
| –vcs-per-vnet | Number of VCs per Virtual Network. |
| –link-width-bits | Width in bits for all links inside the garnet network. Default = 128. |
| Traffic Injection | Description |
|---|---|
| –sim-cycles | Total number of cycles for which the simulation should run. |
| –synthetic | The type of synthetic traffic to be injected. The following synthetic traffic patterns are currently supported: ‘uniform_random’, ‘tornado’, ‘bit_complement’, ‘bit_reverse’, ‘bit_rotation’, ‘neighbor’, ‘shuffle’, and ‘transpose’. |
| –injectionrate | Traffic Injection Rate in packets/node/cycle. It can take any decimal value between 0 and 1. The number of digits of precision after the decimal point can be controlled by –precision which is set to 3 as default in garnet_synth_traffic.py. |
| –single-sender-id | Only inject from this sender. To send from all nodes, set to -1. |
| –single-dest-id | Only send to this destination. To send to all destinations as specified by the synthetic traffic pattern, set to -1. |
| –num-packets-max | Maximum number of packets to be injected by each cpu node. Default value is -1 (keep injecting till sim-cycles). |
| –inj-vnet | Only inject in this vnet (0, 1 or 2). 0 and 1 are 1-flit, 2 is 5-flit. Set to -1 to inject randomly in all vnets. |
Statistics
After the simulation, gem5 produces the following stats file:
./m5out/stats.txt
and configuration file:
./m5out/config.ini
Extracting Network Stats
./my_scripts/extract_network_stats.sh
This creates a network_stats.txt file with the injected flits, received flits, and average latency.
More details can be found on the gem5 wiki:
- Interconnection Network Models in gem5
- Garnet2.0 Microarchitecture and Pipeline
- Running Garnet2.0 in a stand-alone manner with synthetic traffic
Tips for Debugging Garnet
I. Inject one (or more fixed number of) packet(s) into the network from a specific source to a specific destination
This can be done by the following command-line options:
--num-packets-max=<maximum packets to inject> \
--single-sender-id=<sender_id> \
--single-dest-id=<dest_id>
Each of these options works separately by itself too.
Specifying --num-packets-max restricts the total packets injected by every node; --single-sender-id stops injection from every node except single-sender-id, and --single-dest-id sets the destination of every message to single-dest-id instead of that specified by the traffic pattern (uniform random/tornado/bit-complement).
II. Print debug messages
Debug messages throughout the Garnet code are in the following format:
DPRINTF(RubyNetwork, "Debug message is here and is printing value %d", variable)
You can add your own debug messages across the code to track the progress of a flit.
To print the debug messages, add --debug-flags=RubyNetwork as follows:
./build/Garnet_standalone/gem5.debug --debug-flags=RubyNetwork configs/example/garnet_synth_traffic.py ...
III. Running gem5 with gdb
Running gem5 with gdb can help debug Segmentation Faults. gem5 can be run with gdb as follows:
gdb ./build/Garnet_standalone/gem5.debug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
…
(gdb) run configs/example/garnet_synth_traffic.py --network=garnet2.0 ...
On a segmentation fault, getting a backtrace often helps.
(gdb) bt
You can lookup any gdb tutorial online to learn about the various options. Running the code within gdb with breakpoints is another helpful debugging feature.
Implementing your own Synthetic Traffic Patterns
Suppose you are implementing your own traffic pattern called HOT_SPOT.
Step 1: Add hot_spot to configs/example/garnet_synth_traffic.py
parser.add_option("--synthetic", type="choice", default="uniform_random"
choices=['uniform_random', 'tornado', 'bit_complement', \
'bit_reverse', 'bit_rotation', 'neighbor', \
'shuffle', 'transpose', 'hot_spot'])
Step 2: Add HOT_SPOT_ to src/cpu/testers/garnet_synthetic_traffic/GarnetSyntheticTraffic.hh
enum TrafficType {BIT_COMPLEMENT_ = 0,
BIT_REVERSE_ = 1,
BIT_ROTATION_ = 2,
NEIGHBOR_ = 3,
SHUFFLE_ = 4,
TORNADO_ = 5,
TRANSPOSE_ = 6,
UNIFORM_RANDOM_ = 7,
HOT_SPOT_ = 8,
NUM_TRAFFIC_PATTERNS_};
Step 3: Add HOT_SPOT_to
src/cpu/testers/garnet_synthetic_traffic/GarnetSyntheticTraffic.cc
void
GarnetSyntheticTraffic::initTrafficType()
{
trafficStringToEnum["bit_complement"] = BIT_COMPLEMENT_;
trafficStringToEnum["bit_reverse"] = BIT_REVERSE_;
trafficStringToEnum["bit_rotation"] = BIT_ROTATION_;
trafficStringToEnum["neighbor"] = NEIGHBOR_;
trafficStringToEnum["shuffle"] = SHUFFLE_;
trafficStringToEnum["tornado"] = TORNADO_;
trafficStringToEnum["transpose"] = TRANSPOSE_;
trafficStringToEnum["uniform_random"] = UNIFORM_RANDOM_;
trafficStringToEnum["hot_spot"] = HOT_SPOT_;
}
Step 4: Implement HOT_SPOT_ functionality in src/cpu/testers/garnet_synthetic_traffic/GarnetSyntheticTraffic.cc in the
generatePkt() function.
See how other traffic patterns are implemented.
Running the SMART NoC
Pull out the SMART NoC repo:
hg clone /nethome/tkrishna3/teaching/simulators/gem5/repo/gem5_smart
SMART can be enabled by adding the following parameters the command line. HPCmax can be given any value.
--smart --smart_hpcmax=8
Full-System Simulations
Option 1: Pull out the gem5 repo with full system simulation setup.
hg clone /nethome/tkrishna3/teaching/simulators/gem5/repo/gem5_fs
Click here for example run commands.
Option 2: Setup the full-system binaries from scratch.
Setting up Linux and a Disk Image from the gem5 wiki
Setting up Full-System Binaries and Disks for PARSEC2.1 for x86 linux
Step 1:
wget http://www.m5sim.org/dist/current/m5_system_2.0b3.tar.bz2
Step 2:
tar jxvf m5_system_2.0b3.tar.bz2
* creates a m5_system_2.0b3 folder with
disks/
linux-bigswap2.img
linux-latest.img
binaries/
console
ts_osfpal
vmlinux
Step 3:
wget http://www.cs.utexas.edu/~parsec_m5/x86_64-vmlinux-2.6.28.4-smp
Step 4:
mv x86_64-vmlinux-2.6.28.4-smp m5_system_2.0b3/binaries/
Step 5:
wget http://www.cs.utexas.edu/~parsec_m5/x86root-parsec.img.bz2
Step 6:
bunzip2 x86root-parsec.img.bz2
Step 7:
mv x86root-parsec.img m5_system_2.0b3/disks/
Step 8:
Apply the following patch: fullsystem_setup