Teaching Setting up gem5/garnet at Georgia Tech

Setting up gem5/garnet at Georgia Tech

gem5 and garnet

gem5 is a simulation platform for computer-system architecture research. It came as a merger of the m5 simulator from the University of Michigan Ann Arbor, and the GEMS simulator from the University of Wisconsin Madison.

gem5’s on-chip network implementation is called Garnet.

A subset of instructions are given here for installing and running gem5/garnet at Georgia Tech.

The gem5 wiki describes the full details of getting started, running workloads, and the implemented features.

gem5 needs the mercurial version control system.


Software Environment

We strongly recommend using the ECE Machines (ecelinsrvx, ecelinsrvy, ecelinsrvz):

ssh <GTid>@ecelinsrv7.ece.gatech.edu

If you are a CS student, you have been given ECE accounts for this class.

Remember that you need to be on the GT network/VPN in order to connect to the ECE machines (https://faq.oit.gatech.edu/content/how-do-i-install-campus-vpn)


CS Machines
(shuttle3.cc.gatech.edu): 
(Not supported)

Personal Machine:  (Not recommended)

  • g++ version 4.7 or newer or clang version 3.1 or newer.
  • Python, version 2.5 – 2.7 (we don’t support Python 3.X). gem5 links in the Python interpreter, so you need the Python header files and shared library (e.g., /usr/lib/libpython2.5.so) in addition to the interpreter executable. These may or may not be installed by default. For example, on Debian/Ubuntu, you need the “python-dev” package in addition to the “python” package. If you need a newer or different Python installation but can’t or don’t want to upgrade the default Python on your system, see our page on using a non-default Python installation.
  • SCons, version 0.98.1 or newer. SCons is a powerful replacement for make. See here to download SCons. If you don’t have administrator privileges on your machine, you can use the “scons-local” package to install scons in your m5 directory, or install SCons in your home directory using the ‘–prefix=’ option.
  • SWIG, version 2.0.4 or newer.
  • zlib, any recent version. For Debian/Ubuntu, you will need the “zlib-dev” or “zlib1g-dev” package to get the zlib.h header file as well as the library itself.
  • m4, the macro processor.

The following is optional, but highly recommended:

  • protobuf, version 2.1 or newer for trace capture and playback support.
  • pydot, Pythons interface to graphviz, is needed for generation of graphical representations of the simulated system topology.

Example – Commands on Ubuntu:
$sudo apt-get install g++
$sudo apt-get install python
$sudo apt-get install python-dev
$sudo apt-get install swig
$sudo apt-get install zlib
$sudo apt-get install m4

For scons, you need to download it from here.
After extract the compressed file, go to the directory and run the following command.
$sudo python setup.py install.


Downloading gem5

hg clone /nethome/tkrishna3/teaching/simulators/gem5/repo/gem5

Useful hg commands:

  • hg status shows what files have been modified in your repository
  • hg diff shows a diff of the modified files.

Network-on-Chip (Garnet) Simulations

gem5 can be run in a special Garnet_standalone mode, where Garnet2.0, the network-on-chip simulator within gem5, runs with synthetic traffic injectors, without real CPUs or caches.

The following set of instructions are for running gem5 in this mode.

Setting the environment

(run this everytime you log in, OR add it into your .my-cshrc file)
source <path_to_gem5>/my_scripts/set_env.cshrc

Building/compiling gem5 for Garnet_standalone simulations (ECE Red Hat Enterprise Linux Machines) 

(this command only needs to be run once when you start. After that you need to run it only if you change any code).
cd gem5
./my_scripts/build_Garnet_standalone.sh

Running gem5 in Garnet_standalone mode

(run everytime you run a new simulation).
./build/Garnet_standalone/gem5.debug configs/example/garnet_synth_traffic.py \
--network=garnet2.0 \
--num-cpus=16 \
--num-dirs=16 \
--topology=Mesh_XY \
--mesh-rows=4 \
--sim-cycles=1000 \
--injectionrate=0.01 \
--synthetic=uniform_random

This command simulates a 4×4 Mesh for 1000 cycles, and injects uniform random traffic at an injection rate of 0.01. 

Note: The \ is added to add a line break. All the command line options can be written in one line without the \

Command Line Parameters

The list of command line parameters are: [Note there are two dashes in front of each option]

System Configuration Description
–num-cpus Number of cpus. This is the number of source (injection) nodes in the network.
–num-dirs Number of directories. This is the number of destination (ejection) nodes in the network.
–network Network model: simple or garnet2.0. Use garnet2.0 for running synthetic traffic.
–topology Topology for connecting the cpus and dirs to the network routers/switches. More detail about different topologies can be found here.
–mesh-rows The number of rows in the mesh. Only valid when –topology is Mesh_* or MeshDirCorners_*.

 

Network Configuration Description
–router-latency Default number of pipeline stages in the garnet router. Has to be >= 1. Can be over-ridden on a per router basis in the topology file.
–link-latency Default latency of each link in the network. Has to be >= 1. Can be over-ridden on a per link basis in the topology file.
–vcs-per-vnet Number of VCs per Virtual Network.
–link-width-bits Width in bits for all links inside the garnet network. Default = 128.

 

Traffic Injection Description
–sim-cycles Total number of cycles for which the simulation should run.
–synthetic The type of synthetic traffic to be injected. The following synthetic traffic patterns are currently supported: ‘uniform_random’, ‘tornado’, ‘bit_complement’, ‘bit_reverse’, ‘bit_rotation’, ‘neighbor’, ‘shuffle’, and ‘transpose’.
–injectionrate Traffic Injection Rate in packets/node/cycle. It can take any decimal value between 0 and 1. The number of digits of precision after the decimal point can be controlled by –precision which is set to 3 as default in garnet_synth_traffic.py.
–single-sender-id Only inject from this sender. To send from all nodes, set to -1.
–single-dest-id Only send to this destination. To send to all destinations as specified by the synthetic traffic pattern, set to -1.
–num-packets-max Maximum number of packets to be injected by each cpu node. Default value is -1 (keep injecting till sim-cycles).
–inj-vnet Only inject in this vnet (0, 1 or 2). 0 and 1 are 1-flit, 2 is 5-flit. Set to -1 to inject randomly in all vnets.

 

Statistics

After the simulation, gem5 produces the following stats file:
./m5out/stats.txt

and configuration file:
./m5out/config.ini

Extracting Network Stats

./my_scripts/extract_network_stats.sh

This creates a network_stats.txt file with the injected flits, received flits, and average latency.

More details can be found on the gem5 wiki:


Tips for Debugging Garnet

I. Inject one (or more fixed number of) packet(s) into the network from a specific source to a specific destination

This can be done by the following command-line options:
--num-packets-max=<maximum packets to inject> \
--single-sender-id=<sender_id>
 \
--single-dest-id=<dest_id>

Each of these options works separately by itself too.

Specifying --num-packets-max restricts the total packets injected by every node; --single-sender-id stops injection from every node except single-sender-id, and --single-dest-id sets the destination of every message to single-dest-id instead of that specified by the traffic pattern (uniform random/tornado/bit-complement).

II. Print debug messages

Debug messages throughout the Garnet code are in the following format:
DPRINTF(RubyNetwork, "Debug message is here and is printing value %d", variable)

You can add your own debug messages across the code to track the progress of a flit.

To print the debug messages, add --debug-flags=RubyNetwork as follows:

./build/Garnet_standalone/gem5.debug --debug-flags=RubyNetwork configs/example/garnet_synth_traffic.py ...

III. Running gem5 with gdb

Running gem5 with gdb can help debug Segmentation Faults. gem5 can be run with gdb as follows:

gdb ./build/Garnet_standalone/gem5.debug

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

(gdb) run configs/example/garnet_synth_traffic.py --network=garnet2.0 ...

On a segmentation fault, getting a backtrace often helps.

(gdb) bt

You can lookup any gdb tutorial online to learn about the various options. Running the code within gdb with breakpoints is another helpful debugging feature.


Implementing your own Synthetic Traffic Patterns

Suppose you are implementing your own traffic pattern called HOT_SPOT.

Step 1: Add hot_spot to configs/example/garnet_synth_traffic.py

parser.add_option("--synthetic", type="choice", default="uniform_random"
choices=['uniform_random', 'tornado', 'bit_complement', \
'bit_reverse', 'bit_rotation', 'neighbor', \
'shuffle', 'transpose', 'hot_spot'])

Step 2: Add HOT_SPOT_ to src/cpu/testers/garnet_synthetic_traffic/GarnetSyntheticTraffic.hh

enum TrafficType {BIT_COMPLEMENT_ = 0,
BIT_REVERSE_ = 1,
BIT_ROTATION_ = 2,
NEIGHBOR_ = 3,
SHUFFLE_ = 4,
TORNADO_ = 5,
TRANSPOSE_ = 6,
UNIFORM_RANDOM_ = 7,
HOT_SPOT_ = 8,
NUM_TRAFFIC_PATTERNS_};

Step 3: Add  HOT_SPOT_to
src/cpu/testers/garnet_synthetic_traffic/GarnetSyntheticTraffic.cc


void
GarnetSyntheticTraffic::initTrafficType()
{
trafficStringToEnum["bit_complement"] = BIT_COMPLEMENT_;
trafficStringToEnum["bit_reverse"] = BIT_REVERSE_;
trafficStringToEnum["bit_rotation"] = BIT_ROTATION_;
trafficStringToEnum["neighbor"] = NEIGHBOR_;
trafficStringToEnum["shuffle"] = SHUFFLE_;
trafficStringToEnum["tornado"] = TORNADO_;
trafficStringToEnum["transpose"] = TRANSPOSE_;
trafficStringToEnum["uniform_random"] = UNIFORM_RANDOM_;
trafficStringToEnum["hot_spot"] = HOT_SPOT_;
}

Step 4: Implement HOT_SPOT_ functionality in src/cpu/testers/garnet_synthetic_traffic/GarnetSyntheticTraffic.cc in the
generatePkt() function.

See how other traffic patterns are implemented.


Running the SMART NoC

Pull out the SMART NoC repo:

hg clone /nethome/tkrishna3/teaching/simulators/gem5/repo/gem5_smart

SMART can be enabled by adding the following parameters the command line. HPCmax can be given any value.
--smart --smart_hpcmax=8


Full-System Simulations

Option 1: Pull out the gem5 repo with full system simulation setup.

hg clone /nethome/tkrishna3/teaching/simulators/gem5/repo/gem5_fs

Click here for example run commands.

Option 2: Setup the full-system binaries from scratch.

Setting up Linux and a Disk Image from the gem5 wiki

Setting up Full-System Binaries and Disks for PARSEC2.1 for x86 linux

Step 1:

wget http://www.m5sim.org/dist/current/m5_system_2.0b3.tar.bz2

Step 2:

tar jxvf m5_system_2.0b3.tar.bz2

    * creates a m5_system_2.0b3 folder with

        disks/

           linux-bigswap2.img

           linux-latest.img

        binaries/

            console

            ts_osfpal

            vmlinux

Step 3:

wget http://www.cs.utexas.edu/~parsec_m5/x86_64-vmlinux-2.6.28.4-smp

Step 4:

mv x86_64-vmlinux-2.6.28.4-smp m5_system_2.0b3/binaries/

Step 5:

wget http://www.cs.utexas.edu/~parsec_m5/x86root-parsec.img.bz2

Step 6:

bunzip2 x86root-parsec.img.bz2

Step 7:

mv x86root-parsec.img m5_system_2.0b3/disks/

Step 8:

Apply the following patch: fullsystem_setup

Useful scripts to create checkpoints

hg clone /nethome/tkrishna3/gem5_setup/gem5_fs_resources