{"id":47380,"date":"2015-04-02T19:35:11","date_gmt":"2015-04-02T16:35:11","guid":{"rendered":"https:\/\/www.altoros.com\/blog\/?p=47380"},"modified":"2019-11-27T18:09:08","modified_gmt":"2019-11-27T15:09:08","slug":"golang-internals-part-5-the-runtime-bootstrap-process","status":"publish","type":"post","link":"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/","title":{"rendered":"Golang Internals, Part 5: the Runtime Bootstrap Process"},"content":{"rendered":"<p><center><small>All parts: <a href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-1-main-concepts-and-project-structure\/\">Part 1<\/a> | <a href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-2-diving-into-the-go-compiler\/\">Part 2<\/a> | <a href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-3-the-linker-object-files-and-relocations\/\">Part 3<\/a> | <a href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-4-object-files-and-function-metadata\/\">Part 4<\/a> | <strong>Part 5<\/strong> | <a href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-6-bootstrapping-and-memory-allocator-initialization\/\">Part 6<\/a><\/small><\/center><\/p>\n<p>&nbsp;<br \/>\n<img decoding=\"async\" src=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2019\/11\/appenginegopher.jpg\" alt=\"Golang Internals Go Runtime and Bootstrapping\" width=\"200\" class=\"alignright size-medium wp-image-47381\" \/><\/p>\n<p>The bootstrapping process is the key to understanding how the Go runtime works. Learning it is essential, if you want to move forward with Go. So, the fifth installment in our Golang Internals series is dedicated to the Go runtime and, specifically, the Go bootstrap process. This time you will learn about:<\/p>\n<ul>\n<li>Go bootstrapping<\/li>\n<li>resizable stacks implementation<\/li>\n<li>internal TLS implementation<\/li>\n<\/ul>\n<p>Note that this post contains a lot of assembler code, and you will need at least some basic knowledge of it to proceed (here is a quick <a href=\"https:\/\/go.dev\/doc\/asm\" target=\"_blank\" rel=\"noopener noreferrer\">guide to the Go assembler<\/a>). Let\u2019s get going!<\/p>\n<p>&nbsp;<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_79_2 counter-hierarchy ez-toc-counter ez-toc-transparent ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#Finding_an_entry_point\" >Finding an entry point<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#The_starting_sequence\" >The starting sequence<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#Resizable_stack_implementation_in_Go\" >Resizable stack implementation in Go<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#Further_investigation_of_Go_bootstrapping\" >Further investigation of Go bootstrapping<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#Internal_TLS_implementation\" >Internal TLS implementation<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#Returning_to_the_starting_sequence\" >Returning to the starting sequence<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#Further_reading\" >Further reading<\/a><\/li><\/ul><\/nav><\/div>\n<h3><span class=\"ez-toc-section\" id=\"Finding_an_entry_point\"><\/span>Finding an entry point<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>First, we need to find which function is executed immediately after we start a Go program. To do this, we will write a simple Go app.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\npackage main\r\n\r\nfunc main() {\r\n\tprint(123)\r\n}\r\n<\/pre>\n<p>Then we need to compile and link it.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\ngo tool 6g test.go\r\ngo tool 6l test.6\r\n<\/pre>\n<p>This will create an executable file called <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">6.out<\/code> in your current directory. The next step involves the <a href=\"https:\/\/sourceware.org\/binutils\/docs\/binutils\/objdump.html\" title=\"objdump\" target=\"_blank\" rel=\"noopener noreferrer\">objdump<\/a> tool, which is specific to Linux. Windows and Mac users can find analogs or skip this step altogether. Now, run the following command.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nobjdump -f 6.out\r\n<\/pre>\n<p>You should get the output, which will contain the start address.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n6.out:     file format elf64-x86-64\r\narchitecture: i386:x86-64, flags 0x00000112:\r\nEXEC_P, HAS_SYMS, D_PAGED\r\nstart address 0x000000000042f160\r\n<\/pre>\n<p>Next, we need to disassemble the executable and find which function is located at this address.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nobjdump -d 6.out &gt; disassemble.txt\r\n<\/pre>\n<p>Then we need to open the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">disassemble.txt<\/code> file and search for <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">42f160<\/code>. The output that we got is shown below.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n000000000042f160 &lt;_rt0_amd64_linux&gt;:\r\n  42f160:\t48 8d 74 24 08       \t\tlea    0x8(%rsp),%rsi\r\n  42f165:\t48 8b 3c 24          \t\tmov    (%rsp),%rdi\r\n  42f169:\t48 8d 05 10 00 00 00 \tlea    0x10(%rip),%rax        # 42f180 &lt;main&gt;\r\n  42f170:\tff e0               \t\t \tjmpq   *%rax\r\n<\/pre>\n<p>Nice, we have found it! The entry point for my OS and architecture is a function called <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">_rt0_amd64_linux<\/code>.<\/p>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"The_starting_sequence\"><\/span>The starting sequence<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Now, we need to find this function in the Go runtime sources. It is located in the <a href=\"https:\/\/github.com\/golang\/go\/blob\/master\/src\/runtime\/rt0_linux_amd64.s\" target=\"_blank\" rel=\"noopener noreferrer\">rt0_linux_amd64.s<\/a> file. If you look inside the Go runtime package, you can find many file names with postfixes related to OS and architecture names. When a runtime package is built, only the files that correspond to the current OS and architecture are selected. The rest are skipped. Let\u2019s take a closer look at <a href=\"https:\/\/github.com\/golang\/go\/blob\/master\/src\/runtime\/rt0_linux_amd64.s\" target=\"_blank\" rel=\"noopener noreferrer\">rt0_linux_amd64.s<\/a>.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nTEXT _rt0_amd64_linux(SB),NOSPLIT,$-8\r\n\tLEAQ\t8(SP), SI \/\/ argv\r\n\tMOVQ\t0(SP), DI \/\/ argc\r\n\tMOVQ\t$main(SB), AX\r\n\tJMP\tAX\r\n\r\nTEXT main(SB),NOSPLIT,$-8\r\n\tMOVQ\t$runtime\u00b7rt0_go(SB), AX\r\n\tJMP\tAX\r\n<\/pre>\n<p>The <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">_rt0_amd64_linux<\/code> function is very simple. It calls the main function and saves arguments (<code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">argc<\/code> and <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">argv<\/code>) in registers (<code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">DI<\/code> and <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">SI<\/code>). The arguments are located in the stack and can be accessed via the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">SP<\/code> (stack pointer) register. The main function is also very simple. It calls the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.rt0_go<\/code> function, which is longer and more complicated, so we will break it into small parts and describe each separately. The first section goes like this.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n\tMOVQ\tDI, AX\t\t\/\/ argc\r\n\tMOVQ\tSI, BX\t\t\/\/ argv\r\n\tSUBQ\t$(4*8+7), SP\t\t\/\/ 2args 2auto\r\n\tANDQ\t$~15, SP\r\n\tMOVQ\tAX, 16(SP)\r\n\tMOVQ\tBX, 24(SP)\r\n<\/pre>\n<p>Here, we put some previously saved command line argument values inside the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">AX<\/code> and <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">BX<\/code> decrease stack pointers. We also add space for two more four-byte variables and adjust it to be 16-bit aligned. Finally, we move the arguments back to the stack.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n\t\/\/ create istack out of the given (operating system) stack.\r\n\t\/\/ _cgo_init may update stackguard.\r\n\tMOVQ\t$runtime\u00b7g0(SB), DI\r\n\tLEAQ\t(-64*1024+104)(SP), BX\r\n\tMOVQ\tBX, g_stackguard0(DI)\r\n\tMOVQ\tBX, g_stackguard1(DI)\r\n\tMOVQ\tBX, (g_stack+stack_lo)(DI)\r\n\tMOVQ\tSP, (g_stack+stack_hi)(DI)\r\n<\/pre>\n<p>The second part is a bit more tricky. First, we load the address of the global <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.g0<\/code> variable into the DI register. This variable is defined in the <a href=\"https:\/\/github.com\/golang\/go\/blob\/master\/src\/runtime\/proc1.go\" target=\"_blank\" rel=\"noopener noreferrer\">proc1.go<\/a> file and belongs to the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime,g<\/code> type. Variables of this type are created for each <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">goroutine<\/code> in the system. As you can guess, <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.g0 <\/code> describes a root <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">goroutine<\/code>. Then, we initialize the fields that describe the stack of the root <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">goroutine<\/code>. The meaning of <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">stack.lo<\/code> and <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">stack.hi<\/code> should be clear. These are pointers to the beginning and the end of the stack for the current <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">goroutine<\/code>, but what are the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">stackguard0<\/code> and <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">stackguard1<\/code> fields? To understand this, we need to set aside the investigation of the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.rt0_go<\/code> function and take a closer look at stack growth in Go.<\/p>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Resizable_stack_implementation_in_Go\"><\/span>Resizable stack implementation in Go<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>The Go language uses resizable stacks. Each <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">goroutine<\/code> starts with a small stack, and its size changes each time a certain threshold is reached. Obviously, there is a way to check whether we have reached this threshold or not. In fact, the check is performed at the beginning of each function. To see how it works, let\u2019s compile our sample program one more time with the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">-S<\/code> flag (this will show the generated assembler code). The beginning of the main function looks like this.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n&quot;&quot;.main t=1 size=48 value=0 args=0x0 locals=0x8\r\n\t0x0000 00000 (test.go:3)\tTEXT\t&quot;&quot;.main+0(SB),$8-0\r\n\t0x0000 00000 (test.go:3)\tMOVQ\t(TLS),CX\r\n\t0x0009 00009 (test.go:3)\tCMPQ\tSP,16(CX)\r\n\t0x000d 00013 (test.go:3)\tJHI\t,22\r\n\t0x000f 00015 (test.go:3)\tCALL\t,runtime.morestack_noctxt(SB)\r\n\t0x0014 00020 (test.go:3)\tJMP\t,0\r\n\t0x0016 00022 (test.go:3)\tSUBQ\t$8,SP\r\n<\/pre>\n<p>First, we load a value from thread local storage (TLS) to the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">CX<\/code> register (we have already explained what TLS is in the <a href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-3-the-linker-object-files-and-relocations\/\">previous article<\/a>). This value always contains a pointer to the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.g<\/code> structure that corresponds to the current <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">goroutine<\/code>. Then we compare the stack pointer to the value located at an offset of 16 bytes in the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.g<\/code> structure. We can easily calculate that this corresponds to the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">stackguard0<\/code> field.<\/p>\n<p>So, this is how we check if we have reached the stack threshold. If we haven\u2019t reached it yet, the check fails. In this case, we call the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.morestack_noctxt<\/code> function repeatedly until enough memory has been allocated for the stack. The <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">stackguard1<\/code> field works very similarly to <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">stackguard0<\/code>, but it is used inside the C stack growth prologue instead of Go. The inner workings of <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.morestack_noctxt<\/code> is also a very interesting topic, but we will discuss it later. For now, let\u2019s return to the bootstrap process.<\/p>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Further_investigation_of_Go_bootstrapping\"><\/span>Further investigation of Go bootstrapping<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>We will proceed with the starting sequence by looking at the next portion of code inside the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.rt0_go<\/code> function.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n\t\/\/ find out information about the processor we're on\r\n\tMOVQ\t$0, AX\r\n\tCPUID\r\n\tCMPQ\tAX, $0\r\n\tJE\tnocpuinfo\r\n\r\n\t\/\/ Figure out how to serialize RDTSC.\r\n\t\/\/ On Intel processors LFENCE is enough. AMD requires MFENCE.\r\n\t\/\/ Don't know about the rest, so let's do MFENCE.\r\n\tCMPL\tBX, $0x756E6547  \/\/ &quot;Genu&quot;\r\n\tJNE\tnotintel\r\n\tCMPL\tDX, $0x49656E69  \/\/ &quot;ineI&quot;\r\n\tJNE\tnotintel\r\n\tCMPL\tCX, $0x6C65746E  \/\/ &quot;ntel&quot;\r\n\tJNE\tnotintel\r\n\tMOVB\t$1, runtime\u00b7lfenceBeforeRdtsc(SB)\r\nnotintel:\r\n\r\n\tMOVQ\t$1, AX\r\n\tCPUID\r\n\tMOVL\tCX, runtime\u00b7cpuid_ecx(SB)\r\n\tMOVL\tDX, runtime\u00b7cpuid_edx(SB)\r\nnocpuinfo:\t\r\n<\/pre>\n<p>This part is not crucial for understanding major Go concepts, so we will look through it briefly. Here, we are trying to figure out which processor we are using. If it is Intel, we set the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime\u00b7lfenceBeforeRdtsc<\/code> variable. The <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime\u00b7cputicks<\/code> method is the only place where this variable is used. This method utilizes a different assembler instruction to get <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">cpu ticks<\/code> depending on the value of <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime\u00b7lfenceBeforeRdtsc<\/code>. Finally, we call the CPUID assembler instruction, execute it, and save the result in the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime\u00b7cpuid_ecx<\/code> and <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime\u00b7cpuid_edx<\/code> variables. These are used in the <a href=\"https:\/\/github.com\/golang\/go\/blob\/master\/src\/runtime\/alg.go\" target=\"_blank\" rel=\"noopener noreferrer\">alg.go<\/a> file to select a proper hashing algorithm that is natively supported by your computer\u2019s architecture.<\/p>\n<p>Ok, let\u2019s move on and examine another portion of code.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n\t\/\/ if there is an _cgo_init, call it.\r\n\tMOVQ\t_cgo_init(SB), AX\r\n\tTESTQ\tAX, AX\r\n\tJZ\tneedtls\r\n\t\/\/ g0 already in DI\r\n\tMOVQ\tDI, CX\t\/\/ Win64 uses CX for first parameter\r\n\tMOVQ\t$setg_gcc&lt;&gt;(SB), SI\r\n\tCALL\tAX\r\n\r\n\t\/\/ update stackguard after _cgo_init\r\n\tMOVQ\t$runtime\u00b7g0(SB), CX\r\n\tMOVQ\t(g_stack+stack_lo)(CX), AX\r\n\tADDQ\t$const__StackGuard, AX\r\n\tMOVQ\tAX, g_stackguard0(CX)\r\n\tMOVQ\tAX, g_stackguard1(CX)\r\n\r\n\tCMPL\truntime\u00b7iswindows(SB), $0\r\n\tJEQ ok\r\n<\/pre>\n<p>This fragment is only executed when <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">cgo<\/code> is enabled.<\/p>\n<p>The next code fragment is responsible for setting up TLS.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nneedtls:\r\n\t\/\/ skip TLS setup on Plan 9\r\n\tCMPL\truntime\u00b7isplan9(SB), $1\r\n\tJEQ ok\r\n\t\/\/ skip TLS setup on Solaris\r\n\tCMPL\truntime\u00b7issolaris(SB), $1\r\n\tJEQ ok\r\n\r\n\tLEAQ\truntime\u00b7tls0(SB), DI\r\n\tCALL\truntime\u00b7settls(SB)\r\n\r\n\t\/\/ store through it, to make sure it works\r\n\tget_tls(BX)\r\n\tMOVQ\t$0x123, g(BX)\r\n\tMOVQ\truntime\u00b7tls0(SB), AX\r\n\tCMPQ\tAX, $0x123\r\n\tJEQ 2(PC)\r\n\tMOVL\tAX, 0\t\/\/ abort\r\n<\/pre>\n<p>We have already mentioned TLS before. Now, it is time to understand how it is implemented.<\/p>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Internal_TLS_implementation\"><\/span>Internal TLS implementation<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>If you look at the previous code fragment carefully, you can easily understand what the only lines that do actual work are.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nLEAQ\truntime\u00b7tls0(SB), DI\r\n\tCALL\truntime\u00b7settls(SB)\r\n<\/pre>\n<p>All the other stuff is used to skip TLS setup, when it is not supported on your OS, and check that TLS works correctly. The two lines above store the address of the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime\u00b7tls0<\/code> variable in the DI register and call the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime\u00b7settls<\/code> function. The code of this function is shown below.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n\/\/ set tls base to DI\r\nTEXT runtime\u00b7settls(SB),NOSPLIT,$32\r\n\tADDQ\t$8, DI\t\/\/ ELF wants to use -8(FS)\r\n\r\n\tMOVQ\tDI, SI\r\n\tMOVQ\t$0x1002, DI\t\/\/ ARCH_SET_FS\r\n\tMOVQ\t$158, AX\t\/\/ arch_prctl\r\n\tSYSCALL\r\n\tCMPQ\tAX, $0xfffffffffffff001\r\n\tJLS\t2(PC)\r\n\tMOVL\t$0xf1, 0xf1  \/\/ crash\r\n\tRET\r\n<\/pre>\n<p>From the comments, we can understand that this function makes the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">arch_prctl<\/code> system call and passes <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">ARCH_SET_FS <\/code> as an argument. We can also see that this system call sets a base for the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">FS<\/code> segment register. In our case, we set TLS to point to the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime\u00b7tls0<\/code> variable.<\/p>\n<p>Do you remember the instruction that we saw at the beginning of the assembler code for the main function?<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n\t0x0000 00000 (test.go:3)\tMOVQ\t(TLS),CX\r\n<\/pre>\n<p>We have previously explained that it loads the address of the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.g<\/code> structure instance into the CX register. This structure describes the current <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">goroutine<\/code> and is stored in a thread local storage. Now, we can find out and understand how this instruction is translated into a machine assembler. If you open the previously created <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">disassembly.txt<\/code> file and look for the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">main.main<\/code> function, the first instruction inside it should look like this.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n400c00:       64 48 8b 0c 25 f0 ff    mov    %fs:0xfffffffffffffff0,%rcx\r\n<\/pre>\n<p>The colon in this instruction (<code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">%fs:0xfffffffffffffff0<\/code>) stands for segmentation addressing (you can read more on it in <a href=\"https:\/\/thestarman.pcministry.com\/asm\/debug\/Segments.html\" target=\"_blank\" rel=\"noopener noreferrer\">this tutorial<\/a>).<\/p>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Returning_to_the_starting_sequence\"><\/span>Returning to the starting sequence<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Finally, let\u2019s look at the last two parts of the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.rt0_go<\/code> function.<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nok:\r\n\t\/\/ set the per-goroutine and per-mach &quot;registers&quot;\r\n\tget_tls(BX)\r\n\tLEAQ\truntime\u00b7g0(SB), CX\r\n\tMOVQ\tCX, g(BX)\r\n\tLEAQ\truntime\u00b7m0(SB), AX\r\n\r\n\t\/\/ save m-&gt;g0 = g0\r\n\tMOVQ\tCX, m_g0(AX)\r\n\t\/\/ save m0 to g0-&gt;m\r\n\tMOVQ\tAX, g_m(CX)\r\n<\/pre>\n<p>Here, we load the TLS address into the BX register and save the address of the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime\u00b7g0<\/code> variable in TLS. We also initialize the <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.m0<\/code> variable. If <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.g0<\/code> stands for root <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">goroutine<\/code>, then <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.m0<\/code> corresponds to the root operating system thread used to run this <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">goroutine<\/code>. We may take a closer look at <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.g0<\/code> and <code style=\"color: #222222; background-color: #e6e6e6; padding: 1px 2px;\">runtime.m0<\/code> structures in the upcoming blog posts.<\/p>\n<p>The final part of the starting sequence initializes arguments and calls different functions, but this is a topic for a separate discussion. So, we have learned the inner mechanisms of the bootstrap process and found out how stacks are implemented. To move forward, we needs to analyze the last part of the starting sequence, which will be the subject of our next blog post.<\/p>\n<p>&nbsp;<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Further_reading\"><\/span>Further reading<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<ul>\n<li><a href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-1-main-concepts-and-project-structure\/\">Golang Internals, Part 1: Main Concepts and Project Structure<\/a><\/li>\n<li><a href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-2-diving-into-the-go-compiler\/\">Golang Internals, Part 2: Diving Into the Go Compiler<\/a><\/li>\n<li><a href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-3-the-linker-object-files-and-relocations\/\">Golang Internals, Part 3: The Linker, Object Files, and Relocations<\/a><\/li>\n<li><a href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-4-object-files-and-function-metadata\/\">Golang Internals, Part 4: Object Files and Function Metadata<\/a><\/li>\n<li><a href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-6-bootstrapping-and-memory-allocator-initialization\/\">Golang Internals, Part 6: Bootstrapping and Memory Allocator Initialization<\/a><\/li>\n<\/ul>\n<hr \/>\n<p><center><small>This post was written by <b>Siarhei Matsiukevich<\/b> and edited by <a href=\"https:\/\/www.altoros.com\/blog\/author\/sophie.turol\/\">Sophia Turol<\/a> and <a href=\"https:\/\/www.altoros.com\/blog\/author\/alex\/\">Alex Khizhniak<\/a>.<\/small><\/center><\/p>\n","protected":false},"excerpt":{"rendered":"<p>All parts: Part 1 | Part 2 | Part 3 | Part 4 | Part 5 | Part 6<\/p>\n<p>&nbsp;\n<\/p>\n<p>The bootstrapping process is the key to understanding how the Go runtime works. Learning it is essential, if you want to move forward with Go. So, the fifth installment in our Golang [&#8230;]<\/p>\n","protected":false},"author":94,"featured_media":47385,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":"","_links_to":"","_links_to_target":""},"categories":[214],"tags":[895],"class_list":["post-47380","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tutorials","tag-research-and-development"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.6 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Golang Internals, Part 5: the Runtime Bootstrap Process | Altoros<\/title>\n<meta name=\"description\" content=\"This blog post explores the specifics of resizable stacks and internal TLS implementation.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Golang Internals, Part 5: the Runtime Bootstrap Process | Altoros\" \/>\n<meta property=\"og:description\" content=\"All parts: Part 1 | Part 2 | Part 3 | Part 4 | Part 5 | Part 6 &nbsp; The bootstrapping process is the key to understanding how the Go runtime works. Learning it is essential, if you want to move forward with Go. So, the fifth installment in our Golang [...]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/\" \/>\n<meta property=\"og:site_name\" content=\"Altoros\" \/>\n<meta property=\"article:published_time\" content=\"2015-04-02T16:35:11+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-11-27T15:09:08+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2015\/04\/golang-internals-part-5-runtime-bootstrap-process.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"768\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Siarhei Matsiukevich\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Siarhei Matsiukevich\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/\",\"url\":\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/\",\"name\":\"Golang Internals, Part 5: the Runtime Bootstrap Process | Altoros\",\"isPartOf\":{\"@id\":\"https:\/\/www.altoros.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2015\/04\/golang-internals-part-5-runtime-bootstrap-process.png\",\"datePublished\":\"2015-04-02T16:35:11+00:00\",\"dateModified\":\"2019-11-27T15:09:08+00:00\",\"author\":{\"@id\":\"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/5c29ff93db657e3cf6552d5e642003d9\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#primaryimage\",\"url\":\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2015\/04\/golang-internals-part-5-runtime-bootstrap-process.png\",\"contentUrl\":\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2015\/04\/golang-internals-part-5-runtime-bootstrap-process.png\",\"width\":1024,\"height\":768},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.altoros.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Golang Internals, Part 5: the Runtime Bootstrap Process\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.altoros.com\/blog\/#website\",\"url\":\"https:\/\/www.altoros.com\/blog\/\",\"name\":\"Altoros\",\"description\":\"Insight\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.altoros.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/5c29ff93db657e3cf6552d5e642003d9\",\"name\":\"Siarhei Matsiukevich\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2016\/06\/Sergey-Matyukevich-150x150.jpg\",\"contentUrl\":\"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2016\/06\/Sergey-Matyukevich-150x150.jpg\",\"caption\":\"Siarhei Matsiukevich\"},\"description\":\"Siarhei Matsiukevich is a Cloud Engineer and Go Developer at Altoros. With 6+ years in software engineering, he is an expert in cloud automation and designing architectures for complex cloud-based systems. An active member of the Go community, Siarhei is a frequent contributor to open-source projects, such as Ubuntu and Juju Charms.\",\"url\":\"https:\/\/www.altoros.com\/blog\/author\/siarhei-matsiukevich\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Golang Internals, Part 5: the Runtime Bootstrap Process | Altoros","description":"This blog post explores the specifics of resizable stacks and internal TLS implementation.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/","og_locale":"en_US","og_type":"article","og_title":"Golang Internals, Part 5: the Runtime Bootstrap Process | Altoros","og_description":"All parts: Part 1 | Part 2 | Part 3 | Part 4 | Part 5 | Part 6 &nbsp; The bootstrapping process is the key to understanding how the Go runtime works. Learning it is essential, if you want to move forward with Go. So, the fifth installment in our Golang [...]","og_url":"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/","og_site_name":"Altoros","article_published_time":"2015-04-02T16:35:11+00:00","article_modified_time":"2019-11-27T15:09:08+00:00","og_image":[{"width":1024,"height":768,"url":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2015\/04\/golang-internals-part-5-runtime-bootstrap-process.png","type":"image\/png"}],"author":"Siarhei Matsiukevich","twitter_misc":{"Written by":"Siarhei Matsiukevich","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/","url":"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/","name":"Golang Internals, Part 5: the Runtime Bootstrap Process | Altoros","isPartOf":{"@id":"https:\/\/www.altoros.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#primaryimage"},"image":{"@id":"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#primaryimage"},"thumbnailUrl":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2015\/04\/golang-internals-part-5-runtime-bootstrap-process.png","datePublished":"2015-04-02T16:35:11+00:00","dateModified":"2019-11-27T15:09:08+00:00","author":{"@id":"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/5c29ff93db657e3cf6552d5e642003d9"},"breadcrumb":{"@id":"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#primaryimage","url":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2015\/04\/golang-internals-part-5-runtime-bootstrap-process.png","contentUrl":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2015\/04\/golang-internals-part-5-runtime-bootstrap-process.png","width":1024,"height":768},{"@type":"BreadcrumbList","@id":"https:\/\/www.altoros.com\/blog\/golang-internals-part-5-the-runtime-bootstrap-process\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.altoros.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Golang Internals, Part 5: the Runtime Bootstrap Process"}]},{"@type":"WebSite","@id":"https:\/\/www.altoros.com\/blog\/#website","url":"https:\/\/www.altoros.com\/blog\/","name":"Altoros","description":"Insight","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.altoros.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/5c29ff93db657e3cf6552d5e642003d9","name":"Siarhei Matsiukevich","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.altoros.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2016\/06\/Sergey-Matyukevich-150x150.jpg","contentUrl":"https:\/\/www.altoros.com\/blog\/wp-content\/uploads\/2016\/06\/Sergey-Matyukevich-150x150.jpg","caption":"Siarhei Matsiukevich"},"description":"Siarhei Matsiukevich is a Cloud Engineer and Go Developer at Altoros. With 6+ years in software engineering, he is an expert in cloud automation and designing architectures for complex cloud-based systems. An active member of the Go community, Siarhei is a frequent contributor to open-source projects, such as Ubuntu and Juju Charms.","url":"https:\/\/www.altoros.com\/blog\/author\/siarhei-matsiukevich\/"}]}},"_links":{"self":[{"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/posts\/47380","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/users\/94"}],"replies":[{"embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/comments?post=47380"}],"version-history":[{"count":7,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/posts\/47380\/revisions"}],"predecessor-version":[{"id":47848,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/posts\/47380\/revisions\/47848"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/media\/47385"}],"wp:attachment":[{"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/media?parent=47380"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/categories?post=47380"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.altoros.com\/blog\/wp-json\/wp\/v2\/tags?post=47380"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}