Rev 33 | Rev 42 | Go to most recent revision | Only display areas with differences | Ignore whitespace | Details | Blame | Last modification | View Log | RSS feed
Rev 33 | Rev 36 | ||
---|---|---|---|
1 | \chapter{Software} |
1 | \chapter{Software} |
2 | \label{tools} |
2 | \label{tools} |
3 | 3 | ||
4 | During the development of the HelenOS operating system, we came across |
4 | During the development of the HelenOS operating system, we came across |
5 | several types of software tools, programs, utilities and libraries. |
5 | several types of software tools, programs, utilities and libraries. |
6 | Some of the tools were used to develop the system itself while other tools |
6 | Some of the tools were used to develop the system itself while other tools |
7 | were used to faciliate the development process. In some cases, we had a chance |
7 | were used to faciliate the development process. In some cases, we had a chance |
8 | to try out several versions of the same product. Sometimes the new versions |
8 | to try out several versions of the same product. Sometimes the new versions |
9 | contained fixes for bugs we had discovered in previous versions thereof. |
9 | contained fixes for bugs we had discovered in previous versions thereof. |
10 | 10 | ||
11 | Another group of software we have used has been integrated into HelenOS |
11 | Another group of software we have used has been integrated into HelenOS |
12 | to fill gaps after functionality that the genuine HelenOS code did |
12 | to fill gaps after functionality that the genuine HelenOS code did |
13 | not provide itself. |
13 | not provide itself. |
14 | 14 | ||
15 | There is simply too much third party software that is somehow related to |
15 | There is simply too much third party software that is somehow related to |
16 | HelenOS to be covered all. This chapter attempts to present our experience |
16 | HelenOS to be covered all. This chapter attempts to present our experience |
17 | with the key softare tools, programs and libraries. |
17 | with the key softare tools, programs and libraries. |
18 | 18 | ||
19 | \section{Communication tools} |
19 | \section{Communication tools} |
20 | Although the developers know each other in person, the development, with the |
20 | Although the developers know each other in person, the development, with the |
21 | exception of kernel camps, has been pretty much independent as far as locality |
21 | exception of kernel camps, has been pretty much independent as far as locality |
22 | and time goes. In order to work effectively, we have established several communication |
22 | and time goes. In order to work effectively, we have established several communication |
23 | channels: |
23 | channels: |
24 | 24 | ||
25 | \begin{description} |
25 | \begin{description} |
26 | \item [E-mail] --- We used this basic means of electronic communication for peer-to-peer |
26 | \item [E-mail] --- We used this basic means of electronic communication for peer-to-peer |
27 | discussion in cases when the other person could not have been reached on-line at |
27 | discussion in cases when the other person could not have been reached on-line at |
28 | the time his advice was needed or his attention was demanded. E-mail was also |
28 | the time his advice was needed or his attention was demanded. E-mail was also |
29 | used for contacting developers of third party software that we needed to talk to. |
29 | used for contacting developers of third party software that we needed to talk to. |
30 | 30 | ||
31 | \item [Mailing list] --- As almost every open source project before us, also we opened |
31 | \item [Mailing list] --- As almost every open source project before us, also we opened |
32 | mailing list for technical discussion. The advantage of having a mailing list is |
32 | mailing list for technical discussion. The advantage of having a mailing list is |
33 | the fact that it enables multilateral discussions on several topics contemporarily, |
33 | the fact that it enables multilateral discussions on several topics contemporarily, |
34 | without the need for all the participants be on-line or even at one place. We have kept |
34 | without the need for all the participants be on-line or even at one place. We have kept |
35 | our first development mailing list closed to public so that it seemed natural to us |
35 | our first development mailing list closed to public so that it seemed natural to us |
36 | to use Czech as our communication language on the list since Czech, with one exception, |
36 | to use Czech as our communication language on the list since Czech, with one exception, |
37 | is our native language and all of us speak it very well. Besides all the advantages, |
37 | is our native language and all of us speak it very well. Besides all the advantages, |
38 | there are also disadvantages. First, communication over mailing list tends to be rather |
38 | there are also disadvantages. First, communication over mailing list tends to be rather |
39 | slow, compared for instance to ICQ. Second, because of its implicit collective nature, |
39 | slow, compared for instance to ICQ. Second, because of its implicit collective nature, |
40 | it sometimes tends to be so slow that an answer for a given question never comes. |
40 | it sometimes tends to be so slow that an answer for a given question never comes. |
41 | 41 | ||
42 | Apart from the internal development mailing list, we have also used another mailing list |
42 | Apart from the internal development mailing list, we have also used another mailing list |
43 | for commit log messages which proved handy in keeping developers informed about all changes in |
43 | for commit log messages which proved handy in keeping developers informed about all changes in |
44 | the repository. |
44 | the repository. |
45 | 45 | ||
46 | Finally, we have also established a public mailing list for communication |
46 | Finally, we have also established a public mailing list for communication |
47 | about general HelenOS topics in English. |
47 | about general HelenOS topics in English. |
48 | 48 | ||
49 | \item [ICQ] --- Because we divided the whole project into smaller subprojects on which |
49 | \item [ICQ] --- Because we divided the whole project into smaller subprojects on which |
50 | only the maximum of two people out of six would work together, the need for communication |
50 | only the maximum of two people out of six would work together, the need for communication |
51 | among all six people was significantly smaller than the need to communicate between the two |
51 | among all six people was significantly smaller than the need to communicate between the two |
52 | developers who tightly cooperated on a specific task. For this reason, we made the biggest |
52 | developers who tightly cooperated on a specific task. For this reason, we made the biggest |
53 | use of ICQ. |
53 | use of ICQ. |
54 | \end{description} |
54 | \end{description} |
55 | 55 | ||
56 | \section{Concurrent versions systems} |
56 | \section{Concurrent versions systems} |
57 | At the very beginning, when the SPARTAN kernel was being developed solely |
57 | At the very beginning, when the SPARTAN kernel was being developed solely |
58 | by \JJ, there was not much sence in using any software for management of |
58 | by \JJ, there was not much sence in using any software for management of |
59 | concurrent versions. However, when the number of developers increased to six, |
59 | concurrent versions. However, when the number of developers increased to six, |
60 | we immediately started to think of available solutions. |
60 | we immediately started to think of available solutions. |
61 | 61 | ||
62 | We have begun with CVS because it is probably the best known file concurrent |
62 | We have begun with CVS because it is probably the best known file concurrent |
63 | versions system. We have even had repository of HelenOS using CVS for a short time, |
63 | versions system. We have even had repository of HelenOS using CVS for a short time, |
64 | but when we learned about its weaknesses we sought another solution. There are two |
64 | but when we learned about its weaknesses we sought another solution. There are two |
65 | weaknesses that have prevented us from using CVS: |
65 | weaknesses that have prevented us from using CVS: |
66 | 66 | ||
67 | \begin{itemize} |
67 | \begin{itemize} |
68 | \item it is merely a file concurrent versions system (i.e. CVS is |
68 | \item it is merely a file concurrent versions system (i.e. CVS is |
69 | good at managing versions of each separate file in the repository |
69 | good at managing versions of each separate file in the repository |
70 | but has no clue about the project's directory tree as a whole; |
70 | but has no clue about the project's directory tree as a whole; |
71 | specifically renaming of a file while preserving its revision history |
71 | specifically renaming of a file while preserving its revision history |
72 | is next to impossible), |
72 | is next to impossible), |
73 | 73 | ||
74 | \item it lacks atomic commits (i.e. should your commit conflict with |
74 | \item it lacks atomic commits (i.e. should your commit conflict with |
75 | another recent commit of another developer, CVS would not abort the whole operation |
75 | another recent commit of another developer, CVS would not abort the whole operation |
76 | but render the repository inconsistent instead). |
76 | but render the repository inconsistent instead). |
77 | \end{itemize} |
77 | \end{itemize} |
78 | 78 | ||
79 | Being aware of these limitations, we decided to go with Subversion. Subversion |
79 | Being aware of these limitations, we decided to go with Subversion. Subversion |
80 | is, simply put, a redesigned CVS with all the limitations fixed. We were |
80 | is, simply put, a redesigned CVS with all the limitations fixed. We were |
81 | already familiar with CVS so the switch to Subversion was pretty seamless. |
81 | already familiar with CVS so the switch to Subversion was pretty seamless. |
82 | 82 | ||
83 | As for Subversion itself, it has worked for us well and has met all our |
83 | As for Subversion itself, it has worked for us well and has met all our |
84 | expectations. Despite all its pros, there was a serious problem that |
84 | expectations. Despite all its pros, there was a serious problem that |
85 | occurred sometime in the middle of the development process. Because of some locking |
85 | occurred sometime in the middle of the development process. Because of some locking |
86 | issues related to the default database backend (i.e. {\tt Berkeley DB}), |
86 | issues related to the default database backend (i.e. {\tt Berkeley DB}), |
87 | our Subversion repository put itself in a peculiar state in which it became |
87 | our Subversion repository put itself in a peculiar state in which it became |
88 | effectivelly inaccessible by any means of standard usage or administration. |
88 | effectivelly inaccessible by any means of standard usage or administration. |
89 | To mitigate this problem, we had to manually delete orphaned file locks |
89 | To mitigate this problem, we had to manually delete orphaned file locks |
90 | and switch to backend called {\tt fsfs} which doesn't suffer this |
90 | and switch to backend called {\tt fsfs} which doesn't suffer this |
91 | problem. |
91 | problem. |
92 | 92 | ||
93 | Other than that, we are happy users of Subversion. The ability to switch |
93 | Other than that, we are happy users of Subversion. The ability to switch |
94 | the entire working copy to particular revision is a great feature |
94 | the entire working copy to particular revision is a great feature |
95 | for debugging. Once we tracked a bug three months into the past by |
95 | for debugging. Once we tracked a bug three months into the past by |
96 | moving through revisions until we found the change that caused the bug. |
96 | moving through revisions until we found the change that caused the bug. |
97 | 97 | ||
98 | \section{Web tools} |
98 | \section{Web tools} |
99 | On our project website\cite{helenos}, we provided links to different |
99 | On our project website\cite{helenos}, we provided links to different |
100 | web utilities that either functioned to access our Subversion repository |
100 | web utilities that either functioned to access our Subversion repository |
101 | or mailing list or provided another services: |
101 | or mailing list or provided another services: |
102 | 102 | ||
103 | \begin{description} |
103 | \begin{description} |
104 | \item [Chora] is a part of the Horde framework and can be used to comfortably |
104 | \item [Chora] is a part of the Horde framework and can be used to comfortably |
105 | browse Subversion repository from the web. We altered it a little bit to also |
105 | browse Subversion repository from the web. We altered it a little bit to also |
106 | show number of commits per developer on our homepage. |
106 | show number of commits per developer on our homepage. |
107 | 107 | ||
108 | \item [Whups] is another component of the Horde framework. It provides |
108 | \item [Whups] is another component of the Horde framework. It provides |
109 | feature request and bug tracking features. However, in the light of being rather |
109 | feature request and bug tracking features. However, in the light of being rather |
110 | closed group of people, we used this tool only seldomly. On the other hand, |
110 | closed group of people, we used this tool only seldomly. On the other hand, |
111 | any possible beta tester of our operating system has had a chance to |
111 | any possible beta tester of our operating system has had a chance to |
112 | submit bug reports. |
112 | submit bug reports. |
113 | 113 | ||
114 | \item [Mailman] is a web interface to the mailing list we utilized. It allows |
114 | \item [Mailman] is a web interface to the mailing list we utilized. It allows |
115 | to control subsriptions and search mailing list archives on-line. |
115 | to control subsriptions and search mailing list archives on-line. |
116 | \end{description} |
116 | \end{description} |
117 | 117 | ||
118 | \section{Third party components of HelenOS} |
118 | \section{Third party components of HelenOS} |
119 | HelenOS itself contains third party software. In the first place, amd64 and ia32 architectures |
119 | HelenOS itself contains third party software. In the first place, amd64 and ia32 architectures |
120 | make use of GNU Grub boot loader. This software replaced the original limited boot loader |
120 | make use of GNU Grub boot loader. This software replaced the original limited boot loader |
121 | after the Kernel Camp 2005 when {\MD} had made HelenOS Multiboot specification compliant. Because of |
121 | after the Kernel Camp 2005 when {\MD} had made HelenOS Multiboot specification compliant. Because of |
122 | Grub, HelenOS can be booted from several types of devices. More importantly, we use |
122 | Grub, HelenOS can be booted from several types of devices. More importantly, we use |
123 | Grub to load HelenOS userspace modules as well. |
123 | Grub to load HelenOS userspace modules as well. |
124 | 124 | ||
125 | Another third-party piece of the HelenOS operating system is the userspace {\tt malloc()}. |
125 | Another third-party piece of the HelenOS operating system is the userspace {\tt malloc()}. |
126 | Rather than porting our kernel slab allocator to userspace, we have chosen Doug Lea's public |
126 | Rather than porting our kernel slab allocator to userspace, we have chosen Doug Lea's public |
127 | domain {\tt dlmalloc} instead. This allocator could be easily integrated into our uspace tree |
127 | domain {\tt dlmalloc} instead. This allocator could be easily integrated into our uspace tree |
128 | and has proven itself in other projects as well. Its derivative, {\tt ptmalloc}, has been part of the |
128 | and has proven itself in other projects as well. Its derivative, {\tt ptmalloc}, has been part of the |
129 | GNU C library for some time. However, the version we are using is not optimized for SMP and multithreading. |
129 | GNU C library for some time. However, the version we are using is not optimized for SMP and multithreading. |
130 | We plan to eventually replace it with another allocator. |
130 | We plan to eventually replace it with another allocator. |
131 | 131 | ||
- | 132 | Finally, the {\tt pci} userspace task is using the {\tt libpci} library. The |
|
- | 133 | library was simplified and ported to HelenOS. Even though filesystem |
|
- | 134 | calls were removed from the library, it still heavily depends on {\tt libc}. |
|
- | 135 | By porting {\tt libpci} to HelenOS, we demonstrated that applications and libraries |
|
- | 136 | are, given enough effort, portable to HelenOS. |
|
- | 137 | ||
132 | \section{Build tools} |
138 | \section{Build tools} |
133 | Assembler, linker and compiler are by all means the very focal point of attention |
139 | Assembler, linker and compiler are by all means the very focal point of attention |
134 | of all operating system projects. Quality of these tools influences |
140 | of all operating system projects. Quality of these tools influences |
135 | operating system performance and, what is more important, stability. HelenOS has |
141 | operating system performance and, what is more important, stability. HelenOS has |
136 | been tailored to build with GNU {\tt binutils} (i.e. the assembler and linker) and GNU~{\tt gcc} |
142 | been tailored to build with GNU {\tt binutils} (i.e. the assembler and linker) and GNU~{\tt gcc} |
137 | (i.e. the compiler). There is only little chance that it could be compiled and |
143 | (i.e. the compiler). There is only little chance that it could be compiled and |
138 | linked using some other tools unless those tools are compatible with the GNU build tools. |
144 | linked using some other tools unless those tools are compatible with the GNU build tools. |
139 | 145 | ||
140 | As our project declares support for five different processor architectures, |
146 | As our project declares support for five different processor architectures, |
141 | we needed to have five different flavors of the build utilities installed. |
147 | we needed to have five different flavors of the build utilities installed. |
142 | Interestingly, flavors of {\tt binutils} and {\tt gcc} for particular architecture |
148 | Interestingly, flavors of {\tt binutils} and {\tt gcc} for particular architecture |
143 | are not equal from the point of view of cross-binutils and cross-compiler installation. |
149 | are not equal from the point of view of cross-binutils and cross-compiler installation. |
144 | All platforms except ia64 require only the {\tt binutils} package and the {\tt gcc} package |
150 | All platforms except ia64 require only the {\tt binutils} package and the {\tt gcc} package |
145 | for the cross-tool to be built. On the other hand, ia64 requires also some excerpts from |
151 | for the cross-tool to be built. On the other hand, ia64 requires also some excerpts from |
146 | the ia64-specific part of {\tt glibc}. |
152 | the ia64-specific part of {\tt glibc}. |
147 | 153 | ||
148 | Formerly, the project could be compiled with almost any version of {\tt binutils} starting with 2.15 |
154 | Formerly, the project could be compiled with almost any version of {\tt binutils} starting with 2.15 |
149 | and {\tt gcc} starting with 2.95, but especially after we added partial thread local storage |
155 | and {\tt gcc} starting with 2.95, but especially after we added partial thread local storage |
150 | support into our userspace layer, some architectures (e.g. mips32) will not compile even with {\tt gcc} 4.0.1 |
156 | support into our userspace layer, some architectures (e.g. mips32) will not compile even with {\tt gcc} 4.0.1 |
151 | and demand {\tt gcc} 4.1.0. Curiously, ia64 will not link when compiled with {\tt gcc} 4.1.0. |
157 | and demand {\tt gcc} 4.1.0. Curiously, ia64 will not link when compiled with {\tt gcc} 4.1.0. |
152 | 158 | ||
153 | As for the mips32 cross-compiler, {\OP} discovered a bug in {\tt gcc} (ticket \#23824) which caused {\tt gcc} to |
159 | As for the mips32 cross-compiler, {\OP} discovered a bug in {\tt gcc} (ticket \#23824) which caused {\tt gcc} to |
154 | incorrectly generate unaligned data access instructions (i.e. {\tt lwl}, {\tt lwr}, {\tt swl} and {\tt swr}). |
160 | incorrectly generate unaligned data access instructions (i.e. {\tt lwl}, {\tt lwr}, {\tt swl} and {\tt swr}). |
155 | 161 | ||
156 | As for the mips32 cross-binutils\footnote{It remains uninvestigated whether this problem also shows with other cross-tools.}, |
162 | As for the mips32 cross-binutils\footnote{It remains uninvestigated whether this problem also shows with other cross-tools.}, |
157 | we observed that undefined symbols are not reported when we don't link using the standard target. We are still not |
163 | we observed that undefined symbols are not reported when we don't link using the standard target. We are still not |
158 | sure whether this was a bug --- {\tt binutils} developers just told us to use the standard target and then use |
164 | sure whether this was a bug --- {\tt binutils} developers just told us to use the standard target and then use |
159 | {\tt objcopy} to convert the ELF binary into requested output format. |
165 | {\tt objcopy} to convert the ELF binary into requested output format. |
160 | 166 | ||
161 | \section{Virtual environments} |
167 | \section{Virtual environments} |
162 | After the build tools, simulators, emulators and virtualizers were the second focal point |
168 | After the build tools, simulators, emulators and virtualizers were the second focal point |
163 | in our project. These invaluable programs really sped the code-compile-test cycle. |
169 | in our project. These invaluable programs really sped the code-compile-test cycle. |
164 | In some cases, they were, and still are, the only option to actually run HelenOS on certain |
170 | In some cases, they were, and still are, the only option to actually run HelenOS on certain |
165 | processor architectures, because real hardware was not available to us. Using virtual environment |
171 | processor architectures, because real hardware was not available to us. Using virtual environment |
166 | for developing our system provided us with deterministic environment on which it is much easier to do |
172 | for developing our system provided us with deterministic environment on which it is much easier to do |
167 | troubleshooting. Moreover, part of the simulators featured integrated debugging facilities. |
173 | troubleshooting. Moreover, part of the simulators featured integrated debugging facilities. |
168 | Without them, a lot of bugs would remain unresolved or even go unnoticed. |
174 | Without them, a lot of bugs would remain unresolved or even go unnoticed. |
169 | 175 | ||
170 | From one point of view, we have tested our system on eight different virtual environments: |
176 | From one point of view, we have tested our system on eight different virtual environments: |
171 | 177 | ||
172 | \begin{itemize} |
178 | \begin{itemize} |
173 | \item Bochs, |
179 | \item Bochs, |
174 | \item GXemul, |
180 | \item GXemul, |
175 | \item msim, |
181 | \item msim, |
176 | \item PearPC, |
182 | \item PearPC, |
177 | \item QEMU, |
183 | \item QEMU, |
178 | \item Simics, |
184 | \item Simics, |
179 | \item Ski, |
185 | \item Ski, |
180 | \item VMware. |
186 | \item VMware. |
181 | \end{itemize} |
187 | \end{itemize} |
182 | 188 | ||
183 | From the second point of view, we have tested these programs by our operating system. |
189 | From the second point of view, we have tested these programs by our operating system. |
184 | Because of the scope and uniqueness of this testing and because we did find some issues, |
190 | Because of the scope and uniqueness of this testing and because we did find some issues, |
185 | we want to dedicate some more space to what we have found. |
191 | we want to dedicate some more space to what we have found. |
186 | 192 | ||
187 | \subsection{Bochs} |
193 | \subsection{Bochs} |
188 | Bochs has been used to develop the SPARTAN kernel since its beginning in 2001. |
194 | Bochs has been used to develop the SPARTAN kernel since its beginning in 2001. |
189 | It is capable of emulating ia32 machine and for some time also amd64. |
195 | It is capable of emulating ia32 machine and for some time also amd64. |
190 | Bochs is an emulator and thus the slowest from virtual environments capable |
196 | Bochs is an emulator and thus the slowest from virtual environments capable |
191 | of simulating the same cathegory of hardware. On the other hand, it is extremely |
197 | of simulating the same cathegory of hardware. On the other hand, it is extremely |
192 | portable, compared to much faster virtualizers and emulators using dynamic translation |
198 | portable, compared to much faster virtualizers and emulators using dynamic translation |
193 | of instructions. Lately, there have been some plans to develop or port dynamic translation |
199 | of instructions. Lately, there have been some plans to develop or port dynamic translation |
194 | to Bochs brewing in its developer community. |
200 | to Bochs brewing in its developer community. |
195 | 201 | ||
196 | The biggest virtue of Bochs is that it has traditionally supported SMP. For some time, Bochs |
202 | The biggest virtue of Bochs is that it has traditionally supported SMP. For some time, Bochs |
197 | has been our only environment on which we could develop and test SMP code. Unfortunatelly, |
203 | has been our only environment on which we could develop and test SMP code. Unfortunatelly, |
198 | the quality of SMP support in Bochs was different from version to version. Because of SMP |
204 | the quality of SMP support in Bochs was different from version to version. Because of SMP |
199 | breakage in Bochs, we had to avoid some versions thereof. So far, Bochs versions 2.2.1 and 2.2.6 |
205 | breakage in Bochs, we had to avoid some versions thereof. So far, Bochs versions 2.2.1 and 2.2.6 |
200 | have been best in this regard. |
206 | have been best in this regard. |
201 | 207 | ||
202 | Our project has not only used Bochs. We also helped to identify some SMP related problems |
208 | Our project has not only used Bochs. We also helped to identify some SMP related problems |
203 | and {\OP} from our team has discovered and also fixed a bug in FXSAVE and FXRSTOR emulation |
209 | and {\OP} from our team has discovered and also fixed a bug in FXSAVE and FXRSTOR emulation |
204 | (patch \#1282033). |
210 | (patch \#1282033). |
205 | 211 | ||
206 | Bochs has some debugging facilities but those have been very impractical and broken |
212 | Bochs has some debugging facilities but those have been very impractical and broken |
207 | in SMP mode. |
213 | in SMP mode. |
208 | 214 | ||
209 | \subsection{GXemul} |
215 | \subsection{GXemul} |
210 | GXemul is an emulator of several processor architectures. Nevertheless, we have |
216 | GXemul is an emulator of several processor architectures. Nevertheless, we have |
211 | used it only for mips32 emulation in both little-endian and big-endian modes. |
217 | used it only for mips32 emulation in both little-endian and big-endian modes. |
212 | It seems to be pretty featurefull and evolving but we don't use all its functionality. |
218 | It seems to be pretty featurefull and evolving but we don't use all its functionality. |
213 | GXemul is very user friendly and has debugging features. It is more realistic |
219 | GXemul is very user friendly and has debugging features. It is more realistic |
214 | than msim. However, our newly introduced TLS support triggered a bug in the {\tt rdhwr} |
220 | than msim. However, our newly introduced TLS support triggered a bug in the {\tt rdhwr} |
215 | instruction emulation while msim functioned as expected. Fortunatelly, the author |
221 | instruction emulation while msim functioned as expected. Fortunatelly, the author |
216 | of GXemul is very cooperative and has fixed the problem for future versions as well as |
222 | of GXemul is very cooperative and has fixed the problem for future versions as well as |
217 | provided a quick hack for the old version. |
223 | provided a quick hack for the old version. |
218 | 224 | ||
219 | \subsection{msim} |
225 | \subsection{msim} |
220 | msim has been our first mips32 simulator. It simulates 32-bit side of R4000 processor. |
226 | msim has been our first mips32 simulator. It simulates 32-bit side of R4000 processor. |
221 | Its simulated environment is not very realistic, but the processor simulation |
227 | Its simulated environment is not very realistic, but the processor simulation |
222 | is good enough for operating system development. In this regard, the simulator is |
228 | is good enough for operating system development. In this regard, the simulator is |
223 | comparable to HP's ia64 simulator Ski. Another similar aspect of these two is |
229 | comparable to HP's ia64 simulator Ski. Another similar aspect of these two is |
224 | relatively strong debugger. |
230 | relatively strong debugger. |
225 | 231 | ||
226 | Msim has been developed on the same alma mater as our own project. |
232 | Msim has been developed on the same alma mater as our own project. |
227 | All members of our team know this program from operating system courses. |
233 | All members of our team know this program from operating system courses. |
228 | Curiously, this simulator contained the biggest number of defects and inaccuracies |
234 | Curiously, this simulator contained the biggest number of defects and inaccuracies |
229 | that we have ever discovered in a simulator. Fortunately, all of them have been |
235 | that we have ever discovered in a simulator. Fortunately, all of them have been |
230 | eventually fixed. |
236 | eventually fixed. |
231 | 237 | ||
232 | \subsection{PearPC} |
238 | \subsection{PearPC} |
233 | PearPC is the only emulator on which we have run ppc32 port of HelenOS. It has |
239 | PearPC is the only emulator on which we have run ppc32 port of HelenOS. It has |
234 | no debugging features, but fortunatelly its sources are available under |
240 | no debugging features, but fortunatelly its sources are available under |
235 | an open source license. This enabled {\OP} and {\MD} to alter its sources |
241 | an open source license. This enabled {\OP} and {\MD} to alter its sources |
236 | in a way that this modified version allowed some basic debugging. |
242 | in a way that this modified version allowed some basic debugging. |
237 | 243 | ||
238 | \subsection{QEMU} |
244 | \subsection{QEMU} |
239 | QEMU emulates several processor architectures. We have used it to emulate |
245 | QEMU emulates several processor architectures. We have used it to emulate |
240 | ia32 and amd64. It can simulate SMP, but contrary to Bochs, it uses dynamic |
246 | ia32 and amd64. It can simulate SMP, but contrary to Bochs, it uses dynamic |
241 | translation of emulated instructions and performs much better because of |
247 | translation of emulated instructions and performs much better because of |
242 | that. |
248 | that. |
243 | 249 | ||
244 | This emulator seemed to realistically emulate the {\tt hlt} instruction, |
250 | This emulator seemed to realistically emulate the {\tt hlt} instruction, |
245 | which was nice for those of us who use notebooks as their development |
251 | which was nice for those of us who use notebooks as their development |
246 | machine. |
252 | machine. |
247 | 253 | ||
248 | \subsection{Simics} |
254 | \subsection{Simics} |
249 | Virtutech's Simics simulator can be compared to a Swiss-army knife for operating system debugging. |
255 | Virtutech's Simics simulator can be compared to a Swiss-army knife for operating system debugging. |
250 | This proprietary piece of software was available to us under an academic license for free. |
256 | This proprietary piece of software was available to us under an academic license for free. |
251 | 257 | ||
252 | Simics can be set to simulate many different configurations of many different machines. |
258 | Simics can be set to simulate many different configurations of many different machines. |
253 | It has the most advanced debugging features we have ever seen. To highlight some, its |
259 | It has the most advanced debugging features we have ever seen. To highlight some, its |
254 | memory access tracing ability has been really helpfull to us. During device driver |
260 | memory access tracing ability has been really helpfull to us. During device driver |
255 | development, we appreciated the possibility to turn logging of the devices to a specified |
261 | development, we appreciated the possibility to turn logging of the devices to a specified |
256 | verbosity. |
262 | verbosity. |
257 | 263 | ||
258 | We used it to test and develop amd64 and ia32 architectures in SMP mode and mips32 architecture in UP mode. |
264 | We used it to test and develop amd64 and ia32 architectures in SMP mode and mips32 architecture in UP mode. |
259 | 265 | ||
260 | Regardless of its invaluable qualities, it has still contained bugs. One of the most |
266 | Regardless of its invaluable qualities, it has still contained bugs. One of the most |
261 | serious was bug with ticket \#3351. {\OP} discovered that its BIOS rewrites kernel memory |
267 | serious was bug with ticket \#3351. {\OP} discovered that its BIOS rewrites kernel memory |
262 | during application processors start. Another bugs found were related to amd64 and mips32. |
268 | during application processors start. Another bugs found were related to amd64 and mips32. |
263 | As for amd64, Simics did not report general protection fault when {\tt EFER.NXE} was 0 and a non-executable |
269 | As for amd64, Simics did not report general protection fault when {\tt EFER.NXE} was 0 and a non-executable |
264 | page was found (\#4214). As for mips32, Simics misemulated {\tt MSUB} and {\tt MSUBU} instructions. |
270 | page was found (\#4214). As for mips32, Simics misemulated {\tt MSUB} and {\tt MSUBU} instructions. |
265 | 271 | ||
266 | \subsection{Ski} |
272 | \subsection{Ski} |
267 | The ia64 port of HelenOS has been developed and debugged on the HP's IA-64 Ski simulator. |
273 | The ia64 port of HelenOS has been developed and debugged on the HP's IA-64 Ski simulator. |
268 | Ski is just an Itanium processor simulator and as such does not simulate a real machine. In fact, there |
274 | Ski is just an Itanium processor simulator and as such does not simulate a real machine. In fact, there |
269 | is no firmware and no configuration tables (e.g. memory map) present in Ski! On the other hand, the missing parts can be supplied externally\footnote{This |
275 | is no firmware and no configuration tables (e.g. memory map) present in Ski! On the other hand, the missing parts can be supplied externally\footnote{This |
270 | is actually how Linux runs in this simulator.}. The simulator provides means of interaction with |
276 | is actually how Linux runs in this simulator.}. The simulator provides means of interaction with |
271 | host system devices via Simulator SystemCalls (SSC). The simulator itself has graphical interface |
277 | host system devices via Simulator SystemCalls (SSC). The simulator itself has graphical interface |
272 | with pretty powerful, but not as good as those of Simics, debugging facilities. |
278 | with pretty powerful, but not as good as those of Simics, debugging facilities. |
273 | 279 | ||
274 | Ski is a proprietary program with no source code available. Its binaries are available |
280 | Ski is a proprietary program with no source code available. Its binaries are available |
275 | for free under a non-free license. It comes packaged with insufficient documentation |
281 | for free under a non-free license. It comes packaged with insufficient documentation |
276 | which makes the development pretty problematic. For instance, there is no public documentation |
282 | which makes the development pretty problematic. For instance, there is no public documentation |
277 | of all the SSC's. All one can do is to look into Linux/ia64-Ski port, which was written by the |
283 | of all the SSC's. All one can do is to look into Linux/ia64-Ski port, which was written by the |
278 | same people as Ski, and use it as a refernce. We had to look into Linux once more when our kernel |
284 | same people as Ski, and use it as a refernce. We had to look into Linux once more when our kernel |
279 | started to fail in some memory-intensive stress tests. In fact, the problem was that the tests |
285 | started to fail in some memory-intensive stress tests. In fact, the problem was that the tests |
280 | hit the IA-32 legacy videoram area. We fixed the problem, in the light of absence of any memory map, by blacklisting |
286 | hit the IA-32 legacy videoram area. We fixed the problem, in the light of absence of any memory map, by blacklisting |
281 | this piece of memory to our frame allocator. |
287 | this piece of memory to our frame allocator. |
282 | 288 | ||
283 | The way HelenOS is booted on Ski is by simply loading its ELF image |
289 | The way HelenOS is booted on Ski is by simply loading its ELF image |
284 | and jumping to it. The ELF header contains two fields describing where and how to load the program image into memory: |
290 | and jumping to it. The ELF header contains two fields describing where and how to load the program image into memory: |
285 | VMA and LMA. VMA\footnote{Virtual Memory Address} is an address where the program's segment gets mapped in virtual memory. |
291 | VMA and LMA. VMA\footnote{Virtual Memory Address} is an address where the program's segment gets mapped in virtual memory. |
286 | LMA\footnote{Load Memory Address} is the physical address where the segment is loaded in memory. {\JV} discovered |
292 | LMA\footnote{Load Memory Address} is the physical address where the segment is loaded in memory. {\JV} discovered |
287 | that Ski confuses VMA and LMA. This, what we believe to be a bug in Ski, has not shown in Linux since Linux always has |
293 | that Ski confuses VMA and LMA. This, what we believe to be a bug in Ski, has not shown in Linux since Linux always has |
288 | LMA equal to VMA. People from the Ski mailing list had tried to help us but our repeated problem report didn't |
294 | LMA equal to VMA. People from the Ski mailing list had tried to help us but our repeated problem report didn't |
289 | make it far enough for the HP to fix or at least clarify the issue. Finally, we adopted a workaround implemented by {\JJ} |
295 | make it far enough for the HP to fix or at least clarify the issue. Finally, we adopted a workaround implemented by {\JJ} |
290 | that simply swaps LMA and the program entry point in the kernel ELF image. |
296 | that simply swaps LMA and the program entry point in the kernel ELF image. |
291 | 297 | ||
292 | \subsection{VMware} |
298 | \subsection{VMware} |
293 | VMware is the only virtualizer we have used in HelenOS development. It virtualizes the ia32 host machine. |
299 | VMware is the only virtualizer we have used in HelenOS development. It virtualizes the ia32 host machine. |
294 | Since VMware version 5.5, we made use of its possibility to run the guest system (i.e. HelenOS) on multiple processors. |
300 | Since VMware version 5.5, we made use of its possibility to run the guest system (i.e. HelenOS) on multiple processors. |
295 | VMware has no support for debugging but is very useful for compatibility and regression testing because |
301 | VMware has no support for debugging but is very useful for compatibility and regression testing because |
296 | it's closest to the real hardware. VMware, being a virtualizer, is also the fastest of all the virtual environments |
302 | it's closest to the real hardware. VMware, being a virtualizer, is also the fastest of all the virtual environments |
297 | we have utilized. |
303 | we have utilized. |
298 | 304 | ||
299 | 305 | ||
300 | \section{Authoring tools} |
306 | \section{Authoring tools} |
301 | 307 |