Rev 2714 | Go to most recent revision | Details | Compare with Previous | Last modification | View Log | RSS feed
Rev | Author | Line No. | Line |
---|---|---|---|
2714 | cejka | 1 | # $NetBSD: TOUR,v 1.8 1996/10/16 14:24:56 christos Exp $ |
2 | # @(#)TOUR 8.1 (Berkeley) 5/31/93 |
||
3 | |||
4 | NOTE -- This is the original TOUR paper distributed with ash and |
||
5 | does not represent the current state of the shell. It is provided anyway |
||
6 | since it provides helpful information for how the shell is structured, |
||
7 | but be warned that things have changed -- the current shell is |
||
8 | still under development. |
||
9 | |||
10 | ================================================================ |
||
11 | |||
12 | A Tour through Ash |
||
13 | |||
14 | Copyright 1989 by Kenneth Almquist. |
||
15 | |||
16 | |||
17 | DIRECTORIES: The subdirectory bltin contains commands which can |
||
18 | be compiled stand-alone. The rest of the source is in the main |
||
19 | ash directory. |
||
20 | |||
21 | SOURCE CODE GENERATORS: Files whose names begin with "mk" are |
||
22 | programs that generate source code. A complete list of these |
||
23 | programs is: |
||
24 | |||
25 | program intput files generates |
||
26 | ------- ------------ --------- |
||
27 | mkbuiltins builtins builtins.h builtins.c |
||
28 | mkinit *.c init.c |
||
29 | mknodes nodetypes nodes.h nodes.c |
||
30 | mksignames - signames.h signames.c |
||
31 | mksyntax - syntax.h syntax.c |
||
32 | mktokens - token.h |
||
33 | bltin/mkexpr unary_op binary_op operators.h operators.c |
||
34 | |||
35 | There are undoubtedly too many of these. Mkinit searches all the |
||
36 | C source files for entries looking like: |
||
37 | |||
38 | INIT { |
||
39 | x = 1; /* executed during initialization */ |
||
40 | } |
||
41 | |||
42 | RESET { |
||
43 | x = 2; /* executed when the shell does a longjmp |
||
44 | back to the main command loop */ |
||
45 | } |
||
46 | |||
47 | SHELLPROC { |
||
48 | x = 3; /* executed when the shell runs a shell procedure */ |
||
49 | } |
||
50 | |||
51 | It pulls this code out into routines which are when particular |
||
52 | events occur. The intent is to improve modularity by isolating |
||
53 | the information about which modules need to be explicitly |
||
54 | initialized/reset within the modules themselves. |
||
55 | |||
56 | Mkinit recognizes several constructs for placing declarations in |
||
57 | the init.c file. |
||
58 | INCLUDE "file.h" |
||
59 | includes a file. The storage class MKINIT makes a declaration |
||
60 | available in the init.c file, for example: |
||
61 | MKINIT int funcnest; /* depth of function calls */ |
||
62 | MKINIT alone on a line introduces a structure or union declara- |
||
63 | tion: |
||
64 | MKINIT |
||
65 | struct redirtab { |
||
66 | short renamed[10]; |
||
67 | }; |
||
68 | Preprocessor #define statements are copied to init.c without any |
||
69 | special action to request this. |
||
70 | |||
71 | INDENTATION: The ash source is indented in multiples of six |
||
72 | spaces. The only study that I have heard of on the subject con- |
||
73 | cluded that the optimal amount to indent is in the range of four |
||
74 | to six spaces. I use six spaces since it is not too big a jump |
||
75 | from the widely used eight spaces. If you really hate six space |
||
76 | indentation, use the adjind (source included) program to change |
||
77 | it to something else. |
||
78 | |||
79 | EXCEPTIONS: Code for dealing with exceptions appears in |
||
80 | exceptions.c. The C language doesn't include exception handling, |
||
81 | so I implement it using setjmp and longjmp. The global variable |
||
82 | exception contains the type of exception. EXERROR is raised by |
||
83 | calling error. EXINT is an interrupt. EXSHELLPROC is an excep- |
||
84 | tion which is raised when a shell procedure is invoked. The pur- |
||
85 | pose of EXSHELLPROC is to perform the cleanup actions associated |
||
86 | with other exceptions. After these cleanup actions, the shell |
||
87 | can interpret a shell procedure itself without exec'ing a new |
||
88 | copy of the shell. |
||
89 | |||
90 | INTERRUPTS: In an interactive shell, an interrupt will cause an |
||
91 | EXINT exception to return to the main command loop. (Exception: |
||
92 | EXINT is not raised if the user traps interrupts using the trap |
||
93 | command.) The INTOFF and INTON macros (defined in exception.h) |
||
94 | provide uninterruptable critical sections. Between the execution |
||
95 | of INTOFF and the execution of INTON, interrupt signals will be |
||
96 | held for later delivery. INTOFF and INTON can be nested. |
||
97 | |||
98 | MEMALLOC.C: Memalloc.c defines versions of malloc and realloc |
||
99 | which call error when there is no memory left. It also defines a |
||
100 | stack oriented memory allocation scheme. Allocating off a stack |
||
101 | is probably more efficient than allocation using malloc, but the |
||
102 | big advantage is that when an exception occurs all we have to do |
||
103 | to free up the memory in use at the time of the exception is to |
||
104 | restore the stack pointer. The stack is implemented using a |
||
105 | linked list of blocks. |
||
106 | |||
107 | STPUTC: If the stack were contiguous, it would be easy to store |
||
108 | strings on the stack without knowing in advance how long the |
||
109 | string was going to be: |
||
110 | p = stackptr; |
||
111 | *p++ = c; /* repeated as many times as needed */ |
||
112 | stackptr = p; |
||
113 | The folloing three macros (defined in memalloc.h) perform these |
||
114 | operations, but grow the stack if you run off the end: |
||
115 | STARTSTACKSTR(p); |
||
116 | STPUTC(c, p); /* repeated as many times as needed */ |
||
117 | grabstackstr(p); |
||
118 | |||
119 | We now start a top-down look at the code: |
||
120 | |||
121 | MAIN.C: The main routine performs some initialization, executes |
||
122 | the user's profile if necessary, and calls cmdloop. Cmdloop is |
||
123 | repeatedly parses and executes commands. |
||
124 | |||
125 | OPTIONS.C: This file contains the option processing code. It is |
||
126 | called from main to parse the shell arguments when the shell is |
||
127 | invoked, and it also contains the set builtin. The -i and -j op- |
||
128 | tions (the latter turns on job control) require changes in signal |
||
129 | handling. The routines setjobctl (in jobs.c) and setinteractive |
||
130 | (in trap.c) are called to handle changes to these options. |
||
131 | |||
132 | PARSING: The parser code is all in parser.c. A recursive des- |
||
133 | cent parser is used. Syntax tables (generated by mksyntax) are |
||
134 | used to classify characters during lexical analysis. There are |
||
135 | three tables: one for normal use, one for use when inside single |
||
136 | quotes, and one for use when inside double quotes. The tables |
||
137 | are machine dependent because they are indexed by character vari- |
||
138 | ables and the range of a char varies from machine to machine. |
||
139 | |||
140 | PARSE OUTPUT: The output of the parser consists of a tree of |
||
141 | nodes. The various types of nodes are defined in the file node- |
||
142 | types. |
||
143 | |||
144 | Nodes of type NARG are used to represent both words and the con- |
||
145 | tents of here documents. An early version of ash kept the con- |
||
146 | tents of here documents in temporary files, but keeping here do- |
||
147 | cuments in memory typically results in significantly better per- |
||
148 | formance. It would have been nice to make it an option to use |
||
149 | temporary files for here documents, for the benefit of small |
||
150 | machines, but the code to keep track of when to delete the tem- |
||
151 | porary files was complex and I never fixed all the bugs in it. |
||
152 | (AT&T has been maintaining the Bourne shell for more than ten |
||
153 | years, and to the best of my knowledge they still haven't gotten |
||
154 | it to handle temporary files correctly in obscure cases.) |
||
155 | |||
156 | The text field of a NARG structure points to the text of the |
||
157 | word. The text consists of ordinary characters and a number of |
||
158 | special codes defined in parser.h. The special codes are: |
||
159 | |||
160 | CTLVAR Variable substitution |
||
161 | CTLENDVAR End of variable substitution |
||
162 | CTLBACKQ Command substitution |
||
163 | CTLBACKQ|CTLQUOTE Command substitution inside double quotes |
||
164 | CTLESC Escape next character |
||
165 | |||
166 | A variable substitution contains the following elements: |
||
167 | |||
168 | CTLVAR type name '=' [ alternative-text CTLENDVAR ] |
||
169 | |||
170 | The type field is a single character specifying the type of sub- |
||
171 | stitution. The possible types are: |
||
172 | |||
173 | VSNORMAL $var |
||
174 | VSMINUS ${var-text} |
||
175 | VSMINUS|VSNUL ${var:-text} |
||
176 | VSPLUS ${var+text} |
||
177 | VSPLUS|VSNUL ${var:+text} |
||
178 | VSQUESTION ${var?text} |
||
179 | VSQUESTION|VSNUL ${var:?text} |
||
180 | VSASSIGN ${var=text} |
||
181 | VSASSIGN|VSNUL ${var=text} |
||
182 | |||
183 | In addition, the type field will have the VSQUOTE flag set if the |
||
184 | variable is enclosed in double quotes. The name of the variable |
||
185 | comes next, terminated by an equals sign. If the type is not |
||
186 | VSNORMAL, then the text field in the substitution follows, ter- |
||
187 | minated by a CTLENDVAR byte. |
||
188 | |||
189 | Commands in back quotes are parsed and stored in a linked list. |
||
190 | The locations of these commands in the string are indicated by |
||
191 | CTLBACKQ and CTLBACKQ+CTLQUOTE characters, depending upon whether |
||
192 | the back quotes were enclosed in double quotes. |
||
193 | |||
194 | The character CTLESC escapes the next character, so that in case |
||
195 | any of the CTL characters mentioned above appear in the input, |
||
196 | they can be passed through transparently. CTLESC is also used to |
||
197 | escape '*', '?', '[', and '!' characters which were quoted by the |
||
198 | user and thus should not be used for file name generation. |
||
199 | |||
200 | CTLESC characters have proved to be particularly tricky to get |
||
201 | right. In the case of here documents which are not subject to |
||
202 | variable and command substitution, the parser doesn't insert any |
||
203 | CTLESC characters to begin with (so the contents of the text |
||
204 | field can be written without any processing). Other here docu- |
||
205 | ments, and words which are not subject to splitting and file name |
||
206 | generation, have the CTLESC characters removed during the vari- |
||
207 | able and command substitution phase. Words which are subject |
||
208 | splitting and file name generation have the CTLESC characters re- |
||
209 | moved as part of the file name phase. |
||
210 | |||
211 | EXECUTION: Command execution is handled by the following files: |
||
212 | eval.c The top level routines. |
||
213 | redir.c Code to handle redirection of input and output. |
||
214 | jobs.c Code to handle forking, waiting, and job control. |
||
215 | exec.c Code to to path searches and the actual exec sys call. |
||
216 | expand.c Code to evaluate arguments. |
||
217 | var.c Maintains the variable symbol table. Called from expand.c. |
||
218 | |||
219 | EVAL.C: Evaltree recursively executes a parse tree. The exit |
||
220 | status is returned in the global variable exitstatus. The alter- |
||
221 | native entry evalbackcmd is called to evaluate commands in back |
||
222 | quotes. It saves the result in memory if the command is a buil- |
||
223 | tin; otherwise it forks off a child to execute the command and |
||
224 | connects the standard output of the child to a pipe. |
||
225 | |||
226 | JOBS.C: To create a process, you call makejob to return a job |
||
227 | structure, and then call forkshell (passing the job structure as |
||
228 | an argument) to create the process. Waitforjob waits for a job |
||
229 | to complete. These routines take care of process groups if job |
||
230 | control is defined. |
||
231 | |||
232 | REDIR.C: Ash allows file descriptors to be redirected and then |
||
233 | restored without forking off a child process. This is accom- |
||
234 | plished by duplicating the original file descriptors. The redir- |
||
235 | tab structure records where the file descriptors have be dupli- |
||
236 | cated to. |
||
237 | |||
238 | EXEC.C: The routine find_command locates a command, and enters |
||
239 | the command in the hash table if it is not already there. The |
||
240 | third argument specifies whether it is to print an error message |
||
241 | if the command is not found. (When a pipeline is set up, |
||
242 | find_command is called for all the commands in the pipeline be- |
||
243 | fore any forking is done, so to get the commands into the hash |
||
244 | table of the parent process. But to make command hashing as |
||
245 | transparent as possible, we silently ignore errors at that point |
||
246 | and only print error messages if the command cannot be found |
||
247 | later.) |
||
248 | |||
249 | The routine shellexec is the interface to the exec system call. |
||
250 | |||
251 | EXPAND.C: Arguments are processed in three passes. The first |
||
252 | (performed by the routine argstr) performs variable and command |
||
253 | substitution. The second (ifsbreakup) performs word splitting |
||
254 | and the third (expandmeta) performs file name generation. If the |
||
255 | "/u" directory is simulated, then when "/u/username" is replaced |
||
256 | by the user's home directory, the flag "didudir" is set. This |
||
257 | tells the cd command that it should print out the directory name, |
||
258 | just as it would if the "/u" directory were implemented using |
||
259 | symbolic links. |
||
260 | |||
261 | VAR.C: Variables are stored in a hash table. Probably we should |
||
262 | switch to extensible hashing. The variable name is stored in the |
||
263 | same string as the value (using the format "name=value") so that |
||
264 | no string copying is needed to create the environment of a com- |
||
265 | mand. Variables which the shell references internally are preal- |
||
266 | located so that the shell can reference the values of these vari- |
||
267 | ables without doing a lookup. |
||
268 | |||
269 | When a program is run, the code in eval.c sticks any environment |
||
270 | variables which precede the command (as in "PATH=xxx command") in |
||
271 | the variable table as the simplest way to strip duplicates, and |
||
272 | then calls "environment" to get the value of the environment. |
||
273 | There are two consequences of this. First, if an assignment to |
||
274 | PATH precedes the command, the value of PATH before the assign- |
||
275 | ment must be remembered and passed to shellexec. Second, if the |
||
276 | program turns out to be a shell procedure, the strings from the |
||
277 | environment variables which preceded the command must be pulled |
||
278 | out of the table and replaced with strings obtained from malloc, |
||
279 | since the former will automatically be freed when the stack (see |
||
280 | the entry on memalloc.c) is emptied. |
||
281 | |||
282 | BUILTIN COMMANDS: The procedures for handling these are scat- |
||
283 | tered throughout the code, depending on which location appears |
||
284 | most appropriate. They can be recognized because their names al- |
||
285 | ways end in "cmd". The mapping from names to procedures is |
||
286 | specified in the file builtins, which is processed by the mkbuil- |
||
287 | tins command. |
||
288 | |||
289 | A builtin command is invoked with argc and argv set up like a |
||
290 | normal program. A builtin command is allowed to overwrite its |
||
291 | arguments. Builtin routines can call nextopt to do option pars- |
||
292 | ing. This is kind of like getopt, but you don't pass argc and |
||
293 | argv to it. Builtin routines can also call error. This routine |
||
294 | normally terminates the shell (or returns to the main command |
||
295 | loop if the shell is interactive), but when called from a builtin |
||
296 | command it causes the builtin command to terminate with an exit |
||
297 | status of 2. |
||
298 | |||
299 | The directory bltins contains commands which can be compiled in- |
||
300 | dependently but can also be built into the shell for efficiency |
||
301 | reasons. The makefile in this directory compiles these programs |
||
302 | in the normal fashion (so that they can be run regardless of |
||
303 | whether the invoker is ash), but also creates a library named |
||
304 | bltinlib.a which can be linked with ash. The header file bltin.h |
||
305 | takes care of most of the differences between the ash and the |
||
306 | stand-alone environment. The user should call the main routine |
||
307 | "main", and #define main to be the name of the routine to use |
||
308 | when the program is linked into ash. This #define should appear |
||
309 | before bltin.h is included; bltin.h will #undef main if the pro- |
||
310 | gram is to be compiled stand-alone. |
||
311 | |||
312 | CD.C: This file defines the cd and pwd builtins. The pwd com- |
||
313 | mand runs /bin/pwd the first time it is invoked (unless the user |
||
314 | has already done a cd to an absolute pathname), but then |
||
315 | remembers the current directory and updates it when the cd com- |
||
316 | mand is run, so subsequent pwd commands run very fast. The main |
||
317 | complication in the cd command is in the docd command, which |
||
318 | resolves symbolic links into actual names and informs the user |
||
319 | where the user ended up if he crossed a symbolic link. |
||
320 | |||
321 | SIGNALS: Trap.c implements the trap command. The routine set- |
||
322 | signal figures out what action should be taken when a signal is |
||
323 | received and invokes the signal system call to set the signal ac- |
||
324 | tion appropriately. When a signal that a user has set a trap for |
||
325 | is caught, the routine "onsig" sets a flag. The routine dotrap |
||
326 | is called at appropriate points to actually handle the signal. |
||
327 | When an interrupt is caught and no trap has been set for that |
||
328 | signal, the routine "onint" in error.c is called. |
||
329 | |||
330 | OUTPUT: Ash uses it's own output routines. There are three out- |
||
331 | put structures allocated. "Output" represents the standard out- |
||
332 | put, "errout" the standard error, and "memout" contains output |
||
333 | which is to be stored in memory. This last is used when a buil- |
||
334 | tin command appears in backquotes, to allow its output to be col- |
||
335 | lected without doing any I/O through the UNIX operating system. |
||
336 | The variables out1 and out2 normally point to output and errout, |
||
337 | respectively, but they are set to point to memout when appropri- |
||
338 | ate inside backquotes. |
||
339 | |||
340 | INPUT: The basic input routine is pgetc, which reads from the |
||
341 | current input file. There is a stack of input files; the current |
||
342 | input file is the top file on this stack. The code allows the |
||
343 | input to come from a string rather than a file. (This is for the |
||
344 | -c option and the "." and eval builtin commands.) The global |
||
345 | variable plinno is saved and restored when files are pushed and |
||
346 | popped from the stack. The parser routines store the number of |
||
347 | the current line in this variable. |
||
348 | |||
349 | DEBUGGING: If DEBUG is defined in shell.h, then the shell will |
||
350 | write debugging information to the file $HOME/trace. Most of |
||
351 | this is done using the TRACE macro, which takes a set of printf |
||
352 | arguments inside two sets of parenthesis. Example: |
||
353 | "TRACE(("n=%d0, n))". The double parenthesis are necessary be- |
||
354 | cause the preprocessor can't handle functions with a variable |
||
355 | number of arguments. Defining DEBUG also causes the shell to |
||
356 | generate a core dump if it is sent a quit signal. The tracing |
||
357 | code is in show.c. |