bes  Updated for version 3.19.1
README.md
1 
2 ![build](https://travis-ci.org/OPENDAP/bes.svg?branch=master)
3 
4 <a href="https://scan.coverity.com/projects/opendap-bes">
5  <img alt="Coverity" src="https://scan.coverity.com/projects/5060/badge.svg"/>
6 </a>
7 
8 README for the OPeNDAP BES
9 ==========================
10 
11 Updated for version 3.19.1
12 
13 BESInterface/XMLInterface improved
14 
15 Testing/Logging Improvements
16 
17 Updated for version 3.19.0
18 
19 New Server function: range() This function returns the min and max
20 values of a variable and, for a vector, a boolean indicating that it
21 is monotonic.
22 
23 GeoTiff file can now be aggregated.
24 
25 See NEWS for more information.
26 
27 Updated for version 3.18.0
28 
29 New features:
30 
31 - hdf5_handler adds support for sinusodial projections.
32 
33 - ncml_handler adds support for the aggregation and metadata
34  manipulation of remotely located datasets. This new capanbility
35  utilizes the exisiting gateway_handler and allows the ncml_handler
36  to aggregate anything the gateway_handler can read.
37 
38 This release fixes a number of bugs:
39 
40 - Improved error logging (as in errors now get logged)
41  https://opendap.atlassian.net/browse/HYRAX-302
42 
43 - Improved server side function response assembly repaired:
44  https://opendap.atlassian.net/browse/HYRAX-352
45  https://opendap.atlassian.net/browse/HYRAX-342
46 
47 - Stopped prepending "nc_" to netCDF file out responses whose
48  download filename begins with a digit:
49  https://opendap.atlassian.net/browse/HYRAX-341
50 
51 - Repaired various BES timeout issues.
52  https://opendap.atlassian.net/browse/HYRAX-341
53 
54 Updated for version 3.17.4
55 
56 Added support for cached metadata objects.
57 
58 Server functions can now return multiple values using specially-named
59 Structure variables.
60 
61 The code now builds using gcc-6.
62 
63 See NEWS for other details.
64 
65 See the INSTALL file for build instructions, including special
66 instructions for building from a (git) cloned repo. This code can
67 build one of two distinct ways and that requires a tiny bit setup.
68 
69 For more information on Hyrax and the BES, please visit our documentation
70 wiki at docs.opendap.org. This will include the latest install and build
71 instructions, the latest configuration information, tutorials, how to
72 develop new modules for the BES, and more.
73 
74 This version has significant performance improvements for aggregations,
75 particularly those that use the JoinNew directive.
76 
77 Contents
78 * DAP2 versus DAP4 support
79 * What's here: What files are in the distribution
80 * About the BES: What exactly is this
81 * Configuration: How to configure the BES
82 * Testing: Once built and configured, how do you know it works?
83 
84 ## DAP2 versus DAP4
85 
86 The code on the master branch now supports both DAP2 and DAP4
87 responses, and the data format handlers do as well. Because of that,
88 it must be linked with libdap 3.14, not libdap older versions of
89 libdap (libdap 3.14 contains support for both DAP2 and DAP4, while the
90 older 'libdap' supports only DAP2). If you need to get the DAP2-only
91 version of the software, use the branch named 'dap2'. Please commit
92 only fixes there.
93 
94 Each of the handlers, which can be built as submodules within this
95 code, also has DAP2 and DAP4 support on their master branch and a
96 'dap2' branch for the DAP2-only code.
97 
98 ## What's here:
99 
100 Here there's a bes-config script which will be installing in $prefix/bin that
101 can be used to determine where the libraries and their header files have been
102 installed. This directory also contains some documentation files.
103 
104 _dispatch_: This is where the bulk of the BES source code resides.
105 server: The BES server and standalone executables; build using dispatch
106 cmdln: A command line client which can communicate with the bes server.
107 
108 _dap_: A module that implements the (OPeN)DAP access.
109 
110 _modules_: All the modules that are part of a standard Hyrax distribution.
111 The configure script looks for their dependencies and only tries to build
112 the ones that can be built. Note that the HDF4 and 5 modules are pulled in
113 from separate _git_ repositories.
114 
115 _xmlcommand_: The BES/dispatch software is a framework. The basic commands
116 recognized by it are defined and implemented by software in this directory.
117 
118 _standalone_: The 'standalone version of the server; used for testing.
119 
120 _apache_: An Apache module, currently not part of the default build
121 
122 _ppt_: The PPT implementation. The BES uses PPT for its communication
123 
124 _templates_: A collection of source files which can be used as templates when
125 you write your own handlers/modules, et cetera.
126 
127 _conf_: Where the automake and autoconf configuration files live
128 
129 _docs_: Where some bes documentation resides
130 
131 ## About the BES
132 
133 Note see http://docs.opendap.org/index.php/Hyrax
134 for the latest and most comprehensive documentation.
135 
136 ## Configuration
137 
138 ### Basic Configuration
139 
140 Once the BES software has been installed, you will need to make a few
141 changes to the BES configuration file, which is the bes.conf located in
142 _prefix_`/etc/bes/bes.conf`. Module and handler configuration files will
143 be installed in the _prefix_`/etc/bes/modules` directory.
144 
145 Only a few parameters need to be modified to get the BES up and running
146 for the first time. These parameters are located at the top of the
147 configuration file are are:
148 
149 **BES.ServerAdministrator**
150 - set this to an email address that can be used for users to contact if
151  there are issues with your installation of the server.
152 
153 **BES.User=user_name**
154 
155 **BES.Group=group_name**
156 - set these to a valid username and groupname on your system. We
157  recommend that you create a username and groupname called bes that has
158  permissions to write only to the BES installation directory. We'll
159  need to write to the log file, and that's about it.
160 
161 **BES.LogName=./bes.log**
162 - Set this to the full path and file name for where you want the BES log
163  file to be written.
164 
165 With this configuration you will be able to start the BES. No handlers
166 or modules have been installed yet, so you won't be able to serve data,
167 but the BES should run.
168 
169 ### Configuration for Hyrax
170 
171 For the BES to run with Hyrax, additional changes will need to be made
172 to the dap.conf and dap-server.conf files, which are located in the
173 modules directory _prefix_`/etc/bes/modules`. The dap module should be
174 installed by default with the BES. If the dap.conf file is not found, be
175 sure that you have libdap installed. The dap-server module, known as the
176 General purpose handlers, adds responses for the DAP ascii response, the
177 DAP info response, and the DAP html form response. You will need to get
178 the dap-server module. You should see a dap-server.conf file in the
179 modules directory.
180 
181 The changes required for Hyrax are:
182 
183 **BES.Catalog.catalog.RootDirectory** = _path_to_root_data_directory_
184 - Find the BES.Catalog.catalog.RootDirectory parameter in dap.conf. Set
185  this to the root directory of your data. If you're serving data stored
186  in files, this is the place in the file system where those files are
187  stored.
188 
189 **BES.Catalog.catalog.Include**=;
190 - This parameter specifies what files/directories are included in the
191  list of nodes in the catalog. The default is to show everything. After
192  this parameter is looked at, the Exclude parameter is then looked at
193  to see what files you might want to exclude.
194 
195 __BES.Catalog.catalog.Exclude=^\..*;__
196 - This parameter specifies what files/directories to include in the list
197  of nodes in the catalog. The default, shown here, is to exclude any
198  files or directories that starts with a dot (.)
199 
200 The only possible configuration parameter that you may need to change is
201 the one that maps a file to a data handler. This parameter is called
202 BES.catalog.TypeMatch, and is found in each of the data handler
203 configuration files, such as nc.conf. The default values should work.
204 
205 The value of this parameter is a semicolon separated list that matches
206 the name you used in the BES.modules parameter with different datasets.
207 The BES uses regular expressions to identify different types of
208 datasets. In the example given in the file, any dataset name that ends
209 in '.nc' will be accessed using the netcdf hander (because the name 'nc'
210 is used here with the regular expression '.*\.nc'; note that the BES
211 uses regular expressions like Unix grep, not file globbing patterns like
212 a command shell). Since the name 'nc' was associated with the netcdf
213 module in the modules section, the netcdf module will be used to access
214 any dataset whose name that ends in '.nc'.
215 
216 The regular expressions shown in the examples are simple. However, the
217 entire dataset name is used, so it's easy to associate different modules with
218 datasets based on much more than just the 'file name extension' even though
219 that is the most common case.
220 
221 To test your regular expression for the TypeMatch parameter, or the
222 Include and Exclude parameters, use the supplied besregtest program.
223 Simply run besregtest to discover its usage.
224 
225 ### Installing a custom handler/module
226 
227 By default, The BES for Hyrax comes with a suite of handlers that read
228 a number of data formats. However, you can install custom handlers that
229 are not distributed by default. I'll use the SQL handler as an example.
230 
231 Get the SQL handler source code from http://github.com/opendap/,
232 making sure that the version supports Hyrax. Expand the tar.gz file
233 and follow the instructions with the following caveat:
234 If you have installed the BES using a prefix other than /usr/local (the
235 default), make sure that the correct bes-config is being run. If you are
236 having problems compiling or linking the handler, try using not only
237 --prefix=... but also --with-bes=... when you configure the handler.
238 
239 #### Install the newly-built handler.
240 
241 Once built, install the handler using 'make install'. This will install
242 the BES module and a configuration file to use for that module. Each
243 module will have its own configuration file. In this case it is nc.conf
244 and installed in the _prefix_`/etc/bes/modules` directory. The next time
245 the BES is run, this configuration file will be read and the netcdf
246 module loaded. No modifications are necessary.
247 
248 ## Testing
249 
250 To test the server, open a new terminal window and start the bes by using
251 the bes control script besctl, which is installed in _prefix_`/bin`.
252 using the -c switch to name the configuration file. If the server standalone
253 starts correctly it should print something like the following to stdout:
254 
255 ```
256  [jimg@zoe sbin]$ besctl start
257  BES install directory: <prefix>/bin
258  Starting the BES daemon
259  PID: <process_id> UID: <uid of process>
260 ```
261 
262 Go back to your first window and run the bescmdln program. Use the -h (host)
263 and -p (port) switches to tell it how to connect to the BES.
264 
265 ```
266  [jimg@zoe bin]$ bescmdln -p 10002 -h localhost
267 
268  Type 'exit' to exit the command line client and 'help' or '?' to display
269  the help screen
270 
271  BESClient>
272 ```
273 
274 Try some simple commands:
275 
276 ```
277  BESClient> show help;
278  <showHelp>
279  <response>
280 
281  ...
282 
283  </response>
284  </showHelp>
285  BESClient>
286 ```
287 
288 Note that all commands must end with a semicolon except 'exit'.
289 
290 Now try to get a DAP response using a handler you registered (the BES
291 supports a fairly complex syntax which I won't explain fully here):
292 
293 ```
294  BESClient> set container in catalog values n1, data/nc/fnoc1.nc;
295  BESClient> define d1 as n1;
296  BESClient> get das for d1;
297  Attributes {
298  u {
299  String units "meter per second";
300  String long_name "Vector wind eastward component";
301  String missing_value "-32767";
302  String scale_factor "0.005";
303  }
304  v {
305  String units "meter per second";
306  String long_name "Vector wind northward component";
307  String missing_value "-32767";
308  String scale_factor "0.005";
309  }
310  lat {
311  String units "degree North";
312  }
313  lon {
314  String units "degree East";
315  }
316  time {
317  String units "hours from base_time";
318  }
319  NC_GLOBAL {
320  String base_time "88- 10-00:00:00";
321  String title " FNOC UV wind components from 1988- 10 to 1988- 13.";
322  }
323  DODS_EXTRA {
324  String Unlimited_Dimension "time_a";
325  }
326  }
327  BESClient> exit
328  [jimg@zoe bin]$
329 ```
330 
331 If you got something similar (you would have used a different dataset name
332 from "data'nc'fnoc1.nc" and thus would get a different DAS response) the BES
333 is configured correctly and is running.
334 
335 To stop the BES use the bes control script with the stop option:
336 
337 ```
338  [jimg@zoe sbin]$ besctl stop
339 ```
340 
341 Note: Constraints and the bes command line client
342 
343 Constraints are added to the 'define' command using the modifier 'with' like
344 so:
345 
346 ```
347  define d as nscat with nscat.constraint="WVC_LAT";
348 ```
349 
350 If there is a list of containers instead of just one, then there can be a
351 list of <container>.constraint="" clauses.
352 
353 ## COPYRIGHT INFORMATION
354 
355  The OPeNDAP BES code is copyrighted using the GNU Lesser GPL. See the
356  file COPYING or contact the Free Software Foundation, Inc., at 59 Temple
357  Place, Suite 330, Boston, MA 02111-1307 USA. Older versions of the BES were
358  copyrighted by the University Corporation for Atmospheric Research;
359  see the file COPYRIGHT_UCAR.