Laser Range Vision for Tele-Excavation

Michael Greenspan
National Research Council of Canada

Michael Lipsett
Atomic Energy Canada Limited

John Ballantyne
Spar Aerospace Limited

Pam Renton, Eric Gagnon and Nestor Burtnyk National Research Council of Canada

Abstract

This paper presents the development of a teleexcavation system which uses a laser range scanner and a virtual environment to aid a typical remote excavation process. The system includes : a commercial excavator which has been modified for remote operation, a remotely operated sensorized scout vehicle, and a graphical computer workstation. The laser range scanner is attached to a pan-tilt unit on the scout vehicle; the scanner acquires accurate 3D geometric data about the terrain. The scanned data and the excavator telemetry are inputs to the virtual environment that yield an accurate visual model of the worksite that includes the excavator.

One focus of the work is the construction of the virtual environment from a collection of acquired range images, which are processed to register their relative positions within the scene. A number of control schemes have also been implemented to test and compare the effectiveness of various modes of teleexcavation. In particular, virtual barriers are constructed from an analysis of the range data. These barriers can be used both to prevent collisions of the excavator with obstacles, and to guide excavation.

The system has been developed and preliminary integration tests were performed.

Introduction

The remote operation of heavy machinery is a technology which is essential for the cleanup of highlevel waste sites, where there is no safe way to protect an operator on the machine. The issues encountered are

similar to those of telerobotics in general. Because a remote operator is removed from the machine, the absence of direct visual and other sensory queues makes safe and effective operation difficult.

One way to improve the effectiveness of teleoperation is through the use of a virtual environment, a rich 3D graphical model of the worksite and the device under control. The virtual environment provides the operator with an overview of the worksite, to facilitate analysis, planning, and simulation of the operations, with the goal of improving efficiency.

The U.S. Department of Energy has committed to cleaning hazardous waste sites which exist in many nuclear research and weapons production facilities throughout the United States [1]. The high levels of radiation and toxicity make it dangerous for humans to remain very long in the vicinity of these sites, which rules out conventional site restoration practices.

The use of remote machinery, where the operator is physically removed from the device under control, is a promising approach. The issues involved in operating remote machinery are similar to those which have been well addressed in the field of telerobotics [2]. In this case, there is the added difficulty that the out of the line of sight of the worksite and must rely entirely on sensor feedback. Unlike the case of ground operated control of space robotics, there is not necessarily a significant time lag between the operator and the equipment.

The objective of the work described in this paper is to demonstrate the utility of laser range imagery for the effective teleoperation of a remote excavator. Laser

range scanners measure both the intensity and depth of each point in a sensed image, resulting in dense 3D samples over a sensed volume [3]. In addition, range images are qualitatively interpretable, yielding rich visual clues of the scene?s contents. They also have a quantitative range component, allowing direct 3D measurements over the data.

The system operational concept involves acquiring a set of dense range images and constructing a geometric description of the worksite. The images are rendered in a 3D graphical system, known as a virtual environment (VE). The position and status of the excavator and other remote devices is also rendered in the VE, providing an accurate visual description of the worksite. The quantitative aspect of the range data is exploited and geometric CAD models are added to the VE. These models aid in the interpretation of the scene, and are also used to help control the remote operation.

This paper next describes the hardware and software elements and their connectivity, explains some of the modes of operation, and gives test results of a prototype system. The concluding section summarizes the results to date, and outlines some plans for further research and development.

System Hardware

The system hardware has six main elements : the Laser Range Scanner (LRS), the Remote Excavator, the Scout Vehicle, the Pan-Tilt Unit and Boom Mount (PTU/B), the operator workstation (SGI), and the embedded PC. These elements are illustrated in Figure 1. Their connectivity is diagrammed in Figure 2.

Figure 1 : Field Integration Hardware

Remote Excavator

The excavator is a Hitachi LC200 which was previously remotized by Spar Aerospace, RSI Research, and the University of British Columbia .

>=
Range?
>=
Field of View
>=
Acquisition Rate
>=
Image Size
>=
Size
>=
Mass

>=
0.6 to 10 m
>=
30 X 40 degrees
>=
20,000 pixels/second
>=
up to 4096 X 4096 pixels
>=
23cm X 23cm X 13cm
>=
4.95 kg

Table 1 : LRS Specifications

The four hydraulic joints are electrically controlled by a joystick, which was connected to the excavator embedded controller by an RS-232 line. The signals from the joystick are interpreted as rates in a cylindrical coordinate system.

There are a number of additional lines which are bundled with the joystick line in a tether approximately 75 meters in length. An NTSC line connects a real-time video camera which is mounted above the cab, and control lines are included to pan, tilt, and zoom the camera. A customized bucket includes a controllable thumb joint, which allows the grasping of barrels and other objects, and a dedicated line controls the position of the rate of the thumb joint. The tether also includes a dead-man switch cable.

The excavator tracks are also individually rate controlled, although these were not operated remotely at this stage due to the tether.

Laser Range Scanner (LRS)

The LRS is based upon the auto-synchronous scanning method developed at the NRC [3]. Its specifications are listed in Table I. The particular camera used was built by Spar Aerospace and the NRC, and was reinforced for outdoor use.

Scout Vehicle

The scout vehicle is a Robotech Remote Hazhandler, a remote Bobcat tele-driven by wireless link. It transports the LRS to locations of interest in the worksite.

In previous laboratory prototypes, the LRS was mounted directly on the wrist of a Puma 560 robot. The straightforward extension of this onto the current hardware would have been to mount the LRS directly on the Excavator stick. This configuration was not used because of concerns that the excavator dynamics (i.e. vibrations), which were negligible in the laboratory system, might adversely effect the LRS operation. Also, it was conjectured that the use of a more manouverable scout vehicle affords greater flexibility to position the LRS and sense the scene.

The scout vehicle and its suite of sensors is completely untethered, the digital connection being an Arlan wireless Ethernet.

PTU/Boom Mount (PTU/B)

The LRS is mounted to the Scout Vehicle on a 2 degree of freedom pan-tilt unit, and an elevating Boom, with shock mounts to dampen the transmittance of vehicular vibrations. The positions of the PTU/B are controlled at the operator workstation.

The PTU allows a 360 degree pan at a slew rate of 30 degrees/s, and a tilt of greater than +- 60 degrees. There is also a small video camera mounted on the PTU with the LRS. This allows the operator to aim the LRS by viewing the image in its field of view prior to acquiring a range image.

Operator Workstation (SGI)

All of these devices are monitored and operated from a graphical workstation. The richness of the range data is effectively displayed on a high end graphics workstation. For the integration tests, and SGI Impact running IRIX 5.3 was used.

Embedded PC

The real-time requirements of the communication with the excavator controller require an embedded computer as an interface between the joystick, the excavator, and the SGI. A 66 MHz 486 PC running the QNX real-time OS is used.This acts as a central data switch, accepting data from the various devices, and routing them to the proper destinations. The PC also processes and filters some of the signals, to improve the ease-of-use of the joystick, and to provide real-time collision avoidance functionality [6].

Connectivity

Excavator - PC

The excavator and PC are connected by a full duplex RS-232 line. Data packets are transmitted to and received from the excavator every 50 msec.

The message sent to the excavator is a cylindrical coordinate frame rate command, as read from the joystick. Joint rates can also optionally be sent. There are also a number of additional miscellaneous commands, such as emergency stop and standby.

The message sent to the PC is an excavator joint encoder reading.

Joystick - PC

The joystick and PC are connected by an RS-232 line. The joystick is interpreted as cartesian rate commands, and is polled every 25 msec, which is

twice the frequency of the Excavator-PC communication cycle.

Figure 2 : System Hardware and Connectivity

PC - SGI

The PC is connected to the SGI by an Ethernet line.

The message sent to the SGI contains current joint locations as received from the excavator. These are used to update the attitude of the excavator in the VE. The current joystick readings are also included in the message.

The message sent to the PC include operational state information, such as joystick filtering parameters. For example, certain degrees of freedom of the joystick can be toggled on/off, or made more sensitive. The world model containing the obstacle information is also transmitted to the PC for real-time collision avoidance.

SGI - PTU/B

The SGI and PTU/B are connected by a wireless Ethernet.

The message sent to the SGI contain PTU/B position encoder readings.

The message sent to the PTU/B contain motion commands to pan, tilt, and raise or lower the boom.

SGI - LRS

The SGI and LRS are connected by a wireless Ethernet. The PTU and Boom have separate motor controllers.

The message sent to the SGI contains the acquired image data.

The message sent to the LRS contains parameter settings, such as image size, field of view, etc.

System Software

The operator workstation software consists of a node library of C++ classes and C modules. The three main functions of the system software are :

- to register the acquired range images,

- to control devices (LRS, PTU/B and excavator),

- to render and provide interaction with the VE to aid the operator?s understanding of and interaction with the scene.

Range Image Registration

To construct a view of the worksite, the range images must be positioned with 6 degrees of freedom within the VE. The scout vehicle has no automatic telemetry (such as GPS) to give an estimate of its location in the worksite. The contents of the range images themselves must therefore be used to determine their relative locations.

When a range image is acquired, an effort is made to overlap some portion of the new image with a previously acquired image. If there is enough common information in two images, then the newly acquired image can be correctly positioned with respect to the reference image, in a process known as image registration.

The system supports three types of image registration : manual, semi automatic, and fully automatic. In manual registration, the operator selects three corresponding data points in each image. The homogeneous transformation that maps the second set of three points onto the first is calculated and applied to the new image. As long as there are three points in each image that the operator visually recognizes as corresponding, this method will succeed.

In the partially automatic method, the operator provides an initial estimate of the location of the new image by visually positioning the image in the VE using the ?fly-around? controls. Once the initial estimate has been established, a version of the Iterative Closest Point Algorithm (ICPA) [4] is invoked to calculate a more accurate correspondence. As long as the initial estimate is within the global minimum of the error metric used to calculate the residuals, this method will succeed.

In the fully automatic method, known as Signature Search [5], it is assumed that an initial relative position estimate exists. This initial estimate can be less accurate than in the partially automatic case, as it need not be close to the global minimum. A search is applied which refines the initial estimate, following which the ICPA is invoked. The criterion for success of this method is dependant upon the information content of the image and the accuracy of the initial estimate.

When a sequential chain of images are registered, there is an error which accumulates with respect to the initial reference image. This error can be reduced by scanning the initial location, closing the chain and distributing the refined error estimate over the images.

The 6 dof position of the Scout vehicle is determined from its relative position to the acquired range image. If the scout vehicle is not repositioned, and the PTU/B is repositioned to provide a new image view, then an initial estimate exists from the PTU/B encoder readings, and the Signature Search method can be invoked.

If a range image is acquired which does not overlap with any of the previously acquired images, then it can be manually positioned in the VE by the operator, until further overlapping images are acquired.

Device Control

The operator workstation is used to control the LRS, the PTU/B, and the Excavator.

LRS Control

The LRS is a highly flexible ranging mechanism, and there are a number of parameters (such as field-ofview, scan pattern, and scan speed), that can be adjusted, depending upon the scene to be acquired. Typically, a low resolution 64x64 image is acquired first to get an estimate of the field-of-view and exposure settings. Following this a higher resolution 256X256 image is acquired.

As an example, a rectangular region can be selected in an acquired range image to zoom the LRS and limit its field-of-view to a small region of interest.

PTU/B Control

When the Scout Vehicle is in motion, the Boom is lowered to its home position, to reduce the

vibrations transmitted to the LRS. The Boom is raised to an appropriate height when an image is to acquired. The PTU position is also commanded from the operator workstation to capture a suitable scene.

Figure 3 : VE Generated from Field Integration Tests

Excavator Control

Automatic paths can be generated within the SGI and are downloaded to the PC for execution on the excavator. If obstacles are identified in the environment, then a path planning search will generate a collision-free trajectory.

VE Rendering and Interaction

Range images are by far the most expensive items to render in the VE. A single range image may consist of over 64,000 3D points, with associated intensity values, and it is a system requirement to effectively render and interact with 10 to 50 range images simultaneously.

A number of commercial systems exist to simulate and render 3D CAD worlds (IGRIP, ROBCAD, ACT). Range images are relatively novel, however, and are not supported by these systems. It was therefore necessary to develop a VE rendering engine capable of dealing with both range images and articulated CAD models.

There are a number of rendering methods for the range images, depending upon the level of interaction required. The two representations used here are point sets and tile sets. In point mode, each individual point in a range image is rendered in 3D space with its associated intensity value. In tile mode, a neighbourhood operation is applied to create small planar surfaces from adjacent points. Whereas the point mode is closer to the raw data, the tiles are more visually attractive in many cases, as they give the impression of a connected solid surface. Both point and tile representations have a fine and a coarse rendering method, which trades off detail with rendering speed.

Field Integration Tests

Preliminary field integration tests were conducted at AECL Chalk River Laboratories in October, 1995. All of the described equipment was assembled and integrated. A test field was constructed which resembled a general dumpsite. The operator workstation was located in a trailer adjacent to the worksite. For safety reasons, a direct view of the excavator was available through a window of the trailer. During the formal tests, however, the operator used only the video monitors and VE rendering to guide teleoperation.

The worksite was surveyed with 7 images, and the resulting VE as displayed on the operator workstation is illustrated in Figure 3.

The results of the tests were positive. The LRS exceeded nominal specifications, acquiring valid data as far away as 16 meters in daylight conditions. The VE was visually descriptive, and a complete rendering of the range images, plus excavator, scout vehicle, and miscellaneous CAD models, was executed in approx. 2 - 5 Hz.

The major result was working confirmation of the operating principle of the system. The interaction with the VE allowed the operator to better understand the

geometry of the worksite, and to plan and execute remote operations. As an example, working entirely from the visual queues contained in the VE, the operator was able to position the bucket, grasp a barrel, and place it in a dumpster.

Conclusions and Further Work

The results obtained to date demonstrate the benefits that range imagery can add to remote teleoperation. The eventual goal of this work is to increase the effectiveness and the safety of remote teleoperation, which will both reduce the level of skill required by the operator, and increase throughput.

Ultimately, the effectiveness of this system will depend very much on the useability of the interface. One of the continuing areas of research is to improve the user interface to make the excavator operations more complete and intuitive.

Another focus of continued research is improved safety. In particular, real-time collision avoidance has been demonstrated on a laboratory prototype and is currently being implemented on the field system. Objects in the worksite with which contact is hazardous are identified in the VE as obstacles. The low level controller will intervene and disallow any motion commands which would result in an impending collision. The integration tests highly motivated the inclusion of real-time collision avoidance, as one of the excavator operators accidentally contacted and moved the dumpster.

Another area of further research is representations for more effective rendering of range images. Hierarchical t-mesh structures are currently being developed for this purpose.

The system is also being adapted to support other manipulator platforms that require remote teleoperation. One application which is being developed is a small platform with an articulated manipulator for the inspection of indoor nuclear facilities.

Other topcis for continuing research include improved handcontrollers, such as force-reflecting joysticks, automatic updates of the VE as work is executed, and the operational effects of a non-negligible time delay in the communication link between the operator workstation the remote equipment.

Acknowledgements

This work was partially funded by Precarn Associates.

References

[1] Technological Summary, Office of Environmental Management, Technology Development, United States of America Department of Energy, DOE/EM- 248 to EM-255, June 1995.

[2] Sheridan, Thomas B., Telerobotics, Automation, and Human Supervisory Control, MIT Press, Mass., U.S.A, 1992.

[3] Rioux, M., Blais, F., Beraldin, J.A., and Boulanger, P., ?Range Imaging Sensors Development at NRC Laboratories?, Proceedings of the IEEE Workshop on Interpretation of #D Scenes, Austin, TX, Nov. 27-29, 1989, pp. 154-160, NRC 29130.

[4] Besl, Paul K., McKay, Neil D., ?A Method for Registration of 3-D Shapes?, IEEE Transactions on Pattern Analysis and Machine Intelligence?, vol. 14, no. 2, Feb. 1992, pp. 239-256.

[5] Burtnyk, N., Greenspan, M., ?Signature Search Method for 3-D Pose Refinement with Range Data?, IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, Oct. 2-5, 1994, pp. 312-319.

[6] Greenspan, M., and Burtnyk, N., ?Obstacle Count Independent Real-Time Collision Avoidance?, to be published in 1996 IEEE Conference on Robotics and Automation.